How to Fix a Full systemd Journal

The systemd journal stores every log message your server produces, and on a VPS with a small root partition, it can quietly eat through your available disk space. When that happens, services fail to start, writes stall, and you lose the very diagnostics you need to figure out what went wrong. This guide covers how to diagnose the problem, free up space immediately, and configure journald so it doesn't happen again.

Diagnosing the Problem

Start by checking how much space the journal is using:

journalctl --disk-usage

This reports the combined size of all active and archived journal files. Compare that against your available disk space with df -h. On a 20 GB root partition, even 2 GB of logs represents over 10% of total disk, which is enough to cause problems.

Next, find out which services are generating the most noise. This one-liner ranks the top 10 log sources from the past 24 hours:

journalctl --since "24 hours ago" | awk '{print $5}' | cut -d'[' -f1 | sort | uniq -c | sort -rn | head -10

Filter for errors specifically with journalctl -p err -b to see if a single service is stuck in a crash loop and flooding the journal. You can also check whether journald is rate-limiting any services:

journalctl -u systemd-journald | grep -i "suppressed"

Suppression messages mean a service exceeded the default threshold of 10,000 entries within 30 seconds. That's your culprit.

A few things are especially common on VPS instances. If Storage=auto is set and /var/log/journal/ exists, journald uses persistent storage with a default cap of roughly 4 GB. On a small root partition, that fills fast. Unclean shutdowns can also leave corrupted journal files with a .journal~ suffix. These files don't rotate normally and keep consuming space.

Clearing Logs Safely

Before removing anything, rotate the active journal file so current logs are preserved:

journalctl --rotate

Then vacuum archived logs. You can target by age, size, or file count:

sudo journalctl --vacuum-time=1d
sudo journalctl --vacuum-size=100M
sudo journalctl --vacuum-files=5

The first deletes logs older than 24 hours. The second shrinks total journal size to 100 MB. The third keeps only the five most recent archived files. Pick whichever fits your situation.

If the disk is completely full and journalctl commands are unresponsive, you may need to delete journal files manually:

sudo rm -rf /var/log/journal/*
sudo systemd-tmpfiles --create --prefix /var/log/journal

This is a last resort. It wipes all stored logs and recreates the directory with correct permissions. After any cleanup, restart the daemon and verify:

sudo systemctl restart systemd-journald
journalctl --disk-usage

Configuring Journald Size Limits

The default journald settings are generous. On a VPS, they're too generous. Edit the configuration to set hard caps on log storage. Create an override file rather than editing the main config directly, so your changes survive package updates:

sudo mkdir -p /etc/systemd/journald.conf.d/
sudo nano /etc/systemd/journald.conf.d/size-limits.conf

The key directives:

Setting both SystemMaxUse and SystemKeepFree gives you a belt-and-suspenders approach: logs are capped at a fixed size, and the system always has breathing room regardless of what else is on the disk.

Persistent vs. Volatile Storage

The Storage= directive controls where logs go. Set Storage=persistent to write logs to /var/log/journal/ on disk. This is what you want for production servers, because logs survive reboots and you can investigate crashes after the fact. If the directory doesn't exist, create it:

sudo mkdir -p /var/log/journal

The alternative, Storage=volatile, keeps logs in RAM at /run/log/journal/. Logs disappear on reboot. This makes sense for ephemeral containers or servers that forward all logs to an external system.

Disabling Compression on High-Traffic Servers

Journald compresses data objects larger than 512 bytes by default. On servers handling heavy log throughput, this adds CPU overhead. Set Compress=no if you're forwarding logs externally and don't need to minimize local storage. Also set ForwardToConsole=no in production. Console forwarding is synchronous, and a stalled virtual serial console can block the journald daemon entirely.

After any configuration change, restart the service:

sudo systemctl restart systemd-journald

Automating Log Maintenance

Manual cleanup doesn't scale. Create a systemd timer to vacuum logs on a schedule. Set up a service unit at /etc/systemd/system/journal-vacuum.service:

[Unit]
Description=Vacuum old journal logs
 
[Service]
Type=oneshot
ExecStart=/usr/bin/journalctl --vacuum-time=7d --vacuum-size=500M

Then create a matching timer at /etc/systemd/system/journal-vacuum.timer:

[Unit]
Description=Weekly journal vacuum
 
[Timer]
OnCalendar=Sun 02:00
Persistent=true
 
[Install]
WantedBy=timers.target

Enable it with sudo systemctl enable --now journal-vacuum.timer. The timer runs every Sunday at 2 AM and applies both time-based and size-based retention in one pass.

One thing the timer won't catch: corrupted journal files. After an unclean shutdown, journald quarantines damaged files by appending a ~ to the filename. These .journal~ files still count toward disk usage but won't be vacuumed automatically. Periodically check for and remove old ones:

find /var/log/journal/ -name "*.journal~" -mtime +30 -delete

Forwarding Logs Externally

For servers where you need long-term retention without growing local storage, forward logs to a centralized system. The simplest approach is enabling syslog forwarding in journald.conf:

ForwardToSyslog=yes

For structured logs with full metadata (PID, UID, unit names), use systemd-journal-remote to ship JSON-formatted entries to a log management platform. External log storage also protects your audit trail if the local disk fails or the server is compromised.

Conclusion

A full systemd journal is straightforward to fix and easy to prevent. The key steps:

• Check usage with journalctl --disk-usage and identify noisy services.
• Clear space immediately with journalctl --rotate followed by --vacuum-size or --vacuum-time.
• Set explicit size limits in a journald.conf override file.
• Automate cleanup with a systemd timer.
• Forward logs externally for long-term retention.

FDC's VPS and dedicated server plans provide the disk I/O and storage needed for production log workloads.