[ norboten ]
linux · lab linux-01

The Disk That Stays Full

· about 20 minutes · runs on ubuntu-26.04, alpine · unrated

An unrated lab. It runs on your machine with no account and no network, and everything about it — the faults, the checks, the hints and the reference solution — is in the repository. An attempt on it is recorded on your profile and never moves a rating: only rated labs do. Rated and unrated labs.

The briefing

The ledger service writes its transaction log to its own filesystem, mounted at /var/log/app. This morning it reported the disk full.

An hour ago a colleague deleted the biggest file they could find there. df still says the filesystem is almost full. du says there is hardly anything in it. The colleague went home.

What is expected, and graded:

  1. The log filesystem has at least half of its space free.
  2. No space on it is held by files that no longer exist.
  3. The ledger log is rotated automatically, so this cannot happen again.
  4. ledger keeps running, and starts at boot. Stopping or removing it is not a fix.

You have root through sudo. Everything must still hold after a reboot.

What is graded

The machine is checked, rebooted, and checked again. A check passes only if it passes both times.

CheckObjective
01_log_disk_has_spaceFind what is using disk space when df and du disagree
02_no_deleted_files_held_openFind what is using disk space when df and du disagree
03_logrotate_covers_appConfigure log rotation so a service cannot fill its disk again
04_ledger_still_runningFix the cause without taking the service down

Start it

  1. 2Labs
  2. linux-01select this one
  3. udownload it
  4. sstart the machine

Reading

Where the lab's hints send you, level by level, as you ask for them (h, then l opens a journal section in the TUI).

Theory for this lab

3 questions on the same topic, in the lab's Theory tab. They never affect the lab's grade. Here they are, to answer in place:

Question 1

df reports 95% used; du on the same mount adds up to 10%. What is the most likely reason?

Question 2

What does logrotate's copytruncate do?

Question 3

Which command tests a logrotate configuration without rotating anything?