· about 30 minutes · runs on ubuntu-26.04-claude · unrated
An unrated lab. It runs on your machine with no account and no network, and everything about it — the faults, the checks, the hints and the reference solution — is in the repository. An attempt on it is recorded on your profile and never moves a rating: only rated labs do. Rated and unrated labs.
The docs team keeps its handbook in a git repository, ~/handbook, and a nightly job,
~/bin/tidy-handbook, asks Claude Code to fix typos in docs/. Last night the job "tidied" a
little further: the unpublished drafts are gone from the working tree, and the job's log shows the
model opening a file nobody meant it to see. Nobody has committed since, and nobody wants to turn
the job off — the typo fixes are genuinely useful.
This lab runs in a container, as learner with sudo. claude here is the real Claude Code
2.1.270, talking to a scripted model on this machine instead of Anthropic: it replays last night's
run every time the job starts, and Claude Code carries out its tool calls for real. Run the job as
often as you like; ~/tidy-handbook.log is its JSON result.
What is expected, and graded — the grader runs ~/bin/tidy-handbook itself, as learner, against
models that try other things:
docs/, and cannot delete or overwrite anything else, however the
model tries.This lab runs in a container — Docker or Podman, no VM — so there is no boot to survive: the checks run once, against the state you left.
| Check | Objective |
|---|---|
| 01_the_drafts_are_back | Recover work an agent deleted from a git working tree |
| 02_the_job_fixes_typos_and_deletes_nothing | Give a headless Claude Code job exactly the permissions its task needs |
| 03_the_token_never_reaches_the_model | Keep secrets in a repository out of what the model is sent |
| 04_permissions_are_not_bypassed | Give a headless Claude Code job exactly the permissions its task needs |
Where the lab's hints send you, level by level, as you ask for them (h, then l opens a journal section in the TUI).
4 questions on the same topic, in the lab's Theory tab. They never affect the lab's grade. Three of them, to answer here:
A nightly job runs `claude -p "Fix typos in docs/" --permission-mode dontAsk` with no allow rules anywhere. The model asks to Edit docs/guide.md. What happens?
The edit is refused and recorded in permission_denials; the run carries on
dontAsk turns every call that would prompt into a denial, so a headless run never waits for a person. The denial is reported to the model as a tool error and listed in the JSON result's permission_denials; the run goes on. Reads in the working directory need no permission, but edits do. It does not wait (there is nobody to ask in -p) and it does not abort.
https://code.claude.com/docs/en/permission-modes · https://code.claude.com/docs/en/headless
In Claude Code 2.1.270, a job runs `claude -p` in a repository whose committed .claude/settings.json has `"allow": ["Edit(./docs/**)"]`, and nobody has ever run Claude Code interactively there. Edits under docs/ are still denied in dontAsk mode. Why?
In an untrusted workspace, -p ignores allow rules from the project's .claude/settings.json and says so on stderr
A repository can ship a settings file, and allowing tools is exactly what a hostile repository would do, so allow entries from the shared project file are ignored until the workspace is trusted: "Ignoring 1 permissions.allow entry from .claude/settings.json: this workspace has not been trusted." Deny rules from the same file still apply. The same allow rule on the command line (--allowedTools), in --settings, in the user settings or in .claude/settings.local.json is honoured, so dontAsk does respect allow rules. Relative rules like ./docs/** are fine. Verified in the lab image (docs/research/claude-code.md).
https://code.claude.com/docs/en/permissions · https://code.claude.com/docs/en/settings
A job keeps Bash available and adds `"deny": ["Read(./.env)"]`. The model runs four Bash commands. Which one still shows it the token in .env, in dontAsk mode?
grep -r TOKEN .
Read-only commands such as grep and cat run without asking, and a Read deny rule stops one only when the command names the denied path. `cat .env`, `head -1 .env` and `cat ./.env` name it and are refused; `grep -r TOKEN .` names only the directory and prints the token. That is why a job that does not need a shell should not be offered one (`--tools`), rather than counting on rules to anticipate every command.
https://code.claude.com/docs/en/permissions