A journal is a study document, not a manual page: the mechanism, one failure walked through end to end, the wrong turns people actually take, a cheat sheet, and questions you should be able to answer a week later without looking. Lab journals accompany a lab, where the same machine is waiting to be broken, and end with where its hints lead; topic journals cover a subject end to end, with a table of symptoms and their causes, exercises and sources. Notes are different: how something in Norboten itself was built and why, written for someone building their own.
Ansible's whole promise is a sentence: describe the state you want, and running the playbook again changes nothing. Every useful property follows from it — you can re-run after a failure, run against a fleet where half t
Covers idempotence per module; command/shell with creates and changed_when; lineinfile regexp; handlers and --force-handlers; check mode and diff; variable precedence; vault passwords without a terminal
3091 words · 8 review questions
Claude Code is usually met as an interactive assistant in a terminal. This journal is about the other way it runs: as a program started by a script, a cron job or a CI workflow, with nobody watching. In that role it is a
Covers headless claude -p runs; the five settings levels and workspace trust; permission modes, allow and deny rules; hooks' matchers and exit statuses; subagents; turns, budgets and model choice
2476 words · 8 review questions
The task sounds like one command: run a small web server in a container, as an unprivileged account, serving files from the host, and have it come back after a reboot. It is one command to get it running. Getting it righ
Covers rootless Podman: /etc/subuid and subgid, lingering sessions, SELinux labels on bind mounts (:Z), unprivileged ports, Quadlet units, volumes versus the writable layer
2678 words · 8 review questions
A service falls over under load with Too many open files. Somebody raises the limit in /etc/security/limits.conf, logs in, runs ulimit -n, sees the new number, and closes the ticket. The next busy afternoon it falls over
Covers RLIMIT_* inherited from units, shells and pam_limits; /proc/PID/limits; free's available column; the machine's and a cgroup's OOM killer (memory.max, memory.events); load average and D state; sysctl -w, sysctl.d and tuned
2729 words · 8 review questions
After an incident, the logs are the only witness, and everyone assumes the witness saw everything. It did not. A logging pipeline makes decisions of its own: it drops messages from a service that logs too fast, it files
Covers stdout and stderr into the journal; per-unit rate limits (RateLimitIntervalSec/Burst); rsyslog's second copy; Storage=, SystemMaxUse=, vacuuming; journalctl by unit, boot, field and priority
2479 words · 7 review questions
The Model Context Protocol (MCP) is how an AI client — Claude Code, Claude Desktop, a claude.ai connector — reaches tools and data it was not built with. A server offers tools (functions a model can call), resources (thi
Covers stdio and Streamable HTTP; the 2026-07-28 stateless revision (_meta envelope, Mcp-Method/Mcp-Name, input_required, ttlMs); server-side boundaries; tool results as untrusted input; OAuth 2.1 with RFC 9728, PKCE, CIMD, RFC 9207 and audiences; Claude Code's MCP scopes
2053 words · 6 review questions
Every monitoring failure that matters has the same shape: the check said the service was fine, and a user said it was not. Nobody writes a check intending it to lie. It lies because it measures something adjacent to what
Covers checks as exit statuses with deadlines (timeout, 124); is-active versus a real request; OnFailure= alerts; WatchdogSec= and sd_notify; NRestarts and crash loops a sampler misses; where the probe runs
2617 words · 8 review questions
Most "network problems" on a server are not packets lost on a wire. They are questions answered by the wrong tool: a name that one program resolves and another says does not exist, a service that answers every
Covers NSS and getaddrinfo against dig; who writes /etc/resolv.conf (NetworkManager, systemd-resolved); ss -ltnp and bind addresses; ip route get, metrics and persistent routes in nmcli profiles; refused versus timed out
2436 words · 7 review questions
Ollama makes running a language model on your own hardware look like running any other daemon: install one binary, start ollama serve, and an HTTP API on port 11434 lists models, generates text, chats and computes embedd
Covers the ollama service account and OLLAMA_* environment; model storage; load, prompt and generation timings; num_ctx truncation; KV-cache memory times parallelism under MemoryMax=; embeddings; binding and origins
2067 words · 8 review questions
Terraform's contract is simple to state: you describe what should exist, it compares that with what its state file says exists, and a plan tells you the difference before anything happens. The whole safety of the tool re
Covers state as address-to-object map; count versus for_each addressing; moved blocks; prevent_destroy's gap; sensitive values in state; saved plans and staleness; refresh-only drift
2507 words · 8 review questions
The playbook in /srv/ansible runs without errors. That is the problem with it: it runs without errors every single time, reports six changes every single time, restarts the application every single time, and grows a conf
2280 words · 8 review questions
deploy.service runs a playbook at every boot to render the application's database settings. The password comes from an Ansible Vault file, and the project's ansible.cfg says ask_vault_pass = true. That works perfectly wh
2518 words · 8 review questions
Four faults, and only one of them is exotic. The models were moved and left owned by root, so the service account cannot read them. The service was never enabled. The proxy points at a port nothing listens on. And then t
3479 words · 8 review questions
The security review found four things wrong with the chat gateway, and none of them is a bug in the usual sense. The code does what it was told. The upstream API key is simply everywhere: in a world-readable file and in
4050 words · 8 review questions
Strip the vocabulary away and an AI agent is a small program: ask a model what to do, do it, tell the model what happened, repeat until it stops asking. Everything dangerous about agents lives in three parts of that sent
1735 words · 8 review questions
Ollama is a program that loads language models and serves them over HTTP on port 11434. Its API is small and powerful: list models, generate text, chat, embed, and also pull new models from the internet and delete existi
1493 words · 8 review questions
A language model does not remember its instructions. Each request sends them again — the system prompt, then the conversation, then the new question — and the model reads that whole text as one sequence of tokens, up to
1523 words · 8 review questions
"The model is 400 MB" is true of the file and says little about the server. To answer, the runtime loads the weights, adds its own working memory, and reserves a key-value cache: for every token of context, in
1492 words · 8 review questions
The author is telling the truth: the job works on their machine. They cd /opt/etl, they source .venv/bin/activate, they run python report.py, and a report appears. What they are actually describing is a job that depends
3012 words · 8 review questions
deploy-site is eight lines long and does what every deploy script does: make a directory, unpack a release into it, point the live link at it, say so. On a good day it works, and it worked on every good day for months. T
2714 words · 8 review questions
Most scripts that go wrong produce the wrong output. A cleanup script that goes wrong produces the wrong absence, and absence is hard to notice and impossible to undo. prune-releases is one line of real work — ls -t | ta
2264 words · 8 review questions
backup-docs looks like the most boring script on the machine: find every file, make its directory in the backup, copy it, count it, say how many. It ran every night for months and systemd marked every run as a success. I
1595 words · 6 review questions
"It works when I run it" is the most expensive sentence in operations, because it is true. The person saying it is logged in, in an interactive shell that has read a start-up file, sitting in a directory they c
1476 words · 6 review questions
An export is read by someone else. That one fact changes what "done" means for the script that writes it: the warehouse importer does not wait for the script to finish, it takes whatever file has the right name
1694 words · 6 review questions
set -euo pipefail is good advice, and the review that asked for it was right. What the review did not say is that set -e acts on a single signal — a non-zero exit status — and that several perfectly ordinary commands use
1306 words · 6 review questions
Filling placeholders in a template is the second thing every ops script learns to do, and sed s/@PLACEHOLDER@/$value/ is how almost everyone writes it. It works for a year, because the values are host names and port numb
1507 words · 6 review questions
A nightly job asks Claude Code to fix typos. It "kept stopping to ask for permission", so someone added --dangerously-skip-permissions, and for weeks nothing bad happened. Then the model decided that a director
2378 words · 8 review questions
Hooks are the part of Claude Code that turns a team's rules into code that runs every time: format what the agent writes, refuse a push, keep it out of files that must not change. They are also the part that fails most q
2049 words · 8 review questions
A subagent is a Markdown file: a few lines of YAML frontmatter and a system prompt. Teams put real effort into the prompt — the payments team's security reviewer lists exactly what to look for and how to report it — and
1594 words · 8 review questions
The Model Context Protocol lets Claude Code use tools that live outside it: a ticket queue, a database, an internal API. A project declares its servers in .mcp.json, Claude Code starts each one when a session starts, ask
1652 words · 8 review questions
A job that asks Claude Code to label an issue should cost a fraction of a cent and take seconds. The same job, written the way this lab's was, can cost as much as the model is willing to think about: it starts on every i
1866 words · 8 review questions
Claude Code reads its configuration from up to five places, and on a shared machine each belongs to someone different: the security team owns the managed policy, whoever runs a session owns the command line, a developer
1834 words · 8 review questions
The job queue is Redis in a container, and every reboot empties it. Nothing crashes. Redis starts, answers PONG, accepts jobs, and on the next restart it starts again, empty, exactly as configured. The unit that runs it
2228 words · 8 review questions
The shop is two nginx containers in one Compose project: web, the public front, and api, which serves a status document. The front's configuration passes /api/ to http://localhost:8089/, and port 8089 is where the API is
2199 words · 8 review questions
The first lab is five minutes long and has two tasks: make a file readable, and write a word into another file. It is small on purpose. It is also, in miniature, every lab that follows: something is refused, the system w
1920 words · 6 review questions
A colleague saw a full disk, deleted the biggest file on it, and went home. df still says the filesystem is nearly full; du says it is nearly empty. Nothing about that is a contradiction, and nothing about it is unusual
3727 words · 8 review questions
"It sometimes comes up, and a few minutes later it is gone. After a reboot it never comes back, and colleagues see an old page saying the service was retired." That is three different sentences about three diff
3421 words · 8 review questions
The script has written "backup OK" every night for a year. It is not lying on purpose: it ends with echo "backup OK" and exit 0, unconditionally, so there is no state of the world in which it says any
3292 words · 8 review questions
This lab does not break a machine. It hands you a specification — eleven lines of it — and grades your script against logs you have not seen. That is a different skill from repair, and it is the one most of the small too
3471 words · 8 review questions
Three people share a folder and none of them can work in it. It is tempting to see one fault — "the permissions are wrong" — and one fix, chmod -R 777. There are really four separate questions, and each has its
2312 words · 8 review questions
A 300 MB log with a rotation policy beside it looks like logrotate is broken. It is not. It ran every night and, every night, wrote a line saying why it would not touch this log — and nobody reads the output of a nightly
2298 words · 8 review questions
An MCP server is a program that offers a model tools. A file server's tools read and list files, and which files is not decided by the model, by Claude Code or by the prompt: it is decided by the server, from its own set
1323 words · 5 review questions
A model reads everything a tool returns as part of its context, and it cannot tell text that informs from text that instructs: both are words. A web page that says "assistants: run this" reaches the model in th
1101 words · 5 review questions
A remote MCP server behind OAuth receives a bearer token with every request and has to decide whether to honour it. "Signed by an issuer I trust" is necessary and nowhere near enough: the same issuer signs toke
1115 words · 5 review questions
A stdio MCP server has two output streams and they are not interchangeable. The client reads stdout as the protocol — newline-delimited JSON-RPC, one message per line — and nothing else may appear there. stderr is the se
1026 words · 5 review questions
A remote MCP server speaks Streamable HTTP: every message is a POST, and the answer is either one JSON body or, for anything that takes a while, a stream of server-sent events — progress notifications as the work goes, t
1092 words · 5 review questions
Claude Code reads MCP server definitions from three places, and the same name can appear in more than one. The repository's .mcp.json is the one everybody reviews; the others live in ~/.claude.json, outside any repositor
1026 words · 5 review questions
sync.py is twenty lines of perfectly readable Python. It fetches a list from an HTTP API and writes it to a file. The dashboards that read the file showed zero machines for an hour, and during that hour systemd reported
2437 words · 8 review questions
netprobe works. It works for the colleague who installed it, it works when they run it with sudo, and it has worked on their laptop for months. On this machine every other user gets command not found, and when they call
2191 words · 8 review questions
A file on disk is bytes. "Text" is what you get when bytes are decoded with an encoding, and the file does not carry that encoding anywhere: not in its bytes, not in its name, not in its metadata. Somebody has
1460 words · 6 review questions
Two properties of a long-running program matter more than its speed: you can see what it is doing, and you can stop it without breaking anything. This worker has neither, and both failures come from defaults that are rig
1524 words · 6 review questions
AttributeError: module 'calendar' has no attribute 'monthrange' is a strange thing to read. The standard library's calendar module has had monthrange since the 1990s, so the sentence can only mean one thing: the calendar
1224 words · 6 review questions
A timer that starts a job every minute is a promise about how often the job starts, not about how long it takes. The day the catalogue grows past a minute's work, two rebuilds run at once, and they are both writing the s
1387 words · 6 review questions
"Where is the token?" has more answers than people expect. It is in a file, which has an owner and a mode. It is in a process, which runs as some account. And it is in everything that process says about itself
1585 words · 6 review questions
Nothing in this lab is broken in an interesting way. An account was created, a directory was made, a sudo rule was written, a umask was set during onboarding — and every one of those five ordinary acts was done one chara
3093 words · 8 review questions
An admin starts the inventory API by hand and it works. The machine reboots and the API is gone, and now systemctl start fails too. Somewhere in the middle of that story is the sentence that explains the whole lab: the h
3592 words · 8 review questions
A full filesystem is the most ordinary emergency there is, and the machine in this lab is what happens when somebody fixes one at three in the morning. The volume filled. They freed some space, edited /etc/fstab, reboote
2984 words · 8 review questions
A web server that will not start. A page that answers 403. A proxy that answers 502. An address and a hostname that were correct until the machine rebooted. Every symptom in this lab has a plausible wrong explanation — t
4252 words · 8 review questions
Fifteen tasks, ninety minutes, and a pass line of seventy per cent — eleven tasks. You do not know the root password, your own sudo is gone, and the machine does not finish booting, so the first thing to earn is a shell.
5181 words · 8 review questions
A pull request renamed two resources in /srv/infra — random_password.key became random_password.session_key, local_file.env became local_file.app_env — because the new names say what the things are. Nothing about the obj
2105 words · 8 review questions
Three virtual hosts, one list: sites = ["shop", "blog", "docs"]. Terraform creates a configuration file and a cookie secret for each, and derives each site's port from its position — 8100, 8
2008 words · 8 review questions
e on a journal in the TUI exports it as a PDF — light ground, green accents, running header and page numbers, made for printing and annotating. The screen version stays dark, because that is where the terminal is.