How Norboten uses Claude

Claude does its jobs in three places, under three kinds of credentials, and none of them is a model API key on the server. None of them grades a lab: a lab is graded by check scripts reading the machine, and nothing a model says changes a result.

The server calls no hosted model: its one agent, the consultant, runs on its own Ollama, and nothing bills per token. A subscription only ever serves the person it belongs to.

Every Claude Code flag and rule below was checked against the official documentation and, where it matters, run locally; the notes are in the repository at docs/research/claude-code.md.

1. The product's agents

The tutor and the review are in cli/src/norboten/tutor/, the question pipeline in cli/src/norboten/questions/, the consultant in api/src/norboten_api/agents/. Each is a single structured call through norboten.questions.providers — Claude Code headless (no tools, an empty working directory), or the Messages API, OpenAI, Gemini and Ollama over REST, with no vendor SDK — answered into a Pydantic schema. The server's config allows Ollama alone. On a learner's machine tutor/models.py picks the model: Claude Code (sonnet, haiku, opus), then a key (claude-sonnet-5, gpt-5-codex, gemini-3-pro), then what a local Ollama has pulled (qwen2.5:1.5b first) — or the one pinned with m on System.

Every agent is optional. With no model on the learner's machine the tutor falls back to the hint ladder and the review says what would enable it; without Ollama on the server the consultant falls back to the browser-side ranking.

2. Claude Code in CI

Four jobs run claude -p — the table and the workflows are in Architecture → Automation. What they share:

• The model only reads. Triage, the stuck-point summary and release notes run with --tools "" in an empty directory: no shell, no files, no project hooks or MCP servers. The job's instructions are the system prompt (--system-prompt), and the text a stranger wrote — an issue, a commit message — arrives on stdin as data. Triage's answer is held to a JSON-schema enum.
• One job writes, and only one file. lab-author.yml runs Claude Code as the project subagent .claude/agents/lab-author.md in a checkout: --tools Read,Glob,Grep,Write, writes allowed only under labs/_drafts/ (--allowedTools "Edit(/labs/_drafts/**)" with --permission-mode dontAsk, which denies everything else instead of waiting for a person), 20 turns, $1. Then the script checks that exactly one draft appeared before it commits, pushes and opens a draft pull request. The subagent's draft is a starting point; the solvability gate and a review decide.
• The model holds no GitHub token. Labels, issues, branches and pull requests are made by the job's Python, from the model's constrained answer.
• Never --bare. Bare mode is the recommended shape for scripts, but it ignores CLAUDE_CODE_OAUTH_TOKEN. So the lab-author job loads the checkout's .claude/ and CLAUDE.md — which makes that configuration part of what CI runs, reviewed like code, and kept short.
• Pinned and bounded. One Claude Code version (.github/actions/claude-code, the version the rehearsal ran), timeout-minutes and a concurrency group on every job, no background updates.

3. Producing content with Claude Code

Norboten's content — labs, theory, journals, screenshots — has a bar that a model alone does not meet: a lab must pass the gate, a question must survive a blind solve, a journal's walkthrough must have run on a real machine. Claude Code works inside that bar through the norboten-author plugin, which this repository publishes as its own marketplace and enables for its checkout (The Claude Code plugin). Its skills are started by a person (disable-model-invocation); /norboten-author:idea is the way in — it asks what the specs need, plans, and hands over to one of these, in the session or to a subagent:

Questions on a subscription. The pipeline reaches Claude through the local Claude Code when a model id is claude-code/<model>: norboten dev draft-questions <topic> has Opus write, Sonnet and Haiku answer without the key, and Sonnet review — each a headless run with no tools, an empty directory and the schema as --json-schema. Models of one family share blind spots; with OPENAI_API_KEY or GEMINI_API_KEY exported, a mixed set of verifiers is the stronger check. Survivors go to quizzes/_drafts/, never straight into a bank.

Contributor tooling. Like an editor config, and not needed to contribute: a short CLAUDE.md (layout, commands, traps) and .claude/settings.json, whose hook formats an edited Python file with ruff and lints the lab an edited file belongs to, and whose deny rules keep git push and the ignored secrets (.env, the Ansible vault, Terraform state and variables, keys) out of reach.

4. What it costs

Where the spend is bounded.

Request sizes are the bytes Claude Code sent, recorded by the scripted API in the rehearsal (2.1.270). The no-tool jobs used to send Claude Code's default system prompt as well — 27 KB of instructions for tools they do not have — until their own instructions replaced it (--system-prompt): triage's request went from 29.4 KB to 2.6 KB.

The levers, in order of effect:

1. No model where a fact will do. Lab health reads the gate's job results, announcements read the lab.yaml files a release adds, the learner digest is a template, duplicates are a title comparison. Four jobs of eight use a model.
2. Events, not schedules. Three Claude jobs run when something happens; one runs weekly.
3. The smallest model that does the job. Haiku for labelling and summaries; Sonnet only where the model has to read the repository and write.
4. Hard caps. --max-turns everywhere, --max-budget-usd on the one agentic job, timeout-minutes on every workflow.
5. No thinking where there is nothing to think about. Runs without tools set MAX_THINKING_TOKENS=0 (section 5: release notes 111 s → 9 s, $0.070 → $0.021).
6. Small context. No tool schemas or default system prompt for no-tool runs; a CLAUDE.md kept under 80 lines (a test holds it there), because the lab-author job reads it on every turn. Workflows live in skills, which load only when run.

Caching. Claude Code caches the prompt prefix on its own. Issue events are sporadic, so the jobs mostly start cold; the savings that are certain are the ones above, which shrink the prefix itself.

Subscription versus API. CI's Claude Code jobs and question drafting use a subscription token: usage counts against the plan rather than an invoice, and total_cost_usd in each run's summary is Claude Code's list-price estimate, not a bill. The server uses no API key; Ollama answers its agents at no per-question cost. A learner who drafts on keys pays their own vendor.

5. Real runs

Measured on 2026-09-14 with Claude Code 2.1.270 on a subscription login, the real model for every call and stand-ins for everything else (GitHub, Discord and Telegram were a recording sink). Cost is Claude Code's list-price estimate (total_cost_usd), not a bill; wall time includes Claude Code's start.

What the runs changed:

• Thinking off for no-tool jobs. Haiku spent most of its output thinking about a one-field label or a summary of commit subjects. --effort low did not reduce it (four triage runs: 275–445 output tokens, no fewer than without it). MAX_THINKING_TOKENS=0 did: automation/jobs/common.py now sets it for every run without tools, and release notes dropped from 111 s to 9 s.
• The question generator was told its limits. The first two Opus drafts were both rejected by the spec — prompts over 600 characters, each repeating its config snippet inside the prompt. The draft schema now carries the same length limits as the question model, and the rules say a snippet goes in code once. The next two were both accepted: two blind solvers chose the key, the critic found nothing.
• Token counts include the cache. A job's summary line counted only uncached input (14 tokens for the lab author); it now adds cache creation and cache reads.

How good the output was, judged by hand:

• The drafted questions (systemd timers: a started-but-not-enabled timer, and Persistent= on a monotonic timer) are exam-grade scenarios with explanations for every wrong choice. The critic still let through code_lang: yaml for a unit file, and one explanation says the service "must not be enabled", which is stronger than true. This is why drafts go to quizzes/_drafts/ and a person moves them.
• The lab draft is a sound starting point but not a lab: it tagged the lab networking, which it does not exercise, and listed Alpine as a base image for a mail relay no Alpine image has.
• The release notes summarise what the commits say, including an endpoint added and removed again within the same release — the complete commit list the script appends is what stays true.
• Triage held against the injection; bug and question are both defensible for "the site is slow" written inside an injection attempt.

6. How it is tested without spending a token

• make jobs-rehearsal runs every CI job as its workflow does, with the real claude binary, against automation/stand_ins/fake_anthropic.py (a scripted Messages API) and sink.py (GitHub, Discord, Telegram, the API). The scripted model also attempts what it must not — writing README.md, running Bash, writing a second file, all refused — and the rehearsal checks each job's instructions reach the model and its request stays small.
• tests/test_claude_settings.py runs the real claude in a scratch copy of the project: the hook formats a Python file, a broken lab.yaml sends the lint failure back to the model, and reading deploy/.env and git push are denied even with every tool allowed on the command line. Each skill is expanded with its arguments.
• api/tests/test_claude_code_provider.py runs the question pipeline through the real claude: the blind solvers never see the key, a repeated question is a duplicate.