· about 35 minutes · runs on ubuntu-26.04-automation · unrated
An unrated lab. It runs on your machine with no account and no network, and everything about it — the faults, the checks, the hints and the reference solution — is in the repository. An attempt on it is recorded on your profile and never moves a rating: only rated labs do. Rated and unrated labs.
The billing team's support assistant is a custom Ollama model, support-bot, built from
/srv/support-bot/Modelfile on top of qwen2.5:0.5b. Its system prompt lists thirty-nine rules the
assistant must follow — above all, never promise a refund date. Customers reach it through
/opt/support-chat/ask, a small script the web chat calls with the customer's question.
Support leads say the assistant ignores its rules: it promises dates, its tone changes from one answer to the next, and asking the same question twice gives two different stories. Last week someone read that the context window might be too small and raised it in the Modelfile. Nothing changed. The rules themselves are correct.
A small model will never follow rules perfectly, and this lab does not grade what it says. It grades whether the model is given its rules at all.
What is expected, and graded — the grader sends a customer question and reads what Ollama reports about the prompt it evaluated:
support-bot evaluates its whole system prompt, not a truncated piece of it./opt/support-chat/ask.support-bot samples with a temperature of 0.3 or less./srv/support-bot/Modelfile sets the same parameters and the same rules the server holds, so
rebuilding from it gives the same model.You have root through sudo. Everything must still hold after a reboot.
The machine is checked, rebooted, and checked again. A check passes only if it passes both times.
| Check | Objective |
|---|---|
| 01_the_model_sees_all_its_rules | Give a model a context window its instructions fit in, and keep clients from shrinking it |
| 02_the_chat_app_keeps_the_whole_prompt | Give a model a context window its instructions fit in, and keep clients from shrinking it |
| 03_answers_are_steady | Build a custom model from a Modelfile and read back what the server really holds |
| 04_the_modelfile_is_the_source | Keep the Modelfile the source of truth |
Where the lab's hints send you, level by level, as you ask for them (h, then l opens a journal section in the TUI).
4 questions on the same topic, in the lab's Theory tab. They never affect the lab's grade. Three of them, to answer here:
A Modelfile's PARAMETER num_ctx is changed from 256 to 2048, and support-bot still shows num_ctx 256. What was missed?
Rebuilding the model with ollama create support-bot -f Modelfile; the server does not watch the file
ollama create reads the Modelfile once and stores the result as a model on the server. Editing the file changes nothing until the model is created again; `ollama show support-bot --modelfile` prints what the server actually holds.
https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx
A 1,388-token prompt is sent to a model whose num_ctx is 256. In Ollama 0.34 the response says prompt_eval_count 130. What happened?
The prompt did not fit, so Ollama truncated it — keeping a few tokens from the start and the end, most of the system prompt dropped
The server logs "truncating input prompt limit=130 prompt=1388 keep=4 new=130": a prompt longer than the window is cut to about half of it, keeping the first few tokens and the most recent text. The model answers without most of its instructions, and nothing in the reply says so; prompt_eval_count does.
https://github.com/ollama/ollama/blob/main/docs/faq.mdx · https://github.com/ollama/ollama/blob/main/docs/api.md
A model's Modelfile sets num_ctx 4096, and a client sends "options": {"num_ctx": 512} with its request. Which window does that request use?
512: options in a request override the model's parameters for that request
Request options take precedence over the parameters stored with the model, which take precedence over the server's defaults. A client that "makes answers faster" with a small num_ctx silently undoes the model's configuration.
https://github.com/ollama/ollama/blob/main/docs/api.md