Testing Longer Contexts
Part 6 of Oracles, Traces, Triage.
In the previous article, I described getting local models running on the two Macs. A short prompt checked that an installation could generate an answer. The agent experiments from earlier in this series need more than that: the model may have to read source files, a piece of a specification, and the output of several failed test runs before deciding what to try next.
All of that takes space in the context window, the input and output the model can work with in one request. That space is measured in tokens, which are pieces of text rather than necessarily whole words. The code and conversation history use the same budget as the answer.
When I looked at the initial settings, two DwarfStar profiles were limited to 8,192 tokens. Several Orca profiles had no explicit cap in the manager. I wanted to see how much further I could take them on these machines, and to have a predictable failure when a request was too large.
The 8K settings were defaults, not measurements of what the models could handle. Nor could I read an upstream context limit and assume the local installation would manage it. The runtime’s memory use and handling of long requests needed testing with the exact configurations I intended to keep.
A cap needs a test on both sides
The procedure started with a small generation and increased the input in stages. The model had to retrieve specific records from synthetic text, then handle a follow-up asking for those records in a different order. An oversized request had to be rejected. A final small probe checked that the server remained usable afterwards.
For each configuration I checked that:
- The server reports the expected model and context configuration.
- The request contains the measured number of tokens after formatting.
- The answer contains the expected records.
- A follow-up preserves the required information.
- An over-limit request fails instead of silently discarding input.
- The server can still generate after that rejection.
The request and the generated answer share a context budget. A 200,000
token cap is not an invitation to send 200,000 input tokens and then ask
for a long answer. The template and any other inserted tokens count too.
The server’s default output budget is a separate setting; clients should
send max_tokens explicitly.
I precomputed the test inputs before loading the large engine. Earlier, running an auxiliary tokenizer diagnostic alongside a resident model had caused memory pressure severe enough to interfere with SSH access. Moving that preparation out of the inference run avoided making the diagnostic compete with the model for memory.
What passed on the smaller Mac
The following is the accepted M5 Max, 128 GB result. The figures come from the recorded qualification summaries, not from model-card maxima.
| Collection | Model | Context cap | Largest tested input |
|---|---|---|---|
| DwarfStar | DeepSeek V4 Flash 0731 | 200,000 | 197,925 |
| DwarfStar | DeepSeek V4.1 Flash 1017 | 200,000 | 197,906 |
| DwarfStar | GLM 5.3 Flash | 200,000 | 197,918 |
| DwarfStar | Qwen3.8 Flash Next | 200,000 | 197,847 |
| Orca | Qwen3.8 27B | 131,072 | 129,174 |
| Orca | Nex-N2.5-mini | 131,072 | 129,141 |
| Orca | OrcaSAQ-2 Cyber 27B | 131,072 | 129,283 |
| Orca | GLM 5.3 Flash | 131,072 | 129,019 |
Each accepted M5 run used one engine and one sequence or session. The sampled memory-pressure level remained normal. Some runs began with swap already allocated, so I recorded the starting and peak values rather than describing the machine as swap-free.
All eight passed the synthetic retrieval, follow-up, rejection, and recovery checks at these sizes. I would still need real repository tasks to judge whether the extra context helps with an investigation. Finding records in generated text is a much narrower task than understanding the interactions in a large codebase. The separate image smoke tests also leave long-context multimodal use untested.
The second request can be a different workload
The largest Orca requests produced the following elapsed times, including their short answers:
| Model | Initial request (seconds) | Follow-up (seconds) |
|---|---|---|
| Qwen3.8 27B | 447.7 | 454.0 |
| Nex-N2.5-mini | 138.0 | 139.6 |
| OrcaSAQ-2 Cyber 27B | 869.7 | 4.6 |
| GLM 5.3 Flash | 3,208.8 | 4.6 |
The follow-up sent the conversation again and asked for the records in reverse order. The selected MLX serving path recomputed the conversation; the native serving paths reused much more state. A four-second follow-up therefore tells a very different story from a four-second cold request.
Background copies and downloads were active during qualification. These are observations from the recorded workloads, not isolated throughput benchmarks. They should not be used to rank the underlying model families.
The DwarfStar DeepSeek V4.1 streaming configuration made the same point: its largest input took about thirty minutes, followed by a cached request of about five seconds. Whether that is acceptable depends on the job. A long unattended investigation and an interactive editing session have different tolerances.
What I can use these results for
The caps now give the agent client a budget to work within, and the rejection tests give me a way to catch regressions in how the server enforces it. The timings will also affect how I construct requests. A follow-up that reuses earlier computation and one that processes the whole conversation again can have very different costs.
There is still a quality comparison to do. On the M5, smaller quantizations made some of these configurations practical, but the retrieval checks don’t tell me how much reasoning or coding ability I lost. For that I need the same tasks, prompts, tool budgets, and scoring for each candidate. I will keep those results separate from the context qualification table.
For now, I have measured input limits and a repeatable procedure for checking them. Before connecting an agent to these services, I also need to decide what it will be allowed to do with their responses. That became a concrete concern while choosing community-modified models, and is the subject of the next article.
Retrospective entry for October 21. Assembled October 24; the M5 measurements above are from October 23.