After the placement runs I had a practical question left: which models should each Mac load? I wanted both DeepSeek releases in use somewhere, and on the two Mac Studios a small model beside the large one for quick questions. I also wanted prompts closer to my own work than arithmetic and string functions.

Four models

DwarfStar served DeepSeek V4 Flash 0731 and DeepSeek V4.1 Flash 1017. The Llama app served Qwen3.6 27B and Qwen3.8 27B at Q8_0. The Mac Studios stood apart, the laptop had its lid open, and the thermal level was logged. The laptop ran only the two DeepSeeks.

The prompt set is the 40 graded prompts from the first post, plus 13 Ethereum fact questions with one fixed answer each, such as the number of slots in an epoch or the selector of transfer(address,uint256).

The newer DeepSeek release wasn’t better on these prompts. It is larger, 366 GB on the Mac Studios against 153 GB, and slower. On the laptop it doesn’t fit in memory, so DwarfStar streams part of it from disk, and it answered at under half the speed of the older release.

A 27B model matched the large ones on the generic prompts. I read that as a statement about the prompts: they are too easy to rank these models. It doesn’t show the models are equal. Qwen3.8 lost three fact questions by writing past the 256-token limit before reaching its answer line; the facts in its text were right.

The grader was wrong again as well. It wanted a number and marked EIP-155 as a miss, which cost DeepSeek 0731 two fact prompts until I read the failures. The table is after that fix.

Choosing per machine

Score didn’t decide anything, so the choice came from speed, disk and memory.

I checked that the pairing works on the M5 Ultra. With DeepSeek loaded in DwarfStar and Qwen3.8 in the Llama app, 34% of memory stayed free, swap stayed at 4 MB, and each answered at its usual speed: 56 and 44 tokens per second with the older release, 52 and 46 with the newer. That was a short prompt to each, one at a time. I didn’t test both answering at once, or long conversations in both, and I didn’t repeat it on the M3 Ultra. The order matters: DwarfStar’s manager refuses to start while another engine holds a model, so it goes first.

The heat log earned its place here. Standing apart, the M3 Ultra spent 57% of this run at Moderate, all of it during the two Qwen runs in the Llama app. The M5 Ultra spent 41% at Heavy, under the Qwens and also under DeepSeek V4.1 in DwarfStar. In the previous post the M3 Ultra had stayed Nominal apart, and DwarfStar had never taken the M5 Ultra past Moderate; that was a different workload.

A domain test

For prompts nearer my work, I built 40 from a private set of security reports against Ethereum client software. Each prompt is one report, and the task is to classify its severity as the triage team did: Informational, Low, Medium or High, ten of each. Reports that named a severity in their own text were left out, so the label couldn’t be read off the page. The reports are private and none of their content is published here.

The first version gave the models no scale. On the M5 Ultra, where I tabulated the answers, every model called most reports High and none answered Informational once. Reports argue their own impact, and the models believed them. Scores ran from 11 to 14 of 40, where guessing gets 10.

The second version put the public severity scale from the Ethereum bug bounty page into the prompt, with its network-share thresholds, and a note that reporters tend to overstate impact. The models then swung the other way and under-rated.

On the M5 Ultra the four models gave the same answer on 2 of the 40 reports. The same DeepSeek file at temperature 0 scored 15 on one Mac Studio and 12 on the other; I haven’t investigated why the answers differ between machines.

The control

The obvious objection is that these are small local models. So the same 53 Ethereum prompts went to Claude Fable 5.2, each in a fresh session with no tools and a one-line system prompt. Unlike the local models it was allowed to think before answering.

It got 13 of 13 facts and 13 of 40 on the triage prompts. It made the same mistake, calling 14 of the 30 reports above Informational “Informational”, and got one of the ten High reports right.

So the test is broken, and the models are not what it measured. A report’s text and the public scale are not enough to recover the label a triage team assigned, whoever reads them. My guess is that the team uses things a report doesn’t say, such as what was already known and how hard the attack is in practice. I haven’t tested that.

What I take from it

• An oracle that is a human judgement measures agreement with that judgement. Here there was none to measure, and without the control I might have blamed the models.
• Run a strong baseline through a new benchmark before reading anything into the scores. It cost a few minutes.
• The fact questions worked and were too easy. Everyone scored 10 or more of 13.
• Read the failures. This series found a grader bug in every round.

The next attempt needs a task whose answer is in the text or in checkable knowledge: which component a report concerns, with the obvious names removed, or whether a described behaviour violates the specification. That is the remaining entry in the series index, and it is closer to the question this blog keeps returning to: choosing oracles that are real.

Entry dated November 12; the measurements were taken the same day.

The arrangement this post takes apart: Studios stacked, laptop lid closed.

The previous post ended with timings I no longer trusted. The two Mac Studios were stacked, the M3 Ultra underneath and the M5 Ultra on top. During a long session on the M5 Ultra I ran this by hand:

sudo powermetrics --samplers thermal

It reported a thermal pressure level of Moderate, then Heavy, and went back to Nominal about 40 seconds after the load stopped. Thermal pressure is the level macOS reports as it starts limiting the chip to control heat: Nominal, Moderate, Heavy, Critical. The M3 Ultra had worked for hours at the bottom of that stack and nobody had sampled it. Every speed I had written down was now a number without its conditions.

So the runs were repeated with the level logged, and the machines were moved.

Giving the agent one command

powermetrics needs root, and without it macOS reports nothing useful (pmset -g therm printed no level). The runs were driven by Claude Code over SSH, and I didn’t want to hand it the administrator password. I ran this once on each Mac instead, replacing USER with the account name:

echo 'USER ALL=(root) NOPASSWD: /usr/bin/powermetrics' \
    | sudo tee /etc/sudoers.d/powermetrics
sudo chmod 440 /etc/sudoers.d/powermetrics

That lets one account run one program as root without a password. The harness could then log a reading every five seconds beside each run:

sudo -n powermetrics --samplers thermal -i 5000 \
    | grep --line-buffered -E 'Sampled system activity|pressure level' \
    > thermal.log &

This is the narrow version of the access question from Local Models and Tool Permissions: one read-only tool, in place of a password or general sudo. Two caveats belong with it. powermetrics can report more than temperature, including per-process energy use, so the rule grants a read of system activity. And the logger runs as root, so the harness can’t kill it directly; it ends the grep that reads from it, and powermetrics exits by itself. Removing the file undoes the rule.

The runs

Each run is the same 45 prompts per server from the previous post. The table gives the share of five-second samples at each level, and tokens per second for GLM 5.3 Flash and DeepSeek V4 Flash 0731 under DwarfStar.

The first M5 Ultra row covers only its DwarfStar half, because logging started after its Llama half had run. The M5 Max is a MacBook Pro and ran DwarfStar only.

What the first round said

After the side-by-side runs I wrote down a tidy story. Stacking made the Mac Studios warmer and no slower: the M3 Ultra went from Moderate in 32% of samples to Nominal throughout, and its tokens per second didn’t change in the first decimal on any of the four servers. The M5 Ultra’s DwarfStar half went from Moderate to Nominal with 33.3 and 55.8 against 33.8 and 56.6.

The Heavy reading that started all this was not the stack either. The M5 Ultra reached Heavy only while the Llama app read a very long prompt, for two to four minutes at a time, and it did so standing apart as well. DwarfStar, with these two models, never pushed it past Moderate.

The laptop was the machine that heat limited. With the lid closed it sat at Heavy for 96% of the run. Opening the lid, still flat on the desk, moved that only to 91%, yet the run finished in 47 minutes 41 seconds against 52 minutes, and tokens per second rose by about 10%.

Putting it back

One run per placement is an anecdote. The M3 Ultra’s side-by-side run had also started from a rested machine, while its stacked run followed hours of other work, so “apart is cooler” and “rested is cooler” were the same observation. I restacked the Mac Studios, closed the laptop lid, waited for all three to read Nominal, and started the identical runs at the same moment.

The laptop result came back: 52 minutes 52 seconds closed, and tokens per second fell to the closed figures. Two closed runs now agree against one open run.

The M5 Ultra on top was warmer stacked, as before. Its DwarfStar half read Moderate for four and a half minutes where it had read Nominal apart, and Heavy rose from 12% of the run to 20%. Its speed didn’t change.

The M3 Ultra’s Moderate did not come back. At the bottom of the stack, with the M5 Ultra working above it, it stayed Nominal for all 679 samples. The first reading had followed hours of long-input tests, so it was more likely stored heat than the stack. My tidy story was wrong about the machine it started with.

What I take from it

• Speed repeated and heat didn’t. Tokens per second agreed to within half a token across stacked, apart and stacked again on both Mac Studios. The thermal shares depended on what the machine had done in the hours before.
• The thermal level is a coarse signal. Moderate cost nothing I could measure on the desktops, while 91% against 96% Heavy hid a 10% speed difference on the laptop. I now log the level and the speed together.
• The long-input timing I had worried about on the M3 Ultra held up. One repeat, apart and Nominal throughout, read the same 194,819 tokens in 283 seconds against 279 stacked. I repeated only that one of seven.
• Scores didn’t move with heat, placement or machine.

For placement, the laptop runs with its lid open. The Mac Studios can be stacked as far as speed goes; side by side keeps the upper one cooler, and I keep them apart when there is room.

Some things I didn’t measure. I logged the pressure level, without fan speed or temperature in degrees, so I can’t say what the extra heat costs in noise or wear. I didn’t try the stack the other way up, or the laptop on a stand; I expect a stand with the lid open to be no worse than flat, and that is an expectation. And each placement has at most two runs.

A benchmark number without the machine’s thermal state and recent history is incomplete. I had been publishing mine that way. The next post uses the same harness, with the log running, to choose which models each Mac loads.

Next: Four Models and a Test That Measured Nothing

I keep the same model twice on one Mac. DwarfStar serves it from a higher-precision file, and the Llama app, a llama.cpp build with a model router, serves it from a smaller one published by ggml-org. The larger files cost hundreds of gigabytes of disk, and I had assumed they bought better answers. In an earlier post I wrote that I hadn’t done the paired tests needed to say so. This is that test.

The question is narrow on purpose: on the same Mac, with the same prompts, does the higher-precision file get more right?

What was compared

The first runs were on the M3 Ultra Mac Studio with 256 GB of memory, one server loaded at a time.

Those names say how many bits each weight is stored in. A smaller number is a smaller and coarser file: Q2_K keeps about two bits per weight, Q4_K about four, and MXFP4 is a four-bit floating-point format.

Each server received 45 prompts. Forty have an answer a program can check:

• 12 coding tasks, graded by running the returned function against tests in a separate process.
• 14 arithmetic and logic questions, with the answer on a marked last line.
• 6 tasks that must return strict JSON, compared after parsing.
• 8 lookups in a synthetic order ledger, four in about 25,000 tokens of input and four in about 85,000.

The other five are open questions, saved for timing and not graded. Requests used temperature 0 with thinking switched off, and the same request body went to both servers. The expected answers are computed by the harness, and every coding test must pass against a reference solution and fail against an empty one before it is used. The prompts and the grader are published with the series.

One prompt apart

By group, in the same order:

code           11/12  10/12   9/12  10/12
math           11/14  12/14  13/14  14/14
json            5/6    6/6    6/6    6/6
ledger, 25k     4/4    4/4    3/4    2/4
ledger, 85k     4/4    4/4    3/4    3/4

For both models the smaller file scored one prompt higher. With 40 prompts and one run, one prompt is noise. The reading I can defend is that no difference showed, and I can’t claim the smaller file is better.

I expected a gap for GLM, where the Llama file has half the bits. There wasn’t one.

Failures followed the model

The misses clustered by model, and the file made little difference. DeepSeek missed the same four prompts under both engines: two “which city is on the line after order X” lookups, a topological sort, and a Base64 encoder that got the padding wrong on a two-byte input. Both GLM files got the sum of the primes below 60 wrong, with 672 and 769 where the answer is 440. Most of the other misses were arithmetic done without thinking, such as counting the 1 bits in 2026.

DeepSeek returned byte-identical text on 24 of the 45 prompts across the two engines and files. GLM did on 7.

The same four servers also ran on the M5 Ultra Mac Studio, which takes the same files, and scored 35, 36, 35 and 35. The M5 Max MacBook Pro has half the memory and takes smaller DwarfStar files: GLM at Q2 scored 35, and DeepSeek with most experts at Q2 scored

1. Across the three Macs, every run of these two models landed between 34 and 37 on these 40 prompts.

The grader had bugs too

A checker needs checking. I read every failure by hand, and three of them were about the test and not the model.

One coding test rejected any answer whose source contained the word “calendar”, to forbid import calendar. DeepSeek wrote a comment about the Gregorian calendar and failed a correct solution under both engines. The test now looks for the import. The table above is after that fix, which moved both DeepSeek scores up by one.

One lookup answer was the right order ID without the required ANSWER: prefix. I left it as a miss, because the format was part of the task. A looser grader would have passed it.

“Thinking off” also had to be verified. The Llama app’s GLM kept reasoning with the usual switches and stopped only when the request carried thinking_budget_tokens: 0:

body = {'model': model, 'temperature': 0, 'seed': 1,
        'max_tokens': max_tokens, 'reasoning_effort': 'none',
        'think': False, 'thinking': {'type': 'disabled'},
        'chat_template_kwargs': {'enable_thinking': False},
        'thinking_budget_tokens': 0,
        'messages': [{'role': 'user', 'content': prompt}]}

The harness records the length of any reasoning a server returns, and the final runs show zero for all of them.

Speed was a different question

DwarfStar generated DeepSeek at 36.2 tokens per second on the M3 Ultra against 25.9 for the Llama app. For GLM the order reversed, 22.8 against 26.4. Reading the 85,000-token ledger for the first time took between 224 and 317 seconds on every server. Three of the four reused their cache for the next three questions, at 7 to 31 seconds each. DwarfStar with GLM read the ledger again every time, about 300 seconds per question.

When I collected those figures, the two Mac Studios were stacked, and one of them turned out to be running hot. That put every timing in doubt, which is the next post.

What this does and doesn’t show

• It is one run per server and 40 prompts. A gap of two or three answers would still be noise. The test could have shown a large difference and didn’t; it can’t show there is none.
• It compares two stacks as well as two files. The engines differ in chat template and sampling code, so an effect could come from either.
• The prompts are short tasks with one right answer. Nothing here covers long agent sessions, tool use, or the quality of prose.
• The prompts were written by the assistant that ran the test. They are not drawn from my own work.
• With thinking on, the arithmetic misses would probably shrink for every file. I didn’t measure that.

I haven’t deleted the larger files. What changed is my reason for keeping them: I can no longer say it is answer quality, so the choice between the two servers comes down to speed, context handling and convenience.

Next: Where the Machines Sit

In the next couple of posts, I’m going to send the same prompts to the same models on three Macs and write down what the numbers say. Follow along and you’ll:

• see whether a higher-precision model file gets more answers right than a smaller one of the same model
• see what stacking two Mac Studios, or closing a laptop lid, does to a benchmark
• see a test that looked reasonable and measured nothing, and the control that showed it

This picks up where Restoring the Models on a Clean Mac ended: the models install and restore, but I had not compared them on anything. The machines are an M3 Ultra Mac Studio with 256 GB of unified memory, an M5 Max MacBook Pro with 128 GB, both from the earlier posts, and an M5 Ultra Mac Studio with 256 GB that joined them since. The runs were driven by Claude Code with Claude Fable 5.2 over SSH; it also wrote the prompts and the grader, which is a limit I come back to.

Here’s a list of posts to follow:

• Higher Precision, Same Answers
• Where the Machines Sit
• Four Models and a Test That Measured Nothing
• A domain task that can be graded (to come)

Prompts, grader and scores: prompts.py, compare.py, results.json

By this point I had separate model configurations for the two Macs, measured context limits, and several runtime builds to keep track of. The previous post also covered the access settings I wanted to manage separately. I needed a way to preserve that setup without reconstructing it from terminal history.

This matters for the agent experiments that started the series. If I change a model or its settings and get different results, I want to be able to return to the earlier configuration. I also want to be able to reset a Mac without losing the work needed to make those models run.

I put together an offline recovery library on an external SSD, including the model files, pinned source, Python wheels, native runtimes, and Apple toolchain material. Then I reset the M5 and restored from it. I handled the administrator and license prompts myself. The installer finished with:

RESTORED ALL EIGHT PROFILES. No persistent servers started.

I then started each model, checked its status, requested a generation, and stopped it before moving to the next one. All eight passed. That was the first full test of this M5 library on a factory-reset machine.

What the clean restore tested

Before the reset, the qualification records said that offline component restores had passed over independently provisioned files. They explicitly did not claim a factory-reset restore.

The reset removed that uncertainty about the existing installation. It was still a narrower inference test than the earlier qualification: I ran generation smoke tests for all eight models, without repeating the long-input or separate image tests. The evidence summary records which checks ran before and after the reset.

The restore also exercised a dependency that would have been easy to miss on a working Mac. Xcode setup initially couldn’t find the Metal compiler. The installer then imported the archived Metal Toolchain 17F109, checked the compiler, and continued. Keeping the model files and source alone would have left that part of the setup unfinished.

One repository, machine-specific kits

The storage layout settled into this shape:

LLM/
  .git/
  restore-all.sh
  machines/
    m3-ultra-256gb/
      DwarfStar/
      Orca/
    m5-max-128gb/
      DwarfStar/
      Orca/
      optional-models/
    m5-ultra-256gb/       # Reserved; not qualified.

Git lives at LLM/, above the machine directories. It tracks the small material that explains the installation: scripts, configuration, manifests, and reviewed records. The large payloads remain outside Git history.

The model files need their own backup and checksums: recovering the Git repository alone wouldn’t recover those ignored files. Keeping them outside Git lets me review changes to the scripts and configuration without putting hundreds of gigabytes of weights into each kit’s history.

The installed locations under ~/Downloads stayed separate from the SSD layout. The managers run from internal storage, including the streaming model’s installed files. After a successful restore, the recovery SSD can be ejected without taking the service away.

Each kit retained its own required assets. I avoided cross-kit links that would make one machine’s recovery depend on another machine’s directory. The M5 work also preserved the M3 engine pins and existing manager interface. A cosmetic cleanup was not a sufficient reason to disturb a working installation.

Eight accepted models

The M3 had nine profiles. The M5 finished with eight.

An Orca DeepSeek conversion was still incomplete. Cancelling it meant stopping its supervisor as well as the conversion process, after checking their identities. Otherwise the supervisor could start a later step after the visible process disappeared.

I retained the verified original MXFP4 shards as optional future-work assets. The incomplete output and redundant internal download copies were removed only after the SSD originals had been verified.

I also removed that profile from the installer, manifests, manager help, and model tables, and renamed the M5 Orca kit to reflect its four working models. The retained shards are explicitly optional. Restoring the eight accepted models doesn’t depend on finishing that conversion.

Copying the library

I copied the library to the other SSD and used an M2 Mac to inspect both drives. The M2 was a verification host, not a newly qualified inference target.

Both copies had the same Git revision and contained 4,082 filesystem entries and 3,593 regular files. Names, sizes, permissions, and symlink targets matched. The copied Git database passed its integrity check.

I initially started a full content hash. That reads every byte on both drives: about 4.9 TB for these two logical copies. The first implementation was CPU-bound. Switching the large-file hashing to the built-in OpenSSL implementation raised observed disk reads to about 1.5 GB/s per drive.

I then chose the shorter verification scope. It hashed 3,400 small files on each drive, checked all 617 tracked files against Git, and checked the available small-file manifest entries. Those checks passed. The 193 large files on each drive were not all fully rehashed in that comparison.

The quick comparison therefore verified the copied structure, Git state, and small-file contents. It left the large model contents unverified in that copy. Their earlier checksums don’t prove that every byte survived the later copy unchanged.

The copy-check summary records the exact scope. Model weights may be downloadable again from their upstream repositories, but that depends on continued availability and access. It is still worth keeping the archives that already restored successfully.

What I kept separate from restore

The restore does not add firewall exceptions, change the router, configure Tailscale, or leave persistent servers running. Those operations change how the machine is exposed and when it consumes resources. They belong to explicit operating steps.

Likewise, synchronization after independent work needs more care than copying a folder over another folder. The two Git histories must be preserved and reconciled, while payload differences are inspected and verified separately. Replacing .git is not a merge strategy.

The command reference contains the restore and lifecycle commands for each accepted machine. It also calls out the older M3 single-model manager’s different argument syntax.

That finishes the installation work I needed before returning to the agent experiments. I can restore the accepted configurations and check that they generate, and I have records of the longer inputs they handled. I still need to test them on the work described back in February: reading specifications, proposing useful tests, and helping triage the failures. Those results will determine which of these models I keep using.

Reference: Local inference operations

After getting the local models running, I tested their input limits. Those checks were requests to an inference server: text went in and an answer came back. The testing agents from the earlier parts of this series would do more with that answer. They would use it to decide which files to read, which commands to run, and what to investigate next.

While choosing the models, I kept coming back to a concern about the community-modified weights. Could a model emit a tool call asking my agent to scan the drive and upload what it found? Would using an original upstream model, or a different runtime, make that safe?

Running inference locally keeps the prompt processing on my machine. It doesn’t stop a model from proposing an upload, and it doesn’t stop an agent with broad permissions from carrying one out. That applies to an upstream model too. I need to consider both the software I’m loading and the permissions of the program acting on its output.

For the repository work I’ve described in this series, access to a compiler, a checkout, and test results is useful. Access to unrelated files in my home directory usually isn’t. The local installation is a good point to make those choices explicit, before attaching the tools.

Start with what executes

A model installation contains more than weights. It can include model implementation code, tokenizer and processor code, a chat template, native libraries, a Python environment, and an HTTP server. Some of these execute directly. Others supply data to code that executes.

Custom model code can have an ordinary purpose: implementing an architecture that the installed library does not support yet. Its presence does not establish malicious behavior. It does add code to the review and pinning problem. Hugging Face’s custom-model documentation describes this extension mechanism and the explicit trust involved in loading custom code.

File formats matter too. Pickle-based loading can execute code during deserialization; Hugging Face documents that risk in its pickle-scanning guidance. A tensor-oriented format avoids that particular mechanism. It does not prove that a parser is free of bugs, that the surrounding package is benign, or that the generated text is safe to act on.

I therefore recorded the components together:

weights + model code + runtime + template + server configuration

That includes the separate native runtime needed by Orca GLM, as well as the model files. Revisions and hashes let me identify what I used and notice changes later. They don’t establish that the software is benign; the generation tests weren’t a security audit.

Then follow the tool call

Consider the path from a response to a side effect:

model output
    |
    v
tool-call parser
    |
    v
permission checks
    |
    v
tool process ----> filesystem / subprocesses / network

The model can propose a filesystem search. The parser can recognize it as a valid tool request. Neither fact should authorize the search.

The same applies to uploads. A syntactically valid HTTP tool call can still name an unapproved destination or contain data the process should never have read. Giving a model a narrower system prompt does not remove those capabilities from the process that executes its instructions.

For a repository investigation, I would make the checked-out repository and an output directory available, keep unrelated home-directory data out of the sandbox, and restrict network access at the execution boundary. The specific permissions depend on the task. The important property is that they are enforced outside the generated response.

This is a design requirement for the agent harness. The inference tests described in these articles did not validate a complete sandbox or an exfiltration-resistant worker.

Upstream provenance helps, within its scope

Using an upstream checkpoint can remove uncertainty about changes made by another distributor. It does not remove the need to review the runtime or constrain the agent. An upstream model can also produce an unsafe tool request, misunderstand the task, or follow instructions found in a file.

Similarly, switching from MLX-VLM to MLX-LM would change part of the software stack. It would not by itself make the model’s tool requests trustworthy. I would still need to establish compatibility, inspect the loaded implementation, and test the resulting service.

The word “uncensored” in an artifact name describes a publisher’s positioning. It is not evidence about the absence or presence of a backdoor. I had no finding that the Orca artifacts were malicious.

Accessing the server from another device

I also wanted to send requests from other devices, including my phone through a router running Tailscale. That raised a different access question: who could reach the model server?

With the service bound to loopback, an SSH tunnel can carry requests from another machine. With a wildcard bind, the service can accept connections through the machine’s network interfaces, subject to routing and firewall rules. The tested servers did not provide API authentication.

For an already configured SSH host alias named inference-mac, a tunnel can look like this:

ssh -N -o ExitOnForwardFailure=yes \
  -L 127.0.0.1:18000:127.0.0.1:8000 inference-mac

The client then talks to http://127.0.0.1:18000/v1. The explicit local bind keeps this forwarded port on the client’s loopback interface. These are the ordinary OpenSSH local-forwarding semantics. SSH access must already be configured and its host key verified.

For direct LAN access I added macOS application-firewall exceptions manually. The installer deliberately left that decision to the operator. Apple’s firewall controls allow incoming connections by application; those exceptions should not be confused with a rule allowing only one source address to one model port.

A request from another machine returned the model listing on port 8000. That established remote HTTP reachability. The local probe had already established generation. I haven’t yet tested the phone’s route through the Tailscale router.

For a future investigation, I would start by restricting the agent to its checkout, working directory, and approved tools and destinations. Choosing a model or changing the server doesn’t do that work for me.

The restore scripts preserve the model settings but leave firewall changes to the operator. In the next post, I describe what they do restore, and what happened when I tried them after resetting the M5.

Next: Restoring the Models on a Clean Mac

In the previous article, I described getting local models running on the two Macs. A short prompt checked that an installation could generate an answer. The agent experiments from earlier in this series need more than that: the model may have to read source files, a piece of a specification, and the output of several failed test runs before deciding what to try next.

All of that takes space in the context window, the input and output the model can work with in one request. That space is measured in tokens, which are pieces of text rather than necessarily whole words. The code and conversation history use the same budget as the answer.

When I looked at the initial settings, two DwarfStar profiles were limited to 8,192 tokens. Several Orca profiles had no explicit cap in the manager. I wanted to see how much further I could take them on these machines, and to have a predictable failure when a request was too large.

The 8K settings were defaults, not measurements of what the models could handle. Nor could I read an upstream context limit and assume the local installation would manage it. The runtime’s memory use and handling of long requests needed testing with the exact configurations I intended to keep.

A cap needs a test on both sides

The procedure started with a small generation and increased the input in stages. The model had to retrieve specific records from synthetic text, then handle a follow-up asking for those records in a different order. An oversized request had to be rejected. A final small probe checked that the server remained usable afterwards.

For each configuration I checked that:

1. The server reports the expected model and context configuration.
2. The request contains the measured number of tokens after formatting.
3. The answer contains the expected records.
4. A follow-up preserves the required information.
5. An over-limit request fails instead of silently discarding input.
6. The server can still generate after that rejection.

The request and the generated answer share a context budget. A 200,000-token cap is not an invitation to send 200,000 input tokens and then ask for a long answer. The template and any other inserted tokens count too. The server’s default output budget is a separate setting; clients should send max_tokens explicitly.

I precomputed the test inputs before loading the large engine. Earlier, running an auxiliary tokenizer diagnostic alongside a resident model had caused memory pressure severe enough to interfere with SSH access. Moving that preparation out of the inference run avoided making the diagnostic compete with the model for memory.

What passed on the smaller Mac

The following is the accepted M5 Max, 128 GB result. The figures come from the recorded qualification summaries, not from model-card maxima.

Each accepted M5 run used one engine and one sequence or session. The sampled memory-pressure level remained normal. Some runs began with swap already allocated, so I recorded the starting and peak values rather than describing the machine as swap-free.

All eight passed the synthetic retrieval, follow-up, rejection, and recovery checks at these sizes. I would still need real repository tasks to judge whether the extra context helps with an investigation. Finding records in generated text is a much narrower task than understanding the interactions in a large codebase. The separate image smoke tests also leave long-context multimodal use untested.

The second request can be a different workload

The largest Orca requests produced the following elapsed times, including their short answers:

The follow-up sent the conversation again and asked for the records in reverse order. The selected MLX serving path recomputed the conversation; the native serving paths reused much more state. A four-second follow-up therefore tells a very different story from a four-second cold request.

Background copies and downloads were active during qualification. These are observations from the recorded workloads, not isolated throughput benchmarks. They should not be used to rank the underlying model families.

The DwarfStar DeepSeek V4.1 streaming configuration made the same point: its largest input took about thirty minutes, followed by a cached request of about five seconds. Whether that is acceptable depends on the job. A long unattended investigation and an interactive editing session have different tolerances.

What I can use these results for

The caps now give the agent client a budget to work within, and the rejection tests give me a way to catch regressions in how the server enforces it. The timings will also affect how I construct requests. A follow-up that reuses earlier computation and one that processes the whole conversation again can have very different costs.

There is still a quality comparison to do. On the M5, smaller quantizations made some of these configurations practical, but the retrieval checks don’t tell me how much reasoning or coding ability I lost. For that I need the same tasks, prompts, tool budgets, and scoring for each candidate. I will keep those results separate from the context qualification table.

For now, I have measured input limits and a repeatable procedure for checking them. Before connecting an agent to these services, I also need to decide what it will be allowed to do with their responses. That became a concrete concern while choosing community-modified models, and is the subject of the next article.

Next: Local Models and Tool Permissions

Back in February, I left this series with Agents as Fuzzers. The idea was to combine agents that read specifications and propose tests with fuzzers that execute those tests, then check and triage the results. The earlier posts covered the instructions, coordination, and testing practices around that loop. I was using Claude Code, and hadn’t spent much of the series on the model running underneath it.

Eight months later, I’m picking up that part. I’ve been setting up local models on two Macs: an M3 Ultra with 256 GB of unified memory and an M5 Max with 128 GB. I want to find out how much of this agent-assisted work I can run on my own hardware. Before comparing the models on testing tasks, I need an installation I can run reliably and rebuild when something goes wrong.

The series outline originally put self-hosted inference on Runpod next. This experiment ended up on Apple Silicon instead. It is a detour into the machinery behind the agent, over four posts: getting the models running, testing how much input they can handle, deciding what access to give them, and restoring the whole setup after a reset. I haven’t yet measured whether these local models can do the bug-finding work as well as the hosted models I’ve been using.

What I’m moving onto the Mac

In the earlier articles, an agent could read a repository, suggest a test, run it, and inspect the result. The model supplies the responses, which may include requests to use a tool. The agent program handles the conversation and runs the permitted tools. In this setup, the model runs in a separate server process.

Here I’m setting up the model side. An inference server loads the model files and accepts requests from a client, such as an agent. The files contain the model’s weights; the runtime is the software that uses those weights to generate an answer. Hosting that server locally lets me keep this part of the work on the Mac and pin the exact files and software I use. The agent’s tools and permissions still need their own configuration.

This also changes the practical constraints on the experiments from February. A model that takes several minutes to read a prompt will make an agent’s repeated attempts expensive in time. A model that fits alone may not fit alongside another worker. I need to know those limits before trying the parallel-agent setup from Part 2 with these models.

Moving from 256 GB to 128 GB

The M3 installation had nine model profiles. Each profile selected a model and the settings needed to serve it. I kept separate kits for the M5, because copying the M3 configuration to a machine with half the memory wouldn’t tell me which models were actually usable there.

For several models, that meant choosing a smaller quantization: weights stored with reduced numerical precision to reduce their size. Names such as Q2, Q4, and 8-bit identify different choices of representation. Some artifacts use a mixture, keeping selected parts at higher precision. The memory saving needs to be weighed against any loss in answer quality; getting a file to load only answers the first half of that question.

I organized the kits into two collections, called DwarfStar and Orca. These names recur in the tables below. Each kit includes the model files, runtime, settings, restore scripts, and test records. The collection name doesn’t imply that all its models use the same engine.

The M5 ended up with these eight configurations:

The qualification record identifies the selected artifacts and the tests they passed. This isn’t a recommendation of one model over another. I haven’t done the paired coding and reasoning tests needed to compare, for example, the smaller GLM quantization with the one on the M3. DeepSeek V4.1 Flash 1017 is a recent addition: its public repository records its release on October 17, before these experiments.

What stays in memory

The size on disk wasn’t enough to work out what would fit. I also had to account for what the runtime kept in memory, and what it needed while processing a request.

The M5 DeepSeek V4.1 configuration streams expert weights from the internal SSD, keeping a 64GB cache in memory. Qwen has large n-gram tables, described in its model card, which the selected runtime keeps disk-backed. Both arrangements let me use artifacts larger than I could keep entirely resident. They also make the time spent reading and processing a new prompt worth measuring.

The external SSD serves a different purpose. It holds the recovery kit; the installed models run from internal storage. I can disconnect the external drive after restoring them.

I tested one engine and one sequence at a time on the M5. During the earlier work, an auxiliary diagnostic running alongside a loaded model caused severe memory pressure. For now, the agent-swarm experiments will have to respect that restriction. These results don’t establish that I can run several local workers concurrently.

More than one runtime

The model choices also left me with several runtimes to maintain. MLX-LM and MLX-VLM are separate projects using Apple’s MLX framework, with MLX-VLM supporting vision-language models. The Orca Qwen and Nex installations use a pinned MLX-VLM environment. OrcaSAQ and Orca GLM use GGUF files, a format consumed here by native llama.cpp servers. The DwarfStar models use two archived revisions of the DS4 engine.

Even the two llama.cpp installations aren’t interchangeable. Orca GLM needed a separate pinned build. One loading attempt failed a Metal allocation; the configuration that passed used automatic fitting, a single sequence, and --load-mode none. In that particular build, none disables special loading modes, including mmap. It doesn’t disable GPU inference.

I kept those settings with their runtime revisions so that a future restore wouldn’t depend on remembering which build accepted which flags. The operations reference has the machine-specific commands.

Checking that a model can run

Once a server starts, it can answer an HTTP request for its model list without having generated anything. I needed a short generation test as well as that readiness check. The managers therefore expose four actions:

start -> status -> probe -> stop

status checks the owned process and HTTP readiness. probe asks the model to generate an answer and checks the response. The Orca start operation also runs a generation before reporting success. Stopping checks the process identity, including its start time and command, before sending a signal; a saved PID alone could refer to a different process by then.

Eight M5 profiles had passed the qualification procedure. The remaining candidate, Orca DeepSeek, still had an unfinished conversion. I stopped the conversion and its supervisor, kept the verified original shards for possible later work, and left it out of the installer.

A short answer was enough to check that each restored installation could perform inference. For the work in this series, though, the input will include code, specifications, and the results of earlier tool calls. The next article covers what happened when I tested much longer requests, including how long I had to wait for the answers.

Next: Testing Longer Contexts

The previous post proved, for every finite list, that reversing twice gives back the list. Here is another function with the same property:

def revId {α : Type} (xs : List α) : List α := xs

theorem revId_twice {α : Type} (xs : List α) :
    revId (revId xs) = xs := rfl

revId returns its input. It reverses nothing, and the theorem holds for it, with a one-word proof. Lean is correct to accept it. Doing nothing twice does give back the list.

So rev_twice was never a proof that rev reverses. It was a proof of the sentence I wrote down, and the sentence was weaker than what I meant. That is the first of three limits, and none of them is specific to this toy.

Limit one: the statement

The same weakness was in the tests all along. The round-trip property from the first post passes for the identity function in both languages:

// Wrong on purpose: returns its input.
func revId(xs []int) []int {
	return xs
}

func TestIdentityAlsoPasses(t *testing.T) {
	rule := func(xs []int) bool {
		return slices.Equal(
			revId(revId(xs)), xs)
	}
	if err := quick.Check(rule, nil); err != nil {
		t.Fatal(err)
	}
}

A statement that pins the behaviour down says where each element ends up: the result has the same length, and position i holds what was at position length - 1 - i. Here it is in C#, run against RevId, which like the Go function above returns its argument:

var mirrored =
    from xs in lists.ForAll()
    let a = xs.ToArray()
    let b = RevId(a)
    select a.Length == b.Length
        && Enumerable.Range(0, a.Length)
            .All(i => b[i] == a[a.Length - 1 - i]);

mirrored.Check();

Against the identity function, Hedgehog 2.0.4 reports this, with the line that gives the seed for a rerun left out:

*** Failed! Falsifiable (after 1 test and 9 shrinks):

Generated values:
  [-100; -99]

Two different elements are the smallest list on which “unchanged” and “reversed” disagree, and shrinking found it.

On the Lean side, the lemma from the previous post already does this job. rev_append says rev (xs ++ ys) = rev ys ++ rev xs. For the identity function that would claim [1, 2] = [2, 1], so it cannot be proved. The helper turned out to say more about reversing than the headline theorem did.

Moving from tests to proofs did not remove the need to ask whether the statement is the right one. It raised the price of asking late.

Limit two: the thing it is about

rev_twice is about a Lean definition over mathematical lists. The code that runs is this:

static int[] Rev(int[] xs)
{
    var ys = (int[])xs.Clone();
    Array.Reverse(ys);
    return ys;
}

Nothing connects the two. The C# method works on a machine array with a 32-bit length, calls a library routine and allocates memory. The Lean definition does none of that. The proof would be untouched if the C# method were replaced by one that throws.

There are three honest ways to connect them, in rising order of cost:

1. Test the running code against the proved definition. The Lean function becomes the oracle for a property test or a fuzzer. This shows agreement on the inputs tried, and it is cheap.
2. Run the proved code itself, by compiling the Lean definition and calling it. Then the compiler and runtime are trusted instead.
3. Model the C# or Go semantics, including machine integers, and prove the real function equal to the definition. This is the only one that earns the word “verified” for the client code, and it is a project, not an afternoon.

Machine integers deserve their own mention. Lean’s Nat never overflows. A sum that wraps around, or a conversion between integer types that changes the value, simply does not exist in a model written with Nat. A proof over Nat is silent about the exact place such a bug lives.

Limit three: writing it in Lean

Lean is also a programming language, and a program written in it is not proved by being written in it. Take the proved function and add logging:

def revLogged (xs : List Nat) : IO (List Nat) := do
  IO.println s!"input: {xs}"
  pure (rev xs)

def twiceLogged (xs : List Nat) : IO (List Nat) := do
  let ys ← revLogged xs
  revLogged ys

For [10, 20, 30] this prints two lines and returns [10, 20, 30]. The returned list is restored. The two writes still happened. rev_twice says nothing about what was printed, whether printing can fail, or whether those values should be in a log at all. Claims about effects need their own specification and their own proof.

The same holds for the rest of the language. Each of these is ordinary Lean and each sits outside what a theorem about rev covers:

• partial def accepts a function without a termination argument.
• xs[i]! indexes without a bounds proof and panics at run time when the index is out of range.
• sorry closes any goal. #print axioms on a theorem shows whether one is hiding underneath it.
• @[extern] and @[implemented_by] swap a definition for other code that the proofs never looked at.
• Time and memory are not part of the statement. A proved function can still be made to run for an hour.

What a proof covers

A proof establishes its statement, about its definitions, under its assumptions. Within that it is complete in a way no test is, and I would take one over a thousand green runs. Outside that it says nothing.

So when someone tells me a piece of code is formally verified, I ask four things:

1. What is the statement, in one sentence? Would a wrong implementation also satisfy it?
2. What is it a statement about: the code that runs, or a model of it? If a model, what ties the two together?
3. What does it assume? Which axioms, which unproved parts, which compiler and runtime?
4. What is outside it? I/O, resource use, timing, everything upstream of the function.

None of this argues for skipping proofs. It argues for spending them where the answers are good: small, pure pieces with a statement short enough to check by eye, such as an offset calculation, a parser for a fixed format, or a state transition. There the proved definition also gives every implementation the same oracle to test against. The tests, the properties and the fuzzer keep doing what a proof does not: running the real code on real inputs.

A later post in this series will apply all three to a real bug.

The previous post checked reverse (reverse xs) = xs for one list and then for a thousand generated ones. This post proves it for every finite list, in Lean. I assume no Lean; each piece of notation is explained where it first appears.

A list has two cases

[]                 -- Empty.
head :: tail       -- An element followed by a list.

[10, 20, 30] = 10 :: [20, 30]
             = 10 :: 20 :: 30 :: []

head is an element, tail is another list, and :: builds a list from the two. There is no third case. Everything below follows from that: a function on lists has to say what it does with [] and with head :: tail, and a proof about lists has to cover the same two.

Reverse, defined on those cases

def rev {α : Type} : List α → List α
  | []           => []
  | head :: tail => rev tail ++ [head]

α is the element type, ++ joins two lists, and [head] is the list holding only head. For a non-empty list: reverse the tail, then put the head at the end. The recursive call gets a shorter list, so it reaches [] and stops.

Expanding the definition on a small list shows the mechanism:

rev [10, 20, 30]
= rev [20, 30] ++ [10]
= (rev [30] ++ [20]) ++ [10]
= ((rev [] ++ [30]) ++ [20]) ++ [10]
= (([] ++ [30]) ++ [20]) ++ [10]
= [30, 20, 10]

The proof has the same two cases

theorem rev_twice {α : Type} (xs : List α) :
    rev (rev xs) = xs := by
  induction xs with
  | nil => ...
  | cons head tail ih => ...

nil asks for the claim on []. cons asks for it on head :: tail and hands over ih, the induction hypothesis: rev (rev tail) = tail.

Being allowed to assume the claim for the tail can look like cheating. It is not. Establish the rule for the empty list. Then show that whenever it holds for a tail, it still holds after adding a head. That covers length 0, then 1, then 2, and so on. It is one argument for all lengths, not a loop that tries them.

The empty case

  | nil => rfl

rev [] is [] by definition, so rev (rev []) reduces to [] and the goal becomes [] = []. rfl closes an equality whose two sides reduce to the same thing.

The non-empty case needs a helper

Unfolding the definition once gives:

rev (rev (head :: tail))
= rev (rev tail ++ [head])

Now rev is applied to two joined lists, and the definition does not say what that is. It takes a separate fact: reversing joined lists reverses each part and swaps their order.

theorem rev_append {α : Type} (xs ys : List α) :
    rev (xs ++ ys) = rev ys ++ rev xs := by
  induction xs with
  | nil => simp [rev]
  | cons head tail ih =>
      simp [rev, ih, List.append_assoc]

A proved fact used inside another proof is called a lemma. This one is again by induction on the two cases. simp unfolds rev and rewrites with the hypothesis for the tail; List.append_assoc says that the grouping of joined lists does not matter.

With the lemma, the non-empty case is four steps:

rev (rev (head :: tail))
= rev (rev tail ++ [head])
= rev [head] ++ rev (rev tail)     -- By rev_append.
= [head] ++ tail                   -- By the definition and ih.
= head :: tail

The head returns to the front, and the tail is restored by the induction hypothesis. At no point was a length or an element value chosen.

The complete proof

theorem rev_twice {α : Type} (xs : List α) :
    rev (rev xs) = xs := by
  induction xs with
  | nil => rfl
  | cons head tail ih =>
      calc
        rev (rev (head :: tail))
            = rev (rev tail ++ [head]) := rfl
        _ = rev [head] ++ rev (rev tail) := rev_append _ _
        _ = [head] ++ tail := by rw [ih]; rfl
        _ = head :: tail := rfl

Lean checks every step. If one did not follow, the file would not compile.

What was established

This is the gap the previous post ended on, closed. No generator bounds, no sample count, no luck.

One sentence above carries more weight than it seems to: the proof is about rev, the three-line Lean definition. It says that this definition, composed with itself, is the identity. It does not mention the C# method or the Go function from the previous post, and it does not say that rev reverses anything.

Next: The Theorem Holds for the Wrong Function