Self-Hosted Inference on Apple Silicon

Part 5 of Oracles, Traces, Triage.

Back in February, I left this series with Agents as Fuzzers. The idea was to combine agents that read specifications and propose tests with fuzzers that execute those tests, then check and triage the results. The earlier posts covered the instructions, coordination, and testing practices around that loop. I was using Claude Code, and hadn’t spent much of the series on the model running underneath it.

Eight months later, I’m picking up that part. I’ve been setting up local models on two Macs: an M3 Ultra with 256 GB of unified memory and an M5 Max with 128 GB. I want to find out how much of this agent-assisted work I can run on my own hardware. Before comparing the models on testing tasks, I need an installation I can run reliably and rebuild when something goes wrong.

The series outline originally put self-hosted inference on Runpod next. This experiment ended up on Apple Silicon instead. It is a detour into the machinery behind the agent, over four posts: getting the models running, testing how much input they can handle, deciding what access to give them, and restoring the whole setup after a reset. I haven’t yet measured whether these local models can do the bug-finding work as well as the hosted models I’ve been using.

What I’m moving onto the Mac

In the earlier articles, an agent could read a repository, suggest a test, run it, and inspect the result. The model supplies the responses, which may include requests to use a tool. The agent program handles the conversation and runs the permitted tools. In this setup, the model runs in a separate server process.

Here I’m setting up the model side. An inference server loads the model files and accepts requests from a client, such as an agent. The files contain the model’s weights; the runtime is the software that uses those weights to generate an answer. Hosting that server locally lets me keep this part of the work on the Mac and pin the exact files and software I use. The agent’s tools and permissions still need their own configuration.

This also changes the practical constraints on the experiments from February. A model that takes several minutes to read a prompt will make an agent’s repeated attempts expensive in time. A model that fits alone may not fit alongside another worker. I need to know those limits before trying the parallel-agent setup from Part 2 with these models.

Moving from 256 GB to 128 GB

The M3 installation had nine model profiles. Each profile selected a model and the settings needed to serve it. I kept separate kits for the M5, because copying the M3 configuration to a machine with half the memory wouldn’t tell me which models were actually usable there.

For several models, that meant choosing a smaller quantization: weights stored with reduced numerical precision to reduce their size. Names such as Q2, Q4, and 8-bit identify different choices of representation. Some artifacts use a mixture, keeping selected parts at higher precision. The memory saving needs to be weighed against any loss in answer quality; getting a file to load only answers the first half of that question.

I organized the kits into two collections, called DwarfStar and Orca. These names recur in the tables below. Each kit includes the model files, runtime, settings, restore scripts, and test records. The collection name doesn’t imply that all its models use the same engine.

The M5 ended up with these eight configurations:

The qualification record identifies the selected artifacts and the tests they passed. This isn’t a recommendation of one model over another. I haven’t done the paired coding and reasoning tests needed to compare, for example, the smaller GLM quantization with the one on the M3. DeepSeek V4.1 Flash 1017 is a recent addition: its public repository records its release on October 17, before these experiments.

What stays in memory

The size on disk wasn’t enough to work out what would fit. I also had to account for what the runtime kept in memory, and what it needed while processing a request.

The M5 DeepSeek V4.1 configuration streams expert weights from the internal SSD, keeping a 64GB cache in memory. Qwen has large n-gram tables, described in its model card, which the selected runtime keeps disk-backed. Both arrangements let me use artifacts larger than I could keep entirely resident. They also make the time spent reading and processing a new prompt worth measuring.

The external SSD serves a different purpose. It holds the recovery kit; the installed models run from internal storage. I can disconnect the external drive after restoring them.

I tested one engine and one sequence at a time on the M5. During the earlier work, an auxiliary diagnostic running alongside a loaded model caused severe memory pressure. For now, the agent-swarm experiments will have to respect that restriction. These results don’t establish that I can run several local workers concurrently.

More than one runtime

The model choices also left me with several runtimes to maintain. MLX-LM and MLX-VLM are separate projects using Apple’s MLX framework, with MLX-VLM supporting vision-language models. The Orca Qwen and Nex installations use a pinned MLX-VLM environment. OrcaSAQ and Orca GLM use GGUF files, a format consumed here by native llama.cpp servers. The DwarfStar models use two archived revisions of the DS4 engine.

Even the two llama.cpp installations aren’t interchangeable. Orca GLM needed a separate pinned build. One loading attempt failed a Metal allocation; the configuration that passed used automatic fitting, a single sequence, and --load-mode none. In that particular build, none disables special loading modes, including mmap. It doesn’t disable GPU inference.

I kept those settings with their runtime revisions so that a future restore wouldn’t depend on remembering which build accepted which flags. The operations reference has the machine-specific commands.

Checking that a model can run

Once a server starts, it can answer an HTTP request for its model list without having generated anything. I needed a short generation test as well as that readiness check. The managers therefore expose four actions:

start -> status -> probe -> stop

status checks the owned process and HTTP readiness. probe asks the model to generate an answer and checks the response. The Orca start operation also runs a generation before reporting success. Stopping checks the process identity, including its start time and command, before sending a signal; a saved PID alone could refer to a different process by then.

By October 23, eight M5 profiles had passed the qualification procedure. The remaining candidate, Orca DeepSeek, still had an unfinished conversion. I stopped the conversion and its supervisor, kept the verified original shards for possible later work, and left it out of the installer.

A short answer was enough to check that each restored installation could perform inference. For the work in this series, though, the input will include code, specifications, and the results of earlier tool calls. The next article covers what happened when I tested much longer requests, including how long I had to wait for the answers.

Retrospective entry for October 20. Assembled October 24; the M5 results above were obtained on October 23.

Next: Testing Longer Contexts