Same Prompts, Three Macs: Higher Precision, Same Answers

Part 1 of Same Prompts, Three Macs.

I keep the same model twice on one Mac. DwarfStar serves it from a higher-precision file, and the Llama app, a llama.cpp build with a model router, serves it from a smaller one published by ggml-org. The larger files cost hundreds of gigabytes of disk, and I had assumed they bought better answers. In an earlier post I wrote that I hadn’t done the paired tests needed to say so. This is that test.

The question is narrow on purpose: on the same Mac, with the same prompts, does the higher-precision file get more right?

What was compared

The first runs were on the M3 Ultra Mac Studio with 256 GB of memory, one server loaded at a time.

Those names say how many bits each weight is stored in. A smaller number is a smaller and coarser file: Q2_K keeps about two bits per weight, Q4_K about four, and MXFP4 is a four-bit floating-point format.

Each server received 45 prompts. Forty have an answer a program can check:

• 12 coding tasks, graded by running the returned function against tests in a separate process.
• 14 arithmetic and logic questions, with the answer on a marked last line.
• 6 tasks that must return strict JSON, compared after parsing.
• 8 lookups in a synthetic order ledger, four in about 25,000 tokens of input and four in about 85,000.

The other five are open questions, saved for timing and not graded. Requests used temperature 0 with thinking switched off, and the same request body went to both servers. The expected answers are computed by the harness, and every coding test must pass against a reference solution and fail against an empty one before it is used. The prompts and the grader are published with the series.

One prompt apart

By group, in the same order:

code           11/12  10/12   9/12  10/12
math           11/14  12/14  13/14  14/14
json            5/6    6/6    6/6    6/6
ledger, 25k     4/4    4/4    3/4    2/4
ledger, 85k     4/4    4/4    3/4    3/4

For both models the smaller file scored one prompt higher. With 40 prompts and one run, one prompt is noise. The reading I can defend is that no difference showed, and I can’t claim the smaller file is better.

I expected a gap for GLM, where the Llama file has half the bits. There wasn’t one.

Failures followed the model

The misses clustered by model, and the file made little difference. DeepSeek missed the same four prompts under both engines: two “which city is on the line after order X” lookups, a topological sort, and a Base64 encoder that got the padding wrong on a two-byte input. Both GLM files got the sum of the primes below 60 wrong, with 672 and 769 where the answer is 440. Most of the other misses were arithmetic done without thinking, such as counting the 1 bits in 2026.

DeepSeek returned byte-identical text on 24 of the 45 prompts across the two engines and files. GLM did on 7.

The same four servers then ran on the M5 Ultra Mac Studio, which takes the same files, and scored 35, 36, 35 and 35 on its last stacked run. The M5 Max MacBook Pro has half the memory and takes smaller DwarfStar files: GLM at Q2 scored 35, and DeepSeek with most experts at Q2 scored

1. Across the three Macs, every run of these two models landed between 34 and 37 on these 40 prompts.

The grader had bugs too

A checker needs checking. I read every failure by hand, and three of them were about the test and not the model.

One coding test rejected any answer whose source contained the word “calendar”, to forbid import calendar. DeepSeek wrote a comment about the Gregorian calendar and failed a correct solution under both engines. The test now looks for the import. The table above is after that fix, which moved both DeepSeek scores up by one.

One lookup answer was the right order ID without the required ANSWER: prefix. I left it as a miss, because the format was part of the task. A looser grader would have passed it.

“Thinking off” also had to be verified. The Llama app’s GLM kept reasoning with the usual switches and stopped only when the request carried thinking_budget_tokens: 0:

body = {'model': model, 'temperature': 0, 'seed': 1,
        'max_tokens': max_tokens, 'reasoning_effort': 'none',
        'think': False, 'thinking': {'type': 'disabled'},
        'chat_template_kwargs': {'enable_thinking': False},
        'thinking_budget_tokens': 0,
        'messages': [{'role': 'user', 'content': prompt}]}

The harness records the length of any reasoning a server returns, and the final runs show zero for all of them.

Speed was a different question

DwarfStar generated DeepSeek at 36.2 tokens per second on the M3 Ultra against 25.9 for the Llama app. For GLM the order reversed, 22.8 against 26.4. Reading the 85,000-token ledger for the first time took between 224 and 317 seconds on every server. Three of the four reused their cache for the next three questions, at 7 to 31 seconds each. DwarfStar with GLM read the ledger again every time, about 300 seconds per question.

When I collected those figures, the two Mac Studios were stacked, and one of them turned out to be running hot. That put every timing in doubt, which is the next post.

What this does and doesn’t show

• It is one run per server and 40 prompts. A gap of two or three answers would still be noise. The test could have shown a large difference and didn’t; it can’t show there is none.
• It compares two stacks as well as two files. The engines differ in chat template and sampling code, so an effect could come from either.
• The prompts are short tasks with one right answer. Nothing here covers long agent sessions, tool use, or the quality of prose.
• The prompts were written by the assistant that ran the test. They are not drawn from my own work.
• With thinking on, the arithmetic misses would probably shrink for every file. I didn’t measure that.

I haven’t deleted the larger files. What changed is my reason for keeping them: I can no longer say it is answer quality, so the choice between the two servers comes down to speed, context handling and convenience.

Entry dated November 10. The measurements were taken on November 11 and 12, and the post was assembled on November 12.

Next: Where the Machines Sit