Same Prompts, Three Macs: Four Models and a Test That Measured Nothing

Part 3 of Same Prompts, Three Macs.

After the placement runs I had a practical question left: which models should each Mac load? I wanted both DeepSeek releases in use somewhere, and on the two Mac Studios a small model beside the large one for quick questions. I also wanted prompts closer to my own work than arithmetic and string functions.

Four models

DwarfStar served DeepSeek V4 Flash 0731 and DeepSeek V4.1 Flash 1017. The Llama app served Qwen3.6 27B and Qwen3.8 27B at Q8_0. The Mac Studios stood apart, the laptop had its lid open, and the thermal level was logged. The laptop ran only the two DeepSeeks.

The prompt set is the 40 graded prompts from the first post, plus 13 Ethereum fact questions with one fixed answer each, such as the number of slots in an epoch or the selector of transfer(address,uint256).

The newer DeepSeek release wasn’t better on these prompts. It is larger, 366 GB on the Mac Studios against 153 GB, and slower. On the laptop it doesn’t fit in memory, so DwarfStar streams part of it from disk, and it answered at under half the speed of the older release.

A 27B model matched the large ones on the generic prompts. I read that as a statement about the prompts: they are too easy to rank these models. It doesn’t show the models are equal. Qwen3.8 lost three fact questions by writing past the 256-token limit before reaching its answer line; the facts in its text were right.

The grader was wrong again as well. It wanted a number and marked EIP-155 as a miss, which cost DeepSeek 0731 two fact prompts until I read the failures. The table is after that fix.

Choosing per machine

Score didn’t decide anything, so the choice came from speed, disk and memory.

I checked that the pairing works on the M5 Ultra. With DeepSeek loaded in DwarfStar and Qwen3.8 in the Llama app, 34% of memory stayed free, swap stayed at 4 MB, and each answered at its usual speed: 56 and 44 tokens per second with the older release, 52 and 46 with the newer. That was a short prompt to each, one at a time. I didn’t test both answering at once, or long conversations in both, and I didn’t repeat it on the M3 Ultra. The order matters: DwarfStar’s manager refuses to start while another engine holds a model, so it goes first.

The heat log earned its place here. Standing apart, the M3 Ultra spent 57% of this run at Moderate, all of it during the two Qwen runs in the Llama app. The M5 Ultra spent 41% at Heavy, under the Qwens and also under DeepSeek V4.1 in DwarfStar. In the previous post the M3 Ultra had stayed Nominal apart, and DwarfStar had never taken the M5 Ultra past Moderate; that was a different workload.

A domain test

For prompts nearer my work, I built 40 from a private set of security reports against Ethereum client software. Each prompt is one report, and the task is to classify its severity as the triage team did: Informational, Low, Medium or High, ten of each. Reports that named a severity in their own text were left out, so the label couldn’t be read off the page. The reports are private and none of their content is published here.

The first version gave the models no scale. On the M5 Ultra, where I tabulated the answers, every model called most reports High and none answered Informational once. Reports argue their own impact, and the models believed them. Scores ran from 11 to 14 of 40, where guessing gets 10.

The second version put the public severity scale from the Ethereum bug bounty page into the prompt, with its network-share thresholds, and a note that reporters tend to overstate impact. The models then swung the other way and under-rated.

On the M5 Ultra the four models gave the same answer on 2 of the 40 reports. The same DeepSeek file at temperature 0 scored 15 on one Mac Studio and 12 on the other; I haven’t investigated why the answers differ between machines.

The control

The obvious objection is that these are small local models. So the same 53 Ethereum prompts went to Claude Fable 5.2, each in a fresh session with no tools and a one-line system prompt. Unlike the local models it was allowed to think before answering.

It got 13 of 13 facts and 13 of 40 on the triage prompts. It made the same mistake, calling 14 of the 30 reports above Informational “Informational”, and got one of the ten High reports right.

So the test is broken, and the models are not what it measured. A report’s text and the public scale are not enough to recover the label a triage team assigned, whoever reads them. My guess is that the team uses things a report doesn’t say, such as what was already known and how hard the attack is in practice. I haven’t tested that.

What I take from it

• An oracle that is a human judgement measures agreement with that judgement. Here there was none to measure, and without the control I might have blamed the models.
• Run a strong baseline through a new benchmark before reading anything into the scores. It cost a few minutes.
• The fact questions worked and were too easy. Everyone scored 10 or more of 13.
• Read the failures. This series found a grader bug in every round.

The next attempt needs a task whose answer is in the text or in checkable knowledge: which component a report concerns, with the obvious names removed, or whether a described behaviour violates the specification. That is the remaining entry in the series index, and it is closer to the question this blog keeps returning to: choosing oracles that are real.

Entry dated November 12; the measurements were taken the same day.