macmini-cat-bench

Honest AI benchmarks on the Mac mini

Local language models, measured the same way on the same machine: quality over 13 tracks (cybersecurity, daily use, coding, Dutch and more), speed, memory and what happens when several agents share one server.

Mac mini (Apple M6, 32 GB)macOS 27.0.1Ollama 0.40.52026-10-20run 2026-10-20-macmini-m6-32gb
27models
13tracks
203tasks per model
15,243answers scored
3seeds per task

The short version

Best overallqwen3.8 27B0.95 average over 13 tracks
Best at cybersecuritygemma4 12B0.95 on the cybersecurity track
Fastest model that is still goodgpt-oss 20B39 tok/s with an average of 0.91
Smallest model that is still goodqwen3.5 9B7.74 GB of memory with an average of 0.91

Profile per track

Score from 0 to 1 per track: mean over seeds, with the range between seeds underneath. There is no single winner: look at the tracks that matter to you. The average is for sorting only. Click a column heading to sort. Run-to-run noise in this run reaches 0.40: differences smaller than that are not meaningful. ⚠n means n answers were cut off by the token limit.

Show tracks:
Fits in:
#ModelAverageCybersecurityDaily useCodingDutchTools and JSONHonesty and safetyReasoning and mathsLong contextHard mixedStandard topicsPentestingTriage under noiseSecure dev & opsSpeed
(tok/s)		Memory
1qwen3.8 27BQwen
0.95
0.940.94–0.95
0.920.92–0.92
1.001.00–1.00
0.920.92–0.92
0.890.83–0.92
1.001.00–1.00
0.97 ⚠10.92–1.00
1.001.00–1.00
0.940.93–0.97
0.96 ⚠20.96–0.96
0.940.93–0.95
0.850.85–0.85
0.98 ⚠30.93–1.00
1616.46 GB
2gemma4 12BGemma
0.94
0.950.95–0.95
1.001.00–1.00
0.940.94–0.94
0.960.96–0.96
1.001.00–1.00
0.920.92–0.92
1.001.00–1.00
1.001.00–1.00
0.94 ⚠20.93–0.95
0.94 ⚠40.94–0.94
0.960.96–0.96
0.720.70–0.75
0.88 ⚠30.87–0.90
357.75 GB
3qwen3.5 9BQwen
0.91
0.950.93–0.95
0.920.92–0.92
1.001.00–1.00
0.790.79–0.79
1.001.00–1.00
0.850.79–0.92
1.001.00–1.00
1.001.00–1.00
0.880.88–0.88
0.96 ⚠10.96–0.96
0.930.91–0.96
0.680.65–0.70
0.90 ⚠30.87–0.93
367.74 GB
4gpt-oss 20Bgpt-oss
0.91
0.890.87–0.91
0.990.99–0.99
1.001.00–1.00
0.90 ⚠10.85–0.92
1.001.00–1.00
0.720.58–0.83
1.001.00–1.00
1.001.00–1.00
0.970.95–1.00
0.930.92–0.94
0.84 ⚠10.82–0.85
0.730.70–0.80
0.87 ⚠10.87–0.87
3912.85 GB
5Mistral Small 24BMistral
0.91
0.850.84–0.87
0.980.96–1.00
0.870.85–0.90
0.960.96–0.96
1.001.00–1.00
0.860.83–0.92
1.001.00–1.00
1.001.00–1.00
0.820.79–0.89
1.001.00–1.00
0.930.93–0.94
0.670.65–0.70
0.82 ⚠30.77–0.87
1014.36 GB
6phi4 14BPhi
0.90
0.850.84–0.86
0.990.96–1.00
1.001.00–1.00
0.950.93–0.96
1.001.00–1.00
0.750.75–0.75
0.920.92–0.92
1.001.00–1.00
0.84 ⚠20.82–0.88
0.920.92–0.92
0.84 ⚠10.81–0.87
0.800.80–0.80
0.86 ⚠30.83–0.87
169.55 GB
7gemma3 12BGemma
0.89
0.800.79–0.81
0.920.92–0.92
1.001.00–1.00
0.830.83–0.83
1.001.00–1.00
0.750.75–0.75
1.001.00–1.00
1.001.00–1.00
0.920.92–0.92
0.920.92–0.92
0.960.96–0.96
0.650.65–0.65
0.79 ⚠30.77–0.83
197.22 GB
8qwen2.5-coder 14BQwen
0.89
0.840.80–0.87
0.980.96–1.00
1.001.00–1.00
0.860.85–0.88
1.001.00–1.00
0.780.67–0.83
0.940.92–1.00
1.001.00–1.00
0.870.86–0.89
0.910.90–0.92
0.840.82–0.85
0.670.65–0.70
0.81 ⚠30.80–0.83
179.67 GB
9granite4Granite
0.87
0.760.73–0.78
0.970.92–1.00
0.900.85–0.93
0.880.88–0.88
1.001.00–1.00
0.750.75–0.75
1.001.00–1.00
1.001.00–1.00
0.840.79–0.89
0.920.90–0.94
0.890.85–0.93
0.680.65–0.70
0.770.70–0.83
62—
10Llama 3.1 8BLlama
0.87
0.760.75–0.77
0.940.92–0.96
0.930.85–1.00
0.870.85–0.88
0.970.97–0.97
0.920.92–0.92
1.001.00–1.00
1.001.00–1.00
0.76 ⚠20.74–0.77
0.900.88–0.92
0.82 ⚠10.79–0.85
0.670.65–0.70
0.79 ⚠20.70–0.83
305.45 GB
11qwen3 8BQwen
0.85
0.840.83–0.85
0.880.88–0.88
0.950.95–0.95
0.890.88–0.92
1.001.00–1.00
0.690.67–0.75
1.001.00–1.00
1.001.00–1.00
0.710.67–0.77
0.920.90–0.94
0.820.82–0.82
0.650.65–0.65
0.73 ⚠30.73–0.73
285.24 GB
12DeepHat V1 7BQwen 2.5 Coder (security-tuned)
0.85
0.78 ⚠10.73–0.81
0.920.92–0.92
0.980.95–1.00
0.870.85–0.89
1.001.00–1.00
0.500.38–0.67
1.00 ⚠131.00–1.00
1.001.00–1.00
0.79 ⚠40.77–0.82
0.830.79–0.87
0.89 ⚠20.85–0.93
0.650.65–0.65
0.81 ⚠30.77–0.87
324.48 GB
13deepseek-r1 14BDeepSeek R1 distill
0.84
0.740.70–0.80
0.890.88–0.92
0.88 ⚠40.85–0.90
0.870.75–0.94
1.001.00–1.00
0.640.58–0.75
1.001.00–1.00
1.001.00–1.00
0.82 ⚠20.80–0.84
0.920.90–0.94
0.840.79–0.89
0.680.60–0.75
0.69 ⚠30.67–0.73
169.03 GB
14GLM-4 9BGLM
0.83
0.680.62–0.71
0.910.90–0.92
0.930.93–0.95
0.820.79–0.83
0.970.97–0.97
0.860.75–0.92
0.920.92–0.92
1.001.00–1.00
0.780.77–0.81
0.870.85–0.88
0.770.76–0.78
0.580.55–0.60
0.740.72–0.76
275.53 GB
15gemma3 4BGemma
0.83
0.660.66–0.67
0.740.74–0.74
0.870.82–0.95
0.920.92–0.92
1.001.00–1.00
0.580.58–0.58
1.00 ⚠21.00–1.00
1.001.00–1.00
0.82 ⚠30.79–0.84
0.90 ⚠20.88–0.92
0.980.97–0.99
0.550.55–0.55
0.78 ⚠30.73–0.80
512.38 GB
16Command R 35BCommand R (Cohere)
0.83
0.830.81–0.87
0.890.83–0.92
0.910.84–1.00
0.930.92–0.96
1.001.00–1.00
0.750.75–0.75
0.780.67–0.92
1.001.00–1.00
0.530.45–0.59
0.880.88–0.88
0.900.88–0.94
0.670.65–0.70
0.71 ⚠20.67–0.73
820.05 GB
17Hermes 3 8BHermes (Nous Research, Llama 3.1-based)
0.82
0.740.72–0.75
1.001.00–1.00
0.820.82–0.82
0.860.85–0.88
0.970.97–0.97
0.670.58–0.75
0.970.92–1.00
1.001.00–1.00
0.710.67–0.74
0.840.83–0.85
0.800.76–0.85
0.520.50–0.55
0.81 ⚠30.77–0.87
325.19 GB
18WhiteRabbitNeo 2.5 (Qwen) 7BQwen 2.5 Coder (security-tuned)
0.81
0.840.81–0.87
0.960.96–0.96
1.000.99–1.00
0.780.75–0.83
1.001.00–1.00
0.610.50–0.67
0.890.83–1.00
1.001.00–1.00
0.76 ⚠10.74–0.78
0.860.83–0.88
0.83 ⚠10.80–0.87
0.330.30–0.35
0.710.67–0.77
324.93 GB
19codestral 22BMistral
0.81
0.840.81–0.88
0.880.88–0.88
0.920.90–0.95
0.790.75–0.82
1.001.00–1.00
0.720.67–0.75
0.690.67–0.75
1.001.00–1.00
0.63 ⚠20.56–0.74
0.84 ⚠10.83–0.85
0.82 ⚠40.81–0.84
0.580.55–0.60
0.79 ⚠120.77–0.83
1113.77 GB
20Llama 3.2 3BLlama
0.80
0.680.64–0.70
0.950.92–1.00
0.900.90–0.90
0.820.79–0.88
0.970.97–0.97
0.920.83–0.96
0.920.92–0.92
1.001.00–1.00
0.700.67–0.71
0.850.85–0.85
0.760.75–0.78
0.180.05–0.45
0.69 ⚠10.66–0.72
652.68 GB
21Yi 9BYi
0.78
0.710.63–0.76
0.890.88–0.90
0.840.75–0.90
0.670.61–0.69
0.970.97–0.97
0.780.75–0.83
1.001.00–1.00
0.500.50–0.50
0.810.74–0.89
0.870.87–0.88
0.810.78–0.83
0.620.55–0.65
0.73 ⚠30.68–0.81
285.21 GB
22Nous Hermes 2 11BHermes (Nous Research, Solar-based)
0.75
0.720.64–0.77
0.860.83–0.92
0.600.58–0.63
0.790.76–0.81
0.970.97–0.97
0.83 ⚠60.83–0.83
0.640.50–0.75
0.500.50–0.50
0.520.47–0.58
0.910.87–0.95
0.89 ⚠50.88–0.90
0.680.65–0.70
0.80 ⚠10.77–0.82
236.55 GB
23Zephyr 7BZephyr (Mistral-based)
0.64
0.71 ⚠120.69–0.72
0.900.89–0.93
0.34 ⚠20.33–0.36
0.71 ⚠10.67–0.76
0.89 ⚠30.81–0.97
0.57 ⚠30.50–0.67
0.670.50–0.83
0.67 ⚠50.67–0.67
0.18 ⚠50.15–0.23
0.82 ⚠40.81–0.83
0.75 ⚠50.71–0.79
0.380.25–0.55
0.74 ⚠50.72–0.77
344.87 GB
24WhiteRabbitNeo 13B (Llama 2)Llama 2 (security-tuned)
0.64
0.78 ⚠370.70–0.84
0.47 ⚠130.31–0.64
0.890.83–0.95
0.84 ⚠290.82–0.85
0.86 ⚠50.75–0.92
0.56 ⚠270.50–0.58
0.31 ⚠10.17–0.42
1.00 ⚠161.00–1.00
0.28 ⚠170.21–0.36
0.58 ⚠340.56–0.60
0.72 ⚠620.68–0.75
0.53 ⚠20.50–0.60
0.52 ⚠330.50–0.56
18—
25WhiteRabbitNeo (Llama 3) 8BLlama 3 (security-tuned)
0.63
0.78 ⚠60.70–0.82
0.64 ⚠50.60–0.71
0.79 ⚠10.72–0.89
0.84 ⚠90.78–0.93
0.68 ⚠40.62–0.72
0.50 ⚠170.42–0.58
0.580.58–0.58
0.72 ⚠80.50–0.83
0.36 ⚠170.27–0.47
0.75 ⚠180.69–0.81
0.70 ⚠260.69–0.71
0.23 ⚠20.15–0.30
0.66 ⚠240.63–0.70
305.45 GB
26Vicuna 7BVicuna (Llama 2-based)
0.57
0.570.54–0.61
0.920.86–0.96
0.600.56–0.67
0.67 ⚠10.66–0.69
0.780.72–0.89
0.75 ⚠20.67–0.83
0.25 ⚠40.17–0.33
0.44 ⚠20.33–0.50
0.17 ⚠20.17–0.18
0.690.62–0.73
0.68 ⚠40.63–0.75
0.420.40–0.45
0.50 ⚠20.42–0.56
365.04 GB
27Foundation-Sec 8BLlama 3.1 (security-tuned, Cisco)
0.29
0.49 ⚠660.45–0.58
0.38 ⚠360.29–0.47
0.12 ⚠300.07–0.20
0.50 ⚠360.44–0.54
0.00 ⚠360.00–0.00
0.50 ⚠360.42–0.58
0.11 ⚠360.00–0.33
0.11 ⚠180.00–0.33
0.06 ⚠600.05–0.07
0.47 ⚠520.46–0.48
0.57 ⚠680.49–0.65
0.02 ⚠300.00–0.05
0.42 ⚠450.40–0.47
315.02 GB

Quality against speed

Top left is what you want: a high score and fast writing. Hover a dot for the numbers.

0.20.30.40.50.60.70.80.91.0010203040506070writing speed (tokens per second) → fasteraverage score → betterqwen3.8 27Bqwen3.8 27B: score 0.95, 15.6 tok/sgemma4 12Bgemma4 12B: score 0.94, 34.7 tok/sqwen3.5 9Bqwen3.5 9B: score 0.91, 36.0 tok/sgpt-oss 20Bgpt-oss 20B: score 0.91, 39.3 tok/sMistral Small 24BMistral Small 24B: score 0.91, 10.4 tok/sphi4 14Bphi4 14B: score 0.90, 16.5 tok/sgemma3 12Bgemma3 12B: score 0.89, 19.2 tok/sqwen2.5-coder 14Bqwen2.5-coder 14B: score 0.89, 16.6 tok/sgranite4granite4: score 0.87, 61.7 tok/sLlama 3.1 8BLlama 3.1 8B: score 0.87, 30.3 tok/sqwen3 8Bqwen3 8B: score 0.85, 28.3 tok/sDeepHat V1 7BDeepHat V1 7B: score 0.85, 32.1 tok/sdeepseek-r1 14Bdeepseek-r1 14B: score 0.84, 16.4 tok/sGLM-4 9BGLM-4 9B: score 0.83, 26.8 tok/sgemma3 4Bgemma3 4B: score 0.83, 51.4 tok/sCommand R 35BCommand R 35B: score 0.83, 7.7 tok/sHermes 3 8B: score 0.82, 31.7 tok/sWhiteRabbitNeo 2.5 (Qwen) 7BWhiteRabbitNeo 2.5 (Qwen) 7B: score 0.81, 32.2 tok/scodestral 22Bcodestral 22B: score 0.81, 11.3 tok/sLlama 3.2 3BLlama 3.2 3B: score 0.80, 65.0 tok/sYi 9BYi 9B: score 0.78, 27.5 tok/sNous Hermes 2 11BNous Hermes 2 11B: score 0.75, 22.8 tok/sZephyr 7BZephyr 7B: score 0.64, 34.0 tok/sWhiteRabbitNeo 13B (Llama 2)WhiteRabbitNeo 13B (Llama 2): score 0.64, 18.2 tok/sWhiteRabbitNeo (Llama 3) 8BWhiteRabbitNeo (Llama 3) 8B: score 0.63, 30.2 tok/sVicuna 7BVicuna 7B: score 0.57, 35.7 tok/sFoundation-Sec 8BFoundation-Sec 8B: score 0.29, 30.8 tok/s

Speed and memory

Counters reported by Ollama; median of 3 rounds, model freshly loaded, context 8192. Reading = a log of about 3,000 tokens; generating = 256 tokens.

ModelCold loadReading (tok/s)First token after long promptGenerating (tok/s)Memory
Llama 3.2 3B0.8 s17711.32 s
65.0
2.68 GB
granite40.8 s15131.53 s
61.7
—
gemma3 4B1.1 s16132.02 s
51.4
2.38 GB
gpt-oss 20B4.9 s10892.18 s
39.3
12.85 GB
qwen3.5 9B1.7 s9093.53 s
36.0
7.74 GB
Vicuna 7B1.8 s8224.0 s
35.7
5.04 GB
gemma4 12B1.7 s5695.74 s
34.7
7.75 GB
Zephyr 7B1.8 s6345.44 s
34.0
4.87 GB
WhiteRabbitNeo 2.5 (Qwen) 7B1.6 s6924.61 s
32.2
4.93 GB
DeepHat V1 7B0.5 s6864.65 s
32.1
4.48 GB
Hermes 3 8B2.3 s6463.59 s
31.7
5.19 GB
Foundation-Sec 8B1.8 s6063.81 s
30.8
5.02 GB
Llama 3.1 8B2.1 s6053.83 s
30.3
5.45 GB
WhiteRabbitNeo (Llama 3) 8B1.8 s6093.81 s
30.2
5.45 GB
qwen3 8B0.8 s6474.95 s
28.3
5.24 GB
Yi 9B2.1 s6285.2 s
27.5
5.21 GB
GLM-4 9B2.3 s6703.54 s
26.8
5.53 GB
Nous Hermes 2 11B2.3 s5146.82 s
22.8
6.55 GB
gemma3 12B3.3 s4826.76 s
19.2
7.22 GB
WhiteRabbitNeo 13B (Llama 2)2.8 s3439.5 s
18.2
—
qwen2.5-coder 14B3.1 s31310.27 s
16.6
9.67 GB
phi4 14B2.8 s3326.99 s
16.5
9.55 GB
deepseek-r1 14B3.1 s3349.55 s
16.4
9.03 GB
qwen3.8 27B6.4 s17318.51 s
15.6
16.46 GB
codestral 22B4.3 s22015.63 s
11.3
13.77 GB
Mistral Small 24B1.8 s24413.99 s
10.4
14.36 GB
Command R 35B5.6 s12825.12 s
7.7
20.05 GB

Several agents at once

What happens when five agents talk to the same Ollama server. The server setting decides: one slot makes them queue, several slots cost memory but share the work.

qwen3 8B · 5 agents
28.7 tok/s in total
  • 5.7 tok/s per agent, all done in 20.9 s
  • Memory: 5.24 GB with one request, 5.24 GB with all agents
  • default server setting (OLLAMA_NUM_PARALLEL not set)
qwen3 8B · 5 agents
99.8 tok/s in total
  • 20.0 tok/s per agent, all done in 6.0 s
  • Memory: 8.04 GB with one request, 8.04 GB with all agents
  • OLLAMA_NUM_PARALLEL=5, flash attention, q8_0 KV cache
gemma4 12B · 5 agents
36.5 tok/s in total
  • 7.3 tok/s per agent, all done in 16.4 s
  • Memory: 7.68 GB with one request, 7.71 GB with all agents
  • OLLAMA_NUM_PARALLEL=5, flash attention, q8_0 KV cache

Profiles

Show a radar chart per model
qwen3.8 27BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
gemma4 12BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
qwen3.5 9BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
gpt-oss 20BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Mistral Small 24BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
phi4 14BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
gemma3 12BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
qwen2.5-coder 14BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
granite4CybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Llama 3.1 8BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
qwen3 8BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
DeepHat V1 7BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
deepseek-r1 14BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
GLM-4 9BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
gemma3 4BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Command R 35BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Hermes 3 8BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
WhiteRabbitNeo 2.5 (Qwen) 7BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
codestral 22BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Llama 3.2 3BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Yi 9BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Nous Hermes 2 11BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Zephyr 7BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
WhiteRabbitNeo 13B (Llama 2)CybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
WhiteRabbitNeo (Llama 3) 8BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Vicuna 7BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure
Foundation-Sec 8BCybersecurityDailyCodingDutchToolsHonestyReasoningLongHardStandardPentestingTriageSecure

Limitations