Strim benchmarks

What the data supports, and what it does not

Strim against rapid-mlx, mlx-vlm, mlx-lm and oMLX on Qwen3.8-27B 4-bit, one machine. Every claim below is computed from the cells in Results; losses are listed next to the wins.

• Decode

  24-50% faster decode

  than rapid-mlx, the closest rival; 1.8-2.3x the speed of mlx-vlm, 1.9-2.5x the speed of mlx-lm and 1.7-2.2x the speed of oMLX, at 4K to 32K context. At 4K: Strim 75.34 vs rapid-mlx 50.12 vs mlx-vlm 32.69 vs mlx-lm 30.11 vs oMLX 33.53 tok/s.

  Engines differ in speculative decoding (MTP) and prefix caching; see the table of rivals above. Details

• Four agents, one 16K prefix

  2.8-2.9x faster aggregate

  Four clients on one 16K prefix, aggregate tok/s. Median first token: Strim 20.29 s vs rapid-mlx 73.53 s vs mlx-vlm 70.48 s vs mlx-lm 73.4 s.

  Engines without a cross-request prefix cache cannot share the prefix in this setup. Details

• Median first token under concurrency

  29-56% lower, 4 to 8 concurrent

  Against each rival, over 512-token, agent-style and 4K prompts.

  Some rivals report first tokens batch-synchronously, so their median is a weaker number. Details

• Agent replay

  48.93 s 8-turn replay

  Summed prefill. Verdict against rapid-mlx (49.13 s): TIE; mlx-vlm (192.5 s): WIN; mlx-lm (55.02 s): WIN; oMLX (64.84 s): WIN.

  Engines without a cross-request prefix cache re-prefill every turn. Details

Strim vs rapid-mlx, mlx-vlm, mlx-lm and oMLX at a glance

Head-to-head of 2026-10-28 (h2h-final3-20261028): Strim build ee9f9d8 (strim 0.1.1); rivals: rapid-mlx (MTP) 0.15.7, mlx-vlm (no MTP) 0.7.10, mlx-lm (no MTP) 0.32.2, oMLX (no MTP) 0.4.5.dev1. Qwen3.8-27B 4-bit + MTP head on Apple M5 Max, 64 GB, macOS 27.0.1, 2 passes of one rep. Wins and losses are drawn the same way. A verdict is WIN or LOSS only outside a 2% band. Only numbers measured on that build are shown; work done since is not re-benchmarked and not included.

Regimes with at least one LOSS against a rival: Cold prefill, Concurrency, Latency. Cells are counted once (cold prefill by tok/s, not again by first-token time). Every cell is in the full table.

Memory

Peak physical footprint over the whole run: Strim 30.8 GB, rapid-mlx 35.9 GB, mlx-vlm 36.3 GB, mlx-lm 37.5 GB, oMLX 55.2 GB. Claims compare each engine with Strim; the whole-run values are measured maxima, not steady-state memory.

Strim has the lowest peak in 2 of 9 workload cells.

Latency

Median first-token WIN counts (p50): rapid-mlx 6 of 6; mlx-vlm 6 of 6; mlx-lm 6 of 6; oMLX 2 of 6. Tail p95 WIN/LOSS counts: rapid-mlx 6 of 6/0 of 6; mlx-vlm 1 of 6/1 of 6; mlx-lm 6 of 6/0 of 6; oMLX 2 of 6/0 of 6. With four or eight requests per cell p95 is the slowest request, and the rivals deliver first tokens batch-synchronously, so their p50 is a weaker number than Strim's.

Functionality

Strim is a one-model engine, so it does not try to match the rivals on breadth. Rows are checked against the installed packages, their --help, their README or this repository; see the comparison notes for the evidence behind every cell. Rows with a Yes: Strim 16 of 20, rapid-mlx (MTP) 14 of 20, mlx-vlm (no MTP) 11 of 20.

Not found means no evidence was found in what was checked, not proof of absence. Unknown means not verified. Where a rival is ahead of Strim (Strim has No or Partial, a rival has Yes): Runs many models; Reasoning / thinking separation; Memory budget / swap protection.

Wider field: five engines, with two earlier-run points

• Dashed hollow beads are llama.cpp and Ollama from the 2026-10-26 run.

The five filled points are medians from the same 2026-10-28 head-to-head, built from Strim ee9f9d8; llama.cpp and Ollama are hollow dashed beads from the earlier run. They are visual context only. Current run source; earlier run source.

Results

The headline rows first, then every cell. Bars always start at zero. Every number carries its figure id as a hover title and appears in the figure index.

Final head-to-head: Strim vs rapid-mlx, mlx-vlm, mlx-lm and oMLX

Qwen3.8-27B 4-bit served alone over HTTP by each engine, one engine on the GPU at a time, thinking off, greedy decoding, on one Apple M5 Max with 64 GB. 2 passes (pass 1 in the order Strim, rapid-mlx, mlx-vlm, mlx-lm, oMLX; pass 2 reversed), one rep per pass, so each value is the median of two runs, which is their mean. The largest swap growth in any session was 0 MB. Strim is build ee9f9d8; rapid-mlx ran as --no-telemetry serve models/qwen38-4bit --port 8651 --max-num-seqs 4 --speculative-config {mtp,qwen38-mtp-4bit} (MTP: yes (--speculative-config mtp, qwen38-mtp-4bit); prefix cache: yes, default on). mlx-vlm ran as --model models/qwen38-4bit --port 8651 (server defaults) (MTP: no; prefix cache: no cross-request reuse (0% cached in replay)). mlx-lm ran as --model models/qwen38-4bit --port 8654 --decode-concurrency 4 --prompt-concurrency 4 (MTP: no (no speculative decoding); prefix cache: yes, default in-memory LRU prompt cache). oMLX ran as serve --model-dir <scratch>/models (symlink to models/qwen38-4bit) --port 8653, HOME=<scratch>/home, defaults otherwise (memory-guard balanced, max-concurrent-requests 8) (MTP: no (MTP needs oMLX's own checkpoint; stock weights have no MTP head); prefix cache: yes, default paged SSD prefix cache under the scratch HOME (fresh per session, so no cross-session hits)). Verdict: WIN / LOSS when the ratio is outside a 2% band, otherwise TIE. Raw files, launch lines and the full per-rival tables: summary.md.

Strim decodes 24-50% faster than rapid-mlx, the closest rival; 1.8-2.3x the speed of mlx-vlm, 1.9-2.5x the speed of mlx-lm and 1.7-2.2x the speed of oMLX, at 4K to 32K context. rapid-mlx also uses MTP, so the gap to it is the engine, not the speculative head. mlx-vlm, mlx-lm and oMLX have no MTP, so their gap includes it.

Cold prefill (tok/s): within the 2% band of the best rival at none, 2-4% behind it at 4K, 16K and 32K. A tie is not a win.

Strim has the lowest median first-token latency at 4 to 8 concurrent requests, 29-56% lower than each rival. Caveat: rival p50 and p95 are nearly identical (their requests report first tokens batch-synchronously), so their p50 is a weaker number than Strim's per-request TTFT.

Strim is more than 2% behind the best rival in 3 of these six cells, by 2-4%. Batched decode narrows the gap but the remaining cost is the quantized matmul at 4 to 8 rows (below).

Strim's p95 is more than 2% worse than the best rival in 1 of these six cells, by 6%. The same staggering that gives a low median pushes the last request out.

With a shared 16K prefix Strim is 2.8-2.9x faster than each rival in aggregate throughput.

8-turn replay, summed prefill: Strim 48.93 s vs rapid-mlx 49.13 s vs mlx-vlm 192.5 s vs mlx-lm 55.02 s vs oMLX 64.84 s. Verdict against rapid-mlx: TIE; Verdict against mlx-vlm: WIN; Verdict against mlx-lm: WIN; Verdict against oMLX: WIN. Engines with a cross-request prefix cache reuse part of the turn-8 prompt; the others re-prefill it. Replay resends recorded assistant turns, so Strim's token ledger and hot-session reuse do not apply: a lower bound for Strim on a live agent.

Verdicts: WIN means Strim is better by more than 2%, LOSS worse by more than 2%, TIE within 2%. Lower-is-better cells (TTFT, time) win with a ratio below 1.

Valid = results.json claim_eligible for that engine and cell. A measured but ineligible comparison is shown with its value and marked no; aborted oMLX cells show n/a.

Honest losses

• cold 4K prefill: 2% behind mlx-vlm (ratio 0.978×).
• cold 4K TTFT: 2% behind mlx-vlm (ratio 0.978×).
• cold 16K prefill: 3% behind mlx-vlm (ratio 0.975×).
• cold 16K TTFT: 3% behind mlx-vlm (ratio 0.975×).
• cold 32K prefill: 4% behind mlx-vlm (ratio 0.958×).
• cold 32K TTFT: 4% behind mlx-vlm (ratio 0.958×).
• c=4 512-tok prompts aggregate: 3% behind rapid-mlx (ratio 0.966×).
• c=8 512-tok prompts p95 TTFT: 6% behind mlx-vlm (ratio 0.942×).
• c=4 4K prompts aggregate (prefill-bound): 2% behind mlx-vlm (ratio 0.979×).
• c=8 4K prompts aggregate (prefill-bound): 4% behind mlx-vlm (ratio 0.960×).

Whole-run maximum over passes and sessions, not steady state. oMLX measured 55.2 GB and exceeded the watchdog threshold; its abort note is retained in the session flags. Largest swap growth: 0 MB.

Greedy agreement (sanity check only)

8 fixed prompts, 64 greedy tokens. Strim against itself across the two passes: 8/8 identical. Strim against rapid-mlx: 7/8; Strim against mlx-vlm: 6/8; Strim against mlx-lm: 6/8; Strim against oMLX: 7/8. The divergences are bf16 near-ties, where a single flipped token changes every later one; it is a sanity check, not a quality measurement.

Caveats that change how to read the table

• Validity. 28 of 30 primary sessions were clean; 2 were marked polluted. Reruns are retained in the data. Claims use only cells marked claim eligible (highest primary-session 1-minute load 4.7). Method and every cell.
• Two passes, one rep each. Pass-to-pass spread is listed in the raw summary; prefill gaps of a few percent sit near the edge of the 2% tie band, so a single flipped pass can change a verdict.
• Rival configuration. rapid-mlx ran as --no-telemetry serve models/qwen38-4bit --port 8651 --max-num-seqs 4 --speculative-config {mtp,qwen38-mtp-4bit} (MTP: yes (--speculative-config mtp, qwen38-mtp-4bit); prefix cache: yes, default on). mlx-vlm ran as --model models/qwen38-4bit --port 8651 (server defaults) (MTP: no; prefix cache: no cross-request reuse (0% cached in replay)). mlx-lm ran as --model models/qwen38-4bit --port 8654 --decode-concurrency 4 --prompt-concurrency 4 (MTP: no (no speculative decoding); prefix cache: yes, default in-memory LRU prompt cache). oMLX ran as serve --model-dir <scratch>/models (symlink to models/qwen38-4bit) --port 8653, HOME=<scratch>/home, defaults otherwise (memory-guard balanced, max-concurrent-requests 8) (MTP: no (MTP needs oMLX's own checkpoint; stock weights have no MTP head); prefix cache: yes, default paged SSD prefix cache under the scratch HOME (fresh per session, so no cross-session hits)). Defaults were used otherwise; rival tuning was not explored.
• TTFT definitions differ in practice. Rivals report batch-synchronous first tokens, so aggregate throughput is the only fully comparable concurrency number.
• Later commits are not included. Only the build named above was measured.
• One model, one machine, thinking off, synthetic agent prompts and a synthetic 8-turn fixture.

Prefix cache and memory

Coding agents resend the same system prompt, tool schemas and history every turn. The cache stores engine state at chunk boundaries and restores the longest match, so only the new tail is prefilled. The latest cache work added a cut at the start of each assistant turn, so the end of the previous prompt becomes a cache node, and a footprint cap that sheds stale nodes under memory pressure.

Replay: the final head-to-head measured 48.93 s for the same replay on the same build. Cold 16K first token stayed at 17.2-17.3 s (no regression). The earlier footprint growth was by design (the trie filled to its cap), not a leak. The head-to-head numbers already include this work; do not add the two together.

n=1 per row, in-process, no HTTP, earlier build. A hit still pays for the unshared tail. The aligned-chunk policy costs 0.6% versus one-shot prefill at 4K. A trie node is 210.8 MB; the MTP head adds 4.01 MB per node.

How it works

Decoding a 27B dense model is limited by memory bandwidth: every token streams the whole weight set. The gains come from more tokens per weight read, avoiding repeated work, and not wasting time between kernels.

MTP speculative decoding

The Qwen3.8 checkpoint ships a small multi-token-prediction head. Strim loads it (models/qwen38-mtp-4bit), drafts tokens with it and verifies them in one pass of the full model. Rejected drafts are rolled back exactly: the attention cache is sliced and per-token linear-attention (GDN) states are restored. Depth adapts (2-4 tokens). A 48k frequency-ranked draft vocabulary keeps the head cheap; verification uses the full vocabulary, so a missing token costs a rejected draft, never a wrong output. For sampled decoding the accept/residual/bonus rule keeps the output distribution unchanged.

Exact, chunk-aligned prefix cache

A trie over 1,024-token chunks holds KV, GDN state, conv state and the MTP head state per node, with an LRU byte budget. Prefill always runs in aligned chunks of that size whether or not the cache is used, so a hit restores exactly the state a miss would have computed. The token ledger records the exact ids the model produced, so resent histories reuse the model's own tokens and keep hitting the cache.

Blocked GDN prefill kernel

Linear-attention layers need a sequential recurrence. A blocked Metal kernel stages 48-token blocks in threadgroup memory and keeps state in registers. It was ported from rapid-mlx (Apache-2.0, itself adapted from oMLX) and is credited in NOTICE. At 4K it measured 825-846 tok/s before and 915 tok/s after at light load (878 at heavier load). The reduction order differs from the serial kernel, which is the source of the near-tie differences in Quality.

Batched decode kernels

Concurrent requests decode in lockstep through one batched forward pass. Batch-dimension GDN decode and conv Metal kernels (one dispatch for all sequences, per-row arithmetic identical to the single-stream kernel) cut the B=4 step from 49.5 to 42.3 ms against a single-stream floor of 34 ms. Served aggregate throughput went from 57.4 to 63.3 tok/s at c=4 and from 66.2 to 73.1 at c=8 in that sweep.

Those sweep numbers were taken under machine load and are already included in the build used for the head-to-head; they are not additional gains. The remaining c=4 gap is the quantized matmul at 4 to 8 rows. MTP inside the batch was measured and rejected (upper bound 1.09x before draft cost). See PLAN-CONCURRENCY.

Admission control

Every request takes a slot before the engine is touched. Interactive requests go first; background work (x-strim-priority: background, delegated jobs) is capped one slot below the total and re-queues between agent turns while interactive work is active, never mid-token. The default total is 8 slots (an admission cap, not the engine batch size). The engine also admits by memory and refuses requests that would not fit. Waiters are bounded and have a deadline. See ADMISSION.md.

Sandboxed harness

Delegated jobs run the local model as a coding agent in an isolated git worktree under sandbox-exec (no network, no credential reads, scrubbed environment), with contained file tools and a harness-run verify command. See Agents and Security.

Tried, measured, rejected

• MTP inside the batch. Verify cost grows steeply with batch size: at 1K context 34.0 ms/pass for one token vs 120.3 ms/pass for three tokens each across four sequences. Upper-bound speedup over plain batching on prose 1.25× at B=2 and 0.99× at B=4, before draft and rollback costs. No-go. BATCH-MTP.
• A custom 4-bit prefill GEMM. 4-bit matmul already runs at 0.90-0.98 of dense bf16 and dense GEMM plateaus at 62-63 TF/s; the optimistic end-to-end bound is +5-8%, below the plan's gate. No-go. PREFILL-SPIKE.
• Batched prefill. B=4 at 4K ran at 0.95× the serial speed. Not adopted. BATCH-PREFILL.

Quality and exactness

The claim is narrow and has two parts.

• Exact where it says so. A prefix-cache hit equals a miss bitwise (final-position logits and greedy tokens), within one fixed chunking policy: prefill always runs in aligned chunks, so cached and uncached runs follow the same arithmetic. That policy itself differs from a one-shot prefill (it costs 0.6% at 4K).
• Not bitwise where it does not. Batched decode, the blocked GDN prefill kernel, speculative decoding and the cache cuts can change near-tie tokens (bf16 logits that tie within rounding; one flipped token changes every later one). Measured: greedy output over 8 fixed prompts and 64 tokens, Strim against itself across passes 8/8 identical, against rapid-mlx 7/8, against mlx-vlm 6/8. The batch kernels raised agreement with each prompt's own single-stream run from 6/8 to 8/8 at 96 tokens.

Multi-step agent quality has been checked on the corpora described in Agents (a real-client soak with toy tasks, a small real-client delegation run, and a Swift and Go task corpus of 17 tasks), not on a broad benchmark. What is not validated is listed under Limits.

Parity gates against Hugging Face

• Tokenizer. Byte-level BPE checked against the HF tokenizer on encode, decode (including special tokens) and every chunking of streamed decode. Fixture tokenizer_parity.json, frozen; test TokenizerParityTests (skipped when the weights are absent).
• Chat template. The native Qwen3.8 renderer (tool calls, reasoning, generation prompts) against HF-rendered text. Fixture template_parity.json.
• Prompt ids. Template then tokenizer, including incremental multi-turn sessions through the ledger, equals HF ids. Fixture prompt_parity.json.
• Special-token injection. User text (messages, reasoning, tool results, tool names, arguments, schemas) is tokenized as plain text, so a user cannot smuggle control strings into the prompt. Fixture prompt_adversarial.json.

Prefix cache gate

The checkprefix gate compares a cold run against hits on the final-position logit hash and 16 greedy tokens, and checks a cold store against the no-store reference, a second hit (no aliasing), a short prompt hitting a longer stored prefix, a forked tail, eviction under a tiny budget and an unrelated prompt missing cleanly. With MTP it also compares head KV, seed hidden state, logits and 64 speculative tokens. Logs: 4K, 8K, MTP. A hot session continued for 20 tokens equals a cold re-prefill (IDENTICAL over 20 tokens).

Known near-tie caveats

On 8 natural prompts (1K and 4K) the first generated token agreed on 8/8 prompts across all three kernel paths and every run was deterministic. The next 16 greedy tokens agree on most prompts and diverge at a bf16 near-tie on others:

Divergences are at logits that tie within bf16 rounding. Speculative-vs-greedy parity at 4K is not exact: 0/2 prompts matched exactly. Source: longctx-validate. Earlier build; the kernel is unchanged.

Agent harness, MCP and soak

Status in three parts: what is built, what has evidence, and what has not been validated.

Built

• A job harness that runs the local model as a sandboxed coding agent in a git worktree, with a required verify command the harness runs itself (verified, failed or unverified).
• An MCP endpoint on strim serve with 4 tools (strim_task, strim_job, strim_cancel, strim_ask).
• A real-client soak driver (strim soak) and an agent benchmark (strim agentbench) over a corpus of 17 Swift and Go tasks (12 test-fix, 3 mechanical-edit, 2 repo-qa).

Evidence

• Delegation works end to end on trivial tasks. Real Claude Code and Codex delegated through the MCP tools: 4 of 4 runs passed their deterministic checks on the stock release build (n=1 per cell, one-line fixes and questions).
• Token saving is not shown. On a one-line fix the parent used 29.9K input tokens delegated against 25.5K alone, so no saving (n=1).
• Soak, earlier build. One 30-minute run of real claude -p and codex exec clients against the server on toy tasks (below). It shows the client-to-server path was stable, not how well the model codes. Results: pass rates below.
• Agent benchmark, free arms. One run of 120 runs (harness arm and Claude Code arm on the local model) exists, on the earlier corpus. Part of it had an unusable test environment inside the sandboxes, so its pass rates measure the environment and are not quoted here. The corpus was then reduced to Swift and Go and has not been re-run.

Scope. This soak ran on the earlier build with the earlier harness (2026-10-27). The 4 failures were all Claude Code at one task whose test command the harness allow-list denied; fixed afterwards (1/4 before the fix, 4/4 after, in short reruns outside the 30-minute result). A run that failed because of a harness path error is kept as evidence and excluded from every figure.

Not yet validated

• A live soak with real Claude Code and Codex on the current build under the current sandboxed harness (the sandbox path of the new harness has not been exercised live).
• The paid agent-benchmark pilot (parent arms with real subscriptions). It has not been run, so there is no evidence that delegation saves parent tokens. Plan: PLAN-PRODUCT.
• The agent benchmark on the current Swift and Go corpus, with multi-step tasks, for batched decode and the cache cuts against single-stream.

Security posture

Strim is a local, single-user server. It is not designed to be exposed to a network, and it does not defend against another process running as the same macOS user.

Design

• Loopback only, with Host and Origin checks (DNS-rebinding and cross-site defence) and a refusal of cross-site fetches.
• Bearer token by default (~/.strim/token, mode 0600, constant-time comparison), plus a separate control token for the menu-bar app. --no-auth opens only the API side for loopback development.
• The local model is untrusted. Its shell runs under sandbox-exec (no network, writes only to the job worktree, no reads of credential directories), file tools are contained to the job workspace, jobs run in an isolated worktree, and a patch is applied only after an explicit approval carrying the SHA-256 of the exact bytes reviewed.

Reviewed and fixed (2026-10-28)

Three reviews: the server HTTP surface, every process launch and model-influenced code path, and release hygiene (secrets, personal data, licenses, supply chain).

• Server surface: 14 findings, no critical and 1 high (an integer overflow trap from a client-controlled token count). 6 are fixed and one more partly: token-count clamping and saturating admission math, connection cap and head, body and idle deadlines, bounded request pipelining, a bounded response store, a waiter cap and queue deadline, a job-queue cap, and a minimal unauthenticated /health.
• Process launches: 5 findings fixed: credential directories refused as job working directories, sandbox reads of shared temp areas denied, paid benchmark arms without a shell tool, the MCP token passed by environment reference instead of written to a file, and an opt-in for the unsandboxed legacy scripts.
• Release hygiene: personal paths and a hostname scrubbed from committed results, token files ignored by git, license notices completed.

Residual risk, stated plainly

• Any project under a non-denied home directory is allowed as a job working directory by default.
• Paid benchmark arms must let the client read its own login file; file tools are denied it by permission rule, not by the sandbox.
• No per-route body limit (one cap for all routes); no wall-clock limit on a running generation (output length is bounded instead); one API token means one priority class.
• Token-store hardening against symlink races and permissive MCP token defaults are documented but not reworked (same-user only).
• The reviews were read-only: no live server test, no fuzzing of the tokenizer or template parsers, and some sandbox fixes were not executed under sandbox-exec. The low and info items of the process-launch review were not addressed.

Report a vulnerability through GitHub private vulnerability reporting (the repository's Security tab); see SECURITY.md. Reviews: server, process launches, release hygiene.

Supported scope

Strim supports Qwen3.8-27B (dense), 4-bit affine g64 with its MTP head, and nothing else. It has been tested only on a 64 GB M5 Max. Other quantizations are on the roadmap and not shipped; other models (including Qwen4) are not supported.

Computed from the model config with an assumed 5.5 GiB for macOS and apps; weights are 14.09 GiB of text-only 4-bit and the fixed engine overhead is about 17.0 GiB. On the test machine the GPU working-set limit measured 56.0 GiB. Source: RESEARCH-QUANT-TIERS. No smaller Mac has been tested.

16 GB and below are unsupported. The rows for 24 to 48 GB are computed estimates, not measurements; no smaller Mac has run Strim.

Limitations

• One model, one quantization, one machine class (64 GB Apple Silicon), one user. All measurements come from a single M5 Max.
• Two passes with one rep each. Treat differences of a few percent as noise; gaps of a few percent sit at the edge of the 2% tie band.
• Not faster: cold prefill is 2-4% behind it at 4K, 16K and 32K; aggregate throughput at 4 to 8 concurrent requests trails the best rival by more than 2% in 3 of 6 cells (2-4%); p95 first-token latency is more than 2% worse in 1 of 6 cells (6%).
• Only the build named in the run summary at the top was benchmarked; later commits were not.
• The prefix cache is in memory only. Speculative decoding and batching can differ from greedy at bf16 near-ties.
• Not yet validated: live soak on the current build and harness, the paid agent-benchmark pilot, token savings from delegation, multi-step quality of batched decode and the cache cuts, other Mac memory tiers (see Agents and Scope).
• The shipped StrimApp is a menu-bar shell over an older service and does not drive strim serve.

Roadmap

• Qwen4 readiness — planned. Make layer and head dimensions config-driven instead of fixed to Qwen3.8-27B (research note).
• Mixed quantization per memory tier — planned. Other bit widths for other Macs, with a quality gate (research note).
• GPU-index decode — planned.
• Batched-matmul tuning for c=4 — planned: the quantized matmul at 4 to 8 rows is the remaining aggregate-throughput gap.
• Cold-prefill kernels — planned: the cold-prefill gap at long prompts.
• Paid agent-benchmark pilot — planned: the test that decides whether delegation saves parent tokens.
• Menu-bar app and a disk-backed prefix cache, after the benchmark gate.

Reproduce

Hardware: Apple M5 Max, 64 GB, macOS 27.0.1. Versions: strim 0.1.1 (ee9f9d8), rapid-mlx 0.15.7, mlx-vlm 0.7.10, mlx-lm 0.32.2, oMLX 0.4.5.dev1. Run GPU benchmarks one at a time in a foreground shell (the runner takes lockf /tmp/strim-bench.lock). Models live under models/; rival engines must be installed separately.

# head-to-head: OUTDIR [passes] [engines] [cell groups]; pass 2 runs the engines in reverse order
bench/h2h/run.sh bench/results/<dir> 1,2 strim,rapid-mlx,mlx-vlm,mlx-lm,omlx g1,g2,g3
bench/h2h/summarize.sh bench/results/<dir>          # writes summary.md and results.json

# flags used
strim serve models/qwen38-4bit models/qwen38-mtp-4bit --port 8203 --no-auth
rapid-mlx --no-telemetry serve models/qwen38-4bit --port 8651 --max-num-seqs 4 --speculative-config {mtp,qwen38-mtp-4bit}
mlx_vlm.server --model models/qwen38-4bit --port 8651 (server defaults)
mlx_lm.server --model models/qwen38-4bit --port 8654 --decode-concurrency 4 --prompt-concurrency 4
omlx serve --model-dir <scratch>/models (symlink to models/qwen38-4bit) --port 8653, HOME=<scratch>/home, defaults otherwise (memory-guard balanced, max-concurrent-requests 8)

# engine-level gates
swift test --disable-sandbox
.build/release/strim checkprefix models/qwen38-4bit 4096 3 models/qwen38-mtp-4bit
.build/release/strim enginebench models/qwen38-4bit models/qwen38-mtp-4bit

# real-client soak
lockf -k /tmp/strim-bench.lock .build/release/strim soak --duration 30 --arms both

# regenerate this page and the README from site/data/results.json
.build/release/strim site build
.build/release/strim site check

The model-backed tests skip cleanly when models/ is absent. Details: BENCH.md, bench/h2h/run.sh, soak/README.md.

Methodology

• Final head-to-head. Runner bench/h2h/{run,pass,summarize,watchdog,agree,agent-prompts}.sh (shell, jq, awk). Per pass and engine, sessions are short and contiguous; each engine is the only GPU job; a watchdog aborts on memory limits. The figure rows are generated from summary.md by strim site build and re-checked cell by cell against it.
• Thinking off for all engines so the chat template is identical. Decode = (completion tokens − 1) / (last chunk time − first chunk time) from usage.completion_tokens. TTFT = first non-empty output chunk minus send time.
• Cold prompts carry a unique head so no engine can hit a cache across reps. Statistics: medians over two passes; no confidence intervals (the sample is too small).
• Earlier 8-engine run (2026-10-26) not repeated for the final build. Only the llama.cpp and Ollama decode points are shown as hollow dashed earlier-run beads; their full historical scoreboard is not mixed with this run.

Raw data: h2h-final3-20261028, cache-footprint, concurrency plan, soak run.

License and credits

MIT, see LICENSE. Third-party notices: NOTICE and THIRD-PARTY.md. The blocked GDN prefill kernel is ported from Rapid-MLX (Apache-2.0), which adapted it from oMLX (Apache-2.0); the Apache-2.0 text is in LICENSES/. MLX Swift (MIT) is linked through SwiftPM. Contributions: CONTRIBUTING.md.

Figure index

Every number on this page is one of the 592 figures below, defined in site/data/results.json with its source file, method note and caveat. strim site check verifies that each value occurs in its source file.