gemma-3-4b · Q4 quantized · CPU only, 4 cores, no GPU · fully local — no API key, no cloud, no vendor · driven by assay in text mode (no native tool calling, the Gemma 2/3-era path)
A real seeded defect from the bench/workflow benchmark — the customer-facing symptom, not the fix:
The environment's proxy blocked the Ollama installer and GitHub binaries. The model came in anyway: llama.cpp runtime as prebuilt npm binaries, 2.5 GB of GGUF weights straight from Hugging Face, and a 70-line hand-rolled OpenAI-compatible shim. First reply from local silicon:
Every prompt longer than a few hundred tokens segfaulted the prebuilt inference binary — three configs, same native crash. Note what the harness did about it: nothing resembling a result.
Exit 4, "interrupted", zero tokens, resumable. An infrastructure crash is recorded as an infrastructure crash — never scored against the model, never papered over as success. The floor holds even when the floor is the problem.
llama.cpp rebuilt from source against the actual CPU — and the crash survived. Any prompt beyond a few hundred tokens, any role, any batch configuration: same segfault. That narrows it to the Node bindings layer, not the math kernels. Each crash logged with its real signal; still zero fabricated results.
llama-server proper
The source fetch brought the entire llama.cpp tree along, so the battle-tested official server — its own HTTP layer, its own template handling, no bindings, no shim — is compiling right now. The 4B has answered every prompt that ever reached it; the experiment has so far only proven things about runtimes. This page updates when the run completes — pass or honest halt, whichever the gate says.
llama-server from source
The official server took a 3,200-token prompt on the chin (87s on 4 CPU cores) and answered correctly. The bindings were the villain all along; the model never was.
The 2024 method, verbatim: the harness gathers deterministically (runs the repro, isolates the 14-line function), then asks the 4B one narrow question at a time — diagnose in a sentence, hand-trace the failing input, output only the corrected function. The harness applies each patch, runs the gate, rolls back on failure, and feeds the REAL wrong output into the next small prompt. Repeated identical fixes are rejected by name.
Identical harness, identical prompts, identical bug — only the model size changed. Diagnosis on the first try, then a hand-trace the 4B never got right, then a fix that passed the gate on attempt 1:
It replaced a one-shot rollover with a while loop — the correct general fix, not a patched special case — and re-verified with node product/cli.cjs due 2026-01-31 → 2026-03-02, live, after this page was written.
gemma-3-4b (local, CPU): correct diagnosis, correct hand-trace, could not land the fix — two honest halts across two protocol rounds, zero false claims. gemma-3-12b (local, CPU, same box, same protocol): correct diagnosis, correct trace, verified fix on attempt 1. Same harness, same gate, same bug — the only variable was model size, and the gate is what tells the difference apart. Nothing here was taken on anyone's word: on the other side of this same fence, a commercial harness reported fabricated successes for models 5× the size of the one that just passed. This is the model that ran real production timesheets on a MacBook for years, and the receipt above is why that was ever sane to trust.
Same protocol, moved to the cloud (gemma4:31b) — and a real A/B, not a guess, against two unrelated bugs.
The date-rollover bug and a fresh percentage-discount bug, each run through the many-small-prompts protocol AND assay's normal single-context mission mode, same model both times:
The verdict: not capability, cost. Both arms passed both bugs on the first real attempt — a 31B model doesn't need the small-prompt protocol to succeed, it was already strong enough to explore the files itself. What the protocol buys is efficiency: ~30-70× fewer tokens, because the harness does the file-finding and re-verifying deterministically instead of the model spending tool calls on it. For the 4B, small prompts were the difference between an honest halt and a pass. For the 31B, they're the difference between a $ and a ¢.
Re-running the date-bug mission to isolate the token cost, the model repeatedly answered with markdown-fenced pseudo-tool-calls the harness couldn't parse — a format the harness had never been fed before. After one nudge, the harness gave up and accepted the model's unverified prose as the finished answer. The receipt said PASS. The file was never touched.
Root cause: the validation gate only ran when the model had made at least one real tool call. A model that never managed one skipped the gate entirely — so it could never fail, so the turn fell through to "completed." Exactly the fabrication class this whole project exists to catch, found in the one corner case that had never been tested: a model too confused to call a tool at all, rather than one that called tools and got the wrong answer.
The gate now also runs — and can honestly fail — whenever the task clearly needed tools but the model never landed one. Confirmed with three levels of proof: a direct harness test, a test that renders the real Ink terminal UI and reads the actual frame, and a test that drives the real GUI over its actual WebSocket protocol. Both new surface-level tests were sanity-checked against the unfixed code first, to prove they'd actually catch the regression rather than just exercising happy-path code:
Also tightened the context-pruning threshold while in there: a many-small-tool-call mission never crossed the old 700-char floor, so none of it ever got pruned — every tiny result was rebilled in full on every later call. After the fix, the date-bug mission re-ran at 55,683 tokens on 12 tool calls, down from 83,921 on 24, same verified result.
Every line of terminal output above is captured from the real run, unedited. The harness, gates, and benchmark are in the repo; the receipt lands here when it exists. ← back to samples