Failure Triage

The AI Model Benchmark runs coding agents across many harnesses. When a run ends without a deliverable, someone has to decide if it was the model (it gave up or delivered garbage) or the infrastructure (the provider cut the stream, credits ran out, tmux lost the session, the prompt was never delivered). Getting this wrong puts a model DNF on the public grid for a harness bug. failure-triage labels each run's ending from its log and routes confident labels automatically. Unsure ones go to a human.

One request per run, three typed questions on the same state (code-extracted facts plus about 25 filtered log lines):

Why decomposing helps. Code states the facts (tokens, whether the model got the prompt, whether usage was missing, artifact count, HTTP codes). The model makes one closed-set judgment. Code then applies the policy:

• the 0.90 confidence gate;
• "facts beat judgments": a model_* label is sent to a human when the model never got the prompt, or when the endpoint reported no usage.

Only the class question can be wrong. Everything around it can be checked. Pattern sources: tax-doc-classifier (closed-set Choice plus an escape option), Canny (facts can overrule), TypeSafe's jaggedness page (filter the log first).

Real data. The AI Model Benchmark's 6 real _infra_dnf runs (Kimi Code / Kimi K3 stream cut, OpenCode / DeepSeek prompt never delivered, Claude Code / GLM tmux name bug ×2, Grok Build 403 refusal, Claude Code / GLM "stream ended without receiving any events"):

• The reviewer's own diagnosis is used only as the expected label and is never sent to the model.
• Folder names carry the diagnosis, so the model sees only the neutral cell id.

Hand-written fixtures (8 runs, rubric v2): 8/8 correct, 8/8 auto-routed. In 3 nonce runs every confidence was ≥0.996, with no flips.

Failures, stated plainly:

• The Kimi case is hard. The endpoint reported 0 tokens, but the model had reasoned for 17 minutes. Without the harness's own context count as a fact, openjev confidently called it "the model gave up".
• The fix was a code fact plus a guard.
• Both GLM tmux rows are right but only at 0.70–0.74 confidence, so they go to a human.
• On this data, openjev's confidence separates the easy cases (≥0.99) from the hard ones (0.63–0.74) well. The auto gate never fired on a wrong label once the guard was in.

Full details, raw outputs and the mock baseline: RESULTS.md and results/.

Real Jev = jev-1.13-free on OpenCode Zen's limited-time free tier, run 2026-09-24 13:19–13:21 UTC with JEV_PROVIDER=zen. It was $0, and no response carried a cost field. Each result is labelled REAL JEV (OpenCode Zen free, jev-1.13-free). The free tier rate-limited us (HTTP 429 FreeUsageLimitError) after about 288 calls in about 2 minutes. Later calls were paced at ≤4 per minute.

Real Jev lands on the same labels, and is less sure on the genuinely ambiguous tmux rows (as low as 0.39). That is what a calibrated model should do, and those rows go to a human either way. The v2 class and the fact guard were needed on both models.

Raw outputs: results/zen_*.

• Real DNF set: about 1,150 input tokens per run (6,925 for 6).
• Hand-written fixtures: about 1,030 per run.
• Latency: median 360–640 ms per request across runs.
• Cost: $0 on Codiv. At TypeSafe's list price ($0.042 per 1M input tokens) that is about $0.00005 per run, or roughly $0.05 per 1,000 runs.
• Everything in this upgrade: 106 requests, about 118k tokens.

(Written before the real-Jev run. The section above shows what it did.)

• Confidence may behave differently. Real Jev is trained for calibration (RLCD). The 0.90 gate needs re-measuring on it before anything is auto-labelled. Run the nonce test again first (see the Latent Space notes).
• Pin the version. Pin jev-1.13.x in the thresholds. The talk says there is no LTS, so re-tune when the version string changes.
• Keep the guards. The v2 class and the usage-missing fact guard are model-independent and should stay whatever the model.
• Next data. Ask the reviewer for a larger labelled export (about 50 finished runs, not only DNFs) to measure false infra labels on successful runs.

• Latent Space (Diogo Almeida): "verify everything" is one of the use cases he names. Confidence decides escalation, and the nonce test is his robustness check. We ran it: 29/30 stable, and the flips only happen below the gate.
• building-with-jev (dbreunig): its advice is "keep the answer space stable once code depends on it", so v1 and v2 are versioned and reported separately. It also says "a Noul has no confidence; distance from 0.5 plays that role", which is why is_infra and retry_helps are reported as raw probabilities.
• jev-browser-agent (smartdio): escalates below 0.6 to a bigger model. Here, below 0.90 goes to a human reviewer. A bigger-LLM tier in between is a cheap addition if the human queue grows. Sources: ../../research/latent-space-diogo-almeida.md, /dbreunig/building-with-jev-skill, /smartdio/jev-browser-agent, /jyje/pilot-typesafeai-jev.

• tax-doc-classifier: a big closed-set Choice whose criteria are {what, examples, not_for}, plus a not_in_this_list escape and a confidence gate (0.90 here; tax-doc uses 0.95).
• Jaggedness page: filter first. Code keeps only signal lines and the tail of the log, and never asks Jev to read numbers. Token counts, HTTP codes, exit status and minutes idle are code facts.
• Canny rule: facts beat judgments. If model_received_prompt is false, a model_* label is overruled and the run goes to a human.

cd samples/failure-triage && ./run.sh
# = python3 build_mr_fixtures.py <ai_model_benchmark>/results/_infra_dnf fixtures/mr_infra_dnf.jsonl (only if that folder exists)
#   python3 triage.py fixtures/mr_infra_dnf.jsonl   (6 REAL AI Model Benchmark infra DNFs, scrubbed)
#   python3 triage.py fixtures/runs.jsonl           (8 hand-written fixtures)