Free Routing

Python's standard library is sufficient. Set FREE_ROUTING_SCRATCH to an external scratch directory; runtime logs and spill files must stay outside the checkout. The examples assume this variable is set. Tests default to a sibling directory named after the checkout with a -scratch suffix.

nice -n 10 python3 experiments/free-routing/run.py analyse --backend replay \
  --records experiments/free-routing/results/calls.jsonl \
  --output "$FREE_ROUTING_SCRATCH/replay.json" \
  --scratch "$FREE_ROUTING_SCRATCH/context"

nice -n 10 python3 experiments/free-routing/run.py measure --backend mock \
  --output "$FREE_ROUTING_SCRATCH/mock_calls.jsonl" \
  --scratch "$FREE_ROUTING_SCRATCH/context"

nice -n 10 python3 -m unittest discover -s tests -p 'test_free_routing*.py'
nice -n 10 scripts/check.sh

Output collection refuses to overwrite a recording. Replay rejects mixed routes, missing task/model/prompt matches, malformed accounting and parsed answers that disagree with the retained raw text. Mock calls always say MOCK - NOT JEV. freeze.py deterministically rebuilds tasks.jsonl and its provenance manifest.

OPENCODE_BIN must point to an already installed OpenCode executable. Set the scratch argument to its prepared external home. No package or repository install is performed. Only the explicit free model allowlist is accepted; the process inherits no API keys. It uses the CLI, a new session per call, native tool schemas with all permissions set to ask, and headless automatic rejection of tool requests. No browser is used. harness.json requires the API preference; the CLI is the necessary exception because Zen rejects direct free-tier chat API calls.

nice -n 10 python3 experiments/free-routing/run.py calibrate --backend opencode \
  --opencode "$OPENCODE_BIN" --scratch "$FREE_ROUTING_SCRATCH/oc" \
  --output "$FREE_ROUTING_SCRATCH/calibration.jsonl"

nice -n 10 python3 experiments/free-routing/run.py measure --backend opencode \
  --opencode "$OPENCODE_BIN" --scratch "$FREE_ROUTING_SCRATCH/oc" \
  --config "$FREE_ROUTING_SCRATCH/calibration.ladder.json" \
  --output "$FREE_ROUTING_SCRATCH/calls.jsonl"

The saved ladder.json was calibrated before scored calls. Cheaper means lower median wall latency on three small calibration calls, including CLI startup. “Strongest” denotes the final escalation rung; better quality is unverified. Calls are sequential and bounded by 60 seconds. Two consecutive transport failures cost stops the campaign immediately. Missing receipts remain unknown.

Scored model order rotates per task. Every policy is replayed over this same matrix, so the comparison consumes no additional model calls. Router latency sums the calls it uses; it excludes offline analysis and mock-adviser overhead. Latency percentiles include attempted tasks and their timeouts, excluding wholly skipped circuit cells; latency_n states that denominator. Every frozen case stays in the success-rate denominator. Valid wrong labels do not trigger escalation. The grader alone reads fixture answers. Reported token totals retain the provider's tokens.total; component counters are retained separately because they can disagree.

--mode shadow is the default. The mock adviser independently picks the first allowed option; its agreement rate measures plumbing, not Jev quality. off uses the default router without consultation; on acts on adviser choices. Only initial model selection and failure-driven escalate/stop choices consult. Parsing is counted as a local decision. Forced stops are queued and recorded as code-only, with no consultation. Agreement excludes forced and local decisions.

The published pending_jev.jsonl is QUEUED — REAL JEV NOT RUN. To validate it later, use the existing external Jev credential file and strict zero-budget client:

JEV_PROVIDER=zen JEV_MAX_USD=0 JEV_MAX_CALLS=48 \
nice -n 10 python3 experiments/free-routing/run.py drain \
  --queue experiments/free-routing/pending_jev.jsonl \
  --output "$FREE_ROUTING_SCRATCH/jev_results.jsonl" \
  --scratch "$FREE_ROUTING_SCRATCH/context"

The shared client enforces its Zen lock, pacing, timeout and free-only guards. The drain resumes successful IDs, stops on the first error, and will not retry a retained refusal automatically. It reports REAL agreement only for verified live Zen receipts. Keep mock and live drain results in separate files. No drain has been executed against real Jev for this milestone.

The separate v2 queue has 57 stakes rows: 33 potential consultations and 24 forced stops. To validate that trace later, use its own receipt file:

JEV_PROVIDER=zen JEV_MAX_USD=0 JEV_MAX_CALLS=33 \
nice -n 10 python3 experiments/free-routing/run.py drain \
  --queue experiments/free-routing/pending_jev_v2.jsonl \
  --output "$FREE_ROUTING_SCRATCH/jev_results_v2.jsonl" \
  --scratch "$FREE_ROUTING_SCRATCH/context-v2"

This command is a queued follow-up, not part of the completed measurement.

Pass --jev-results "$FREE_ROUTING_SCRATCH/jev_results.jsonl" and --queue to analyse to replay completed real advice in shadow or on mode without more Jev calls. Advice must match the exact decision, including its default choice. If active advice reaches an unvalidated decision, replay fails closed; agreement on the original default path is not a complete active-policy evaluation.

The classifier and escalation gate consume bounded JSON handoffs containing only routing features, prompt hashes, model and failure status. Oversized tool logs are saved intact under <scratch>/tool-output/; the consumed log includes that path and capped head/tail text. Classification evidence and rubrics stay complete in every model prompt. Caps are configurable in harness.json.

The comparison uses the same prompts, calls and handoff payloads with lean rules off and on. The off baseline inlines full task/history/log payloads at each handoff. The shared estimate_tokens helper estimates JSON characters divided by four. These are harness-context estimates, not measured model input-token savings. A smaller OpenCode system prompt remains untested.

Collection is complete at the 100-new-attempt cap: 60 planned fixed cells, 32 fresh router calls and eight excluded resource interruptions. The repaired fixed scores are MiMo 21/24, Space Bunny 24/24, Ultra 12/24 and Lightning 19/24. The primary v1 router remains 24/24; secondary live v2 is 19/24, with median 10.81 s and p90 193.80 s. All 24 v2 tasks were attempted; one additional escalation was blocked to reserve initial calls for remaining tasks. The commands below document the collection procedure; the published campaign budget is exhausted.

DESIGN.md and round2-plan.json register the extension before new calls. Run 1 remains the primary result; its original analysis is byte-preserved in results/run1/analysis.json, and its original recordings remain unchanged. Round 2 retries the 36 timeout/circuit-skipped cells, adds 24 Ultra fixed-arm cells, then runs a fresh router after the second ladder is frozen and committed.

All new calls use 180-second deadlines, run sequentially, and share a durable 100-attempt budget. Five consecutive errors stop one model; a free-tier, quota, transport and refuses mixed mock/real resumes or unresolved reservations. Keep the same ledger across phases; use unused output paths for each export.

nice -n 10 python3 experiments/free-routing/round2.py fixed --backend opencode \
  --opencode "$OPENCODE_BIN" --scratch "$FREE_ROUTING_SCRATCH/oc" \
  --ledger "$FREE_ROUTING_SCRATCH/round2-ledger.jsonl" \
  --output "$FREE_ROUTING_SCRATCH/round2-fixed.jsonl"

nice -n 10 python3 experiments/free-routing/round2.py freeze --backend opencode \
  --opencode "$OPENCODE_BIN" --scratch "$FREE_ROUTING_SCRATCH/oc" \
  --ledger "$FREE_ROUTING_SCRATCH/round2-ledger.jsonl" \
  --output experiments/free-routing/ladder-v2.json

freeze reads existing receipts and makes no calls. It uses original calibration and round-2 status, parseability and latency; no scoring answer is read. Commit the frozen ladder before grading new answers or running the next command. The router verifies the committed file and evidence hashes before making calls.

nice -n 10 python3 experiments/free-routing/round2.py router --backend opencode \
  --opencode "$OPENCODE_BIN" --scratch "$FREE_ROUTING_SCRATCH/oc" \
  --ledger "$FREE_ROUTING_SCRATCH/round2-ledger.jsonl" \
  --queue "$FREE_ROUTING_SCRATCH/pending-jev-v2.jsonl" \
  --output "$FREE_ROUTING_SCRATCH/router-v2.json"

The router reserves one initial call for each remaining task before escalation. The measured fixed arm for the final rung also serves as always-strongest, with all 24 task receipts reported directly and counted once. This is one measured arm, not an independent duplicate sample. Retained run-1 cells and new repairs remain distinguishable by provenance and deadline. Real Jev remains queued.

Memory guard and interrupted collection

Before every new call the runner reads /proc/meminfo. It launches only when 1 - MemAvailable / MemTotal is below 75%; otherwise it logs a wait and polls every 30 seconds inside the runner. Waiting is recorded separately from model latency. Memory above 85% during a call terminates the owned process group and ends that invocation with resource_interruption.

Resume the same phase with the same ledger and a new output filename. Completed cells are reused; resource-interrupted cells receive a new attempt sequence. interrupted_by_supervisor and aborted_memory_pressure consume budget slots but are excluded from accuracy, latency, reliability and model-error streaks. An unresolved reservation requires an explicit interruption receipt before resuming; it must never silently become a timeout or a successful call. A

Offline round-2 recount

After the frozen ladder is committed, the reporter reads the append-only ledger and retained run-1 recordings. It makes no model calls. The original analysis remains at results/run1/analysis.json; the combined result uses schema version 2 and separates primary_run1, secondary_run2 and live_router_v2.

nice -n 10 python3 experiments/free-routing/round2_report.py \
  --ledger experiments/free-routing/round2-ledger.jsonl \
  --router experiments/free-routing/results/run2/router-v2.json \
  --output "$FREE_ROUTING_SCRATCH/recount-round2.json"

The published ledger and results/run2/calls.jsonl are two representations of the same attempts. Count each call identity once. The ledger also retains reservations and protocol identity; the flattened file supplies top-level route labels for the result manifest. Source hashes and collection accounting are in results/run2/metadata.json.

Recounting the exported ledger and live summary reproduces the published results/analysis.json byte-for-byte. Acquisition totals include 769,493 known tokens and 14 missing usage/cost receipts; all received costs are $0. The report separates the mixed-deadline fixed comparison from each router version and explains the limits of this small authored set.