Sift

Lots of useful data is unlabelled text: tickets, prompts, transcripts, reviews. sift keeps the rows that match a plain-language condition ("the customer wants to close their account"), or ranks them. It's a reusable tool for everything else in this repo.

Several rows are packed into ONE request as a shared state {condition, rows}, with one Noul per row pointing at rows[i] (pg-jev's recipe). Cheap exact filters (--where) run in code first. The per-row cache makes re-thresholding free. --label reports precision and recall, and --sweep reports precision, recall and F1 across 11 thresholds.

Why decomposing helps. Each row is its own yes/no question, so a threshold is one number you can sweep on labelled data. There's no prompt to rewrite and no prose to parse.

A bigger labelled set. Two sets were built from BANKING77 (PolyAI, CC-BY-4.0, test split, via the Hugging Face datasets server). The labels are human intent labels, not ours:

• close_account: 220 rows, 40 positive (terminate_account), with hard negatives cancel_transfer, request_refund and edit_personal_details, plus 60 random rows.
• lost_card: 240 rows, 40 positive (lost_or_stolen_card), with hard negatives compromised_card, lost_or_stolen_phone, card_not_working and card_swallowed, plus 40 random rows.

Only the text field is sent. The intent and label never are.

What the sweep found:

• On openjev, the batch size mattered far more than the threshold.
• At 20 rows per request, recall collapsed. Blunt positives like "I am highly unsatisfied with this company and want to delete my account!" scored 0.01, which looks like answers drifting off their row index.
• At 5–10 rows, close_account was perfect at any threshold, and lost_card went from F1 0.61 to 0.86 at 0.5.
• Smaller batches did not cost more tokens per row (70–77 vs about 90). One row per request cost about twice as much and was no better.
• So on openjev the default batch is 5. On real Jev it stays 20, pg-jev's measurement, which the real-Jev run below confirms (F1 0.99 / 0.82 at 20 rows).
• Remaining errors on lost_card are the genuinely confusable hard negatives: "someone used my card" (compromised_card) and "the ATM took my card" (card_swallowed).
• Mock caveat: the mock beat openjev-at-batch-20 on close_account. That shows batch 20 was broken. Word overlap is still weak: mock lost_card is 0.35.

Full details, raw outputs and the mock baseline: RESULTS.md and results/.

Real Jev = jev-1.13-free on OpenCode Zen's limited-time free tier, run 2026-09-24 13:19–13:21 UTC with JEV_PROVIDER=zen. It was $0, and no response carried a cost field. Each result is labelled REAL JEV (OpenCode Zen free, jev-1.13-free). The free tier rate-limited us (HTTP 429 FreeUsageLimitError) after about 288 calls in about 2 minutes. Later calls were paced at ≤4 per minute.

pg-jev's batch-20 claim holds on real Jev and fails on openjev. Real Jev at 20 rows per request is close to its own 5-row numbers. So the default of 5 is an openjev workaround. Real Jev does not need it. The hard negatives (compromised_card, card_swallowed) are still where both models make most of their mistakes. Real Jev tokens: about 91 per row at batch 20, 132 at batch 5; median latency 420–480 ms per request.

Raw outputs: results/zen_*.

• Tokens: 70–90 input tokens per row in batches, 142 per row one at a time.
• Latency: median 250–530 ms per request. Larger batches were slower per request but faster per row.
• Everything in this upgrade: 401 requests, about 143k tokens, $0. At list price, 10,000 rows is about 750k tokens, roughly $0.03.

(Written before the real-Jev run. The section above shows what it did.)

• Verify the batch knee. pg-jev measured real Jev at 100% for 1–20 rows per request. If that holds, real Jev keeps batch 20 and costs fewer requests. Measure it with --batch 5/10/20 --sweep on these same two sets before trusting it.
• Threshold depends on the set. Best thresholds differed by set (1.0 at anything on close_account, 0.9 on lost_card), so thresholds must come from labelled examples of the actual job, not a default.

• Latent Space: "dark data" is one of the two biggest money makers he names. "Choose thresholds from real examples" (about 1:07) is what --sweep does.
• building-with-jev (dbreunig): "accuracy falls as inputs grow → filter in code, send only the needed fields" matches both --fields text and the batch-size finding. Sources: ../../research/latent-space-diogo-almeida.md, /dbreunig/building-with-jev-skill, /smartdio/jev-browser-agent, /jyje/pilot-typesafeai-jev.

• pg-jev: 20 rows share one state with one Noul per row (rows[i]). pg-jev measured 100% accuracy at ≤20 rows per batch, 92–98% at 40 and 77–94% at 80, and ~175 tokens/row at 20 vs ~435 for a single row. Batch size is fixed at 20 until we re-measure it ourselves.
• Answers are cached per row content (live mode only, so mock answers can't poison the cache). --max-rows and the call cap act as spend guards.
• jev-curate: cheap host-side filters run before the model call.

cd samples/sift && ./run.sh
# = sift.py fixtures/banking77_close_account.jsonl "the customer wants to close or delete their account" --fields text --label wants_to_close_account --sweep --batch 5
#   sift.py fixtures/banking77_lost_card.jsonl "the customer says their card has been lost or stolen" --fields text --label is_lost_or_stolen_card --sweep --batch 5
#   sift.py fixtures/tickets.jsonl "the customer threatens to cancel or leave" --label threatens_to_leave --sweep