Evals & accuracy

Workflow evals, not public benches

❓ What is TypeSafe measuring, and why did it skip MMLU?

Almeida’s 11 September 2026 essay “Lies, Damned Lies, and Benchmarks” argues that public evals get benchmaxxed: once a score is famous, training data and experimental search move toward it, and the score stops measuring what you thought.[5] TypeSafe’s stated policy is no standard benchmark table at release; new evals are “dated snapshots and immediately retired once posted rather than hill-climbed.”[5]

What they published instead is evals.typesafe.ai: four workflows (security incidents, agent-trace observability, invoice processing, customer service). Each workflow is a harness of Noul / Choice / Score questions plus code. Every model, including Jev, runs the same harness. The reference labels are the average of GPT-6 Astra and Claude Fable 5.1 at high thinking — not human ground truth, and not a held-out test set from TypeSafe’s customers.[30][1]

TypeSafe says this on purpose: “Instead of debating the correctness of the harness and labels, we assume that the code is correct, and measure against the current smartest large models.”[30] That makes the dashboard a agreement-with-frontier-judges plot, not a correctness plot. It also, as the launch post admits, “biases answers towards OpenAI and Anthropic’s models.”[1]

What the dashboard actually shows

❓ Does Jev win accuracy, or only cost and time?

OrcaRouter’s 16 September 2026 reading of that dashboard, which this research did not independently re-scrape as a pixel table, reports:[19]

• Aggregate: Jev 67.8% vs 74.1% for the best comparator
• Security incidents: 61.7% vs 66.2% (Opus 5)
• Agent-trace observability: 71.6% vs 76.6%
• Invoice processing: 61.8% vs 79.1%
• Customer service: 76.0% vs 78.3%

Jev owns the cheap/fast corner. It does not own the accuracy corner. TypeSafe’s homepage still titles a chart “nothing is both cheaper and more accurate” as a Pareto claim against prompted one-shot LLMs, which is a different comparison than “Jev vs the same workflow on Astra.”[2][30] Keep those two plots separate.

The launch post’s 193.6× / 444.6× pair is taken from this workflow family and labeled by TypeSafe as the high end.[1][2]

Every’s independent tests

❓ What happened when someone who does not work at TypeSafe called the API?

Mike Taylor (Every, head of evals) on 15 September 2026:[17][19]

• 27 Every articles + 10 AI-styled counterparts, 21 questions each, concurrently: 777 judgments in under 0.7 seconds, about a quarter of a cent.
• A second, CEO-run test: 12 synthetic passages (six clean, six with planted defects), four writing checks. Jev median 0.35 s vs Claude Fable 5.1 high-effort 8.83 s (~25×). Cost about 1/580th. Jev caught six of seven planted defects; Fable caught all seven.

Every’s own line, as quoted by OrcaRouter: “good but not perfect,” and they would want a much more thorough accuracy check before production.[19] The Every page itself is paywalled past the lede in this research fetch; the numbers above are those OrcaRouter attributed to that article plus the public lede.[17][19]

That is the entire independent accuracy record as of 16 September 2026. It is small. It is also the only non-vendor accuracy record.

Consistency cookbook

❓ Is Jev more stable across repeats than a temperature-zero LLM?

TypeSafe’s noul self-consistency cookbook (sampled on the production API, jev-latest, 2026-09-11, before the public launch) runs a 14-question insurance rubric 15 times.[26] Reported TypeSafe mean per-question probability standard deviation: 0.0102, below the LLM probability conditions in that notebook. Some answers still sit on a 0.5 threshold (covered spanning 0.43–0.53). The notebook’s point is that you should treat a band around 0.5 as uncertain and route to a human, rather than rounding.

Cost table in that cookbook uses TypeSafe ($0.042, $0.00) per million input/output as of August 2026, against Haiku 4.5 ($1 / $5), GPT-5.4-mini ($0.75 / $4.50), GPT-5.5 ($5 / $30), Opus 4.8 ($5 / $25).[26] Those LLM prices are dated mid-2026 in the notebook and will drift.

How a buyer would test calibration

❓ What would actually support the RLCD claim?

A reliability diagram: bin predictions by stated probability, plot observed frequency. A well-calibrated model sits on the diagonal. TypeSafe describes this in the primer and does not publish the plot for Jev.[8]

The cheap version, which OrcaRouter also walks through:[19]

1. Label a few hundred of your cases.
2. Ask Jev the same Noul/Choice you will ship.
3. In each probability bucket (0.0–0.1, …, 0.9–1.0), measure how often the label matches.
4. Set act / confirm / escalate thresholds from that curve, not from the docs’ 0.9 example.[9]

Until that plot exists for a workload, “calibrated” is a training objective, not a property of the deployed model on that workload.