The Scientist

Coverage moved as far as the pass rate did

The most diagnostic number in the Harvey/Baseten results is the one nobody quotes. Under GRPO on Qwen3.5-122B-A10B, held-out rubric pass rate went from 30% to 63% and document coverage went from 62% to 96%, per AINews's account. Both moving together points at a search-and-recall failure that the reinforcement learning repaired. The agent was not reading enough of the data room. Formatting cleanup does not move coverage, and a single scalar pass rate would have hidden the mechanism entirely.

Where the write-ups agree, and where they stop

Daily Dose of Data Science and MIT Technology Review both land where Harvey does: the measurable gain sits in the scaffolding, not the weights. Daily Dose of Data Science documents the post-training half collapsing to a config-level task. Qwen3-4B-Base loaded through Unsloth, LoRA applied, GRPO via HuggingFace TRL's GRPOConfig/GRPOTrainer, trained on Open R1 Math with four deterministic reward functions (exact format, approximate format, answer check, number check) on a single GPU. No critic network, no preference labels; the reward signal is four string comparisons. MIT Technology Review supplies the selection rule: the pattern works where a programmatic oracle already exists, query-execution equivalence for text-to-SQL, unit tests for transforms, schema validation for extraction, and it breaks on summarization and judgment, where LLM-as-judge is not an oracle.

Rigor runs one way here. The 23%-to-62% figure is a mean across models on a synthetic benchmark, LAB Diligence, with no per-model breakdown, no variance, a vendor-authored rubric and no independent replication. A swing that size is also what a badly configured baseline tool loop produces, and nothing published rules that out. The zero-label tutorial is thinner. It calls the result a reasoning powerhouse and reports no pass@1, no held-out split and, critically, no SFT control arm on the same data, which leaves the RL gain confounded with the format conditioning the system prompt supplies on its own. The recipes are public; the ablations that would let anyone attribute the gain are not.

What this changes in the harness

Harness is a factor in these results, not a constant. A model x harness factorial with fixed seeds and at least three repeats per cell is the only design that separates orchestration from weights; it reports main effects and the interaction separately, and it puts a number on how much of the last quality win was orchestration. Pass rate and coverage belong in two columns, since Harvey's data shows they move independently and one number cannot triage three failure surfaces. A frozen golden set and a pinned baseline model are the conditions under which a quality change can still be attributed to a cause. Practitioners are already reporting that they had not finished migrating off one model before the next release landed.

The largest measured lever in these results was the scaffolding, not the weights. Most production agent evals are measuring their own harness ceiling and calling it model quality.

The README is the acquisition thesis

Applied AI's read is that the price is positional rather than financial, and the supporting evidence sits in product documentation rather than deal terms. Hugging Face's new $400 walking robot, Microduck, ships documentation stating that the simulations used to teach it skills must run on an Nvidia GPU with CUDA. Every unit sold is a compute on-ramp: 15,000-plus in roughly one month at $6M, all figures sourced to a co-founder speaking during his own company's acquisition. The thing that $6M doesn't tell you is margin, returns, or how many buyers already owned the GPU. Benedict Evans supplies the capital context, reading $127bn of trailing free cash flow as spending to fragment adjacent layers of the stack while entrenching the one Nvidia owns. The CUDA default will propagate through the rest of the tooling.

Mutable references are the exposure

Most teams reference external artifacts by mutable names. Branch and tag references resolve to whatever the upstream publisher last pushed, so a rerun labelled as a reproduction is frequently running different bytes. The container layer makes the drift measurable. The Hacker News reports Chainguard at 675,000 image versions across 3,000-plus images, roughly 225 rebuilds per image, with over a billion build manifests and a doubling in six months. Continuous rebuilding is good security and terrible determinism. Training Dockerfiles that pin CUDA or PyTorch bases by tag move underneath the experiment silently, and no dashboard reports it.

A second provider still needs benchmarking

The provider layer is genuinely diversifying, and the reporting on it has not converged. MIT Technology Review describes Mistral's raise as a $3.5B European record on an explicit open-weights thesis, followed by a pivot toward infrastructure and services. Bloomberg Technology describes a EUR3B round led by Samsung at a post-money above EUR21B. Both are as-reported figures with no disclosed revenue behind them, which makes the raise a measure of months of runway at an undisclosed burn. Our own practice is to bench the weights on an internal suite, tool-calling reliability and structured-output conformance included, behind an adapter that keeps a provider swap a config change.

The same coverage forces a cost-model correction. ASML's newest lithography systems run up to $400M apiece with only TSMC and Samsung committing, and Amazon has mandated banks for a debut sterling bond to fund AI capex. Capital intensity is rising into the next node and capacity is being financed with leverage. That financing arrives in the inference bill rather than the training budget, which is the opposite of the setup for reflexive hardware-driven price declines. We removed the assumed annual $/token drop from our 2027 forecast and replaced it with line items we control: task-specific distillation, FP8 or INT4 quantization behind a quality gate, speculative decoding, and prefix-cache hit rate as a first-class dashboard metric.

What changed hands is the default import statement in every training script — and an unpinned reference is now a vendor-controlled dependency with a close date attached.

Refusal and uncertainty appear to share machinery

The abliteration result is not capability loss. John McIntosh ran abliterated open-weight builds against their base models on a vulnerability-research task; the most aggressive build graduated 96% of candidates to VALID against 65% for base Gemma, emitted 138 false positives, and never found the real CVE that base Gemma caught, per Clint Gibler's writeup. The chain-of-thought correctly identifies why a finding is invalid, then confirms it anyway. Across the broader test, abliterated builds confirmed findings three to four times more often regardless of evidence. That reads as a calibration failure rather than a guardrail failure: editing away a model's willingness to refuse appears to edit away its willingness to say I am not sure. For anyone running LLM-as-judge over vulnerability candidates, model outputs or labeling disagreements, the consequence is precision collapse with a coherent reasoning trace attached. Conditions travel with the claim: one tool, one codebase, one CVE, one model family, no confidence intervals and no total candidate count. Worth a one-hour spike on an in-house judge set.

The same defect one layer up the stack

Refactoring published its growth agent's weekly report verbatim: an LLM with warehouse access, a cron schedule and no inferential tooling. The agent narrated a collapsing funnel across three metrics. Audited, almost none of it is there.

Nothing in the agent's toolchain computes a noise floor. Scan three metrics weekly at alpha=0.05 and chance alone manufactures an alarming narrative roughly one week in seven, on schedule and in confident prose.

The pattern, and the quantity that closes it

The guardrail coverage supplies a third instance. TLDR IT argues that frontier labs treat prompt-injection defenses as largely solved despite high failure rates, favoring detection over containment while publishing no operating point at all. A scalar blocked-percentage measures how often the classifier fired. The thing it doesn't tell you is how often an attack succeeded. Three domains, one missing quantity: the judge ships a verdict with no abstention rate, the reporting agent ships a delta with no interval, the detector ships a count with no attack-success rate at a fixed false-positive budget. None of the three fixes is a model change.

The prescriptions diverge, which tells you who owns the work. The judge failure is model-side: audit checkpoints for refusal-stripping, plant known-negatives. The reporting failure is tooling-side: expose the statistics library as a callable tool the agent must use before it emits a number. The detector failure is harness-side: score on a held-out adversarial split, version the result per model. Treat all three as prompt problems and none of them get fixed.

A metric without a confidence interval is a plausible narrative about noise, delivered on a schedule with your credentials.