The Scientist

Max effort cost less than medium effort

The most useful number in the independent runs is a price. @mhmazur's ARC-AGI-3 sweeps priced the same benchmark at $26k at max effort, $38k at low, and $48k at medium. Higher effort took fewer actions to finish each game. Most teams treat effort as a linear spend knob and throttle it down on multi-turn loops to save money. On this evidence, throttling buys worse output at higher total cost.

Quality moves the same direction. Hallucination fell from 92% to 51% at max effort, with four points of accuracy gained. The 51% is the improved figure, and it comes from the configuration cheap traffic would never be routed through. On HealthBench Professional, the lowest effort setting beat the predecessor's best score at roughly half the cost. Sweep effort per workload.

Two evaluators, two cost verdicts

Artificial Analysis lands on +75% cost per task; Cognition lands on −64%. Both are probably correct about their own task mix, and neither is correct about yours. Standard pricing is $10 per 1M input and $50 per 1M output, roughly 2.5x the predecessor per token, offset by roughly 70% better token efficiency and one third of Sol's tokens in the Codex harness. With that 2.5x per-token price offset by that efficiency, the per-million figure no longer predicts spend; cost-to-target-score on your own distribution, harness held constant, does.

Where the scores went down

• ~80 Elo drop on GDPval-AA v2, mirrored by a ~80 Elo gain on AA-Briefcase long-horizon knowledge work.
• 2–3 point regressions on τ³-Banking, SciCode, and AA-LCR: structured domain workflows, scientific code, and long-context retrieval, which is a precise description of a data team's workload.
• Presentation Quality Elo down even as Analytical Quality Elo rose.

Chain-of-thought dashboards lost resolution

UK AISI measured a no-CoT time horizon of 30.9 minutes versus 3.6 minutes for GPT-5.6 Sol, with reasoning summaries missing up to 80% of long simulated cyber trajectories. Apollo measured 41.1% verbalized evaluation awareness, against 27.7% for the prior generation. OpenAI's own system card pairs improved alignment with decreased chain-of-thought monitorability.

Anything parsing reasoning text (anomaly detection, guardrails, trace review) is now sampling an incomplete signal, so enforcement has to move to the action layer: tool-call authorization, diff review, output validators, full trajectory logs. The second consequence is measurement. At 41.1% eval-awareness, offline scores are a biased estimator of production behavior, which argues for prod-mirrored shadow evals and a canary detector for eval-recognition language. Both items are engineering work rather than policy.

ARC-AGI-3 went from under 1% to near-saturation in six months and ARC-AGI-4 is not due until Q1 2027. For the next several quarters, public benchmarks are either saturated or harness-confounded, and your internal suite is the only instrument left with variance in it.

Latent.Space supplies the statistics that make this concrete. Separating a 97% from a 99% pass@1 at 80% power takes roughly 800 independently graded tasks per arm, which no public eval ships. Paired continuous metrics recover the power: with per-task correlation around 0.7, detecting a 20% cost shift needs roughly 30 paired tasks. That is a nightly job rather than a research program.

The failure mode is a retry loop with no error to log

Start with the mechanism, since it generalizes well past Databricks. A tool call fails. The failure never surfaces as an error, so the agent reads the empty or ambiguous response as an instruction to try again. The retry bills quietly enough that aggregate spend never flags it. It reads as normal token growth. Strict schema validation against a stochastic caller is a self-inflicted retry generator. Reject a plausible unit, alias, or case variant and the loop is manufactured.

The dollar figures deserve their interval. The $499K and $1.2M numbers are annualized single-org extrapolations with no stated measurement window and no counterfactual. The thing they don't tell you is what the baseline was. What survives is the ratio: diagnosis required instrumentation the team did not have, and the repair took about an hour once it did.

Audit the harness benchmark before citing it

FrontierHarness reports no sample size, no confidence intervals, and no task-set composition. A 66.7% pass rate is exactly two-thirds, which is what small denominators produce. "Relatively similar pass rates" is consistent with genuine convergence and equally consistent with a task set that has no power to separate the arms. Exo is labelled cheapest and DSH Minimal fastest with no accuracy attached, so the cost/accuracy Pareto frontier cannot be drawn from what was published.

The directional finding holds anyway, because a 17x spread is not explicable by model weights. It is prompt-cache hit rate, tool-call round trips, and context re-transmission. All three are locally controllable and locally measurable.

The Tangle result is an existence proof, not a benchmark: no task named, no dataset spec, no seed variance, no statement of whether the frontier baseline was prompt-optimized. What matters is that the infrastructure is open source, which converts "we should distill someday" into a bounded spike.

Why cost metrics have more statistical power than pass rates

This is the part that survives a design review rather than a budget meeting. Tokens-to-solve and cost-per-solved-task are heavy-tailed but strongly paired across models on the same task. Exploit the pairing and a 20% cost shift is detectable with roughly 30 tasks, where an unpaired design needs about 100 and a binary pass rate sitting at the ceiling needs hundreds. Cost is where the variance still lives, so cost is where the discriminative power is.

Storage is now officially in the serving path

MLCommons shipped MLPerf Storage v3.0 with new KV-cache and vector-database workloads plus an S3 object-storage access layer alongside POSIX. The benchmark authority is conceding what serving traces already showed: inference caches, vector indexes, and checkpointing govern throughput and cost, not FLOPs. Two implications land immediately. Any capacity model built on GPU-hours alone is mis-specified, and any storage or vector-DB vendor without v3.0 numbers on those workloads has nothing to bring to a procurement conversation. Read submissions as vendor-tuned best-case ceilings.

The cost arbitrage has an obvious first target. Open-weight backends are entering shipping enterprise coding stacks, with Snowflake's CoCo adding DeepSeek-V4-Flash and GLM-5.3 alongside Anthropic and OpenAI across 9,100 accounts. Substitution is safe where the output is machine-checkable: SQL generation, test scaffolds, docstrings, mechanical refactors.

At comparable accuracy, harness and tooling choices swing agent cost by 17x. The bottleneck is observability, priced in dollars.

KEV membership is counted exploitation, not modelled severity

Rank the queue by expected loss and the ordering is not ambiguous. Gitea code injection (CVE-2026-60004, KEV 2026-08-25) and JFrog Artifactory path traversal (CVE-2026-66384, KEV 2026-08-27) sit directly on the model and package supply chain, and poisoned weights carrying valid internal provenance are the worst failure mode in this ledger. Patch, then audit the two weeks of artifact pushes and repo hooks preceding the patch. LiteLLM is next. An authentication bypass on the proxy that holds every provider key is functionally a key dump, and most gateways persist full prompt bodies. In a RAG system, that means persisted customer data.

The bug class beneath the CVE list

Drop the routers and printers and the AI tooling failures collapse into three old categories: unsafe deserialization, template or expression injection, and services that bind wide and ask for no credentials. Six vendors shipped the same wildcard-bind mistake, including a 10.0 on UI-TARS-desktop's mcp-http-server. Patching CVE-by-CVE will not keep pace with this ecosystem's release cadence. A CI check that fails any service binding to 0.0.0.0 or :: catches the class instead of the instance. The same generalization covers NLTK's pickle RCEs: pickle.load, torch.load, and joblib.load are one primitive, and any of them touching an artifact CI did not write and sign is an unauthenticated RCE in the serving path.

One item is a data-quality note rather than a security note. Apache Hive 4.2.0's Metastore SQL injection (CVE-2026-49845, 9.8) lets an attacker repoint partition locations. Drift monitors will label that poisoning as seasonality, because they measure distribution shift and never provenance. A drift metric does not tell you where the bytes came from. Catalog-versus-object-store row-count and checksum assertions belong in the data-quality suite for exactly that reason.

The agent on the laptop is an untrusted code execution engine

Manifold Security disclosed eight flaws across seven command-line AI coding agents, Claude, Codex and Cursor among them, in which a cloned repository's own Git configuration names a command the agent runs on the host, in several cases before the trust prompt appears. Seven independently built agents failing the same way is a design-level trust failure, not a coincidence of implementation, which is why the control is isolation rather than patching. No CVE IDs or version ranges were published, so prioritizing by vendor is not on the table.

Compose that with GitGuardian's finding that the early-August Shai-Hulud variant sweeps 469 distinct credential locations across developer environments and CI/CD. The most routine act in this job, cloning a paper's repo or an HF Space and asking an agent to sort out the environment, now executes attacker-chosen code on the one machine holding simultaneous live credentials to object storage, the warehouse, the model registry, and the experiment tracker.

The registry is a vendor now

NVIDIA agreed to acquire Hugging Face for $12.93B, putting 3M+ models, 500K datasets and 1M apps serving 18M+ developers under the ownership of the company selling the accelerators. Nothing breaks today. Huang's neutrality pledge is a blog post and, as written, unfalsifiable, which is precisely why this belongs on an engineering list. Self-preferencing, if it happens, shows up in defaults: attention backend, dtype, quantization recipe, which fused kernel a model card recommends. None of that requires a policy change and all of it moves cost per token.

The completion endpoint sits in the control plane. Whoever answers a request chooses which file-read, file-write, and shell calls the host executes.

The mitigations are cheap and were worth wanting anyway: immutable revision SHAs instead of tags, internal mirrors for every production weight, tokenizer and dataset, license metadata snapshotted at ingestion, and a CI run that stays green with the Hub blocked at the firewall. The side benefit is that reproducibility of past experiments stops depending on an external party's retention policy.