The Engineer

Why the catalogue broke before the traffic

The ML Engineer's back-of-envelope math explains where the wall landed. A tool schema is a few hundred tokens, so Uber's catalogue costs an agent roughly a million tokens of definitions before any request. That estimate is the author's, not Uber's figure. Gateway calls are cheap, so request volume never bound. The context window did, because every candidate tool competes for it.

This is a service mesh, and meshes already proved the split. An MCP registry is the control plane, cataloguing servers and tools. A proxy gateway is the data plane, translating MCP calls into HTTP, gRPC or TChannel at runtime. That translation is the underrated win. Existing microservices become agent-reachable without an MCP rewrite. AutoCrawler continuously scans Uber's API definitions, generates MCP servers and has an LLM enrich tool descriptions.

Three fixes, three new dependencies

CorpusMap reaches the same answer from the retrieval side

TheSequence covers a Microsoft and KAIST paper attacking the same budget in RAG. CorpusMap builds Entity Pages offline, each linking every document that mentions a recurring entity, and agents walk those links as paths. On EnterpriseRAG-Bench, WixQA and HERB, overall quality rose 6.45 points with GPT-5.5 and up to 11.74 with Luna. Input tokens fell to 0.43–0.65× baseline. Those two usually trade off. Both improved because cross-document work moved to index time.

Same pattern twice. Both move work off the context window into something cheaper: discovery calls and files at Uber, an offline build in CorpusMap. Both buy a failure mode. Uber's discovery recall becomes a correctness dependency. CorpusMap operators own entity resolution and rebuild cadence, and stale pages on fast-moving corpora like tickets or chat logs send agents down outdated paths.

The control that makes auto-generation survivable

Every Uber tool is registered disabled until someone enables it. Otherwise the crawler would make each new endpoint, destructive ones included, agent-callable the moment it read the definition. The ML Engineer names the catch, and I have watched approval gates fail exactly this way. With no named owner and no enablement SLA, disabled-by-default becomes a queue that teams bypass by wiring agents to APIs directly. The LLM-written descriptions need code review too, since the description decides which tool the agent picks.

One gap matters. Tools are authorized at tool level, but Uber's write-up never says whose identity the gateway evaluates: end user, agent or gateway. Zalando's open-sourced Agentic Identity Broker, walked through with KAOS, separates three principals: a user delegating to an agent, the agent's own tool and model calls, and an autonomous agent with no user that must stay inside its grant. KAOS is maintained by the institute that published the walkthrough, so treat it as a reference design, not a validated dependency.

The ablation is the result

Mid-Harness leaves the generator and harness alone. Each step samples candidate terminal actions, N=8 in the headline run, and a verifier vets them before anything executes. TheSequence's ablation is the part worth keeping. Distilling the frontier verifier into a smaller model recovered only about 2.4 of the 13.3-point gap between self-checking and the frontier judge.

• Pairwise self-verification, the 9B model judging its own candidates: +4.76 points.
• Distilled verifier: +7.14 points.
• GPT-5.6 Sol verifier: +18.03 points.

The gain needs a verifier that is stronger and different from the generator. Likeliest reason: a model reviewing its own proposals shares their blind spots.

The cost shape is asymmetric

A cheap generator proposes. An expensive model judges short candidates. Each step costs N samples plus verifier calls. Per TheSequence, verifier calls reuse the trajectory prefix, so at current frontier pricing most of their input bills at the cached rate. Mid-Harness plus Best-of-T (T=3) hit 66.33%, below the plain verifier. Its compute budget is unpublished. Read it as another point on the cost curve, not a regression.

So gate the steps that cannot be undone: shell commands, file writes, API calls with side effects. Reads and reversible edits run unverified. The metric is cost per successful task, not cost per call.

RRSI: the same judge problem in harness tuning

Same principle, applied to automated optimization. An LLM editing prompts, tools and control flow around a frozen model, scored against the same evolve set each round, overfits that set. RRSI, from Google Cloud AI Research, Stanford and others, keeps every component editable and constrains the search instead:

• an annealed edit budget: bundled edits early, single edits late;
• a diff critic that rejects edits encoding task names or benchmark answers;
• a noise-adjusted acceptance floor and a cost rule;
• a pruner that deletes components that stopped contributing.

The two outlets agree on direction, not framing. The ML Engineer reports the best out-of-distribution average, 43.6 versus 39.7 for the base harness, on about a third fewer tokens, with the smallest evolve-set gain. TheSequence reports +4.0 on evolve suites and +3.4 held-out, all six held-out benchmarks improving around a frozen Claude Opus 4.8, at roughly 36% fewer policy tokens per trial. TheSequence also flags that the benchmark counts in its source don't reconcile, so check the paper before quoting either figure internally. Ranked by evolve-set gain alone, the best method would have come last.

Where this still breaks

• The diff critic only catches crude leakage. Prompts that mimic the eval set's phrasing pass. The held-out set has to stay frozen and get refreshed.
• The verifier sits on the critical path of every irreversible step. Our inference: its latency and availability now bound the agent's. Budget it like any other synchronous dependency.

A backlog is a dependency chain

Remove the valuation and Nscale's contracted capacity is a sequence of steps, each of which can fail on its own. Financing closes, the site gets built, power is energized, chips arrive, racks burn in, the cluster validates, and capacity is handed over. The Information reports that financing sits upstream of every step and is getting harder and costlier to raise. The anchor tenants on Nscale's ramp are Anthropic, Microsoft and Figure AI, scheduled for 2027–2028. Microsoft buys Nscale capacity, so some hyperscaler GPU supply in that window also depends on these sites landing on time.

The systems term for the sector-wide risk is correlated failure. Neoclouds borrow from the same credit markets, so tight credit can slip several build-outs at once. Three neoclouds are less diversification than they look. Two replicas on one power feed were never redundant either.

The financing signals point the same way

The FT reports Nvidia's insurer talks, which are preliminary, are meant to protect lenders whose pledged GPUs might not resell for enough to repay the loans. The FT also reports Amazon is exploring moving $8B of Nvidia chips into an outside vehicle and leasing them back. The Information reports credit for data center debt is tightening while Oracle borrows heavily for its AI buildout. None of this says any service is failing. Read together, it says lenders doubt how well GPUs hold value. The GPU under a smaller provider's reservation is also someone's collateral. A default can make that capacity disappear on short notice.

CPUs join the list

Horizontal pod autoscaling and node autoscaling share one assumption: capacity is acquirable on demand. The CPU shortages Gergely Orosz describes break that assumption at the worst moment, during a traffic spike or when recovery needs scale-out. Cloudflare's post on saving another ~100TB of RAM with math and Rust shows the other supply source, which is capacity reclaimed from the operator's own fleet. Its most reusable section covers changing the hashing scheme in a massively distributed cache without invalidating it. Get that step wrong and a memory win becomes an origin stampede.

Right-sizing has a cost under scarcity. Tighter requests pack workloads densely and remove burst slack. Keep reclaimed capacity reserved instead of releasing it. Re-acquiring it during a shortage may not be possible.

A delivery tier for every reservation

Nscale's backlog sits almost entirely in T2 and T3. The reporting doesn't separate the two. The price risks point in opposite directions. Costlier debt lands in GPU-hour prices, which argues for budgeting 2027 against more than one price scenario. Distressed capacity reaching the market could instead push short-term prices down. Long fixed-price commitments on current-generation hardware lose in both cases.