The Scientist

Both recipes train inside the production harness

Both new training recipes put the deployment harness inside the reinforcement-learning loop instead of training against a simplified environment. Hugging Face and Liquid AI place an OpenEnv capture proxy between the harness and vLLM. The proxy records token ids and log-probabilities. TRL's asynchronous GRPO, an RL method that scores groups of sampled answers against each other, then trains on exactly what the agent did. Harbor supplies the tasks and sandboxes. The reward is correctness plus a small bonus for using fewer tool calls.

Microsoft Research Asia's Agent Lightning v1.0, as TheSequence describes it, does the same thing through an OpenAI-compatible LLM proxy. OpenHands, Claude Code or Codex can join training without being rewritten. The whole RL control plane is about 3,500 lines of code, and agents run as Kubernetes jobs. In our view the main contribution is less the algorithm than the removal of train/serve skew, the mismatch a feature store prevents by computing features the same way in training and serving.

What the numbers can carry

TheSequence estimates the 95% interval on a 500-task benchmark like SWE-bench Verified at roughly ±4.4 points. Agent Lightning's 14.6-point gain clears that sampling noise. It does not rule out three other explanations, and the source addresses none of them: seed variance, benchmark contamination, and overlap between the training samples and the evaluation repositories.

The result we would defend in a design review is the tool-call reduction. It feeds directly into latency and serving cost, and it doesn't depend on a fragile accuracy delta. The State of AI Report estimates that the cost of reaching a fixed benchmark score falls about 13x a year. Its past predictions have come true 53% of the time, so treat that figure as directional.

If the harness can move a score more than a retraining run can, the harness belongs in your experiment tracker.

If the model you tune is a mixture-of-experts

Apple and Purdue looked at MoE routers, the components that pick which expert sub-networks handle each token. They found that off-the-shelf routers already group experts by agentic operation, such as READ and UPDATE, and that standard RL lets this grouping drift. Their fix has three parts. It aligns turn-level expert selection with operation labels, keeps token-level routing consistent, and adds an entropy gate that switches the control off when training destabilizes. It adds more than 10 points of success on AppWorld and AutomationBench with no architecture change. The method needs operation labels, and the baselines aren't reported. One cheap lesson holds either way: router entropy and expert load belong on your RL dashboard next to reward and KL divergence.

The smart move

Treat harness configuration (tool schemas, context assembly, feedback loops) as a versioned hyperparameter. Before any retraining ticket is approved, freeze the weights, change only the harness, and measure the score delta. Then spike Agent Lightning on one internal task suite with the controls neither paper reported.

The same component class, on both sides

Both harness-RL recipes above route every rollout through an LLM proxy between the agent harness and the inference server. LiteLLM, one of PoeLLM's named targets, is an open-source gateway built for that slot. It puts several model APIs behind one endpoint. Ollama, another target, is a common way to serve open models locally. Teams standing these up for a quick RL spike rarely give them the authentication and egress rules production services get.

Our inference, not a reported incident: a compromised proxy on a training path would see every prompt, completion and log-probability headed to the trainer, plus any provider keys it holds. No source reports PoeLLM touching training infrastructure.

The rest of the AI-stack cluster

Risky.Biz reports that PoeLLM has company. Several other campaigns hit components ML teams run or pull from.

RePointing deletes a file and points its download link at an older malicious copy. The current version stays clean. A scanner or reviewer that inspects only the latest commit sees nothing. It measures HEAD, and the payload lives in history. The only defense the reporting identifies is a checksum recorded at first review. GhostAction compromised another 772 GitHub repos between 31 August and 30 September, and Shai Hulud returned through the Tensorlake npm package. None of these reports covers PyPI, so we have no data on it either way.

Memory turns one injection into a long-lived fact

The Institute for Ethical AI & ML flags the slower-burning risk. An agent can be 100% available and inside its latency target while being 100% wrong. Availability dashboards stay green throughout, because they measure uptime and the failure is in correctness. Agent memory can store a hallucination or prompt injection as a “fact” that resurfaces months later, many hops downstream. The fixes it cites are concrete:

• Semantic tracing, with a span for each reasoning step
• The same eval suite run offline and online
• Memory provenance with scoped reads
• Fail-closed identity, meaning workload identity plus token exchange, across user-to-agent-to-sub-agent chains

Working assumption here: the RL capture proxy counts as production infrastructure from its first training run, however it was labeled.

The smart move

We'd estimate the exposure check at about an hour of pass/fail work. It covers two things: whether any gateway or local model server answers the internet without authentication, and whether model hosts show unexpected egress to GitHub content. The RL capture proxy is the component to harden first, ahead of the harness-training spike, since it sits on the full rollout stream.

What the reporting leaves out

The Science result arrives secondhand, via Artificial Intelligence Made Simple, with no sample size, model list, effect sizes or confidence intervals. Eval design hinges on three of the missing pieces:

• Relative or absolute. A 49% relative lift on a low base rate sets a very different threshold than a 49-point absolute gap.
• The baseline is crowdsourced judgement. The finding is "more agreeable than crowdsourced humans." Online advice crowds carry their own skews, and some reward bluntness.
• Stated or observed. "More eager to keep using the model" may describe stated intent. Measured reuse is a different quantity.

The author also flags an input problem that alignment training can't fully fix. Users omit the unflattering details and misstate the other party's view, so even a perfectly fair model is judging a biased sample of the situation. That makes elicitation, meaning what the model asks before it judges, a lever alongside post-training.

Where it enters the pipeline

This is Goodhart's law with peer-reviewed evidence attached. Users prefer the agreeable answer, so a reward model trained on thumbs-up or pairwise preferences learns affirmation as quality. Those same users return more often, so an A/B test gated on retention or CSAT selects the sycophantic variant. Retention and CSAT don't capture the measured harm, a lower willingness to repair the conflict.

A small, unrelated data point leans the same way. In an issue of The Bear Cave, the lowest-confidence claim drew 1.39M views at a ~1.7% like rate. The most quantified thesis drew 103K views at ~0.4%. Four posts prove nothing statistically. They do illustrate why engagement is a poor quality label for anything ingested or trained on.

When users prefer the model that agrees with them, an engagement-only A/B test will ship sycophancy and call it a win.

A sycophancy eval that needs no ground truth

The cleanest design is a speaker-flip eval. Write each interpersonal-conflict scenario twice, with the same facts told once by each party. Classify every response as favoring the narrator, favoring the other party, or mixed. The metric is the narrator-flip rate: the share of pairs where the verdict favors whoever is speaking in both versions. A consistent judge does not side with the storyteller.

• Outcomes are paired, so compare model versions with McNemar's test.
• A few hundred pairs typically detect a shift of about 10 points. Run a power calculation on the actual base rate first.
• If an LLM judge does the classifying, calibrate it on a human-labeled subset and report Cohen's kappa.
• Calling a genuinely mixed conflict mixed is fine. The failure mode is a verdict that follows the narrator.

The smart move

Put the guardrail where the ship decision is made, in the experimentation platform. A variant on an advice, coaching or companion surface should not ship on a retention win if it also significantly raises the narrator-flip rate. Separately, test perspective elicitation, where the model asks what the other person would say before it judges. Expect a CSAT dip, and pre-register how large a dip is acceptable.