EvoTrainer: Fixing the Training Harness While Tuning the Policy Is a False Economy

Most agentic RL pipelines treat the training harness as fixed infrastructure. The policy evolves; the harness that interprets rollouts, shapes rewards, and diagnoses failures does not. That split is not a neutral engineering choice. It is the reason so many agentic fine-tuning runs plateau in ways that are genuinely hard to diagnose.

The core problem is that scalar rewards compress everything. A model that fails because it misunderstands the task structure looks identical in the loss curve to a model that fails because the reward signal is too sparse for the current task horizon. A static harness cannot distinguish these cases, so it cannot adapt to them. The result is a pipeline that optimizes confidently against failure modes it cannot see.

EvoTrainer breaks this by making the harness itself a participant in the training loop. Instead of separating policy optimization from harness design, it runs both as co-evolving processes, each informing the other through empirical feedback. The mechanism has four stages: rollout-level diagnosis, where the system inspects trajectory evidence to identify what is actually failing; diagnostic revision, where those diagnoses are updated as new rollouts arrive; backtest-gated intervention, where candidate harness changes are evaluated against historical rollouts before being promoted; and skill accumulation, where strategies that survive backtest are retained as reusable components for later search. Think of it as the difference between a static test suite and a QA process that rewrites its own test cases when it finds a class of bugs the original suite missed. The harness is not just scaffolding. It is a hypothesis about what good training looks like, and that hypothesis needs to update.

The backtest gate is the structural detail worth examining closely. Without it, the system risks promoting harness changes that score well on recent rollouts but generalize poorly. By replaying candidate interventions against retained trajectory data before committing them, EvoTrainer filters out high-scoring but brittle branches. Trajectory analyses confirm this: retained strategies diverge meaningfully across domains, which means the system is not converging on a single universal harness but discovering domain-specific ones. That is a qualitatively different outcome from recipe search.

Evaluated across mathematical reasoning, competitive-programming code generation, and repository-level software engineering, EvoTrainer matches or exceeds human-engineered RL references under identical data, codebase, and evaluation protocol. The largest gain appears on long-horizon agentic SWE, which is exactly the setting where shifting bottlenecks and sparse rewards make a static harness most expensive. For teams running agentic RL fine-tuning on complex, multi-step tasks, the takeaway is direct: the harness design deserves the same iteration budget as the policy itself.

We're thinking: We read EvoTrainer as an indictment of a widely shared assumption in agentic RL: that the training harness is a solved problem you configure once and leave alone. The paper's trajectory analyses make this concrete. Retained strategies diverge across domains, meaning there is no universal harness that works for math reasoning and SWE simultaneously. Most teams today share a single harness across task types, which means they are almost certainly leaving domain-specific signal on the floor. The deeper issue is that backtest-gated promotion changes the economics of harness iteration: you can now explore harness changes speculatively without risking a bad training run, which removes the main reason teams avoid touching the harness mid-training. That operational unlock may matter more than any single benchmark number.

Key takeaways:

• EvoTrainer co-evolves both the LLM policy and the training harness through a four-stage loop: rollout diagnosis, diagnostic revision, backtest-gated intervention, and reusable skill accumulation, replacing static recipe search with joint adaptation.
• It matches or beats human-engineered RL baselines across math, code, and SWE under identical experimental conditions, with the largest gains on long-horizon agentic tasks; the main caveat is that domain-specific harness divergence means results may not transfer directly to tasks outside these three evaluated domains.
• Teams running agentic RL fine-tuning on multi-step or long-horizon tasks should treat harness design as an iterable component, not fixed infrastructure, and consider backtest-gated evaluation before promoting any harness change mid-training.

Source: EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic RL