Agentic RL Training Actively Degrades Tool Judgment: A Fix in 18% Fewer Calls

Standard agentic RL training assumes more training means better tool use. The assumption is half right. Models do learn to call tools more reliably. They also lose the ability to decide when a tool is unnecessary, inflating call counts, latency, and cost without proportional accuracy gains.

The failure has a structural cause. During agentic RL, every training trajectory either calls a tool or does not, but the reward signal treats these choices coarsely: did the final answer come out correct? That framing cannot distinguish between "correctly called a tool" and "called a tool redundantly when parametric knowledge was sufficient." Reward shaping patches that try to penalize excess calls end up suppressing tool use indiscriminately, which is reward hacking, not calibration.

AKBE (Agentic Knowledge Boundary Enhancement) fixes this by probing the model's knowledge boundary on every training instance rather than inferring it from outcome alone. During training, each question gets two rollouts: one with tools available, one without. Comparing correctness across these dual paths sorts each instance into one of several trajectory categories: cases where tools are genuinely required, cases where parametric knowledge already suffices, and cases where the model calls tools but would have been correct without them. Each category receives a distinct supervisory signal targeted to that specific failure mode. The boundary is defined per instance, not globally, which means the training signal is fine-grained enough to teach restraint without suppressing necessary calls.

This is architecturally closer to a diagnostic loop than a reward modifier. Instead of adjusting a scalar penalty and hoping the model generalizes, AKBE generates labeled contrast evidence at training time and feeds it back directly. The dual-path rollout adds compute, but the signal quality improvement is what separates it from prior reward-shaping approaches.

Across seven QA benchmarks, AKBE lifts task accuracy by +1.85 points on average over standard agentic RL while cutting tool calls by 18%, yielding 25% higher tool productivity with no accuracy-efficiency trade-off. The method is plug-and-play across different RL algorithms. For ML infrastructure teams running agentic pipelines in production, the takeaway is direct: the accuracy-efficiency trade-off you are managing may not exist at the model level, it may be an artifact of how the training loop constructs its supervision.

We're thinking: We find the directionality here worth naming clearly: agentic RL training, as currently practiced, is not a neutral process that improves tool use and leaves everything else intact. It actively degrades the model's ability to distinguish parametric knowledge from tool-required knowledge. That is a training-induced regression, not a capability ceiling. The implication for teams shipping agentic systems is uncomfortable: if your pipeline shows rising tool call rates over training iterations without corresponding accuracy gains, the training loop itself may be the source. AKBE's dual-path design is one concrete answer, but the broader point is that knowledge boundary integrity should be a first-class training metric, not an afterthought addressed by downstream cost monitoring.

Key takeaways:

• AKBE probes the model's knowledge boundary per training instance using dual-path (with-tool and no-tool) rollouts, then constructs category-specific supervisory signals rather than a single coarse reward penalty.
• Across seven QA benchmarks, AKBE delivers +1.85 average accuracy and 18% fewer tool calls over standard agentic RL, yielding 25% higher tool productivity; caveat is that dual-path rollouts increase per-step training compute.
• Teams training LLM agents with tool-use RL should instrument knowledge boundary degradation as a training metric and consider AKBE-style dual-path supervision before reaching for reward-shaping heuristics.

Source: AKBE: Agentic Knowledge Boundary Enhancement