GUI Agents Top Out at 21% on Multi-App Tasks That Mirror Real Work

Single-application benchmarks have been the de facto standard for measuring GUI agent progress. The assumption embedded in that choice is that competence on one app generalizes to coordinated work across many. WindowsWorld tests that assumption directly, and the answer is no.

The benchmark covers 181 tasks drawn from 16 occupational profiles, spanning 17 common desktop applications, with 78% of tasks requiring at least two applications in sequence. Each task is broken into sub-goals, averaging 5.0 per task, with intermediate inspection checkpoints so failure can be localized to a specific step rather than just logged as a final outcome. That process-centric design is the structural difference from prior work: instead of asking whether an agent completed a task, it asks where and why the agent stopped making progress. Think of it as the difference between grading a student's final answer and reading their scratch work. The intermediate checkpoints reveal that agents are not failing late in complex workflows; they are stalling early, before the cross-application coordination even begins.

The numbers are direct. Every leading model and agent evaluated scores below 21% success rate on multi-application tasks. Performance on simpler single-app tasks sits meaningfully higher, confirming the gap is not a general capability floor but a specific failure mode triggered by cross-app coordination. Agents also consistently exceed human step limits by a wide margin before failing, meaning they are not stopping because they ran out of attempts; they are spinning. Conditional judgment tasks requiring state tracking across three or more applications produce near-zero completion rates. For teams building enterprise automation tools, the takeaway is direct: if your eval suite is single-application, your benchmark is not measuring the workflows your customers actually run.

We're thinking: The deeper problem WindowsWorld surfaces is not that agents are bad at multi-app tasks. It is that the field has been rewarding the wrong capability. We have been optimizing for isolated task completion while real professional workflows require state continuity across application boundaries, something closer to working memory than instruction following. The benchmark's 21% ceiling should read as a calibration signal: any team claiming production-readiness for enterprise automation based on OSWorld-class scores is measuring a proxy that does not transfer. The more uncomfortable read is that architectural changes, not more fine-tuning on single-app data, are probably required to close this gap.

Key takeaways:

• WindowsWorld shifts evaluation from final-outcome scoring to process-centric sub-goal inspection, exposing where cross-application coordination breaks down rather than just whether it does.
• All tested agents score below 21% on multi-app tasks and consistently exceed human step limits before failing; the benchmark covers 181 tasks across 17 applications, though the task count is modest enough that per-occupation variance deserves scrutiny.
• Teams evaluating GUI agents for enterprise or professional automation workflows should add cross-application task suites to their benchmarks before drawing conclusions from single-app scores.

Source: WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents