Agent work is only valuable if the surrounding organization can absorb it safely.

A coding agent’s change still needs review, approval, deployment, monitoring, and a rollback path. A sales agent’s closed deal still needs account setup, permissions, billing, and contract updates. A recruiting agent’s hire still needs identity, devices, access groups, and policy checks. As models get better at producing work, the bottleneck shifts to the operational systems that turn that work into durable state.

Enterprise Worlds is a family of executable enterprise environments for measuring that problem directly. Each release is a world with persistent state, tools, policies, and simulated users. The aim is to grow evaluation from isolated tasks to operational jobs: agents that do not just assist with work, but become accountable for completing it correctly.

ITSMBench is the first benchmark in that family, and currently the only one released: an IT service management environment for testing whether agents can carry out policy-governed work inside a system of record. A task runs against an executable ITSM tenant — 900+ interconnected records across 16 entity types, a policy rulebook, 93 typed tools, and a simulated operator who surfaces facts only when asked. Reward is computed after the run from the final environment state, not from the transcript.

Everything below is ITSMBench. Further Enterprise Worlds releases are planned — scaling the worlds, lengthening task horizons, broadening the tool surface, and introducing workflows where external systems fail mid-run.

ITSMBench leaderboard

Each model is evaluated on 53 ITSMBench tasks with 4 independent trials per task. A trial passes only if the final database state matches the expected final state.

  • pass@k — did at least one of k independent attempts solve the task? This matches coding workflows, where failed candidates can be discarded and a test runner or reviewer can select the working attempt.
  • pass^k — did all k independent attempts solve the task? This is the enterprise reliability view: every failed attempt may write records, send notifications, miss approvals, or leave stale access behind.
Columns
Sorted by pass^1, high to low — click a column to change it, click again to reverse.
#Model
1
Claude Opus 4.8medium adaptive
0.698
0.563
0.467
0.396
0.698
0.833
0.873
0.887
2
GLM-5p2
0.547
0.365
0.274
0.208
0.547
0.730
0.821
0.887
3
GPT-5.6Luna
0.505
0.349
0.274
0.226
0.505
0.660
0.741
0.792
4
Muse Spark 1.1
0.500
0.302
0.203
0.151
0.500
0.698
0.797
0.849
5
Qwen3.7-plus
0.476
0.318
0.231
0.189
0.476
0.635
0.708
0.736
6
Nemotron-3 UltraNVFP4
0.222
0.104
0.061
0.038
0.222
0.340
0.415
0.472

6 models · 53 tasks · 4 independent trials per task. Muse Spark 1.1 and Nemotron-3 Ultra NVFP4 are published figures; no trajectories were released for them, so they have no viewer entry.

Capability vs. repeatability
pass^4 — all four attempts correctpass@4 — at least one correct
  • Claude Opus 4.80.3960.887
  • GLM-5p20.2080.887
  • GPT-5.60.2260.792
  • Qwen3.7-plus0.1890.736
  • Muse Spark 1.10.1510.849
  • Nemotron-3 Ultra0.0380.472

The wider the bar, the more of a model’s apparent capability depends on which attempt you happen to look at. Claude Opus solves most tasks at least once (pass@4 = 0.887) but fewer than half in every attempt (pass^4 = 0.396). GLM shows the same pattern more sharply: the same 0.887 at pass@4, but 0.208 at pass^4.

How reliability decays with k
0.00.20.40.60.81.0k=1k=2k=3k=40.3960.2080.2260.1890.1510.038
  • Claude Opus 4.8
  • GLM-5p2
  • GPT-5.6
  • Qwen3.7-plus
  • Muse Spark 1.1
  • Nemotron-3 Ultra

Every model loses more than a third of its pass^1 score by k=4. The ordering is not stable either: GPT-5.6 starts below GLM-5p2 at k=1 and finishes above it at k=4, because a model that solves fewer tasks can still solve them more consistently.

Every trial, browsable
Trajectory viewer

The full agent–operator conversation, every tool call and its result, and for each failure the exact database fields that diverged from the expected final state.

Cite us

If you use Enterprise Worlds, ITSMBench, or the released task data in your work, please cite:

BibTeX
@misc{enterprise_worlds_itsmbench_2026,
  title        = {Enterprise Worlds: ITSMBench},
  author       = {Ragas},
  year         = {2026},
  howpublished = {\url{https://github.com/vibrantlabsai/Enterprise-Worlds}},
  note         = {Executable benchmark for policy-governed enterprise IT service management tasks}
}

References

32 leaderboard cells are recomputed from the raw trial files at build time and checked against the published table; the build fails on any disagreement. Published figures only, no released trajectories: Muse Spark 1.1, Nemotron-3 Ultra.