Enterprise Worlds: ITSMBench

Enterprise Worlds is a family of executable enterprise environments. Each release is a world with persistent state, tools, policies, and simulated users. The aim is to grow evaluation from isolated tasks to operational jobs: agents that do not just assist with work, but become accountable for completing it correctly.

ITSMBench is the first benchmark in that family, and currently the only one released: an IT service management environment for testing whether agents can carry out policy-governed work inside a system of record. A task runs against an executable ITSM tenant — 900+ interconnected records across 16 entity types, a policy rulebook, 93 typed tools, and a simulated operator who surfaces facts only when asked. Reward is computed after the run from the final environment state, not from the transcript.

How a model is scored

Each model is evaluated on 53 ITSMBench tasks with 4 independent trials per task. A trial passes only if the final database state matches the expected final state.

  • pass@k — did at least one of k independent attempts solve the task? This matches coding workflows, where failed candidates can be discarded and a test runner or reviewer can select the working attempt.
  • pass^k — did all k independent attempts solve the task? This is the enterprise reliability view: every failed attempt may write records, send notifications, miss approvals, or leave stale access behind.

Leaderboard

  1. 1Grok 4.60.703
  2. 2Claude Opus 4.8medium adaptive0.698
  3. 3DeepSeek V4 Pro08130.698
  4. 4DeepSeek V4 Flash07310.604
  5. 5DeepSeek V4 Pro0.604
  6. 6Claude Fable 50.599
  7. 7Qwen3.7-max0.599
  8. 8Gemini 3.5 Flash0.590
  9. 9Kimi K30.590
  10. 10GLM-5p20.547
  11. 11Claude Opus 50.538
  12. 12GPT-5.6Luna0.505
  13. 13Gemma 4 31Binstruct0.505
  14. 14Muse Spark 1.10.500
  15. 15GPT-OSS 120B0.481
  16. 16Qwen3.7-plus0.476
  17. 17Nemotron-3 UltraNVFP40.222

17 models · 53 tasks · 4independent trials per task. Colour follows the vendor, so a maker’s models share one. Six of these rows are our own runs, recomputed from the raw trials at build time; the other eleven were contributed by a third party and carry scores only, with no trials behind them and no viewer entry. The two sets come from different harnesses and have not been reconciled, so compare across them with care.

Capability vs. repeatability
Two bars per model on one scale: what it solved at least once in four attempts, and what it solved every time.
  • Grok 4.6
    at least once0.925
    every time0.472
  • Claude Opus 4.8
    at least once0.887
    every time0.396
  • DeepSeek V4 Pro
    at least once0.962
    every time0.377
  • DeepSeek V4 Flash
    at least once0.792
    every time0.321
  • DeepSeek V4 Pro
    at least once0.849
    every time0.321
  • Claude Fable 5
    at least once0.868
    every time0.321
  • Gemini 3.5 Flash
    at least once0.830
    every time0.321
  • Kimi K3
    at least once0.887
    every time0.302
  • Claude Opus 5
    at least once0.811
    every time0.245
  • GPT-5.6
    at least once0.792
    every time0.226
  • Qwen3.7-max
    at least once0.868
    every time0.226
  • GLM-5p2
    at least once0.887
    every time0.208
  • Qwen3.7-plus
    at least once0.736
    every time0.189
  • Gemma 4 31B
    at least once0.811
    every time0.189
  • Muse Spark 1.1
    at least once0.849
    every time0.151
  • GPT-OSS 120B
    at least once0.792
    every time0.132
  • Nemotron-3 Ultra
    at least once0.472
    every time0.038

The difference between a model’s two bars is the gap: work it can do but cannot be relied on to repeat. Claude Opus solves 0.887 of tasks at least once and 0.396 in all four attempts; GLM-5p2 reaches the same 0.887 and holds 0.208. Colour follows the vendor. The two sets of runs come from different harnesses and have not been reconciled, so compare across them with care.

Every trial, browsable
Trajectory viewer

The full agent–operator conversation, every tool call and its result, and for each failure the exact database fields that diverged from the expected final state.

Beyond the released trials
Access full dataset

This site publishes the trajectories released for four of the six models. Book a short call to talk about access to the complete dataset.

Cite us

If you use Enterprise Worlds, ITSMBench, or the released task data in your work, please cite:

BibTeX
@misc{enterprise_worlds_itsmbench_2026,
  title        = {Enterprise Worlds: ITSMBench},
  author       = {Vibrant Labs},
  year         = {2026},
  howpublished = {\url{https://github.com/vibrantlabsai/Enterprise-Worlds}},
  note         = {Executable benchmark for policy-governed enterprise IT service management tasks}
}

References

Contact

Questions about the benchmark, the released trajectories, or evaluating a model on it: shahul@vibrantlabs.com