Stateful enterprise agent evaluation
Where agents meet
business rules.
Multi-turn tasks. Nine business domains. Executable contracts that check what agents say, do, and leave behind.
A research benchmark for tool-using agents in stateful enterprise environments.
01 / The benchmark
Business workflows with state.
Each task binds a user goal and an initial world to a task-specific executable contract. Trials run against synthetic records and deterministic tools, without a live CRM backend.
- 1
Reset the business world
Start with fresh records, domain policies, and the task’s declared initial state.
- 2
Let the interaction unfold
An on-policy simulated user responds to the agent’s visible replies. The reference scaffold routes each turn to scoped tools.
- 3
Check the executed contract
Grade the recorded dialogue, tool trace, and final state. Strict success requires every task-declared check to hold.
Tasks and their contracts are separate from the reference agent. Latency and cost are reported alongside correctness.
Nine domains
Number of tasksEvery system faces the same domain task sets.
Inside the benchmark
From a task contract to an executed verdict.
An anonymized funds-transfer example: a $1,000 request fails validation without moving funds; after the user corrects the amount to $50, the agent validates, executes, and confirms the transfer.
View full-size SVG02 / A closer look
A correct reply can hide
a duplicated transaction.
A $15.50 checkout should award 155 loyalty points once. In an observed trace, the agent repeats the checkout and leaves 310 points, while reporting 155.
Compare the expected behavior with the observed duplicate. Only the declared state check catches the doubled balance.
See the trace in the paper“You earned 155 loyalty points.”
Observed duplicate: two checkouts, twice the loyalty credit.
03 / Domain workflows
Follow a business workflow.
Explore the paper’s reference-agent diagrams, from a user request through scoped tools, business-state changes, policy gates, and declared evaluation checks.
Payments
A representative payment journey in the reference scaffold.
Each diagram illustrates one journey. Tool lists span the turns shown; evaluation checks apply when declared by the task contract.
On a narrow screen, scroll within the diagram to inspect its labels, or open the full-size SVG.
04 / Results
One benchmark. Different strengths.
Explore the official first-round leaderboard and the two additional executions. The mean gives each round equal weight.
Cost and latency describe the first round. Cost is estimated tested-agent cost for the full suite; “n/a” denotes an unmetered self-hosted run.
How to read these scores
Strict success is the conjunction of the checks declared by each task. It measures the complete evaluated agent system.
Three-round mean averages the three round-level success rates. Sample standard deviation is in percentage points. All-three success is the percentage of tasks that succeed in all three executions, restricted to tasks eligible for evaluation in every round.
Success rates use tasks with an eligible verdict; unavailable verdicts are excluded.
Across executions
Order can persist.
Task outcomes still change.
Across the three reported rounds, pairwise rank correlations range from 0.869 to 0.934.
Gemini 3.8 Flash reaches 86.49% strict success in both rounds 2 and 3, while 5.41% of task outcomes switch between those executions. Its all-three success is 78.20%.
These findings describe the published paper snapshot. Imported data changes the table preview, not the paper’s reported conclusions.
05 / Results by domain
Domain success across three rounds.
Each cell is the equal-weight mean of the three round-level strict-success percentages within a business domain. Model search and family filters above also apply here; domain values remain three-round means.
Values are percentages. On a narrow screen, scroll horizontally to compare domains.
06 / Resources
Read, inspect, and build on the benchmark.
The paper
CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
Code & benchmark
Access the benchmark, task definitions, tools, and evaluation code.
Open repository