02 · Enterprise Intelligence
Evaluating enterprise agents under real conditions
What does a trustworthy evaluation of an enterprise agent look like when the stakes are actual?
- Area
- Enterprise Intelligence
- Status
- In progress
- Models in scope
- ClaudeGPTGeminiLlamaMistral
The question
Benchmarks measure what is easy to score. Enterprise work is long, ambiguous, and full of context that never makes it into a test set. This note explores evaluation methods that reflect how agents are really used.
Approach
- Design task suites from real workflow shapes, not synthetic prompts
- Separate capability from reliability across repeated runs
- Compare closed and open models under identical conditions
- Record where human review changes the outcome
What we're watching
- How much does run-to-run variance matter in practice?
- Which signals predict failure before it happens?
- What should be scored automatically, and what needs a person?