We're hiringCareers at Moonlit Labs
Research

02 · Enterprise Intelligence

Evaluating enterprise agents under real conditions

What does a trustworthy evaluation of an enterprise agent look like when the stakes are actual?

Area
Enterprise Intelligence
Status
In progress
Models in scope
ClaudeGPTGeminiLlamaMistral

The question

Benchmarks measure what is easy to score. Enterprise work is long, ambiguous, and full of context that never makes it into a test set. This note explores evaluation methods that reflect how agents are really used.

Approach

  • Design task suites from real workflow shapes, not synthetic prompts
  • Separate capability from reliability across repeated runs
  • Compare closed and open models under identical conditions
  • Record where human review changes the outcome

What we're watching

  • How much does run-to-run variance matter in practice?
  • Which signals predict failure before it happens?
  • What should be scored automatically, and what needs a person?
NextOversight and trust in autonomous workflows