Evals
A test for an AI system: give it an input, apply grading logic to its output and measure success (Anthropic's definition). A set of tasks is re-run after every change to the model, the instructions or the tools.
AI and GrowthTerm 69 / 81
In practice
Evals are the agent-world counterpart of an experimentation culture: a growth team without A/B tests decides on instinct, an AI team without evals decides on complaints (our reading). Anthropic describes the loop of a team without evals: wait for complaints, reproduce manually, fix the bug, hope nothing else regressed. Its advice is to start small: 20-50 simple tasks drawn from real failures is a great start, and evals get harder to build the longer you wait. The growth test: does your set measure the work the customer pays for, or the model's general ability?