# Testing with evals

> Write cases, choose a scoring method and a threshold, and compare two versions of a prompt with numbers. Who runs the model in the pre-beta.

Source: https://prompt-verse.io/docs/testing-with-evals/

An **eval** answers one question: is this version of the prompt better than that one?

## Anatomy of an eval

- **Cases**: a list of inputs, each with an id and what you expect.
- **A method** that scores an answer from 0 to 1.
- **A threshold** from 0 to 1. The default is 0.8.
- **The target**: a prompt, workflow or agent.

```json title="An eval with two cases"
{
  "entity_type": "prompt",
  "entity_id": "<id>",
  "title": "Reply stays within the rules",
  "method": "contains",
  "threshold": 0.8,
  "cases": [
    { "id": "angry-refund", "input": "I want my money back NOW. Order A-1043.", "expected": "next steps" },
    { "id": "not-found", "input": "Where is order Z-9?", "expected": "could not find" }
  ]
}
```

## The five methods

| Method | A case passes when |
|---|---|
| `exact` | the answer equals what you expect |
| `contains` | the answer includes what you expect |
| `structural` | the answer has the shape you expect, such as valid JSON with certain keys |
| `judge` | a model reads the answer and scores it against your expectation |
| `hybrid` | a mix of rules and a judge |

## Writing good cases

- Start with the cases that hurt: the angry customer, the missing order, the odd input.
- Ten good cases beat a hundred similar ones.
- Add a case whenever a bug teaches you something.

## Who runs the model

> [!HONEST]
> Prompt Verse runs no model in the pre-beta, so `evals run` refuses with `AI_DISABLED`. Your own AI does the testing: it reads the cases with `evals get`, renders and runs your prompt for each one, scores the answers and tells you the result in the conversation. Keeping scored runs inside the app belongs to the AI features of Premium and Max, which are switched off for now.

Ask your AI something like:

```text title="Test a prompt with your AI"
Use Prompt Verse. Get my eval "Reply stays within the rules". For each case render my prompt "Support reply" with the case input, answer it yourself, and score each answer from 0 to 1 against the expected text. Give me the mean and say whether it beats 0.8.
```

## Compare versions

Run the same cases on two commits and compare the means. Use the result in your commit message when you merge.