Skip to main content
Evals replay customer conversations against your AI agent and score the result. Save the questions customers ask most, and the ones your agent got wrong, then run them after every change to catch regressions before customers see them. Each run checks up to three things:
  • Success check: did the agent do what you described? An AI judge decides, and explains why.
  • Backed by sources: is every claim in every reply supported by an article or procedure the reply cited? Unsupported claims are highlighted in the transcript.
  • No handoff: did the agent answer without handing the customer to a human?
Evals never run actions, never create tickets, and never contact anyone. They don’t appear in Chat activity, metrics, or QA Reviews.
Evals test your support AI agent. To score past conversations against quality rules, see AutoQA. To test the LLM calls your own apps make, see AutoEval.

​
Create an eval

Open Quality → Evals and click New eval. Fill in: Click Save, or Save and run to run it right away.
Describe the outcome, not exact wording. “Explains how to start a return and the time limit” holds up across model and content changes. “Says ‘You can return items within 30 days’” breaks on the first rephrase.

​
Turn a real conversation into an eval

The fastest way to build evals is from conversations your agent already had.
  1. Go to Activity → Conversations and open a chat.
  2. On any AI reply, click Turn into eval (next to Revise).
  3. IrisAgent drafts the name, first message, customer goal, and success check from the conversation up to that reply. Personal details such as names, emails, and order numbers are replaced with made up values.
  4. Review the draft, especially the success check, and click Save or Save and run.
The success check is based on what the customer needed, not on what the agent said, so a draft from a wrong answer still tests for the right one.

​
Import evals from a CSV file

  1. Click Import CSV to see the expected format and download a template.
  2. Put one customer message per row in the first column. A header row such as first_message is optional, and other columns are ignored.
  3. Choose your file. You see how many messages were found and a preview of the first five.
  4. Click Import.
Each row becomes a one question eval. IrisAgent drafts a name and success check for each row, so check them before you run. A file can hold up to 50 rows.

​
Run evals

  • Run one eval: click the run button on its row, or Run again in its results.
  • Run several: select them and click Run selected.
Runs happen in the background, and results appear as each eval finishes. You can leave the page while they run. Each run uses your default chatbot configuration, its answer guidelines, and your enabled procedures and knowledge, just like a live chat.

​
Read the results

The Evals list shows each eval’s latest result: Click an eval’s name to see its last run:
  • Checks: each check with Passed or Failed and the judge’s reason.
  • Transcript: the AI customer’s messages and your agent’s replies, with the sources each reply cited and the procedure it matched. Claims that no cited source backs are highlighted in red and listed under the reply.
  • Skipped actions: if a procedure reached the point where a live chat would run its action (such as a refund or order lookup), the reply shows Action skipped in this eval. The judge knows the action didn’t run.
  • Handoffs: a reply that would have handed off shows Would have handed off to a human.

​
How grounding is checked

For each reply, the judge compares every factual claim with the text of the articles the reply cited and the procedure it matched. A claim passes only if one of those sources states it or clearly implies it. Asking a question, acknowledging the customer, or saying that the agent started a skipped action are not claims. See Understanding grounded answers for how IrisAgent grounds answers in the first place.

​
Test a procedure change before you save it

When you edit a procedure, the Test your changes panel has a Run your evals section.
  1. Make your edits. You don’t need to save.
  2. Click Run evals on this draft. All your saved evals run against the agent with your unsaved procedure in place of the live one.
  3. Compare each eval’s Live result with This draft:
If you edit the procedure again, the results are marked out of date until you run them again. Evals still running against an earlier draft are skipped, and the panel tells you to run again once they finish. Live is each eval’s latest live result. If your live agent changed since then, run the evals on live first for an accurate comparison.

​
Edit and delete evals

  • Edit: click the edit button on the row, or Edit in the results. Saving clears the eval’s old results, since they no longer describe it.
  • Delete: click the delete button on the row. The eval and its results are removed.

​
Limits and credits

The Evals page shows how many eval conversations you’ve used today. On the free plan, each agent reply in an eval uses 1 credit, the same as a test chat in the dashboard. An eval without a customer goal uses 1 credit, and an eval with a goal uses up to 4. If a run could use more credits than you have left, it doesn’t start. Only admins can create, edit, delete, import, and run evals. Everyone with dashboard access can view them and their results.

​
FAQ

No. Evals don’t run API actions, create tickets, hand off, send messages, or save chat history. They don’t appear in Chat activity, AI Feedback, procedure metrics, or AutoQA.
Read the judge’s reason in the results. A common cause is a success check that asks for something the customer never asked about, or that names exact wording. Rewrite the check to describe the outcome, and run it again. If Backed by sources fails, check whether the cited article actually states the highlighted claim.
The conversation or its scoring didn’t finish, for example because a model was briefly unavailable. Nothing is wrong with your agent. Run it again in a few minutes.
Your agent and the AI customer are language models, so wording can vary between runs. If an eval flips between pass and fail, tighten its success check or the procedure it tests.
Your default chatbot configuration, with its answer guidelines, product filter, and the procedures mapped to it.