AutoEval: run evals on your own LLM traffic

Overview

AutoEval runs evals on the LLM calls your applications make, so you know whether each workflow works, which model to run, and what it costs. For each call it takes the input (the system prompt and the user prompt or conversation) and the output, and grades it against two kinds of rules:

Default rules. On for every call from day one: is the output a correct, complete answer to the input it was given? A judge model returns a verdict, a score from 0 to 5, and an explanation.
Custom rules. Checks your team writes in plain English for each use case, such as “always include a link to a help article” or “reply in the language the user wrote in”. Each rule returns pass, fail, or not applicable, with an explanation.

Use the results three ways:

Measure quality in production. See the correctness rate and the pass rate of every rule, per use case, model, and tenant, and find the calls that failed before your customers report them.
Keep a golden eval set. Build a set of inputs from your own traffic that stays current as your product and your users change.
Compare models. Replay the golden eval set on other models, including your own in-house or fine-tuned models, grade every output the same way, and compare the results with cost and latency before you change models.

AutoEval is part of Iris Control. It works on a copy of your logs, so it never sits on your request path and adds no latency to your app.

Try it before you connect anything: the AutoEval tab of the Iris Control live demo compares models on IrisAgent’s own support eval set, with price.

How it works

Collect calls. AutoEval needs the input and output of each call. Calls arrive from the Iris Control endpoint, from a log import, or one at a time through the evaluate API.
Grade each output. A judge model reads the system prompt, the user prompt or conversation, and the output, and returns the correctness verdict from the default rules and a result for each of the use case’s custom rules.
Keep a golden eval set. AutoEval builds the golden eval set for each use case from your call logs and keeps it current. You can also add calls from the call log or upload a file.
Compare models. AutoEval sends every input in the eval set, with its original system prompt, to each model you choose, and grades each output with the same default and custom rules.

AutoEval never changes the models your application uses. You decide whether to switch.

Rules

Default rules

Every output is graded for correctness without any setup: is it a correct, complete answer to its own system prompt and user prompt? The result has a verdict (correct or defect), a score from 0 to 5, and an explanation. See Results.

Custom rules

Custom rules are checks written in plain English, in the same format as AutoQA rules. Each use case has its own rules, and every output in that use case is graded against all of its active rules. To create a rule, open AutoEval in the Iris Control dashboard, select the use case, and click Create rule. Each rule has:

Field	Description
Description	What the output must or must not do, in plain English.
Category	Groups rules in reports, such as Format, Accuracy, Compliance, Tone, or Custom.
Severity	`critical`, `warning`, or `info`. Reports show critical failures first.

Example rules:

Category	Rule
Format	The output must include at least one link to a help center article.
Format	The output must be valid JSON with the fields `intent` and `priority`.
Language	Reply in the same language, or mix of languages, the user wrote in.
Compliance	Never promise a refund amount or a delivery date.
Accuracy	Never mention a product feature that is not described in the system prompt.

A rule returns not_applicable when it doesn’t apply to that call, for example a rule about refunds on a question about passwords. Not applicable results are left out of the rule’s pass rate.

Results

Every graded output has one result:

{
  "id": "call_8f2a",
  "model": "gpt-5.6-sol",
  "judge": "gemini-3.7-flash",
  "correctness": {
    "verdict": "defect",
    "score": 2,
    "explanation": "The user asked how to export invoices; the answer explains how to download a single receipt."
  },
  "rules": [
    {
      "ruleId": "include-link",
      "outcome": "fail",
      "explanation": "The output contains no link."
    },
    {
      "ruleId": "same-language",
      "outcome": "pass",
      "explanation": "The user wrote in English and the reply is in English."
    }
  ]
}

Field	Description
`correctness.verdict`	`correct` or `defect`.
`correctness.score`	0 to 5. Higher means a more correct and complete answer to the input.
`correctness.explanation`	Why the judge gave that verdict.
`rules[].outcome`	`pass`, `fail`, or `not_applicable`.
`rules[].explanation`	Why the rule passed or failed.

The AutoEval page of the dashboard rolls results up by use case, model, tenant, and rule: correctness rate, average score, the pass rate of each rule, and the failed calls behind each number.

Golden eval set

A model comparison is only as good as the inputs it runs on. Eval sets written by hand for a pilot stop matching production within months: new workflows are missing, common questions repeat, and the hard cases drop out. AutoEval keeps a golden eval set for each use case that stays live:

Built from real traffic. Inputs come from your call logs, so the set reflects what your users ask today.
No redundancy. Duplicate and near-duplicate inputs are removed, so one common question can’t skew the score.
Diverse. The set is balanced across workflows, tenants, languages, and edge cases, and includes calls that failed.
Current. When you launch a new use case or workflow, it gets its own slice of the set. Inputs that no longer match your product are retired.

You can add specific calls from the call log or upload a file of inputs, for example cases your team wants every model to pass.

Compare models

Open AutoEval, select a use case, and open Compare models.
Choose the eval set: the use case’s golden eval set, calls selected from the call log, a filter (for example the last 30 days, or only failed calls), or an uploaded file.
Choose the models. Any model from your providers can be compared, including a self-hosted model with an OpenAI-compatible endpoint.
Run the comparison.

AutoEval sends each input to each model with its original system prompt and grades every output. The report shows, per model:

Correctness rate and average correctness score
Pass rate of each rule, critical rules first
Cost per call and total cost, from each provider’s price
Typical and slowest latency

Open any row to see the input and every model’s output side by side, with their grades.

Import logs

Use a log import to run evals on calls your applications already made without routing traffic through Iris Control. Send one record per LLM call as JSONL:

{
  "id": "call_8f2a",
  "timestamp": "2026-10-15T18:04:11Z",
  "use_case": "support-answers",
  "model": "gpt-5.6-sol",
  "messages": [
    {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
    {"role": "user", "content": "How do I export last month's invoices?"}
  ],
  "output": "Go to Billing, then Invoice history, and select Export.",
  "latency_ms": 1840,
  "metadata": {"tenant": "acme-corp"}
}

Field	Required	Description
`id`	Yes	Your ID for the call.
`messages`	Yes	The input in OpenAI message format: the system prompt and the user prompt, or the whole conversation before the output.
`output`	Yes	The model’s output for that input.
`model`	Yes	The model, or model version, that produced the output.
`use_case`	No	Selects the rules the output is graded against. Defaults to `default`.
`timestamp`, `latency_ms`	No	Used in reports and for filtering.
`metadata`	No	Tags you filter reports by, such as tenant, product, or language.

For a conversation, send one record per assistant turn: the conversation up to that turn in messages, and the turn in output. For voice, use the transcript. Send a one-time file to your Iris Control point of contact, or set up a recurring export to a storage bucket with them. Exports from logging tools such as Langfuse, LangSmith, or Helicone can be converted to this format.

Evaluate API

Grade one output on demand, for example from your own test suite or CI. The endpoint grades the output against the rules you send, or against the use case’s rules when you send use_case instead.

curl https://api1.irisagent.com/v1/control/autoeval/evaluate \
  -H "Authorization: Bearer $IRIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
      {"role": "user", "content": "How do I export last month'\''s invoices?"}
    ],
    "output": "Go to Billing, then Invoice history, and select Export.",
    "rules": [
      {"id": "include-link", "description": "The output must include at least one link to a help center article.", "category": "Format", "severity": "critical"}
    ]
  }'

The response has the correctness and rules fields described in Results. Send up to 100 rules per request.

Data handling

Inputs and outputs are redacted for PII and secrets before they are stored. Set how long they are kept with the use case’s logging.retention_days. If prompts and outputs can’t leave your environment, Iris Control and AutoEval can run in your own VPC; see the customer VPC reference architecture.

Questions? Contact us at contact@irisagent.com or book an Iris Control demo.

Getting started

Connecting Data Sources

Deploying IrisAgent

Configuring AI and Automation

User Guides

Security and Compliance

Account Management

Iris Control

AutoEval: run evals on your own LLM traffic

Overview

How it works

Rules

Default rules

Custom rules

Results

Golden eval set

Compare models

Import logs

Evaluate API

Data handling

​Overview

​How it works

​Rules

​Default rules

​Custom rules

​Results

​Golden eval set

​Compare models

​Import logs

​Evaluate API

​Data handling

Overview

How it works

Rules

Default rules

Custom rules

Results

Golden eval set

Compare models

Import logs

Evaluate API

Data handling