Skip to main content

Pull eval scores

When continuous evals is on, a sample of an Automation’s runs is graded against its eval file. The Evals tab charts each criterion’s average score over time. The evals endpoints return the same numbers, so you can pull them into your own dashboards and alerts. The endpoints are read-only. They need an access token for a service account, and they answer not_found when the organization has traces or continuous evals turned off.

1. Find the Automation

Evals endpoints address an Automation by its id. Automation names are not unique, so list Automations with a name filter and pick the one you mean:

2. List its criteria

Each criterion is one chart of the Evals tab. Pass the same range the tab offers: 24h, 7d or 30d.
The list matches the tab: every criterion the current eval file declares, graded yet or not, and every criterion with a score in the range. A criterion removed from the eval file stays listed while it has scores in the range. A criterion’s key stays the same when its text is edited. Use it to ask for that criterion’s scores.

3. Pull the scores

Ask for all criteria or a comma-separated list of keys:
Each row is one point of the chart:
  • time is where the bucket starts. Buckets are UTC hours for 24h, and UTC days for 7d and 30d.
  • deployment_version is n in the tab’s “Deployment #n”. While a new version rolls out, a bucket can have a row for each version.
  • score is the average score from 0 to 10 of the runs graded in the bucket.
  • samples is how many graded runs that average covers.
Buckets with nothing graded are left out.

Score at a point in time

Pass at instead of range to get the one bucket that holds a moment:
  • A date, such as at=2026-11-18, returns that UTC day.
  • A date-time with a UTC offset, such as at=2026-11-18T14:00:00Z, returns the UTC hour it falls in.

List the graded executions behind a chart

The table under a chart lists the executions graded on that criterion. Get the same rows, newest first, a page at a time:
  • reason is the judge’s overall reason for the score. It can quote the execution.
  • checks holds the judge’s verdict on each check. Only pass and fail count toward the score.
  • execution_id is the id the Automation’s executions endpoint uses, so you can fetch the execution or its trace next.
To get the next page, pass meta.next_page_token as page_token, with the same criterion and range. next_page_token is null on the last page.

Errors

  • bad_request: a range the tab does not offer, an at that is not a date or an RFC 3339 date-time, both range and at, or an empty criteria.
  • not_found: the Automation does not exist or you cannot see its traces, the organization has traces or continuous evals off, or a criteria key has no chart in that range or at. List the criteria for the same window first.