Evaluation guide

Run structured evaluations to confirm your agent calls the right tools, produces quality responses, and handles edge cases. Grading uses the GenAI Eval SDK in Gemini platform. Evaluation works with any framework template: eval run, eval grade, eval compare and eval analyze are framework-agnostic, and a template only changes how traces are generated. See First-party frameworks.

How do I run and fix evals?#

Your project includes a default dataset at tests/eval/datasets/basic-dataset.json and a metrics configuration at tests/eval/eval_config.yaml. In a Go project (adk_go template), the same files live under eval/ instead of tests/eval/. The examples on this page use the Python layout.

1. Write one or two core eval cases for the most important behavior.
2. Run agents-cli eval run. The command executes the agent over the dataset, grades the traces, and writes results_<timestamp>.json and .html files to artifacts/grade_results/.
3. Read the Failures: list above the RESULT: line, then open the HTML report to see why those cases failed.
4. Fix the agent's instruction, tools, or logic.
5. Re-run agents-cli eval run, then check the change with agents-cli eval compare <before>.json <after>.json.
6. Once core cases pass, add edge cases and new scenarios.

Expect 5 to 10 or more cycles before your agent consistently passes. To walk one failing case through the playground and its trace, see Debug a failing agent.

# Run for a custom dataset and different metrics
agents-cli eval run --dataset tests/eval/datasets/custom-dataset.json --metrics general_quality

# Evaluate a deployed agent instead of running one locally (ADK templates)
agents-cli eval run --url https://my-agent.run.app --app-name app

To keep traces in a custom location, or to re-grade existing traces without re-running the agent, split the run:

agents-cli eval generate --dataset tests/eval/datasets/custom-dataset.json --output custom_traces/
agents-cli eval grade --metrics general_quality --traces custom_traces/

How do I choose metrics?#

Two metrics need extra fields on the case: final_response_match needs a reference, and final_response_reference_free needs rubric_groups.

Only the three multi_turn_* metrics accept multi-turn traces. The other built-ins reject them. Run agents-cli eval metric list for every built-in metric with a description. For case schemas and the full metric reference, see the Gemini platform evaluation documentation.

How do I define custom metrics?#

The eval_config.yaml file selects the metrics to run and defines custom ones. The scaffolded file runs custom_response_quality, a local LLM judge defined in tests/eval/response_quality.py. You can pass a different YAML or JSON file with --config.

• The metrics_to_run list selects the metrics for a run. A name resolves to a custom_metrics entry when one matches, otherwise to the built-in metric of that name.
• The custom_metrics list defines metrics without running them. A metric defined here runs only when it's listed in metrics_to_run or passed with --metrics.

metrics_to_run:
  - response_under_500_chars

custom_metrics:
  - name: response_under_500_chars
    custom_function: |
      def evaluate(instance: dict) -> dict:
          response = instance.get("response") or {}
          text = "".join(
              p.get("text", "") for p in (response.get("parts") or []) if p.get("text")
          )
          passed = len(text) <= 500
          return {
              "score": 1.0 if passed else 0.0,
              "explanation": f"Final response is {len(text)} chars (limit 500).",
          }
  - name: response_quality_rubric
    prompt_template: |
      Rate the agent's response 1-5 for helpfulness and accuracy.
      Prompt: {prompt}
      Final response: {response}
      Full trace (for tool-call and reasoning context): {agent_data}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model: gemini-3.8-flash
    judge_model_sampling_count: 3

How do I gate CI on eval scores?#

Set the minimum mean score for each metric, in that metric's own scale, under thresholds. The eval run and eval grade commands end with a verdict line such as RESULT: FAIL (response_quality_rubric 3.20 < 4.00) and exit 1 when a metric misses its threshold, so CI can gate on the exit code. A run also fails when any case errored, with or without thresholds.

thresholds:
  response_under_500_chars: 1.0
  response_quality_rubric: 4

Override one threshold for a run with --fail-under response_quality_rubric=4. Repeat the flag for more metrics. The flag always names a metric because metrics score on different scales. When a threshold is set and the eval service reports no pass_rate (local custom metrics), pass_rate in the results is the share of cases at or above the threshold.

What other eval commands are there?#

The eval run command covers the inner loop. The CLI marks eval dataset, eval analyze, eval optimize, eval submit, and eval results as experimental.

For eval case schemas, metrics, and user simulation, see the Gemini platform evaluation documentation.

To keep eval spend down, see Control cost.