An AI Agent Evaluation Framework for Production Workflows

You don’t need another generic rubric. You need to know whether the agent can finish real work, what it’s allowed to touch, and how it’s watched after go‑live. Here’s a production AI agent evaluation framework: build evaluation datasets from real workflows, score task completion and factual grounding, check tool selection and argument correctness, enforce permissions and clarification behavior, and track latency and cost. Separate offline evaluation for regression from production monitoring and human review. Below, concrete graders for two agents: a warehouse‑query agent and a change‑authoring agent that can open tickets or PRs. For background on query trust, see Enterprise Text‑to‑SQL and the platform patterns in Internal AI Platform Architecture.

What “evaluation” means once an agent touches production

Evaluate agents by outcomes, not vibes. In production, the question is whether the workflow completed correctly, safely, and within budget. That requires an evaluation approach beyond traditional LLM evaluation of answer quality; you’re evaluating execution quality for agentic AI. Our core evaluation framework scores three layers:

• Task layer: did the end‑to‑end goal complete? Are outputs correct, grounded by sources, and formatted for downstream systems?
• Decision layer: were the right tools selected with correct arguments and retries? Did the agent ask for clarifications when inputs were ambiguous?
• Guardrail layer: did it respect permissions, PII rules, rate limits, and escalation policies? Can you safe‑stop and reinstate cleanly?

Metric names matter less than determinism and coverage. Favor checks you can compute repeatedly without a human in the loop and reserve reviewer time for borderline calls. For LLM signals you can’t make deterministic, freeze prompts, models, and tool mocks to reduce variance and keep a reproducible baseline. This agent evaluation focuses on agent behavior observed across steps, not just final text—especially important with generative models that plan and call tools. If you run Slack‑native ai agents, we apply the same evaluation method in threads; we expand the UI and orchestration details in Building a Slack AI Agent That Can Actually Query Your Warehouse.

But first, what is an AI agent?

An agent is an LLM‑driven program that can decide, call tools, and take actions in your stack. Most teams use agents to interact with software via APIs, not just chat. Typical types of agents: warehouse query, PR/ticket author, data quality triage, and Slack concierge. A real‑world agent is more than a prompt; it’s a policy, a set of tools, and logs you can audit. The agent is part of your ai system once it’s wired to production tools; treat it like any other service.

Build an eval set from real workflows (warehouse‑query example)

Skip synthetic trivia. Your evaluation suite should mirror real tickets and ad‑hoc asks. Start with 30–50 prompts and accepted outputs for a warehouse‑query agent. Include constraints: row limits, cost caps, required citations, and freshness expectations. Pull cases across schemas and time windows. Include joins against a 40M‑row orders table on an X‑Small warehouse so you catch plan regressions early. This is your evaluation dataset for repeatable, automated evaluation and offline evaluation in CI.

Capture gold SQL and expected result contracts you can check deterministically. Store fixtures with the warehouse and role to test permission scoping, and include common patterns you see in production (rolling windows, slowly changing dimensions, top‑N by segment):

-- fixture: expected.sql
-- role: ANALYST_READONLY
-- warehouse: XS
-- requirement: cite sources in commentary; cost < $1; runtime < 30s
WITH daily AS (
  SELECT order_date::date AS d,
         COUNT(*) AS orders,
         SUM(amount) AS revenue
  FROM analytics.orders
  WHERE order_date >= current_date - 30
  GROUP BY 1
)
SELECT d, orders, revenue
FROM daily
ORDER BY d DESC
LIMIT 30;

Pair that with a compact result contract and a grounding set: table definitions, metric semantics, and known caveats. If you have a semantic layer, anchor the eval to those entities; if not, write minimal docs that behave like one. We explain why a thin semantic layer reduces prompt complexity in Why AI Agents Need a Semantic Layer Before More Prompts.

# fixture: contract.yml
expected_columns:
  - name: D
    type: DATE
  - name: ORDERS
    type: NUMBER
  - name: REVENUE
    type: NUMBER
checks:
  - name: has_30_rows
  - name: non_negative_revenue
  - name: cites_tables

This grounded eval lets you measure whether the agent selected the right tables, wrote performant SQL, cited sources, respected limits, and returned the data shape your BI expects. List the agents to test, the roles they assume, and the environments they can use; this keeps comparisons fair and reproducible. When the agent is in production, promote new fixtures from incidents and near‑misses; that feedback loop is what will improve ai agent performance over time.

Deterministic graders for the warehouse‑query agent

Deterministic graders beat subjective rubrics. Grade what you can run: SQL compilation, execution, shape, and cost. Then grade behavior: tool/argument choices and clarifications. Example harness (Python) that runs agent output against a mocked or read‑only warehouse and collects evaluation results for automated evaluation:

import time, json
from typing import Dict, Any

class SQLAgentGrader:
    def __init__(self, client, cost_model):
        self.client = client
        self.cost_model = cost_model

    def grade(self, prompt: str, sql: str, contract: Dict[str, Any]) -> Dict[str, Any]:
        started = time.time()
        try:
            plan = self.client.explain(sql)
            compiled_ok = True
        except Exception as e:
            return {"compiled_ok": False, "error": str(e)}

        try:
            df = self.client.query(sql, role="ANALYST_READONLY", wh="XS")
            runtime_s = time.time() - started
            cost = self.cost_model.estimate(plan, runtime_s)
        except Exception as e:
            return {"compiled_ok": True, "ran_ok": False, "error": str(e)}

        shape_ok = list(df.columns) == [c["name"] for c in contract["expected_columns"]]
        checks = {
            "has_30_rows": len(df) == 30,
            "non_negative_revenue": (df["REVENUE"] >= 0).all(),
        }
        cites_tables = "analytics.orders" in sql.lower()
        return {
            "compiled_ok": True,
            "ran_ok": True,
            "shape_ok": shape_ok,
            "checks": checks,
            "cites_tables": cites_tables,
            "runtime_s": runtime_s,
            "estimated_cost": cost,
        }

Layer in decision and safety graders and wire them into simple evaluation pipelines:

• Tool selection: assert the first tool was “query_sql” not “explain_only” when the task required data; penalize detours.
• Clarification: when the prompt is underspecified, the agent must ask a question before running any query over 100k rows.
• Permissions: reject any attempt to switch to a role outside the fixture; log and safe‑stop.
• Agent calling correctness: verify arguments to external tools (warehouse, cache, lineage API) are complete and consistent.

Some SDKs offer built-in evaluation hooks; use them if they export raw events you can trust, but keep graders in your repo so you can change models without rewriting tests. Deterministic graders are natural for data and coding agents because software is straightforward to check: does it run, and do the checks pass? Tie this to BI acceptance by replaying charts in a headless run and diffing image hashes or CSV outputs. For a deeper example agent that runs in Slack, see Building a Slack AI Agent That Can Actually Query Your Warehouse.

Evaluate a change‑authoring agent (tickets and PRs)

For a ticket/PR agent, evaluate three axes: correctness of the change, policy compliance, and ergonomics for humans‑in‑the‑loop. The llm agent should open an issue or pull request only after generating a minimal diff and linking to evidence. Whether the individual agent is allowed to touch production depends on policy checks you can grade offline. Define clear evaluation criteria so reviewers don’t guess.

Fixture: a small dbt change to add a NOT NULL test and fix a join. The gold PR includes a passing model build and an updated test. Your grader can run dbt in CI and assert that the PR turns red/green as intended—this makes a robust evaluation obvious to trust.

-- models/orders_enriched.sql
SELECT o.id, o.user_id, i.sku, i.qty
FROM {{ ref('orders') }} o
LEFT JOIN {{ ref('order_items') }} i
  ON o.id = i.order_id -- agent must correct this if wrong key used

# models/schema.yml
models:
  - name: orders_enriched
    tests:
      - not_null:
          column_name: id

# grader pseudocode
result = run_dbt_pr_checks(pr_branch)
score = {
  "build_ok": result.models["orders_enriched"].status == "success",
  "tests_ok": result.tests["orders_enriched.not_null_id"].status == "pass",
  "policy_ok": pr.targets_only_models(["orders_enriched"]) and pr.no_secrets(),
  "linked_issue": pr.description_contains("Closes #"),
}

Argument correctness: validate tool calls to GitHub/Jira have the minimum fields and correct repository/project. Permission compliance: block label or merge actions outside a sandbox org. Clarification behavior: when CI fails, the agent may propose a fix but must request approval before pushing more commits. Evaluate how the agent handles flaky tests, required reviewers, and branch protection. Deterministic graders here run your CI in a container and parse statuses. You can also snapshot PR diffs for lightweight checks:

diff --git a/models/orders_enriched.sql b/models/orders_enriched.sql
- ON o.user_id = i.order_id
+ ON o.id = i.order_id

Evaluate reinstatement: safe‑stop mid‑run (e.g., after opening the PR) and measure how easy it is to restart with the same trajectory and context. Log all human overrides. For an autonomous agent, keep the override UI one click from the PR or ticket system; guardrails beat wishful thinking. We cover rollout tactics and adoption pitfalls in AI for Data Teams: Workflows That Actually Stick.

Offline regressions, production monitoring, and latency/cost SLOs

Separate concerns. Offline agent evals catch breakages before deploy; production monitoring catches reality; human review catches nuance. You need all three for credible production evaluation and defensible sign‑offs.

Mode	Goal	Signals	Cadence	Who acts
Offline regression	Prevent drift	Deterministic checks, fixture pass rate, latency deltas	CI on PR, nightly	Agent devs
Production monitoring	Catch incidents	Error rates, safe‑stops, cost spikes, tool misuse, trace gaps	Continuous	On‑call
Human review	Improve judgment	Sampled transcripts, override reasons, disputed outcomes	Weekly	Domain owners

Do you need tracing to evaluate ai agents? Yes, if you want debuggability and guardrail proofs. Log prompts, tool calls, arguments, returns, costs, and model IDs. Evaluate by checking random samples of logs—are any missing or incomplete? Missing traces mean you’re blind during incidents. Publish evaluation results where people look: your BI or PR comments. Use evaluation tools that make these checks one command locally and in CI.

Latency, cost, and reliability under real load

Measure end‑to‑end latency and cost the way SREs do: controlled load, realistic data sizes, and failure injection. For a query agent, create fixtures that hit a 40M‑row table on an X‑Small warehouse, a medium‑granularity join, and a cold‑cache run. Record wall time, queue time, and warehouse auto‑suspend/auto‑resume penalties. Tie cost to your provider’s model and warehouse billing. Record token counts per step so you can attribute spend to planning vs action.

• P50/P95 latency per workflow and per tool chain, including clarification turns.
• Token usage by step and total estimated dollars per successful task.
• Queue and concurrency behavior: how many tasks pile up before you degrade?
• Retry and backoff efficacy; error‑class distribution across agent architectures.

Reliability tests to automate: mock tool outages (e.g., Jira 500s), revoke permissions mid‑run, and force a cold start. Assert graceful degradation, a clear user‑facing message, and that reinstatement is one click with context restored. These are service SLOs; treat them like any microservice. Where you need alerts without noise, borrow patterns from Anomaly Detection for Data Pipelines Without Alert Fatigue.

Scoring design, custom metrics, and component-level checks

Define evaluation metrics that roll up to a composite score and a go/no‑go gate. Keep weights transparent. Start with task completion, factual grounding, tool/argument correctness, permission compliance, clarification ratio, latency, and cost. Add domain‑specific metrics (e.g., row‑count plausibility) as custom checkers. This is where you evaluate ai agents consistently across models and prompts, and where ai agent evaluation metrics become real levers—not vanity scores.

# eval_config.yml
workflows:
  - name: warehouse_query
    weights:
      task_completion: 0.35
      factual_grounding: 0.20
      tool_correctness: 0.15
      permission_compliance: 0.15
      clarification_behavior: 0.05
      latency: 0.05
      cost: 0.05
    gates:
      min_fixture_pass_rate: 0.9
      max_p95_latency_s: 20
      max_daily_cost_usd: 50

Can I write custom AI agent evaluation metrics? Yes—most of the value comes from them. Examples: “asked_before_scan” (did the agent request clarification before scanning a large table), “sensitive_term_blocked” (was PII masked), and “retry_backoff_ok.” Component-level evaluation is critical: unit‑test prompt/chain pieces (SQL generation, PR diff creation) separately from execution. That shortens debugging and improves agent behavior across runs.

For LLM evaluation cases that require judgment (narrative tone in tickets), freeze the model and prompt, then use paired comparison with a human tie‑breaker. That keeps variance in check while you accumulate adjudicated gold outputs. When using LLM graders or evaluating subjective qualities, document instructions and inter‑rater rules. Calibrating LLM graders or evaluating ambiguity needs a pilot: run small, adjudicate, adjust, then lock. Your evaluation must include a paper trail so you can determine whether an agent changed behavior after a model update.

Governance, tooling, and integration with your stack

Autonomy isn’t binary. Employees are comfortable with more autonomous AI when it’s clear what the agent can do, how to stop it, and how to fix it. Governance policies should be executable code. Define allowed tools, max scope, and escalation paths per workflow and role; then assert them in graders and in runtime guards. These best practices prevent unclear ownership during incidents and make your evaluation strategies enforceable.

# policy.yml
workflows:
  ticket_pr:
    allowed_tools: ["plan_change", "open_issue", "open_pr"]
    forbidden_tools: ["merge_pr", "delete_branch"]
    require_human_approval: ["push_commits_after_ci_fail"]
    max_changeset_lines: 50
    escalation:
      - condition: ci_failed_2x
        action: notify_oncall

Tooling integration: most agent frameworks (LangChain‑style planners, tools‑first routers, custom loops) can emit standardized events: thought, tool_call, tool_result, message. Instrument those and you can monitor any runtime. Minimal tracing fields that make comprehensive evaluation practical: run IDs, parent/child IDs, prompt hash, model name, temperature, tool name and arguments (redact secrets), durations, result hashes, token counts, cost estimates, safe‑stop reasons, and a reinstatement token. Popular libraries often provide hooks; use them, but keep a thin wrapper so you can swap models and architectures based on evaluation results without rewriting logs.

Multiple agents introduce new failure modes: if one agent optimizing a sub‑goal hurts another’s progress, the orchestrator must catch conflicting actions (e.g., two PRs racing). Evaluate scenarios where conflicts occur and assert that the coordinator de‑duplicates or serializes appropriately. For agent systems with shared resources, require locks around schema‑altering actions and route conflicting intents to a human. Evaluate the UI for humans to override: it should be a single button in Slack or the PR system that safe‑stops the run and records who and why. For a Slack‑native front‑end with shared integrations and review UX, see our Slack AI Agents and how we assemble internal platforms in Internal AI Platforms.

What breaks in week two: permissions, grounding, and drift

The second week is when shortcuts surface. Permissions drift after a role change; grounding docs fall behind schema changes; latency spikes when a warehouse auto‑suspends mid‑query. Build checks for these early.

• Permissions: put role and warehouse in the fixture; fail any deviation. Add a “privilege_changed” monitor that compares yesterday’s grants to today’s.
• Grounding: store the docset or semantic YAML used during eval; alert if tables or columns in fixtures change without doc updates.
• Drift: track p95 latency and cost per workflow; alert on deltas, not absolutes. Evaluate continuous evaluation in production by replaying a 10% sample nightly.

Most teams roll out ai agents for narrow tasks first. That’s correct. Specialized agents reduce blast radius and make scoring crisp. As agent development expands into more ai applications, keep the same backbone: fixtures, deterministic graders, production monitors, and a weekly review. If you need a shared orchestrator UI, we’ve outlined one in Internal AI Platform Architecture: Connect Agents to Your Stack.

Answering the “can it do X?” question without wishful thinking

To determine whether an agent can perform a task correctly under expected conditions, write a fixture that encodes the task, the constraints, and the acceptance checks. Then add a failure‑mode variant (tool down, permission revoked, ambiguous prompt) and require a graceful path. That’s the evaluation method you can defend in change review and postmortems.

Examples:

• Warehouse query: required to cite source tables and limit to 30 days. Fails if it scans >100k rows without asking for clarification. Passes only if the shape matches contract and runtime < 30s.
• Ticket PR: required to add a not_null test and fix join key. Fails if it touches files outside the model folder or tries to merge. Passes only if CI is green and the PR links to the original issue.

For coding agents, deterministic graders are straightforward: does the code run and do the tests pass? For narrative outputs, set up paired comparisons with a small rubric and a human adjudicator. Your goal isn’t a perfect score; it’s enough signal to ship safely. If you need to evaluate ai agents across models, pin models for a week, compare, then rotate—avoid mixing experiments with prod. Publish “what changed” notes so stakeholders have the judgment to grade these agents over time.

Putting it together: a production‑ready evaluation regime

Bringing the framework together, here’s how we evaluate ai agents in practice and ship with confidence:

1. Inventory workflows and pick high‑value, low‑blast‑radius candidates. Start with specialized agents (warehouse query, PR author). Document the evaluation criteria per workflow.
2. Create 30–50 fixtures per workflow from real tickets. Save prompts, gold SQL/PRs, contracts, and policies. Tag them by complexity and cost; these become your evaluation datasets.
3. Write deterministic graders for task, decision, and guardrail layers. Add subjective checks only where necessary. Keep a simple CLI to run the full suite locally. Include built-in evaluation hooks where your runtime provides them.
4. Wire tracing in your agent frameworks. Emit everything you’ll need to debug and score. Store traces where you can query them with SQL. Evaluate by checking random samples for gaps.
5. Run CI nightly and on PRs. Gate deploys on pass rate, p95 latency, and policy compliance. This is your production evaluation gate. Keep agent evals fast enough that engineers run them before pushing.
6. Deploy with monitors and budgets. Sample transcripts weekly; log and review overrides. Track evaluation results over time in BI. Adjust architectures based on evaluation results when you spot systematic errors.
7. Test scenarios: tool outage, permission revocation, orchestrator conflicts, reinstatement. Promote new fixtures from real bugs and near‑misses.
8. Repeat per individual agent. Compare models and prompts side‑by‑side; avoid regressions masked by averages. Use results to improve ai agent performance deliberately.

If you want concrete runbooks for text‑to‑SQL trust and Slack review UX, skim Enterprise Text‑to‑SQL: What It Takes to Trust the Answer and our AI agents articles. If you need help wiring the platform, we build the policies, traces, and review loops that evaluate ai across agent systems and ship safely.

---

Need an evaluation harness you can ship this quarter? We’ll stand up fixtures, graders, and monitors for your top workflows—warehouse queries and change authoring first. Start a project with Vertex.