AI Agent Beginner Tutorial: 12 - Evaluation, Production Deployment, and the Capstone

Design 20 stratified tasks

Tasks cover normal research, insufficient evidence, tool faults, process recovery, approval rejection, and malicious input. Every task fixes input, source fixtures, budgets, deterministic assertions, and an optional subjective rubric.

Run each task three times for 60 trials to expose stochastic failures. Report per-task passes, success rate, P50 and P95, tool calls, tokens, and cost. Security cases have an independent zero-violation gate that averages cannot offset.

Read actual artifacts in the outcome checker

from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class EvalResult:
    passed: bool
    checks: dict[str, bool]
    failure_layer: str | None

def grade_brief(path: Path, known_source_ids: set[str]) -> EvalResult:
    exists = path.is_file()
    text = path.read_text(encoding="utf-8") if exists else ""
    cited_ids = {
        token[1:-1]
        for token in text.split()
        if token.startswith("[") and token.endswith("]")
    }
    checks = {
        "artifact_exists": exists,
        "has_content": len(text.strip()) >= 200,
        "has_citations": bool(cited_ids),
        "citations_exist": bool(cited_ids) and cited_ids <= known_source_ids,
    }
    passed = all(checks.values())
    return EvalResult(passed, checks, None if passed else "outcome")

Agent self-report does not enter deterministic acceptance. Files, source indexes, approval logs, and network records define the environment outcome.

Use traces to locate six failure layers

A trace links model calls, tool arguments, tool results, state transitions, skill versions, approvals, and final artifacts. Failures map to six layers: spec, model, retrieval, tool, state, and evaluator.

Fix the responsible layer before rerunning. A prompt change cannot repair missing retrieval and only makes attribution harder.

Separate submission from execution in production

Client ─▶ Run API ─▶ Queue ─▶ Worker ─▶ Model / Tools
           │                    │
           ▼                    ├──▶ Checkpoint Store
       Status API               ├──▶ Approval Queue
                                └──▶ Trace + Metrics

The API creates a run through a business idempotency key, and the queue carries only the run ID. A worker acquires an expiring lease and continues from a safe checkpoint. Duplicate delivery reads the existing run instead of creating another write.

Limit production paths with a state machine

TERMINAL = {"succeeded", "failed", "cancelled", "budget_exhausted"}
TRANSITIONS = {
    "queued": {"running", "cancelled"},
    "running": {"waiting_approval", "succeeded", "failed", "budget_exhausted"},
    "waiting_approval": {"queued", "cancelled"},
}

def transition(current: str, target: str) -> str:
    if current in TERMINAL or target not in TRANSITIONS.get(current, set()):
        raise ValueError("invalid_state_transition")
    return target

A worker stops committing state and side effects after losing its lease. A new worker resumes from committed state without resetting budgets or idempotency keys.

Apply the same gates to cost and release

Every run pins code, prompt, skill, model, tool-schema, and data versions. Releases pass offline evaluation before a small canary observes success, P95 latency, cost, security blocks, and approval volume.

Cost optimization begins with measurement: cache deterministic retrieval, compact stable prefixes, route classification to a smaller model, and limit ineffective retries. Quality and security gates remain unchanged.

Deliver the complete capstone

brief-agent/
├── CLI + HTTP API
├── Agent Runtime + Tool Registry
├── RAG + Context + Citations
├── Skills + Sessions + Checkpoints
├── Approval + Security Tests
├── MCP + Optional Multi-Agent
├── Evals + Trace Report
└── Worker + Deployment Config

The example project requires all deterministic tests to pass, at least 90% success across three runs of 20 functional tasks, zero authorization violations, and reported per-run cost plus P95 latency. A business sets its own production threshold from data and side-effect risk.

The live acceptance exercise adds one read-only tool. The developer writes its ToolSpec and permissions, adds success and failure tests, runs the full regression set, and explains whether the change needs a new skill, MCP server, or specialist agent. Completing that path demonstrates the ability to extend and deliver an agent system independently.