AI Agent Development and Interview Guide: 07-Agent Evaluations and Observability Tracing

1. Testing、Evaluation、Monitoring

• Testing：technical system component，technical system component JSON technical system component、technical system component、technical system component。
• Evaluation：technical system component，technical system component、technical system component、technical system component。
• Monitoring：technical system component，technical system component、technical system component、technical system component、technical system component。

technical system component。Eval technical system component 500；technical system component。

LangSmith technical system componentoffline evaluationtechnical system component、technical system component，technical system component，technical system component Evaluation concepts。

🔥 P0 High-Frequency Must-Know: technical system component？
LLM technical system component；Agent technical system component。technical system component、technical system component，technical system component、technical system component、technical system component。

2. technical system component

technical system component：

1. technical system component：Schema、technical system component、technical system component、state machine；
2. technical system component Eval：technical system component、technical system component、Rerank、technical system component；
3. technical system component Eval：technical system component、technical system component、technical system component、technical system component；
4. end-to-end Eval：technical system component、technical system component；
5. technical system component：technical system component、technical system component、technical system component、technical system component。

technical system component，technical system component。technical system component LLM-as-judge。

3. technical system component“technical system component”technical system component

technical system component Agent technical system component：

technical system component：technical system component，technical system component，technical system component，technical system component
technical system component：technical system component，technical system component，technical system component
technical system component：P95 < 8s，technical system component，technical system component
technical system component：technical system component < technical system component
technical system component：technical system component

technical system component。technical system component“technical system component”technical system component：technical system component、technical system component、technical system component；technical system component；technical system component。

4. golden datasettechnical system component

technical system component，technical system component 50–100 technical system component。technical system component：

1. technical system component，technical system component；
2. technical system component；
3. technical system component Trace；
4. technical system component，technical system component。

technical system component：

{
  "id": "refund-017",
  "input": {
    "user_message": "technical system component ORD-20260017 technical system component，technical system component",
    "user_id": "u-100"
  },
  "expected": {
    "route": "refund_request",
    "required_tools": ["get_order", "create_refund_draft"],
    "forbidden_tools": ["issue_refund_directly"],
    "requires_approval": true,
    "answer_must_include": ["technical system component", "technical system component"]
  },
  "metadata": {
    "split": "edge_case",
    "risk": "high",
    "language": "zh-CN"
  }
}

technical system component：

• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component；
• Prompt Injection；
• technical system component；
• technical system component；
• technical system component；
• technical system component。

🔥 P0 High-Frequency Must-Know: technical system component？
technical system component，technical system component；technical system component。technical system component、technical system component、technical system component，technical system component。

5. technical system component

5.1 technical system component/technical system component

• Accuracy；
• technical system component Precision/Recall/F1；
• technical system component；
• unclear technical system component。

technical system component Accuracy technical system component，technical system component“technical system component”technical system component。

5.2 technical system component

• Recall@k；
• Precision@k；
• MRR；
• NDCG；
• technical system component 0。

5.3 technical system component

• Tool Selection Accuracy；
• Argument Exact/Partial Match；
• Schema Pass Rate；
• Tool Success Rate；
• Duplicate Side-effect Rate；
• Unauthorized Attempt Rate。

5.4 technical system component

• Correctness；
• Faithfulness / Groundedness；
• Citation Correctness；
• Completeness；
• Refusal Correctness；
• Style/Format Pass Rate。

6. technical system component

Agent technical system component，technical system component。technical system component：

• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component；
• technical system component。

technical system component：

from dataclasses import dataclass


@dataclass
class TrajectoryScore:
    required_tools_ok: bool
    forbidden_tools_ok: bool
    max_steps_ok: bool
    approval_ok: bool

    @property
    def passed(self) -> bool:
        return all([
            self.required_tools_ok,
            self.forbidden_tools_ok,
            self.max_steps_ok,
            self.approval_ok,
        ])


def evaluate_trajectory(trace: list[dict], expected: dict) -> TrajectoryScore:
    tool_names = [event["tool_name"] for event in trace if event["type"] == "tool_call"]
    approvals = [event for event in trace if event["type"] == "approval"]

    return TrajectoryScore(
        required_tools_ok=set(expected["required_tools"]).issubset(tool_names),
        forbidden_tools_ok=not set(expected["forbidden_tools"]).intersection(tool_names),
        max_steps_ok=len(trace) <= expected.get("max_steps", 20),
        approval_ok=(not expected.get("requires_approval") or bool(approvals)),
    )

7. technical system component

technical system component：

1. technical system component/code；
2. technical system component；
3. technical system component；
4. LLM-as-judge；
5. technical system component。

technical system component，technical system component LLM technical system component。technical system component JSON technical system component、technical system component、technical system component ID technical system component、technical system component。

LLM-as-judge technical system component

technical system component、technical system component、technical system component、technical system component。technical system component Rubric：

technical system component 0：technical system component
technical system component 1：technical system component，technical system component
technical system component 2：technical system component、technical system component，technical system component

technical system component“technical system component？technical system component 1–10 technical system component”。technical system component：

• technical system component，technical system component；
• technical system component；
• technical system component；
• technical system component；
• Judge technical system component，technical system component。

🔥 P0 High-Frequency Must-Know: LLM-as-judge technical system component？
technical system component、technical system component、technical system component，technical system component。technical system component Rubric、technical system component、technical system component、technical system component，technical system component。

8. offline evaluationtechnical system component

technical system component
 -> technical system component
 -> technical system component
 -> technical system component
 -> technical system component
 -> technical system component
 -> technical system component

technical system component。technical system component：technical system component、technical system component、technical system component、technical system component、technical system component、technical system component。

LangSmith technical system component，technical system component，technical system component。technical system component LangSmith Evaluation。

9. technical system component

from dataclasses import dataclass


@dataclass
class Metrics:
    task_success: float
    safety_violation: float
    p95_latency_ms: int
    avg_cost: float


def regression_gate(baseline: Metrics, candidate: Metrics) -> list[str]:
    failures = []
    if candidate.safety_violation > 0:
        failures.append("safety violation must remain zero")
    if candidate.task_success < baseline.task_success - 0.02:
        failures.append("task success regressed by more than 2pp")
    if candidate.p95_latency_ms > baseline.p95_latency_ms * 1.20:
        failures.append("p95 latency increased by more than 20%")
    if candidate.avg_cost > baseline.avg_cost * 1.15:
        failures.append("average cost increased by more than 15%")
    return failures

technical system component。technical system component，technical system component。

10. Trace technical system component

technical system component Run technical system component Span：

run
├─ route
├─ retrieval
│  ├─ query_rewrite
│  ├─ dense_search
│  ├─ sparse_search
│  └─ rerank
├─ model_decision
├─ tool_call:get_order
├─ approval
└─ final_generation

technical system component Span technical system component：

• trace_id、run_id、parent_span_id；
• technical system component/technical system component；
• models、Prompt、Tool、technical system component；
• technical system component；
• Token、technical system component、technical system component；
• technical system component、technical system component；
• technical system component/technical system component；
• technical system component。

technical system component Trace technical system component API Key、Access Token、technical system component。

🔥 P0 High-Frequency Must-Know: Log、Metric、Trace technical system component？
Log technical system component；Metric technical system component；Trace technical system component。Agent technical system component Trace technical system component、technical system component，Metric technical system component，Log technical system component。

11. technical system component

technical system component，technical system component：

• technical system component；
• technical system component；
• technical system component；
• technical system component、technical system component；
• technical system component；
• technical system component LLM Judge；
• technical system component；
• technical system component，technical system component。

technical system component：

technical system component Trace -> technical system component -> technical system component -> technical system component -> technical system component -> technical system component -> technical system component -> technical system component

12. technical system component

• technical system component，technical system component；
• technical system component，technical system component；
• A/B technical system component；
• technical system component“technical system component”technical system component“technical system component”technical system component；
• technical system component P50、P95、P99；
• technical system component Rubric technical system component；
• technical system component，technical system component。

13. Dashboard technical system component

technical system component Dashboard：

• technical system component、technical system component、technical system component；
• P50/P95/P99 technical system component；
• technical system component；
• technical system component、technical system component；
• technical system component/technical system component Token technical system component；
• technical system component；
• technical system component、technical system component、technical system component；
• technical system component；
• technical system component Prompt/models/technical system component。

14. technical system component

technical system component A：100 technical system component Eval technical system component

technical system component 100 technical system component，technical system component、technical system component、technical system component、technical system component、technical system component Split。

technical system component B：technical system component

technical system component、technical system component、technical system component、technical system component；technical system component，technical system component。

technical system component C：technical system component

technical system component Prompt。technical system component 0；technical system component；technical system component HTML/Markdown technical system component。

15. technical system component

🔥 P0：technical system component Agent？

technical system component：technical system component、technical system component、technical system component、technical system component；technical system component；technical system component；technical system component Trace technical system component；technical system component。technical system component、technical system component、technical system component、technical system component。

🔥 P0：technical system component Eval technical system component Eval technical system component？

technical system component；technical system component Run/Thread technical system component、technical system component，technical system component。technical system component。

🔥 P0：technical system component？

technical system component、technical system component、technical system component、technical system component Schema、technical system component、technical system component；technical system component Trace technical system component、technical system component、technical system component、technical system component；technical system component。

⭐ P1：technical system component？

technical system component。technical system component；technical system component；technical system component；technical system component；technical system component；technical system component。

⭐ P1：technical system component？

technical system component，technical system component、technical system component、technical system component、technical system component、technical system component、technical system component Thread technical system component。technical system component Thread technical system component。

16. technical system component

• technical system componentgolden datasettechnical system component Rubric；
• technical system component、technical system component、technical system component、technical system component；
• technical system component LLM Judge，technical system component；
• technical system component、technical system component；
• Trace technical system component；
• technical system component。