Contents7 sections
“It runs” still lacks a measuring stick
Chapter 4's run_agent() can already follow a search-refine-finish path, but “the answer is good” cannot become a test. Two reviewers may disagree about one brief, and a developer cannot tell whether a change improved quality or merely changed wording.
Pause feature work and convert the vague requirement into checkable input and output constraints. The specification does not prescribe model reasoning. It defines valid input, available tools, resource budgets, output shape, and acceptance.
Split one requirement into checkable fields
The research-brief task uses the following contract.
Input
├── topic: non-empty string
├── audience: engineer | manager
└── max_sources: 1..8
Permissions
├── allowed: search_local_sources, get_document
└── forbidden: file writes, network access, unregistered paths
Budget
├── max_turns: 6
└── max_tool_calls: 8
Output
├── status: succeeded | insufficient_evidence | failed
├── summary: 200..800 words
└── source_ids: at least 1 and all must existinsufficient_evidence is a valid result. When the corpus has no reliable evidence, the system reports the gap instead of fabricating sources.
Store the task specification in data classes
from dataclasses import dataclass
from typing import Literal
@dataclass(frozen=True)
class Budget:
max_turns: int = 6
max_tool_calls: int = 8
@dataclass(frozen=True)
class BriefTask:
topic: str
audience: Literal["engineer", "manager"]
max_sources: int
allowed_tools: tuple[str, ...]
budget: Budget
def validate_task(task: BriefTask) -> list[str]:
errors = []
if not task.topic.strip():
errors.append("topic_required")
if not 1 <= task.max_sources <= 8:
errors.append("max_sources_out_of_range")
if set(task.allowed_tools) - {"search_local_sources", "get_document"}:
errors.append("tool_not_allowed")
return errorsInvalid tasks return before a model call, avoiding inference cost for deterministic errors. The runtime and tests share the same specification object so prompts, code, and evaluators do not maintain separate limits.
Use a fake model to run control-flow tests offline
A fake model does not imitate language ability. It returns scripted output to verify that the runner dispatches tools, appends results, and stops correctly.
from dataclasses import dataclass
from typing import Any
@dataclass(frozen=True)
class FakeOutput:
output: list[Any]
output_text: str = ""
class FakeResponses:
def __init__(self, script: list[FakeOutput]):
self._script = iter(script)
self.calls: list[dict] = []
def create(self, **kwargs) -> FakeOutput:
self.calls.append(kwargs)
return next(self._script)
class FakeClient:
def __init__(self, script: list[FakeOutput]):
self.responses = FakeResponses(script)The test script controls three turns precisely without network access, API balance, or model sampling. The live client and fake client both provide responses.create, so the runner does not need to know which one it uses.
Test behavior without freezing prose
Tests focus on terminal status, tool order, and sources rather than complete natural-language text.
def test_research_path_uses_search_then_read(fake_client, tool_schemas, tools):
result = run_agent(
client=fake_client,
model="fake-model",
question="How do agent loops work?",
tools_schema=tool_schemas,
tools=tools,
max_turns=6,
)
assert result["status"] == "succeeded"
assert [event.get("tool") for event in result["events"] if "tool" in event] == [
"search_local_sources",
"get_document",
]Add four more classes: an empty topic fails before any model call; an unknown tool returns a stable error; the model can repair invalid tool arguments once; and repeated tool requests eventually exhaust the budget.
Move from unit tests to business acceptance
┌───────────────────────────────────────┐
│ Business acceptance: sources exist, │
│ length passes, claims have evidence │
├───────────────────────────────────────┤
│ Behavioral tests: tool order, state, │
│ budget exits │
├───────────────────────────────────────┤
│ Unit tests: retrieval, validation, │
│ error conversion │
└───────────────────────────────────────┘The lowest layer has the most tests and runs fastest. Business acceptance has fewer cases and checks final artifacts. Keep only a small number of live-model smoke tests so routine code changes do not depend on stochastic output.
Completed code inventory
After five chapters, the repository contains one live model call, one read-only tool, one stoppable loop, input and output constraints, a fake client, and offline tests.
Next, refactor the tool layer: add get_document(source_id), write two failure tests, and register it with search_local_sources in the tool registry. Chapter 6 keeps current behavior unchanged while centralizing argument validation, permissions, timeouts, and errors. Chapter 7 adds RAG only after that boundary is stable.
REFERENCES
References
Series
AI AGENT Beginner's Guide