Hacker News AI · 10/23/2026, 06:18:40
CATT 0.2.1 Released: Test Agents Against Simulated Assistants in CI
Open-source tool CATT v0.2.1 enables developers to test MCP servers and APIs against simulated mainstream AI assistants (e.g., Muse/Instinct) within CI pipelines. Using a surrogate emulation mechanism with confidence-labeled behaviors, it addresses the challenge of black-box assistant testing while clarifying that passing tests do not certify official compatibility.
SOURCE COVERAGEFull Original Report
Consumer Agent Testing Tool (CATT)
Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds.
Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated.
Contributing | Security | Changelog | MIT license
A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis:
documented,unofficial,assumedorunknown. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.
What you can test today
| Lane | Command | What it checks | Status and scope |
|---|---|---|---|
| MCP connectors | connector-run SCENARIO | Use the server's tools, apply account and approval rules, protect private values, and check the service's recorded outcome and the agent's answer. | Muse and Instinct surrogates: --emulation-target muse|instinct. See connector lane. |
| Connector extensions | connector-run SCENARIO with extensions: | Routes, memory, custom API clients, browser actions, wallet cards, background jobs, untrusted content, channel limits and simulated users. | Supported with the lane. user_simulator is exploratory and cannot run in scripted tests or decision contracts. See extensions. |
| Scenario scaffolding | scaffold URL --output FILE, scaffold --from-run RUN --output FILE | Read a tool catalog or turn an agent run into scripted calls. Calls no service tools. Replace every TODO(scaffold) before running. | Supported. One Streamable HTTP connector; no OpenAPI input. |
| Muse connector readiness | check URL --suite muse-guidelines | Review the catalog and optional manifest against the pinned connector guidelines without calling tools. | Documented constraints, not admission. See readiness checks. |
| Shared surrogate readiness | check URL --suite surrogate --emulation-target instinct | Check the catalog, allowlist, schema syntax and selected projection without calling tools. | CATT's contract for either target; see shared contract. |
| Personal Agent Protocol | check ORIGIN --suite poppy | Read poppy.json. With a manifest, probe sessions, sign-in and one declared operation on the company's hosts. | Draft 0.1; no production Muse evidence. |
| Muse gadget registration | check FILE --suite gadget | Check a link.register message and command specs against the pinned SDK limits. Opens no connection. | Documented limits, not admission. The SDK's terms say it is not a developer platform. |
| MCP protocol contracts | check TARGET | Check schemas, version negotiation, sessions and operation contracts; run the pinned conformance suite. | Supported. Generic MCP evidence; see readiness checks. |
| Embedded MCP apps | surface-run SCENARIO | Render the app in Chromium and check forms, tool calls, resources and service responses. An agent driver chooses from the scenario's scripted steps. | Supported. Generic MCP Apps host, not Muse. See host scope. |
| Websites | browse POLICY, run --surface browser | Act on accessibility snapshots under request rules. Takeover pauses the run (exit 3); the CLI has no handback. | Native Chromium, without OS isolation. Browser scenarios use only the packaged reservation portal. |
| HTTP APIs | run --surface api-tools, inspect, check | Bind approval to each proposed call and check backend outcomes. | Supported with a mapping. Your API also needs a loopback deployment and private control adapter. See API scenarios and API trials. |
| Evidence | report, export, replay, diff | Inspect reports, redact exports, verify playback and compare runs. | Supported. Playback and diff require sealed runs. |
| Development extensions | --development-extensions COMMAND | CLI binaries, isolated browsers, cloud workers, local Glimmer, generated OpenAPI and Prism mocks. | Outside the supported release. See development extensions. |
The connector lane records the basis for each Muse-specific behavior. Tool classes
and approval prompts follow documented guidelines. The muse-cli projection and
template client follow unofficial mirrors; the native projection is an assumed
generic baseline. The neutral cli projection supports both targets as a CATT
treatment. Instinct runs have a separate source record and label the
shared host algorithms as assumptions. Native/CLI MCP is CATT's test adapter;
no public Instinct MCP integration contract was found.
Run the local booking example under Instinct without a model or account:
uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml \
--driver scripted --script examples/connector/fixture-booking.calls.json \
--start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"Set emulation_target: instinct in your scenario or pass
--emulation-target instinct. Both targets use native or cli projection and the generic
MCP client. Instinct defaults to plain-text replies and does not run Muse guideline
checks. Its local wallet fixture covers payment behavior; live payment rails,
trusted-network protocol, Vault and Files remain outside the
shared contract.
Three quickstarts follow. The connector lane and the popular MCP server quickstarts run on loopback with no account; the Coverage Cat sandbox quickstart calls a hosted sandbox. Test your own service turns them into a workflow, and
（注：更多技术实现细节与完整 API 文档请查阅原项目 README）
作者发帖说明（HN）：
Hey HN, we recently open sourced our Consumer Agent Testing Tool to help teams iterate against proprietary assistants like Muse and Instinct. We built this to reduce our own frustrations with building an MCP/API for these agents to consume.
The problem: testing against these tools is hard! Agent memory + account statefulness make reproducibility, in the ways required for repeatable/reliable outcomes, between agents and the APIs/MCPs they interact with very challenging.
The solution: CATT lets devs test their services against a surrogate Muse or Instinct agent (not affiliated with either of these companies). Testers define starting conditions, approvals, and success conditions for their service's state. CATT then records the full interaction (incl. tool calls, responses) so it can diff what happened from what the agent reports. Runs can be tested in local builds or via CI, with or without LLM backing.
CATT is designed to emulate the important parts of personal assistant interactions. Its got:
- Tool calls and command-line tool access - Simulated approvals for service actions - Outcomes tracking and comparison - Optional MCP profile based on an unofficial Muse template
There are limitations to how much we know about different assistant specific behaviors so the tool labels its assumptions as documented, unofficial, assumed or unknown for developer review. We'll do our best to keep the tool updated as new insights about the internal workings of these assistants come...