Production AI Agents Face Semantic Drift and Hidden Data Leaks

AI agents in production are exhibiting new failure modes: non-crashing errors, semantic drift, and implicit data leakage. Text-to-SQL agents cause analytical bias due to domain concept confusion, bypassing traditional observability mechanisms. Approval and review processes have not yet adapted to the velocity and risk profile of agent-generated code.

Semantic drift leads to domain concept mismatches, causing downstream analytical bias

Prompts and context windows constitute a new surface for sensitive data leakage

LLM pipelines lack observability solutions tailored to semantic errors

Target audience: AI engineers, MLOps engineers, platform architects

In a recent InfoQ webinar titled “AI in Production: What Fails, What Works, and Who Approves Deployment?”, a panel of senior engineering practitioners delved into the real-world challenges encountered when deploying agentic systems. The session was moderated by Hien Luu, featuring guests such as Zichuan Xiong and Prem Chandrasekaran from ThoughtWorks, privacy expert Katharine Jarmul, and financial AI practitioner Lan Chu.

Luu opened by identifying three core drivers: AI failure modes differ from traditional software, generating plausible but inaccurate results rather than crashing outright; coding agents produce production code faster than humans can review it; and sensitive data now has new leakage vectors, including prompts, context windows, and retrieved documents.

Xiong, Head of Managed Services AI Operations at ThoughtWorks, highlighted the pervasive issue of “semantic drift” in LLM pipelines. He described how a text-to-SQL translation agent frequently confused domain-specific concepts—for example, treating “member” in sales contexts identically to “member” in customer support contexts—resulting in subtle errors in downstream business analytics that evaded standard distributed tracing detection.

Similarly, Chu shared a severe incident caused by “embedding drift” in a financial compliance system. After deploying an agent that used vector similarity thresholds to block non-compliant investment advice, a background update to the underlying embedding model altered the vector distance distribution. This caused false positive and false negative rates in live traffic to spike dramatically, while all infrastructure health checks remained completely green (normal).

Furthermore, Chandrasekaran noted that RAG pipelines might inadvertently retrieve sensitive documents, while Jarmul pointed out that models inherently memorize training data, which could be exploited to de-anonymize information or leak API keys. Chandrasekaran also referenced the GitSpawn vulnerability, illustrating how malicious repository configurations could trigger unauthorized code execution during routine commands.

As the volume of production code generated by coding assistants grows exponentially, traditional manual code review processes face unsustainable bottlenecks. Chandrasekaran, Head of Technology and Business Services at ThoughtWorks, emphasized that line-by-line human code reviews cannot keep pace with high-velocity agent output. Engineering teams must automate tactical style and syntax checks within CI/CD pipelines, reserving human review for high-level architecture, business intent, and non-functional requirements.

Xiong added that teams must “shift left,” evaluating intent at the harness level before pull requests are even generated. Chandrasekaran also highlighted the resurgence of mutation testing as a critical practice. By introducing deliberate code mutations, teams can verify whether automated test suites truly validate underlying logic, rather than merely satisfying superficial code coverage metrics.

When evaluating agentic systems, Chu stressed that evaluation frameworks should function as long-term, continuously operating infrastructure, not static checklists performed just before release. Effective evaluation requires continuous online tracking at the span level, log filtering, and using LLMs to classify failure modes, enabling teams to feed edge cases from production back into offline benchmark suites.

To address privacy and security risks, Jarmul warned against relying on prompt engineering for system defense. She advocated for systematic red teaming, continuous threat modeling, defining strict functional scopes for autonomous agents, and enhancing observability to handle larger volumes of incidents. To limit the blast radius of autonomous agent behavior, Chandrasekaran suggested managing each AI agent like a newly onboarded engineer.

Enterprises should assign distinct non-human identities to agents, implement tiered access controls distinguishing read-only operations from rollback-capable write operations, version-manage third-party MCP dependencies, and strictly isolate API keys within sandboxes separate from the agent runtime environment. Ultimately, while engineering, security, and privacy teams are responsible for validating safeguards, deployment approval remains a strategic business decision.