YC Session: Harness Architecture Defines Agent Performance Ceiling

"Harnesses are just wrappers, scaffolding, or prompt engineering." This sentiment pops up on Reddit every few days, with some even claiming that "context engineering isn't a research problem at all." However, the reality is starkly different: swapping out the harness for the same model weights can boost ARC-AGI scores from 30% to 95%, or even 100%.

During the "Harness Night" session of YC Paper Club, host and VC Visiting Partner Francois Chaubard joined forces with several paper authors—Seth Karten (Prime Agent), Jon Saad-Falcon (OpenJarvis), and Josh France and Regan Bell (QM team)—for three talks centered on the question: "Why does the Harness matter more than the model itself?" Viewed together, these talks form a coherent narrative: Seth discusses what expressive power a harness should possess; Jon explores the hardware boundaries within which this system can operate; and Josh and Regan examine how permissions, trust, and collaboration become the primary bottlenecks when deploying such systems in real-world organizations. This article is based on the video recording, edited by InfoQ.

Key Takeaways:

Seth Karten: Return to first principles to understand what a Harness actually is and how it should be built.

• Context exists in three layers: L1 is model weights, L2 is active context plus REPL, and L3 is scheduling state like the file system. The harness's job is to perform compression, CRUD operations, and refinement between these layers—essentially acting as a cache combined with garbage collection.
• An analogy: A bare LLM is like a Turing machine, while a harness is like a von Neumann computer. Because it can read from and write to external memory, it possesses significantly greater expressive power.
• The metric for evaluation is expressive power. Fixed-loop models like ReAct are already natively understood by LLMs, so there is no need to explicitly impose them. However, capabilities such as compression, Python REPL, programmatic sub-agents, state access, and feedback mechanisms must be provided. Removing any one of these removes a capability that cannot be replicated through other means.
• Specific design choices: Treat sub-agents as persistent sub-sessions (which can be woken up or unloaded), enable direct messaging between any two agents, and borrow from continual harness designs for real-time CRUD operations. Evaluation should focus on long-term performance and identifying where plateaus occur.

Jon Saad-Falcon: Personal AI shouldn't be trapped in the cloud.

• The current status quo carries four costs: Annual API fees can reach thousands or even tens of thousands of dollars; the most private data must be sent to the cloud; we are "renting intelligence" rather than owning it; and energy consumption is orders of magnitude higher.
• Local LLMs lag behind frontier models by only 6 to 12 months. For instance, Qwen 3.8 27B is roughly equivalent to Claude 4.6 Opus from August 2025, and hardware accelerators are catching up rapidly.
• The proposed solution defines the entire stack using five primitives: Interface, Agent Logic, Model, Inference Engine + Hardware, and Tool Memory & Learning. These are written into a specification that allows the entire stack to be optimized.
• Minimize changes to the LLM itself. Instead, use cloud-based LLMs to diagnose and gate local configurations, ensuring that deployment inference incurs no additional cloud costs.

Josh France and Regan Bell: Reverse-engineer what an Agent Harness should look like from lessons learned across three iterations.

• What is QM? It is a Claw-like assistant for every YC employee, accessible via Slack or Web. It features personal context, sandboxed files, and scheduled tasks, supporting multi-user collaboration. Its use cases include email classification, legal and financial workflows, document editing, internal data retrieval, and launching internal web applications.
• Design Iteration 1: Extract the agent's "brain" from the sandbox and centralize everything in Postgres. The sandbox shifts from being a "home" to an on-demand resource. Crucially, the decision to use resources is delegated to the agent, not the harness.
• Design Iteration 2: Keep the harness minimal. The core consists of only three tools: remote sandbox execution, object storage read/write, and publishing internal apps. Features like memory and scheduled tasks are treated as temporary patches. Additionally, allow agents to choose their own runtime and sandbox provider.
• Using an LLM as a judge leads to "protagonist syndrome," where it only sees part of the elephant. Therefore, Human-in-the-Loop remains critical. Agents have poor understanding of both their own context and social contexts, making privileged information prone to leakage. Consequently, what can actually be fed into the agent's brain is limited by the robustness of the permission system.
• The solution is to "sharpen the tools": Set budgets for goals. Do not allow abandonment until actual time or token consumption limits are reached, resulting in better research outputs.

Why the Harness Matters More Than the Model Itself

Francois Chaubard: Why focus on the Harness? Isn't it just a wrapper, scaffolding, or prompt engineering? A Reddit post from a month ago stated: "I'm unsure if this kind of prompt engineering deserves a top-tier ML conference." Another classic take was: "Context engineering is not a research problem." For a long time, harnesses have been dismissed as unworthy of serious research. Yet, they delivered an 18% improvement—the gap between Harness 1 and Harness 2. As Seth will discuss, this difference determines whether ARC-AGI can be solved, proving it is clearly worthy of study.

Look at this classic chart. The x-axis represents release time, and the y-axis represents the duration an agent can sustain operation. You'll notice that much of the progress is attributable to the harness. I call this the "Era of Static Harnesses," because the harness itself did not self-improve. Over the last six months, however, all progress has concentrated on self-improving harnesses.

This chart comes from a presentation by Trajectory's CEO. He discussed perplexity, which we have always measured and which correlates somewhat with intelligence. We keep pushing intelligence higher but fail to effectively leverage test-time experience. We generate massive amounts of test-time experience in new domains but do not adapt quickly. We have long known that when increasing online samples, how do we truly learn from a batch size of 1? There is no structure for this. In-context learning (ICL) saturates after 40 to 50 samples, showing no further improvement on validation sets. Then we resort to small-rank LoRA, large-rank LoRA, or SFT. The existence of so many different training pipelines is strange, and this is precisely where the harness shines.

The core insight revealed by ARC-AGI is how quickly a model can adapt to new problems and new distributions. Significant thought has been invested in these games to ensure that the skills required for Game 1 are orthogonal to those for Game 2, thereby isolating fluid intelligence. Claude Opus was one of the earliest models validated on the private holdout set, accessible only to Greg and Chalet, achieving a best score of 30%. Yet, solely through Harness—often dismissed as mere packaging and scaffolding not worth researching—we reached 95%. Nvidia’s AVO hit 100%, with Prime Agent and Nvidia’s results being very recent releases.

There is another point. In March, Karpathy released auto-researcher. After forking it, I initially intended to build just a small interface to monitor progress, but inadvertently constructed a Harness. Its workflow is as follows: you specify a goal. My example was "Diffusion LMs cannot beat Autoregressive (AR) LMs," but if we use ensembling—sharding the Diffusion LM into multiple instances—it might perform better overall because its arithmetic intensity per GPU is much higher than that of AR models.

I passed this goal to my agent swarm along with some seed ideas: varying ensemble sizes, sharding 100 times, 10 times, or 5 times. I also specified validation metrics, such as the GSMAK plus GPT2 setup. Then, scoping agents searched papers and GitHub repositories, handing off to a PI agent named Chris Ray, who then delegated to a research agent named John South Khan. He convened a council; Yaso and I provided feedback. Chris Ray nudged every hour. Finally, an author agent froze the ideas, wrote ablation studies, and drafted the paper.

It is now genuinely publishing papers. I have read them, and they are well-written. The output was poor in March and April but improved significantly by May. Currently, I hand eight ideas to eight H100 nodes (each with eight GPUs) and let them run, checking in occasionally to collect the papers. The real change lies in the Harness layered on top. I didn’t spend much time on it, but this is essentially how I conduct research.

I spent a weekend reading a batch of classic literature to provide a five-minute historical review. This will not be chronological, and many papers will be skipped.

The original Harness was simply GPT-2, back in February 2019: a while loop, end-of-sequence detection, top-P sampling, and an environment. That’s it. No tool calling, no skills. This was version V0. Taking GSMAK as an example: the system prompt sets the role as a math teacher, the context is "Susie has five dollars, she spends three, how much does she have now?" There is no chain of thought; we just append four hash marks (####) followed by two end-of-state tokens and an end-of-sequence token, measuring accuracy after the four hash marks.

For the next six years, we kept adding features to this static Harness. Placing a bunch of examples back into the context became few-shot learners in July 2020. Then came Chain of Thought: instead of directly predicting ## 2, spread the computation across more tokens. These were innovations in context, output space, and action space. Next were WebGPT and Toolformer, which introduced tools that could be called. For subtraction, rather than computing $5 - 2$ within the weights, you simply call Python’s sub function. This is cool; you expose all tools in the system prompt, and that’s how tools originated.

Then came MemGPT. One of the coolest tools is the ability to read and write its own context. Previously, you could only append indefinitely. Now, granting CRUD permissions over the context creates a small block called "memory." Voyager took this further: tools can chain together to complete tasks, but once learned, how do you distill them back into the system prompt so the agent remembers them forever? This is what defines "skills," demonstrated in Minecraft, and remains the definition today. InterCode represents an innovation in action space: directly outputting code equates to instant generation of tools or skills. ReAct emerged first, followed by self-refine and reflection. The idea involves multiple agents with different roles helping each other improve at the context level. For instance, if I swap 3 and 5, I send it to an internal evaluator asking, "Is this correct?" The response is, "No, it looks wrong." Reflection then says, "You swapped these two; go back and fix it." This was the earliest concept of multi-agent systems.

Later, a single tool could generate sub-agents residing in persistent loops, which I could invoke or add to continuously. RLM went even crazier, recursively calling RLM queries to solve broader categories of problems. This constitutes Harness v1. It remained entirely static; I did not improve the system prompts nor update the Harness itself.

We must place GStack at the top; he enabled us to do all this. Thank you, Gary. To summarize V1: there are agent specifications and system prompts defining allowed rounds and tool calls, listing tools, skills, and sub-agents, all placed within the loop. It can be triggered by prompts in Slack—which QM, discussed today, uses daily—or run on cron jobs, waking up hourly to do work. It includes session management, loops, and context compilation, packing everything into the LM call, retrieving actions, calling tools, and appending results back to the context.

The most exciting development is enabling the Harness to learn from itself, either learning the system prompt or the Harness itself. DSPy exemplifies this: Demonstrate, Search, Predict. It takes a small subset of examples to learn optimal system prompts. Since I cannot backpropagate, I use genetic programming: find candidates, merge them, evaluate them. This effectively gives me CRUD operations on the system prompt itself. Darwin Machines goes further, modifying even the Harness code: an archive stores various combinations of Harnesses and system prompts. We sample, run fitness functions, and add results back to the archive. There is even a meta-Harness allowing agents to modify their own Harness, creating a snowball effect. The meta-Harness generates Harnesses on top of the main Harness—very meta.

Tonight, the lead author of the Continual Harness paper is present; make sure to talk to him. They added extra dimensions and history logs to memory categories. I think the coolest part for traditional RL researchers is Dagger-style online learning, which can update weight files themselves based on test-time training using a small number of recently learned examples. This is an important direction we need to implement.

Prime Agent: Designing the Harness as an Agent Operating System

Seth Karten: Tonight, I want to convince you to think about how to build a Harness from first principles.

The raw LLM is essentially a sequential processor with fixed weights and visible context: tokens in, tokens out. But we no longer view it that way. We now have a set of files it can access, infinite programs and tools at its disposal, and the ability to message other LLM sessions or create sub-agents. Fundamentally, it’s still tokens in, tokens out. The Harness is the layer between the LLM and the world, adding persistent state, tools, and computation.

From a human perspective, opening Prime Agent feels like using Claude Code, Codex, or Cursor. You enter the Agent view, see an overview of all parallel sessions, and click into the root session, which acts as the project orchestrator for all sub-agents. You don’t need to explicitly instruct it to launch sub-agents; it does so autonomously when useful. All interactions are programmatic because everything is based on recursive language models—tools, memory, and sub-agents reside within an IPython shell. Behind each agent is a persistent daemon. Even if you close your laptop or hit Control-C, it continues running in the background and must be explicitly stopped. We also borrowed real-time CRUD (Create, Read, Update, Delete) capabilities from the Continual Harness to manage memory, skills, and sub-agents, allowing us to persistently modify our own system prompts.

I like to view the context we build as hierarchical layers—L1, L2, and L3—similar to a cache. The most readily accessible information resides in the model weights, but stuffing all information into weights means every update requires fine-tuning, which is prohibitively expensive. Therefore, we have active input context, where in-context examples enhance capability, though this eventually fills up. The earliest form of Harness was compression, enabling agents to summarize their own history to exceed the context window limits.

Once we move beyond direct model input/output into L2 and L3, the dynamics change. L3 resembles scheduling state, such as a file system, reading from and writing to main memory. L2 sits between active context and the file system. You have a live REPL, similar to an IPython shell in Jupyter, where variables reside in RAM. Agents can programmatically manipulate these variables and run programs on them, saving significant token usage. You can also create sub-agents to conserve context.

Compression allows context to evolve over time, but beyond a certain threshold, we must consider how to update it. We discuss CRUD operations: how to go beyond simple creation and reading to enable updating and deleting state at any time. I view the REPL as Agentic garbage collection—cleaning up variables and deciding which sub-agents remain available to prevent RAM exhaustion and crashes. Further up is refinement: updating and removing skills, memories, and prompts within the system to prevent disk space saturation.

Another perspective I appreciate: the raw LLM resembles a Turing machine—a single tape where instructions enter, operations occur, and outputs emerge. The Harness resembles a von Neumann architecture, capable of reading and writing to external memory, making it more powerful and able to handle a broader range of problems.

How do we build a good Harness? It must be as expressive as possible. Early Harnesses were rigid, following fixed steps like plan, act, and critique. However, current models natively understand these patterns, so there’s no need to explicitly impose ReAct loops. What they lack is expressive power for compression, the ability to use a Python REPL to run programs, programmatic creation of sub-agents, access to state, and diverse feedback mechanisms. Each of these expressive features represents a capability that cannot be substituted by another method; removing one removes that specific capability.

You may be familiar with the RLM paper co-authored by my colleague Alex. Our extension treats sub-agents as persistent sub-sessions: the parent session creates and launches RLM sub-agents. After completing tasks and reporting their final state, these sub-agents enter an idle state but remain in RAM. The parent session can send messages to resume them at any time, preserving previously accumulated context without losing information. Of course, we don’t want to consume excessive RAM, so we can offload them to an inactive state and wake them via messaging when needed.

The Continual Harness was the subject of my previous paper, discussing the encoding of the entire Harness state. We aim to expose this capability to the agent, allowing it to use its full history to shape future Harness configurations: Should the system prompt be modified? Should new skills be created? I define skills as sets of instructions or programs achieving specific goals; memory serves as long-term storage for important items, alongside persistent specifications for sub-agents. Some sub-agent contexts are valuable and worth reusing later. This capacity for reflection and refinement is incredibly powerful for models. They aren’t yet proficient at it, but when building a Harness, we should design it to be slightly ahead of current model capabilities. This allows us to capture reasoning traces that drive the next iteration of model development, enabling models to handle the Harness and guide themselves toward higher performance.

One of my favorite features in Prime Agent, included from the beginning, is the ability for any two agents to message each other. I’ve been searching for the best way to manage all my agents, which work for me across five directions and billions of different tasks daily. If they could directly share contact information and coordinate, it would be far more efficient. As it turns out, this is extremely useful for software engineering and long-horizon tasks.

When designing the Harness, we consider long-horizon performance. I want to run tasks without constantly monitoring them, checking progress only upon return. This perspective is often missing in evaluations: if one model runs for insufficient time or stops after a budget limit, while another uses more budget to continue working, you’re neither comparing equal expenditures nor revealing true performance differences. I want to observe the actual plateau—identifying the point where increasing test-time tokens yields only marginal improvements.

Next, I’ll discuss how we built Prime Agent, covering test-time scaling and results. We’ll examine whether information management truly aids REPLs in handling long contexts and what actually happens during the refinement process in ultra-long-horizon tasks lasting more than a week.

You may have seen these ARC-AGI results. I added a data point not present in the original results to facilitate cross-Harness comparisons. We wanted to find an evaluation metric that people care about, so we chose ARC-AGI. I recall previously achieving 20% using Continual Harness with Gemini Flash, so I expected at least 20%. The result was that our general-purpose Harness, which wasn't designed specifically for ARC-AGI, performed surprisingly well.

I found a community leaderboard called Prolong online and simply fed their system prompt into Prime Agent. On the first run, it hit 99.9%, but when I checked the logs, I realized I was cheating. So, I properly implemented sandbox isolation, which took another day. Afterward, using GPT-5.6 Sol, I achieved 78%, which I consider a great result.

The prompt is quite generic: use world models to solve ARC-AGI-3, list the actions you can take, and combine this with Prime Agent's general system prompt. You have a REPL and can invoke sub-agents. I reviewed the trajectories; it was making various coding calls to check scenarios, analyzing images, performing image processing, and seemingly using the REPL for reasonable inference. Then we ran GPT-5.6 Terra, achieving 25.7%. This is good relative to the previous week's results and already surpasses some high-end versions of GPT-5.6 Sol. Running Opus yielded 95.5%.

We also compared against other Harnesses. People asked if I had run it with Claude Code. I did, but the results weren't good, so I directly cited Claude Code's original results. Later, others achieved better scores using configurations similar to Prime Agent. Interestingly, many popular Harnesses do not necessarily perform well where Prime Agent excels. For example, Hermes Agent quickly consumed a lot of budget; I spent $5,000 with no significant performance improvement and had to abort. This doesn't mean it's their ceiling, but the cost is indeed high. Cost-performance ratio is crucial, and one way to save money is to handle context programmatically.

For long-horizon evaluations, we ran Oolong and some coding Emulator Benchmarks. It was released quickly as a replacement for Program Bench. The results were mostly comparable or slightly better, such as comparing GLM-5.2 to Opus 5 and GPT-5.6 Sol against Pimono, Claude Code, etc. Emulator Bench is a simulator that reproduces entire computer systems, like creating a Game Boy Color. Because it has a REPL and RLM, it allows for out-of-loop design experiments using these programs before submitting the final solution, offering more freedom to try various approaches. We also tried GPU kernels; results were roughly comparable across different models—one was better, one was worse—and we didn't overfit to any single evaluation.

On the long-horizon front, we also conducted automated research, extending NanoGPT Speedrun by giving it 8 H200 GPUs to run for a week. You might expect Prime Agent to perform much better because it can program. However, task variance is high, making it difficult to attribute results to either the Harness or the model. We observed that newer models like DeepSeek-V4, GLM-5.3, and Kimi K3 engage in what we call "out-of-loop experimentation": running experiments on the CPU, examining parameters, performing hyperparameter searches, and analyzing data. This avoids spending all time on expensive H200 experiments. They aren't running the main experiment but rather optimizing it. This is cool and indicates we are on the right track in shaping expressive capabilities. It excels at automated research, and collaboration with humans works even better; Human-in-the-Loop helps guide experiments more effectively.

Finally, we conducted a 7-day factory run with 633 agents and 23 million output tokens, aiming for continuous progress on the tech tree. Sub-agents divided tasks among research, construction, and resource collection, and could use refinement mechanisms to leverage past experiences without getting stuck in ultra-long contexts. The most interesting aspect was that it never got stuck; it continued progressing until the end of the phase, somewhat resembling the conclusions from Gemini playing Pokémon.

If you remember only one thing: think about Agentic context management, think about swarm intelligence, dive deep into RLM, and run standardized evaluations.

OpenJarvis: Bringing Personal AI Back from the Cloud to Your Computer

Jon Saad-Falcon: Personal AI is ubiquitous, but today it is largely trapped in the cloud. Many Harnesses and projects handle daily writing, research, coding, and scheduling, but projects like OpenClaw and Hermes Agent rely heavily on cloud LLMs for most intelligent work and queries. The costs are high—annual API fees can reach thousands or even tens of thousands of dollars. It’s also not private; your most personal data is sent to the cloud, and you don’t know where it flows. Furthermore, you are renting intelligence rather than having it out-of-the-box, and energy consumption is orders of magnitude higher than running locally on a laptop.

Local LLMs are finally good enough. Today’s LLMs lag behind previous frontier models by only 6 to 12 months. For instance, Qwen 3.8 27B is roughly equivalent to Claude 4.6 Opus from August 2025. With accelerators in laptops and workstations improving, this gap is narrowing. Apple just released a new Mac Mini this week, and both Apple and Nvidia are refocusing on accelerators for personal use cases.

The core question we want to explore is: Can we move the core of the personal AI stack—inference, agent execution, memory, learning, and other components primarily dependent on the cloud—entirely onto the device while remaining competitive with pure cloud stacks? That’s why we built OpenJarvis. We want to know how much functionality can be run for free on-device while ensuring security, privacy, and quality. To this end, we defined a set of simplest primitives.

The first is the user interface. The second is the actual Agent logic, centered around composable reasoning using different intelligences and tools. The intelligence refers to the LLMs running everything—such as Qwen, GPT-OSS, or Gemma 3N. Then there are the inference engines: Ollama, Llama.cpp, vLLM, and SGLang, along with the hardware, such as Apple Silicon or Nvidia. Additionally, we have tools and memory operating via the standard MCP protocol. Finally, there are learning primitives: prompt-based methods like JAPA and DSPy, or weight-based methods like GRPO, SFT, and LoRA. You need Agents that improve over time, becoming truly personalized and effective.

In terms of presentation, we use interfaces people are already accustomed to: desktop applications run as usual but display savings in cost and energy; they also support various persistent Agents running standard protocols day after day as cron jobs. The goal is plug-and-play integration with existing workflows, allowing users to experience on-device LLMs for the first time, much like when ChatGPT or Claude first emerged.

To optimize, we minimize interference with the LLM itself. We simply write a concise specification for these five primitives and let them be optimized through this spec. By completing the full optimization loop, we achieved significant reductions in cost and latency, alongside quality improvements. This configuration covers the five aspects most critical to engineers building LLM Harnesses: intelligence, engine, Agentic logic, tools or learning systems, and overall optimization.

Another interesting aspect is using cloud LLMs to manually and automatically optimize the entire local stack. This leverages the capabilities of cloud models to diagnose issues, propose changes, gate modifications, and generate improvement plans for local LLMs, without incurring any cloud costs during actual deployment and inference. We found that Open Jarvis configurations optimized by Claude or ChatGPT were far more effective than out-of-the-box deployments, because you can tailor specific LLMs and Harnesses to different workloads.

As a result, even across diverse workflows such as personal AI, individual usage, coding, and Agents, today’s edge-side models can compete with cloud models. There are still many tasks that local models cannot handle, but as models are better distilled and accelerators improve, the gap is narrowing remarkably every month. Even with current models, we achieved an 800x reduction in cost while significantly decreasing latency. Furthermore, using any major cloud model for optimization yields benefits. While the Opus series, Opus 5, and GPT-5.6 Sol are naturally superior, larger model families like Gemini, Kimi, and GLM also perform well.

The optimization cost for the entire Harness is also lower than alternative approaches, which often require more data or more model calls. These primitives and specifications are most effective for local models because they prevent the optimization loop and model abstractions from obstructing the cloud model, allowing the cloud model to focus solely on optimizing the system as a whole.

Looking ahead, we believe that in the near future, a large portion—or even the majority—of daily inference calls will shift to local devices and workstations.

QM: From a Fleet of 50 Mac Minis to an Open-Source System Available to All YC Staff

Josh France: We built QM, YC’s open-source AgentHarness. QM provides every YC employee with a Claw-like assistant that is fully customizable and accessible via Slack or a web interface. Each person has their own personal context, sandboxed files, and scheduled tasks, and can collaborate in multi-user online environments, such as Slack channels. Its uses are broad: email classification, automation of legal and financial workflows, document editing, retrieving data from internal databases, and launching real-time internal web apps, such as those helping plan events.

Why did we build this? It was the culmination of a series of internal Agent projects, all benefiting from the tailwind of improving model capabilities. The first was "General Agen" around January 2025—a system prompt with tools running in a loop, one-size-fits-all, where everyone interacted with the same thing. The architecture was simple, yet it performed surprisingly well at answering data questions. Interestingly, as the underlying models improved, its scope of proficiency expanded. Later, we integrated it into Slack, adding scheduled tasks and more tools.

By June 2025, many engineers started using Claude Code and Codex. We found it easy to run them in virtual machines, so we added a Slack tag. This proved powerful for those who just wanted to execute one-off code changes. We also used it to run CI pipelines and spin up development environments for testing. People would describe a bug, and the bot would actually solve it. For individuals who had never changed code in their lives, this was incredibly empowering. We also established a small feedback loop, observing how the bot failed and where it made errors, then updating the Agents.md file in the codebase accordingly.

This past January, many partners began using OpenClaw. YC partners are extremely busy, receiving a high volume of inbound emails between office hours and constantly reading applications, so any tool providing additional leverage is valuable. OpenClaw was particularly useful because it was the first Agent with its own computer for many partners. Its customizability far exceeded previous paradigms, functioning almost like a personal assistant. So the question in April became: Can we provide this to every employee without buying each one a Mac Mini? Ultimately, we deployed a cluster of over 50 Hermes Agents running on virtual machines. They were helpful but required extensive configuration and were inherently difficult to manage—it felt like playing whack-a-mole, requiring me to go in and fix issues one by one. Thus, the challenge evolved: We had already extracted significant value from these systems; now we needed to build something that addressed the drawbacks of large clusters while preserving the personalization and features people genuinely appreciated.

Regan Bell: From Josh’s account, a clear trend emerges: models are improving exponentially, and giving them more capabilities yields increasingly substantial returns. OpenClaw gave Agents their own computers, and the payoff was astonishing. So in May, we started wondering: How far can we push along this line?

We really liked the perspective of "unleashing potential." Around 2024, during the era of Situational Awareness, test-time compute was just becoming a reality. When we first gave Agents tools, we had an intuition: There is more intelligence within the models than we were actually utilizing. If we boldly pushed the boundaries of what Agents could do, magic might happen.

The first thing we did with QM was to extract the system's "brain" from the sandbox. When using Hermes and OpenClaw, each Agent had its own computer—powerful, but trapped within that single machine. Even with just a few dozen instances, management became nearly unmanageable almost immediately, and all sessions were confined locally. So, we offloaded everything to Postgres, centralizing it all—including every conversation with the Agents—and then exposed this aggregated context back to the Agents themselves. Another key shift was treating the sandbox not as the Agent’s home, but as a resource it can access on demand, with far fewer restrictions.

This approach also opens up a possibility: you accumulate a large evaluation set containing traces of all human-Agent interactions. In principle, this enables hill-climbing optimization to form an automated improvement loop. Our results here have been mixed. If you deploy a swarm of Agents to fix every bug they encounter, using LLMs as judges, you run into "protagonist syndrome"—fixes address only part of the elephant, missing the systemic view. Thus, Human-in-the-Loop remains crucial, although we are eager for full automation in the long run.

Another major initiative was connecting Agents to every available resource within the company. YC already had a CLI connected to many systems; for those not included, we allowed arbitrary API keys. We also wanted equivalent capabilities on employee laptops, such as importing keys via device code OAuth and refreshing them, mimicking the human experience on a computer as closely as possible. Databases remained primarily read-only, while writes went through manually reviewed batch updates: the Agent proposes an edit plan, a human quickly reviews it to ensure nothing crazy is happening, and then executes it. We’ve observed a tendency toward mechanical approval, similar to how early users of Claude Code carefully scrutinized tool usage before gradually building trust. This is something to monitor closely over the next few months.

We treat sandboxes as resources for the Agent, not its home. In QM, the default is the sandbox assigned to the conversational user, though there are environments where Agents can gather and collaborate. For heavy development workloads, it can request a more powerful machine; for simple tasks, it selects a lower-performance sandbox. Pushing this decision-making to the Agent rather than the Harness is incredibly powerful.

Similarly, allowing Agents to access their own runtime means they can switch providers if needed. This helps avoid situations common when using models like Fable, where requests for AI research or cybersecurity often result in refusals; the Agent can simply jump to another model. Switching between different sandbox providers is also frequently useful. The general rule is that the Harness should be extremely lean, with three core tools: executing in remote sandboxes, reading/writing from object storage, and publishing internal apps via a simple Git-based system. Tools for memory and scheduled tasks are temporary measures to patch rough edges in the system.

We aim to become a proactive Harness designed for AGI. However, since we haven’t reached that stage yet, we face challenges. One issue is that Agents, despite having a powerful environment and all necessary tools, often give up prematurely. Over the past month, we’ve experimented with "grinding tools": setting budgets for goals so that abandonment isn’t allowed until a certain amount of real time (e.g., several hours) or token consumption is reached. This yields better research outputs and reports. It’s interesting to see OpenAI and Anthropic using very similar techniques to tackle open mathematical problems; the same applies to ordinary office work.

Another common problem is that while the Harness supports multi-user collaboration and Slack integration, due to training artifacts, Agents remain confused about their context even when explicitly instructed in the system prompt. Therefore, localized assistance mechanisms are essential.

Josh France: Another issue is that Agents don’t understand social contexts. Specifically, if I tell Regan a piece of information, he intuitively knows what occasions warrant sharing it. Building this judgment in an Agent requires actual work, as privileged information easily leaks into inappropriate contexts. Consequently, the information you can put into the "brain" is limited by how robust your permission system is. YC is fortunate to have years of accumulated fine-grained permission management, but many others lack this and need to do the groundwork to enable nuanced knowledge sharing.

Original video link: https://www.youtube.com/watch?v=n9xKblqyQ28