OpenAI 自研 Git 后端 Pando 支撑 Codex 并发开发

Over the past year, the size and demands on our monorepo have increased at a staggering pace. It was clear that we needed to revamp our source control system to scale to the level of concurrent work that Codex makes possible.

We built Pando, a new backend for Git repositories, without interrupting any engineering work or changing how our team uses Git. Pando now powers all development in OpenAI’s monorepo. Pando is designed to meet two ambitious goals: first, horizontally scalable writes, optimized for concurrent pushes to many branches. And second, maximize read throughput, so clients can download data fast enough to fully saturate their available network bandwidth.

We met these goals while retaining compatibility with the Git protocol and client, enabling both our engineers and agents to seamlessly adopt Pando into their existing workflows. In production, Pando cut median push latency by nearly 5x. It now delivers nearly a petabyte of Git data per day.

Here, we’ll cover our motivations for building a new source-control system, along with why we chose to retain compatibility with Git. We’ll compare our system with common approaches to scaling Git, describing why Pando is uniquely capable of meeting OpenAI’s scalability needs. We also discuss what’s next, and how we’ll continue contributing to Git to support agent-scale work.

Why we needed a new source control system

Codex has changed the scale and pace of development at OpenAI, enabling engineers to do far more work in parallel. That means more branches and pushes, more merges, and more fetches and clones from development environments, research jobs, and CI.

These demands compound. One engineer now has several changes ready for CI at once, and each push can set several jobs fetching the same commit.

Starting around the end of last year, the systems supporting our monorepo were struggling to keep up. We suffered slow fetches and pushes along with failed merges, with reliability at times falling below 90%. Had we reached a limit of Git itself, or of the way we served our monorepo? Git grew out of a community that exchanged patches to the kernel via email. For more than two decades, developers have adapted their tools around Git as the way they build software has changed. When we started to feel friction in our own monorepo, it mattered to us that we could keep building on that work.

We saw a way to keep that familiar workflow while rebuilding the backend to handle pushes to different branches in parallel and serve the same history to many readers concurrently. Building on Git as the foundation of our source-control workflow meant that improvements to the client could be useful beyond OpenAI. We wanted to contribute those improvements from our copy of Git⁠(opens in a new window) to upstream, sharing them with the broader community.

Engineering for Git at scale

Scaling a source control system has two essential dimensions: many writers pushing to different branches, and many readers fetching the same history. Pando optimizes for both.

First, Pando coordinates updates to each branch independently. Updates to unrelated branches can proceed in parallel, and a reader fetching a branch gets the history needed for that revision. The tradeoff is that a reader looking at several branches can see them at different points of time.

Second, Pando reuses packfiles (Git’s format for compressing multiple objects together) across fetches instead of generating a n individual packfile for each client. A client may download some objects it already has, but the server can serve the same bytes to many clients with little computation.

These choices let us scale writes across branches and serve reads from caches. To see how they fit together, let’s start with how Git stores and transfers history.

Git object storage primer

Git must be able to keep two parts of a repository in sync: objects and references.

A Git repository records snapshots of files and the history connecting them as a graph of objects. An object may be either a blob, tree, commit, or tag. Blobs hold the content of an individual file, and trees associate names with blobs or other trees to form a directory structure. Commits point to a root tree and zero or more parent commits, forming the history of your repository.

Objects are identified by a hash of their type and contents. Objects are also immutable: recording a changed file creates a new blob, while unchanged files and directories can reuse existing objects.

Branches and tags are references, or “refs”, that point to commits. Committing on a branch creates a new commit and updates the branch’s ref to point to it.

Git objects reference each other by their ID, commits to trees, and trees to trees and blobs. The branch refs/heads/main points to commit C2, whose parent is C1. C1 points to root tree T1, where parser.py refers to blob B1. C2 points to root tree T2, where parser.py refers to blob B2. In both root trees, docs/ refers to the same tree, T3. Its README.md entry points to blob B3, and usage.md points to blob B4.

Git stores objects in either a “loose” or “packed” state. Loose objects are stored in individual files identified by their object ID. Packed objects are stored together in a packfile, enabling similar objects to be stored as deltas against one another.

Git can store blobs as separate files or pack them together, encoding one blob as a delta against another.

In Git, an object is reachable from a commit if Git can find it by following links from that commit through its parents, trees, and blobs. During a fetch, the client tells the server which commits it wants and which it already has. The server finds the objects reachable from the wanted commits, excludes those it knows the client already has, and sends the remainder as a packfile in the same format Git uses on disk. A full clone starts without any objects, so it needs all the objects reachable from the history being cloned.

A push transfers objects in the other direction by asking the server to update one or more refs. The client uses the server’s set of advertised refs to avoid sending objects the server already has. The pack being sent can therefore depend on existing objects: receiving the pack alone is not enough to reconstruct the new history.

Replicating a Git repository means keeping both objects and references available across multiple servers. Before a replica can serve a reference’s new value, it must be able to supply the objects reachable from that value. The service also needs to coordinate competing reference updates, i.e., pushes that try to move the same branch from the same old tip to different new commits. Systems that impose a single order on all writes to a repository can make pushes to unrelated branches wait on one another.

Existing distributed Git service patterns

While designing Pando, we evaluated conventional approaches for scaling source control using Git: one that coordinates ref updates across repository replicas using three-phase commit (3PC, a protocol for coordinating a commit across multiple nodes), and one that orders updates through a shared write-ahead log (WAL, a record of changes written before they’re applied) in blob storage.

Three-phase commit (3PC)

In the three-phase commit design we modeled, the Git service keeps a complete copy of the repository on several replicas. When any server receives a push, the incoming packfile is copied and sent to all replicas in parallel. To publish the push, a majority of replicas must accept the ref update. The server managing this is the coordinator, and it requires multiple rounds of communication between the coordinator and each other replica.

To accept a reference update, each replica locks its copy of the ref and checks that it still points to the commit the client expects. The coordinator then sends a pre-commit message to the replicas that accept. Once a majority acknowledges the pre-commit message, the coordinator tells them to perform the update. Any up-to-date replica can then advertise the new branch tip from its local copy of the repository.

Keeping complete replicas makes it possible to serve reads locally, and guarantees that clients get a consistent view of all references no matter which replica they pick. As a consequence, references are updated one at a time, and each push requires multiple round trips to coordinate the ref update among replicas.

Waiting for a quorum puts replica disk performance and communication latency on the push path. As write volume grows, the replicas must keep up with incoming packs as well as read traffic.

Write-ahead log (WAL)

In the WAL-based design we modeled, the Git service uses a shared manifest in blob storage to order updates across the repository. When a server receives a push, it uploads the incoming packfile to a blob store and prepares the ref update in its local copy of the repository. To publish the push, it adds an entry to a shared manifest stored in the blob store that records the ref update and points to the packfile. The Git server then updates the manifest using a compare-and-swap operation. This atomically replaces the manifest, or otherwise rejects the change. If another writer has updated it in the meantime, the server reads the intervening entries, checks that its ref update is still valid, and retries using the packfile it already uploaded.

Writers can upload their packfiles in parallel, but even pushes to unrelated branches require updating the same manifest. The round-trip time for a successful compare-and-swap limits how often the repository can publish a new manifest. A low-latency blob store makes this step fast. Batching lets one manifest update several refs at the same time, at the cost of queueing updates and increasing latency.

The blob store can receive packfiles in parallel, but updates to the shared manifest still compete with one another. Increasing the blob store’s capacity alone doesn’t resolve that bottleneck. Each replica still needs to read and index packs from the blob store before it can serve them locally. If incoming writes outpace that work, replicas fall behind and clients wait longer to fetch newly pushed commits.

Architecting Pando

That brings us to Pando. Pando is a new architecture for serving Git repositories with the goal of scaling to many clients pushing simultaneously. Here’s how it works:

Rethinking constraints

Prior to developing Pando, we had a hunch that we could meet our requirements without compromising on Git, but we did need to compromise on some of the things that most Git systems assume to be requirements:

• Repository-wide consistency a consistent, point-in-time snapshot of all refs requires coordination across the repository. In a system that orders all ref updates together, pushes to unrelated branches contend on a single coordination point.

• Unique packfiles for every client two clients with different local histories fetch the same branch, the server may need to generate a different pack for each to avoid sending objects they already have. Creating these responses costs CPU and disk I/O, making it difficult to scale to high volumes of readers.

Instead, Pando provides a relaxed consistency model (causal consistency) to scale writes, and reuses packfiles to scale reads.

Causal consistency (CC)

When a client pushes to a branch, Pando stores the incoming packfile as an immutable blob in a blob store. Each time the branch moves, Pando records a ref version containing the new tip, the pack that supplied it, and any earlier ref versions the pack depends on in a metadata store. Both the blob store and metadata store are horizontally scalable.

Fetching a commit or reference from Pando requires replaying a set of ref versions (packfiles) to the client, reconstructing the state of the client that created the ref version being fetched. This provides a weaker form of consistency across the entire repository, while providing serializability for updates to individual refs. Clients only conflict when pushing to the same ref simultaneously. A client reading several branches cannot assume those reads describe a single point in time across the repository.

A push to trunk adds an incremental pack, containing one or more commits. Each push to a side branch includes the topic’s earlier commits again, so a client with the advertised trunk version can fetch the topic in one pack.

This is a form of causal consistency. In order to fetch a reference, the reader observes the history of the writer that produced it. During a push, Pando controls what shared history the client and server agree on, ensuring clients produce more easily shared packfiles.

As a consequence, Pando doesn’t provide a point-in-time state of all references during Git operations or an ordering of ref updates within a clone or fetch. Concretely, suppose that branches foo and bar are both updated during the negotiation of a full clone. During the negotiation, one client may observe foo at version 1 and bar at version 2, while another client observes foo at version 2 and bar at version 1. In practice, users and agents are agnostic to the global ordering of ref updates, and instead depend on the order of updates to a single reference.

Comparing Approaches

In this simulation of the three consistency models we configure 1,000 clients to each select a distinct branch and then repeatedly push unique commits, measuring total wall clock duration and push to fetch latency. The latter is an important metric for workloads, as it measures round trip time for users and agents to collaborate or to use other systems which rely on version control. The simulation models hardware ranging from commodity to premium virtual machines and blob storage systems. In this simulation, we’ve relaxed repository-wide consistency for three-phase commit.

Simulating Git Server Implementations

Follow each writer through repeated pushes, and separately measure when an independent reader can check out each commit.

Run this simulation comparing Pando with two other approaches. Choose infrastructure from commodity to premium hardware, number of clients, and more settings to see how they compare.

The modeled three-phase commit system shows significantly greater write latency under load as their disks saturate; meanwhile, write-ahead log systems show significantly greater read latency for the same reason, as servers struggle to keep up with reading from the journal. In both real world testing on our monorepo and in this simulation, Pando supports many thousands of clients able to push concurrently without impacting performance under load.

On handling a push, Pando validates the incoming packfile while streaming it to blob storage, without unpacking it into a local repository. When a client begins a fetch Pando selects the stored packs the reader needs, and the reader downloads them. No server needs to apply packfiles to a local repository or perform costly unpack and packing operations. Because these servers primarily perform negotiation and packfile selection, Pando servers can run on commodity hardware.

This allows Pando to scale to the limits of clients, the network between clients and Pando, and the blob storage service. With enough capacity, clients working on independent branches can each transfer packs as fast as their own connection allows. To serve more clients at that rate, we can add Pando servers and scale the metadata store and blob store alongside them.

Pushing and Fetching a Branch in Pando

Pando stores the writer’s packfile in blob storage and records the ref version in Turtle. On fetch, Pando selects the packs the reader needs, and Express serves them.

An agent edits files and creates a commit locally and runs git push, which creates a packfile of those changes to update a branch. Pando receives the pack and streams it to blob storage. After validating the incoming push with a single pass decode, Pando uses a multi-key transaction to record a ref version in our metadata store and atomically update the ref pointer. This enables Pando to provide “read-your-own-write” consistency. This entire write operation requires $O \left(\right. 1 \left.\right)$ metadata reads and writes.

When another client fetches the branch, Pando selects a set of stored packs using the history the client already has. This operation requires $O \left(\right. log ⁡ \Delta \left.\right)$ metadata reads, $\Delta$ being the shortest distance in ref versions between what the client has and what the client wants. The server returns packfile URIs, and the client downloads them through a service, Express, that stores packfiles in regional caches.

This design relies on the three central services of Pando’s architecture: pushes, fetches, and compaction.

Scaling pushes

When a client pushes, a conventional Git server is like a poker player showing all of its cards, revealing to the client every branch it has. The client then gathers the commits, trees, and blobs to create a packfile that excludes objects it knows the server already has.

Advertising more refs can make the push smaller. It can also make the pack harder to reuse, because another client may be missing objects the writer left out.

Pando takes a different approach. Instead of showing its full hand, it only advertises a select set of branches to the client. This is done for two reasons:

1. Providing a consistent view of every ref is expensive. With millions of branches, the ref advertisement alone can exceed 100 MiB. Providing this requires each server to either maintain a current mapping, requiring significant coordination overhead, or a database query that scales with the number of refs.

2. By providing only a subset of the refs to the client, the client will generate a pack that depends only on what the server explicitly allows the client to exclude. Tracking the dependencies allows Pando to reuse the packfile a client generates in one push by serving it as the response to a subsequent fetch.

As a Git server advertises more refs, clients pushing have more sources from which to exclude objects, enabling them to produce smaller packs. However, reusing these packfiles requires the server to be able to prove the clients fetching have each of the dependencies. Otherwise, the packfile may omit objects the client needs.

On the other hand, if a Git server advertises none at all, the client would generate a packfile containing the repository’s full history. That packfile can be used by any other client, but at the significant cost of making fetching any branch effectively a full clone.

Advertising more refs results in smaller packfiles. Most of the size reduction can come from advertising just one.

By default, when a client pushes a new feature branch, Pando advertises only a designated trunk branch as a base. This keeps the advertisement the same size as the repository gains branches, but comes with tradeoffs. When an engineer either pushes to a single branch many times, or stacks one feature branch on top of another the client has to resend everything outside of the trunk.

This causes successive branch pushes to be larger, containing potentially-redundant objects, and uses additional client bandwidth. In our monorepo, the median packfile produced this way was under 20 KiB, and the 99th percentile was under 3 MiB. For us, this was a sensible trade-off because a reader that already has the advertised trunk history can fetch the branch with that one packfile.

After choosing the set of refs to advertise to the client, Pando accepts the push and updates its metadata store. Pando splits the metadata update into synchronous and asynchronous portions. The synchronous metadata update ensures clients can immediately fetch the newly updated commit. This writes a new ref version and updates the ref pointer mapping to that new version storing that new tip, the dependencies used to negotiate the packfile, and a link to the packfile that was pushed.

At the same time, asynchronous work is recorded in the same transaction, such as scheduling compaction. For very large pushes, only a subset of commits are written synchronously. This allows us to require only a constant number of reads and writes to the metadata store, regardless of the size of the push.

Importantly, Pando servers can accept pushes to unrelated refs independently. The only contended key is the ref pointer, which we guard using an atomic compare-and-swap-like operation. We can scale up write throughput by adding Pando servers, metadata store scaling units, or blob stores.

Line-rate fetch and clone performance

When a client performs a fetch, it provides the server with two sets of commit IDs, known in the protocol as haves and wants. In a conventional Git server, the server computes the set difference among the objects reachable by the haves and those reachable by the wants, compresses them, and sends the result to the client. While the open-source Git server has a variety of indexes locally to accelerate this operation, these packfiles can be expensive to compute and are generally resistant to caching.

Instead, Pando uses the commit mappings recorded on pushes to look up the client’s haves and wants, resolving them to a set of ref versions that contain those commits. Each of those ref versions links to a packfile previously received by Pando. The server uses this to solve for what the client needs, selecting the packfiles that cover the set of objects the client needs.

Planning fetches in Pando

Pando compares what a client has with what it wants, then selects the stored packfiles needed—including any missing branch dependencies.

Already held Fetch Missing branch dependency

Fetch 27 trunk packs.

Select a branch to include its trunk dependencies.

Pando maps the Git protocol’s haves and wants to a sequence of ref versions. Wanting a side branch implies wanting up to its dependencies on the trunk.

The Git protocol only allows a single packfile to be sent in the HTTP response on a clone or fetch. However, an under-appreciated feature of the Git protocol, packfile-uri, allows a server to return links to download additional packs separately. Pando uses this exclusively, and transmits an empty packfile inline. To support this feature, OpenAI contributed a number of improvements to packfile-uri upstream.

OpenAI systems operate in many geographic regions, so we maintain Pando clusters with local caches in each. A pack delivery service, Express, caches our blob store, and a separate service caches the metadata store. These caches enable Pando to serve requests for a commit ID or a special read of a ref entirely from the cached state in that cluster. We use consistent hashing to route requests for a packfile to Express servers, with cache hit rates of over 99%.

This system of caches provides extremely high throughput, capable of saturating the network limit between clients and Pando. Using these packfile URIs and local caches, we’ve validated that using only standard cloud provider virtual machines, we see clone and fetch operations saturate 10 Gbps network links.

Compaction and time to first byte

For most Git servers, repacking and garbage collection are necessary to maintain the performance of a growing repository. Both of these steps reclaim space, but are compute- and I/O-intensive.

In a fast-moving repository like OpenAI’s monorepo, it is infeasible to send an individual packfile URI for every push that has occurred since the last fetch. Each day, a client would need to download tens of thousands of packfiles to catch up their trunk.

Instead, Pando combines packs in the background to reduce this overhead, adopting a design similar to compaction in log-structured merge-trees. Each push produces what Pando refers to as an “L0” packfile, representing the smallest unit of data Pando can transmit. Over time, background workers collect these packfiles into covers spanning a sequence of ref versions, scaling geometrically from L0. Today, our monorepo uses up to L4 packfile covers.

These covers serve a dual purpose. After a cover is sufficiently full, a compaction worker will claim it and produce a single packfile. This improves the compression ratio by reorganizing the objects inside the packfile. But it also improves fetch negotiation performance by acting as skip list over ref versions. By scanning through indexes at each level simultaneously, Pando can select a set of packs to send to a client in logarithmic time. These packfile covers, repacked or not, enable a more efficient solution to fetch planning, reducing both throughput and latency requirements for Pando.

Compaction in Pando

Small packfiles are grouped into larger covers across levels, improving compression and speeding up fetches.

L2

L1

L0

Compaction in progress Published cover

Compaction combines packfiles from many ref versions in the background, enabling more efficient fetches with fewer files.

Unlike many Git servers, packfiles are kept at every level and not discarded or garbage collected. Packs at every level are used to serve clients continuously, based on the haves and wants they request in a fetch. And for clients using Pando as an API, reading an arbitrary object from a packfile tends to require reading less data from an L0 than an L4, because it’s less likely to be compressed.

Compaction in Pando significantly reduces both the total transfer size sent to clients and enables pack planning to scale sub-linearly in how far behind the client is. This is especially important for large and quickly changing repositories, which we expect to be a new normal in software development.

Building Pando

We built Pando with Codex, moving from kick-off to production, in just 10 weeks. The core of Pando was designed by a single engineer, implemented by teams of agents, and deployed by a handful of engineers. Pando began serving production traffic in April of this year, and is one part of a larger system of source control management tools used at OpenAI.

We employed a wide variety of approaches when validating Pando’s design. We applied our principles for building more secure software foundations⁠. As a greenfield project, we chose to implement Pando in Rust and without legacy dependencies. We wrote a conformance test suite so agents could test their changes against standard Git behavior. We also developed deterministic simulation testing harnesses allowing us to check invariants and properties of the system, enabling us to find and mitigate subtle race conditions and regressions in negotiation and pack planning.

Many models contributed to this system; as new models became available, we used them to improve Pando’s performance and reliability. Using GPT‑6 Astra, agents wrote Lean and TLA+ to formalize a “theory of Pando” that we used to find and fix bugs in the implementation, and prove properties of the system. Multiple continuous sessions ran for hundreds of hours finding and fixing edge cases in Pando. We also run agents around the clock to red-team Git and Pando using Daybreak-capable models.

Deploying Pando

Our initial deployment of Pando was as a read- and write-through proxy to our existing source control system. This allowed us to keep Pando and the previous system in sync, so we could safely move traffic onto Pando without downtime.

As part of its caching design, Pando can also deduplicate and combine requests before proxying. This allowed us to keep our existing tools healthy as we moved systems to Pando one at a time. As we migrated more workloads to Pando, their pushes populated Pando’s cache and improved the reliability of our previous backend.

In order to optimize performance when using Pando, we began shipping our public distribution of Git to developer machines and within the ChatGPT desktop app. To migrate clients to Pando, we used an internal wrapper around the Git CLI to migrate clients in phases. We created many other tools for handling change requests, code review, and APIs for automations. We migrated hundreds of integrations that depended on legacy APIs, webhooks, and repository downloads alongside Git traffic.

Codex itself helped developers and teams make the move. We created skills for preparing existing checkouts, and for guiding agents to use the new system. As a result, developers could simply ask Codex to migrate their development environments and services.

Switching to Pando improved merge reliability and reduced the latency of the median push by nearly 80%.

Pando can deliver a petabyte of Git data per day with just a handful of CPU cores. Pando has served more than 57 terabytes of Git packs in a single hour, with more than 99% of those bytes from local disk cache.

Reusing packs lets many clients download the same repository history with little computation on the serving nodes. This example shows a single repository, but Pando is designed to support many more. As demand grows, we can scale data delivery and Git negotiation independently.

Contributing back to the Git project

Building Pando on the Git protocol and client helped us identify bugs and opportunities to improve Git’s own performance.

For example, Pando serves reusable packs directly to clients, so downloading and indexing those packs became a central part of making the system fast and reliable. That work surfaced edge cases in concurrent downloads, cleanup after interrupted transfers, and resuming downloads that had already completed. We fixed those paths, added validations when accepting an existing pack for a fetch, and contributed that work back upstream.

We also needed to see where time and bandwidth were going, so we added tracing for packfile-uri downloads and byte counts for pack generation, making it easier to tell whether a slow fetch was spending its time generating packs or transferring them.

In OpenAI’s Git build⁠(opens in a new window), the client can download and index independent packs in parallel. We added the concept of “explicit haves” so a server can advertise a branch’s current tip separately from the history a pushing client may leave out of its pack. On Pando, this enables clients to use --force-with-lease to check that a feature branch hasn’t moved and build its pack against trunk alone. Anyone with that trunk history can then fetch the branch using the same pack.

Several of these improvements are already in upstream Git, and many others in our public repository. We’re continuing to work with the Git community to make these improvements useful beyond Pando. You can follow that work in our public patch tracker⁠(opens in a new window).

What’s next

By rethinking conventional wisdom on how to scale Git repositories, Pando enables us to scale our monorepo without changing the Git workflows our engineers and agents already use. In our monorepo, Pando has unlocked a greater degree of parallelism, improved merge reliability, and reduced the time it takes to publish code changes. Beyond our monorepo, we’re also exploring other uses for Pando within OpenAI, including workloads where large numbers of agents need to collaborate on a shared Git repository.

As engineers and agents produce more software, more teams beyond OpenAI will encounter the limits we hit in source control. We’re currently exploring ways to share the technology that makes Pando possible with our users.

Pando builds on two decades of experience from users and operators of Git repositories. We’ll continue to develop Pando while working with the open-source Git community to share any improvements that benefit Git users more broadly.