CMU Prof. Pavlo on Agent DB Deletions and Guardrails

Every month, a chilling story circulates on X: A developer asks an AI Agent to help build something, and the Agent immediately rushes into the database, wiping out production files. This has happened with Replit and other platforms. While people exclaim, "AI is too dangerous," they continue handing over production database credentials to Agents.

However, within the database community, the absurdity of this situation lies in the fact that these disasters could have been prevented. Mechanisms designed to stop humans from making foolish mistakes have existed for decades.

Professor Andy Pavlo of Carnegie Mellon University (CMU) is one of the most revered figures in the database field. His courses have educated an entire generation of engineers and, incidentally, fed today's AI models. Recently, he joined ClickHouse, a real-time analytics database company, as Vice President of Database Research. Their core product is a column-oriented OLAP (Online Analytical Processing) database management system.

Recently, Andy Pavlo appeared on The MAD Podcast with host Matt Turck to clarify four major shifts happening in databases during the Agent era: the retreat of vector databases, guardrail issues regarding Agents deleting data, whether AI can build databases from scratch, and the decade-long journey of autonomous driving databases. This article is based on the podcast video, edited by InfoQ.

TL;DR

Q: Why do Agents keep deleting databases?

A: Agent-induced data deletion isn't a new problem; it’s an old issue being re-exposed. Permissions and control mechanisms have existed for decades. People are just eager to let Agents "get work done" and forget to ask, "What if they mess up?"

Q: How should we add guardrails for Agents?

A: Treat them like toddlers learning to walk. You wouldn’t let a toddler run around the house without stair gates or outlet covers. Set proper permissions for Agents so they simply cannot access production databases or delete tables they shouldn’t touch. Combine this with branching to create sandbox environments. Don’t try to sit next to each Agent clicking "approve"—you can’t babysit hundreds of Agents simultaneously.

Q: Are vector databases still worth investing in?

A: The moat isn’t deep. Ultimately, it’s just an index. Within a year, all database vendors added vector capabilities, turning a differentiating advantage into a baseline entry requirement. They won’t disappear but will evolve in two directions: either becoming general-purpose systems like Postgres or remaining standalone add-ons like Elasticsearch.

Q: Is text-to-SQL reliable now?

A: Off-the-shelf solutions have an accuracy rate of about 60%. One major bank spent years and millions of dollars building semantic and context layers to raise accuracy to 99.5%. So, it is feasible, but only if you have the budget and time like that bank. Ordinary users downloading ready-made tools won’t reach that level.

Q: Can AI really build a database from scratch?

A: Yes. In CMU’s database course projects, Agents couldn’t complete everything a year ago. After the release of Opus 4, the floodgates opened; they can now complete almost everything with minimal prompting. Moreover, over 60% of open-source databases already show commits involving Agents. AI is already building.

Q: What is the advantage of LLMs tuning database parameters compared to traditional auto-tuning?

A: Traditional models require hours or longer to train and need sufficient training data to produce "purely optimal" configurations. LLMs can provide "good enough" results in 15 minutes, achieving about 85% effectiveness. For most people, "good enough" is actually good, especially since many run Amazon’s out-of-the-box configurations, which are often terrible.

Q: Has the database market stagnated?

A: If you use AI as a reference point, everything seems stagnant. But the relational model won’t disappear, just as you won’t invent new arithmetic to replace 1+1=2. SQL might become the new assembly language, but the underlying logic remains unchanged. Often, what truly matters is the scaffolding above the database: UI, developer experience, data ingestion, and interoperability. Part of Snowflake’s success comes from excelling at these peripheral aspects.

Frequent Data Deletions by Agents: How to Protect?

Matt: Two years ago, the topic of databases and AI emerged during the era of vector databases. The prevailing idea was that you needed RAG (Retrieval-Augmented Generation), combining chatbots with search. Clearly, we are now in the Agent era. What is different?

Andy: From the perspective of "AI and Databases" becoming buzzwords, roughly two years ago was indeed when RAG and vector databases surged. However, vector databases themselves largely existed since 2016. It’s just that their utility became obvious recently—you can use them to enhance and improve what Agents do.

The vector database market is interesting. They entered the space early, pushing a specific concept: performing approximate similarity searches on vectors or embeddings, rather than the exact searches performed by traditional database systems. This is particularly useful when processing natural language because what you’re looking for may not be precisely definable semantically. For example, "Find me all albums in New York City related to a certain activity"—it’s hard to describe exactly what that means.

So, when chatbots exploded and everyone realized this technology existed, vector databases were in the right place at the right time. But the moat wasn’t that large. Because ultimately, it’s just an index. Within a year, almost every database vendor had its own vector index; it has now become standard.

This doesn’t mean vector databases can’t perform better than traditional data systems in certain areas; it just means everyone now has that basic functionality. So, you’ll still use vector search to enhance chatbots and Agents, but you no longer need a specialized database dedicated solely to that task.

Matt: About a year ago, when Databricks acquired Neon, there was a statistic: The proportion of Neon databases created by Agents rose from 30% to 80%. What does "an Agent creating a database" mean? Why would an Agent need a database?

Andy: I think that statistic primarily refers to Agents creating branches. Branching occurs when you have a database and want to create a copy-on-write snapshot, making it logically appear as two independent databases.

Why do we need this? You have a production database running your business and applications—your system of record—where all changes are written. Yet you want human developers or Agents to experiment with new features and modify the application, ideally against the real database. The traditional approach is to copy the database or use a smaller sample version, because full replication is too expensive. So you set up a staging or development database, make changes there, apply them to production, and hope everything works when scaled to real-world conditions.

Branching allows you to take a snapshot of the actual database. All your changes go into the snapshot without affecting production. Once you’re satisfied with schema or application code changes, you promote them to production. Often, these branches are created during GitHub Actions runs for PRs and tests.

Matt: Let’s simplify further: Agents need to create databases because they take actions in the world. If you ask an Agent to build an application, that application ultimately requires a database.

Andy: When you give a Coding Agent a prompt like “build this app with these features,” it immediately creates a database. Most of the branches—or new databases—are created for development purposes. But yes, Agents can now genuinely create databases. Conversely, they can also delete them. People often get into trouble because Agents perform unintended actions.

Matt: Every year, there’s an interesting incident that goes viral on X. Jason Lemkin, founder of SaaStr, tried to build something using Replit, but Replit’s Agent accessed the database and deleted a large number of files. Replit subsequently made adjustments. Clearly, we’re still in the Wild West (a state where rules are incomplete, constraints are lacking, and experimentation leads to frequent mishaps).

Andy: We’ve actually had the capability to prevent people from doing foolish things for a long time; control mechanisms have always existed. It’s just that everyone wants to reinvent the wheel, so people learn painful lessons by repeating mistakes that were already solved decades ago.

Matt: In the world of Agents, what capabilities must databases have that don’t exist today?

Andy: Guardrails. Think of it this way: if you have a toddler at home, you wouldn’t let them fall down the stairs or stick their fingers into electrical outlets without installing safety gates. Treat Agents similarly—you must view them as toddlers. They are more sophisticated than toddlers, but you cannot let them run wild on production databases, doing whatever they please. Of course, you can sit nearby constantly approving actions, as seen in Claude Code. But the problem is, this doesn’t scale.

The correct approach is to configure permissions and controls so that Agents simply cannot access production databases or delete tables they shouldn’t touch. Similarly, restrict read access to data they shouldn’t see. If you set up proper guardrails combined with branching or other sandboxing techniques, disasters can be avoided. Currently, every month we see news about someone losing their production database because an Agent deleted critical data. With adequate guardrails, these incidents could have been prevented.

Matt: Are these guardrails the same ones used for humans today?

Andy: This raises a broader question: Do Agents behave fundamentally differently from humans, aside from operating 24/7? I haven’t seen evidence suggesting that how Agents develop software or generate inaccurate data differs from humans. Ultimately, data systems expose a limited set of capabilities—the APIs you interact with remain the same. Therefore, if you can constrain human usage of these APIs, you can constrain Agents in the same way.

The granularity of guardrails depends on the product you’re using. I appreciate open-source databases, but enterprise-grade commercial traditional databases have already implemented robust protection mechanisms over decades for human users. Now, we simply need to expose those same mechanisms to Agents.

Four Eras of Database Traffic

Matt: You mentioned volume and scalability. If billions of Agents start taking actions and interacting with databases in the future, will this change everything?

Andy: It’s too early to conclude. Volume is certainly a concern because Agents operate 24/7, unlike humans. However, database history has already witnessed similar leaps in scale.

The first era, roughly the 1970s to 1980s, marked the birth of databases themselves. For the first time, organizations could store information, execute transactions, and manage data digitally—previously, everything was done with paper and pen. During this period, database traffic grew significantly, primarily driven by humans interacting with databases in work contexts. No one used databases at home, as personal computers weren’t common in households during the 1980s.

The second era, spanning the 1990s to early 2000s, saw the rise of the internet. Everyone began interacting with databases through websites and home networks, causing another surge in traffic.

The third era was mobile. Today, everyone runs various apps at any given moment, interacting with websites to read and write data from databases.

Now, Agents represent the next chapter. Each person may not only have a smartphone but potentially hundreds of Agents acting on their behalf, reading and writing data in databases. Fundamentally, behind every application lies a database; all reads and writes eventually reach the database layer.

However, I don’t believe we’ve reached that point yet, since not everyone currently has hundreds of Agents running 24/7. What’s interesting is that when humans interact with databases via phones, laptops, websites, or apps, they do so only while awake and typically one task at a time. Humans cannot simultaneously operate a phone and a laptop; our capacity is limited. Imagine a world where Agents continuously interact with databases 24/7, behaving like humans. The resulting increase in volume would be substantial.

You might ask: How do Agents differ fundamentally from current IoT devices or smartphones? IoT devices primarily stream measurement data, while smartphone apps mostly read data. But Agents perform more complex operations involving combinations of reads and writes, making traffic patterns more intricate. Consequently, I anticipate query volumes hitting databases could grow by 10 to 100 times in the future.

One thing I am still uncertain about is the complexity of Agent interactions: are they more complex or simpler than what humans currently do? Some argue that queries and SQL generated by Agents are actually shallower, as an Agent might simply read one or two tables to see what’s inside. In contrast, when humans use dashboards or BI tools like Tableau or MicroStrategy, they often perform very complex joins across multiple tables. I don’t yet know if Agents have reached that level of sophistication, or if they are merely accessing database data in a simpler way.

Will Agents change everything? I believe so. At least in terms of volume, it is certainly happening now, and this trend will continue to grow. Of course, once Agents start running, they generate their own telemetry data, which you must store and understand—this presents another challenge. However, how significantly their behavioral patterns differ remains to be seen.

Matt: There is a concept called "Agent Memory" that seems to be becoming a battleground. On one side is the file system, and on the other is the database. What is your take?

Andy: A file system is a database. Ultimately, the question is: what form should the input or context provided to the Agent take? As far as I know, it is all human-readable text right now. It doesn't really matter whether this human-readable text comes from a bunch of JSON files in your directory or is extracted from a database system and converted into the correct input format.

When you use a data system, you obviously gain certain advantages and safety guarantees; it protects the data. SQLite is well-known for this. Why would you use a file system and worry yourself about whether writes are correct, safe, or capable of handling multiple modifications before committing, instead of just using SQLite to get it right? So, at the end of the day, everyone should put everything into a database system. Even some file systems, like WinFS, are essentially database systems under the hood.

Matt: Does the world of multi-Agent, sub-Agents, and Agent clusters require databases even more?

Andy: If you need to share state across multiple machines or multiple Agent instances, regardless of whether they run on the same machine, Database Management Systems (DBMS) provide exactly this capability: you can have multiple readers accessing shared state. I haven't seen any scenario where an Agent pushes filter conditions down into the database when querying memory; usually, these are provided as context. This brings us back to RAG or vector databases: you compare against your memory based on what has already been retrieved to provide context.

So, from my understanding, the API input for context is human-readable text, and it doesn't matter whether it comes from a file system or a database system. But data systems offer guarantees and conveniences that file systems cannot, such as shared state across multiple Agents.

AI-Recommended Databases: The New SEO

Matt: Have you observed the Agent world converging on specific types of databases?

Andy: Whether it's used as memory or when an Agent chooses one data system over another while building an application—perhaps it's more the latter. Models are trained on what people write, so if I say I want to build a new application today, it will likely recommend Postgres because that is what exists in the pre-training corpus.

Two database companies have already approached me asking how to make their databases the preferred recommendation for Agents instead of Postgres. I told them: first, that's not my job; second, go talk to the people building the models, as they clearly have influence over this. Essentially, this is SEO: how do you teach an Agent to learn "I should use Database X instead of Postgres"?

Matt: I suspect the logic is this: if your articles are highly indexed by search engines, then "pitching" yourself to search engines becomes a way of "pitching" yourself to the models.

Andy: I don't mean to brag, but I actually face a problem: when I ask questions about databases, the model cites my own content back to me because all my course materials are online. But this isn't good. For example, when teaching a database course, I might say, "I want to know about data systems that implement a certain type of indexing." Now I have to specify: "I already know which systems do this; show me the ones that don't." Then I also have to add: "Do not cite anything I wrote."

Another issue is that all our course materials are open-source, so anyone can use them. Consequently, the model might say, "Okay, I won't cite Carnegie Mellon University; I'll cite another university." But that is actually my slides being reused by another course. So I have to be extremely specific to clarify what should not be returned to me.

Regarding OLTP, I haven't tried it recently, but typically it spits out "You should use Postgres," unless you are very specific. For instance, if you say, "I need to run on an embedded device," it might suggest DuckDB or SQLite. Or if you say, "I only care about key-value stores," it might return RocksDB. Then it might follow up with recommendations: Do you want a hosted version or self-deployment?

For OLAP, I am curious if there is some kind of ranking. It will definitely give an initial answer, but you might need to refine it further by saying, "My workload looks like this, my data looks like this," to get a more nuanced response. I haven't tried it in about a year, but previously I saw it say: Do you have money? Yes. Then you can use Snowflake or Databricks. No? Then you can use DuckDB. Do you want local deployment? Then you can use ClickHouse, and so on.

Matt: This might involve a context layer or ontology concept, which seems to be pushing everyone toward graph databases. Does this make sense?

Andy: I don't think you need graph databases. I haven't seen any benchmarks proving that native graph databases outperform relational database systems, provided those relational systems have the correct constructs in SQL or APIs to allow graph traversal on the server side rather than requiring round trips. What you use for the context layer depends on your needs: how many Agents you have, and whether you are sharing state among multiple Agents. For many of these scenarios, Postgres should often be the first choice, especially for OLTP. If you are doing analytical workloads, that is a different story.

Breakthroughs in MCP and Text-to-SQL

Matt: It seems like over the past year or so, every database vendor has added an MCP server to their product. What’s your take on this? A year or two ago, there was a wave of hype around text-to-SQL, where everyone wanted LLMs to run directly against databases, especially for analytics scenarios, but that wave didn’t really succeed. Have we reached that tipping point now?

Andy: Text-to-SQL, or natural language to SQL, is something people have been trying since the 1970s. Every once in a while, there’s a breakthrough in NLP, and everyone says, “This time it’ll finally work,” but it never did—until LLMs came along. However, I’ve always been skeptical of academic papers and various framework benchmarks because they often rely on well-known test sets like Spider, and these models are essentially trained on those benchmarks. So, of course, they can memorize the correct answers.

I just spoke with someone from a very large bank yesterday. They said they already have a text-to-SQL interface. The initial implementation used off-the-shelf solutions, and then they made some internal improvements, but the accuracy was only about 60%, which aligns with numbers I’ve seen in other reports. But they mentioned that by building richer semantic layers and context layers, spending years and millions of dollars, they’ve now boosted accuracy to 99.5%. That’s insane.

I believe that if you’re willing to invest the time to build the right semantic layer and provide the necessary contextual information, enabling Agents to derive the correct SQL query from your question, it is feasible today. But again, that’s a huge bank with seemingly unlimited budget—they managed to achieve this. If you download a standard tool today, an average person will find that these tools simply don’t reach that level of accuracy.

As for MCP, I understand why they’re doing it. It originated from Anthropic, and everyone quickly adopted it as a standard. Every so often, someone posts on Hacker News saying things like “MCP is dead,” but I don’t think it’s going away. Essentially, it’s just a REST interface—a standardized way to describe what an Agent wants to request from your data system. The data system receiving the MCP request needs to translate that request into the appropriate API call to fetch data. For database systems, that means generating SQL queries to retrieve data. But typically, it’s just a strict translation of what you pass in, rather than trying to infer the semantic meaning of the MCP request, because the MCP request itself must be defined clearly enough.

I have no issues with MCP because it encourages more use of data rather than relying on homegrown hacks. Can it be improved? I think it’s inefficient because you’re sending human-readable text over the network to retrieve data. There are obviously better approaches using binary protocols, but ultimately, it works well enough, and it seems to have become the standard everyone is using.

Matt: Should the security layer be handled by that same layer you mentioned earlier?

Andy: Yes. But this isn’t fundamentally different from requests coming from traditional application clients via JDBC or ODBC. Whether it goes through the network protocol or terminal of these data systems, or through an MCP REST request, it’s essentially the same: Okay, I received this request; I need to figure out who is making the request, what they want to do, and whether it should be allowed. So, the safeguards we discussed earlier apply equally, whether the input is SQL or MCP.

Matt: Let’s flip the discussion from “databases as the memory or execution layer for Agents” to “what AI can do for databases.” Clearly, we live in a world where AI can write almost any code. Does this hold true for databases too?

Andy: Yes. I teach a course on database systems at Carnegie Mellon University (CMU). Our project involves giving students a scaffold for a database system, and they have to implement parts like indexing and the query engine. A year ago, Agents could only handle part of it, not all of it. Then Anthropic released Opus 4, and the floodgates opened. Agents can basically complete all our assignments now, requiring minimal prompting because there’s so much training data.

All our projects are open-source, providing massive amounts of training data for learning how to implement these components. I believe Agents can now essentially implement any database system you want to build. Of course, you need to prompt them correctly, guide them step-by-step, ensure they produce the right design first, and then generate the implementation based on that design.

Matt: But my experience as a venture capitalist building databases is that it’s a decade-long painful journey. Nothing happens for the first three years, and then the smartest people in the world gather to solve every problem that seemed insurmountable at the time. Now you’re saying it can build the whole thing?

Andy: There’s an old saying in the data systems field: Building a data system takes ten years, but you can finish the first 90% in three years, and the remaining 10% takes another seven years. So, I believe that with enough tokens and sufficient guidance, people can Vibe Code an entire database system.

I have a side project I particularly love called "database of databases," located at dbdb.io. One thing we do is catalog every data system I know of. For the open-source ones, we pull down all the latest commits from GitHub every night and track which commits were actually "signed" by tools like Claude or Codex. Many people disable this feature, so you can’t be 100% sure if a piece of code was AI-generated, but most people don’t turn it off. And by now, I believe over 60% of open-source database systems have seen commits from Agents.

The Road to Autonomous Databases

Matt: If writing code can be automated, what about running the database itself? You’ve been working on the concept of "autonomous databases" for ten years—it’s a big project. Can you walk us through this journey? What exactly was impossible back then but possible today?

Andy: The first thing I did after arriving at Carnegie Mellon University was to spend the early years visiting various companies to understand what problems they were actually encountering with their databases. What I found was that the recurring themes were strikingly consistent: running, maintaining, and optimizing these systems is itself a massive struggle. Since the invention of the relational model in the 1970s, people have been trying to automate tuning—indexes, partition keys, sharding keys, all sorts of configuration knobs have been tried. Microsoft Research even had a project called Auto Admin in the early 2000s, which built a suite of tools for automatically managing and optimizing data systems.

Why is this so difficult? Because there are thousands of possible configuration combinations for a single database. For example, deciding which indexes to create on a table to speed up queries—if you create too many indexes, write performance degrades, and disk space gets consumed rapidly. Then there are runtime behavior controls, such as how much memory is allocated to specific memory pools versus index caches. This exceeds the reasoning capacity of any single human, especially when you are dealing with thousands or millions of databases; manual management is simply impossible.

So we spent considerable time researching how to use machine learning to build automation tools. We essentially pursued two research tracks. The first was a "black-box" approach: without modifying the database internals, we used only the exposed APIs to see how much automatic optimization could be achieved for existing systems like MySQL, Postgres, and Oracle. The second was a "rebuild from scratch" approach: assuming the database is natively controlled by machine learning tools—or what we now call Agents—should the system design be different? Many of the lessons we learned from optimizing MySQL were eventually incorporated into our own custom-built systems.

The biggest headache in building automation tools back then was the lack of sufficient production database data, and the inability to transfer knowledge learned from one deployment to another. Most organizations have two environments: a staging or development environment, and a production environment. Obviously, you don't dare experiment recklessly in production, because if you introduce a performance regression, the business suffers. So you validate changes in staging first, then migrate them to production. But the problem is, staging and production never share the same hardware, nor do they experience the same workload patterns. This was the core challenge we faced for a long time: the training data required to feed models so they can make good decisions in production environments was always insufficient.

Then LLMs emerged, fundamentally changing the solution to this problem. Because LLMs are essentially trained on all the blog posts, documentation, and best practice guides about database tuning available on the internet. At the research level, we found that LLMs can help you achieve roughly 85% of optimal tuning, though there are still some quirky edge cases that require customized algorithms to handle.

Matt: Does that mean databases can largely self-tune and self-optimize now?

Andy: The answer is yes, but it depends on how much improvement you are seeking. LLMs can get you to the 80–85% mark. Is there always more performance to squeeze out? Yes. Some people will always care about those edge cases, so the automated methods we built in the past remain important for them. But for most people, a finding that surprised me greatly was this: when we were running our startup and trying to auto-tune MySQL, we discovered how many users were running with the out-of-the-box configurations provided by Amazon or Google Cloud platforms. They told us, "I thought Amazon was handling the tuning for me." No, they weren't. So a significant number of people are running on default configurations, which are often terrible—please don't do that. LLMs can step in here, address the obvious issues, and deliver meaningful improvements in a very short time.

Matt: In a world where Agents invoke and manage databases, and databases themselves are becoming autonomous, what work truly requires a deep understanding of databases?

Andy: Even if you don't plan to become a database system engineer—someone who writes the kernels for ClickHouse, CockroachDB, or Postgres—you should still understand the internal principles of these systems. Everyone interacts with database systems. If you understand their internal implementations and grasp what a data system is trying to do (or not do) given a specific query, you will be in a much stronger position to understand the entire technology stack.

Do you still need to take my course to figure this out? Not necessarily. Agents have already ingested all my materials; they can reproduce everything I teach verbatim. But ultimately, whether you learn through traditional university courses, self-study, or via language models, once you understand what data systems are attempting to accomplish, much of the mystery disappears. You will understand why a query runs slowly, or why it doesn't, placing you in a better position at the application layer.

If you are pursuing a degree in computer science or engineering focused on data systems, I don't believe the era of "writing a system from scratch yourself" will continue indefinitely. However, database companies are still hiring people to do this work, because someone must understand what the Agent is doing and be able to judge its actions.

Matt: Are we entering an era where database management and deployment no longer require a Human-in-the-Loop?

Andy: The traditional DBA role, as we know it, will likely be downgraded, with a large portion of its tasks being automated. This doesn't mean these jobs will disappear; rather, professionals can shift to higher-level tasks, such as data modeling, which remains perpetually important. Agents can perform much of the modeling work, but you still need humans to judge: does this data model truly capture what our organization or business intends to represent in the database? However, specific tasks like parameter tuning and index selection can be delegated to Agents.

Why Establish a Research Lab within ClickHouse

Matt: You recently joined ClickHouse to launch ClickHouse Labs. Why establish a lab within a commercial company rather than conducting this work in academia?

Andy: Doing research in universities has become quite challenging lately; research funding in the US is no longer what it used to be. There is also a larger question: When an Agent can already perform many tasks, what does recruiting a PhD student actually mean? When Agents can produce results and co-author papers with me as a professor, do you still need as many PhD students to do this work? I certainly still value educating students and enjoy doing so, but I am unsure how to secure enough funding to support the number of students I would ideally want to mentor.

Some companies reached out to me, asking if I was interested in leaving academia to work on something interesting with them. But those offers always felt more like, "Do you want to come be an engineer here?" or a diluted version of the research I wanted to pursue. Then I talked to the people at ClickHouse, and the opportunity seemed almost too good to be true. I said, "You want me to do research?" They said, "Yes, do research." I replied, "Research doesn't always yield results; I don't know what will happen." They said, "Great, go for it."

This is insane because, as you know, they aren't an early-stage startup, but they also aren't a publicly traded company. Typically, at least in the database world, companies wait until they are well-established before setting up a research lab. The fact that they were willing to take this risk—or have this 'luxury'—to let someone like me come in and conduct peer-reviewed research is remarkable.

I am currently in an observation period, getting to know what they have built so far, because they haven't published much or written extensively about their internal work. They have done some incredible work, and my current task is to help them translate that work into papers that can be disseminated and published to the world, while simultaneously kicking off longer-term, more exploratory projects.

During my time at university, I worked on very applied aspects of database systems, not the kind of theoretical research that only ten people in the world understand and has no connection to reality. What I want to do is directly relevant to what ClickHouse wants to achieve—not just the ClickHouse system itself, but also their Postgres product line. I am super excited because I can have a direct impact without having to write a paper first and hope someone happens to see it and adopt our findings. We can evaluate whether an idea is good on real production systems and real workloads, and if it makes sense, we can push it out in a very short time. This opportunity is incredibly tempting to me.

Matt: In 2016, you worried that ClickHouse might be "vaporware" because it sounded too good to be true. Can you elaborate on that?

Andy: Databases are a fascinating area of research because I'm not just competing with other university researchers; I'm competing with big tech companies like Microsoft, Google, and Amazon, plus a slew of startups that all have their own data systems. There aren't many operating system startups, but there are tons of data system startups. So, my job is, when a new system is released, to figure out what it does differently from my research? Where does it perform better? What can I learn from how it solves problems?

When ClickHouse was released in 2016, my reaction was, "This thing popped out of nowhere," appearing seemingly out of thin air yet possessing all these capabilities: vectorized execution, columnar storage, compression, compaction... It seemed almost too good to be true, or it must be a fork of an existing project. But back in 2016, there simply wasn't an open-source OLAP engine that combined all these capabilities, which left me very confused. I initially assumed it was fake. Of course, it turned out to be real, and clearly brought significant benefits.

Matt: ClickHouse is very fast and very real-time. But why do databases, especially analytical databases, typically struggle in this regard?

Andy: ClickHouse's architecture and core foundations aren't novel by 2026 standards, but they were absolutely novel in 2016. At its essence, it stores data in a columnar layout. Instead of storing all column values for a tuple contiguously as in row-oriented storage, it stores all attributes and data for a single column contiguously. It then uses CPU vector instructions to process multiple data points in a single instruction, rather than writing inefficient for loops that iterate one by one.

Add to that built-in compression mechanisms and its log-structured storage approach: appending a batch of changes first to make writes as fast as possible, then performing compaction in the background to organize the data into a more efficient storage format. Many of these core architectural ideas actually originated from work done by Snowflake. Before Snowflake, there was a system called VectorWise, originally known as X100, which was commercialized by CWI as VectorWise—the same institution behind DuckDB. So, these ideas for column-oriented systems, analytical systems, and real-time analytics have been passed down through generations.

But in 2016, no open-source project had implemented all of these together. So when ClickHouse came out saying, "We've incorporated all these capabilities based on prior research," it was truly stunning. Since then, DuckDB has similar features, the team in Amsterdam built Polars, Firebolt started as a fork of ClickHouse, rewrote a lot of internals, and now has these capabilities too. Databricks has its own version of the engine, so nowadays everyone has this kind of architecture.

What I think ClickHouse has done particularly well over the years is continue pushing performance forward while constantly expanding the types of workloads it can genuinely support.

Matt: From an internal perspective, what are customers using ClickHouse for?

Andy: People use it to observe telemetry data generated by Agents, allowing them to react to this data in real time and query it. There are also numerous applications in financial markets, fraud detection, and any scenario where you need to ingest data quickly and then ask very complex questions against it extremely fast, getting results in milliseconds. I think combining these two aspects puts ClickHouse in a unique position.

Matt: ClickHouse has been on a path of expanding its product line. It started as that powerful real-time OLAP analytical database, and now it has observability offerings and acquired Langfuse. How do you understand this evolution from the inside? Do these things really fit together?

Andy: Before I decided to join ClickHouse, they were doing things at the business level, such as offering a managed Postgres service, rather than just providing a single, excellent OLAP engine. They didn't start in the OLTP space, yet they built out the entire stack for monitoring, visualization, and observability—not just for traditional BI or traditional data, but now covering Agents and LLMs as well. Everything they are doing is about expanding into different vertical domains, enabling those verticals to build on top of the ClickHouse platform.

Matt: There seems to be a trend: Databricks acquired Neon, and Snowflake acquired Crunchy last year. Is the convergence of analytical and transactional databases the future? Is it driven by commercial and business reasons—essentially trying to sell more to the same customer—or is there an underlying product logic at play?

Andy: There are actually two questions here: First, should a database company offer both OLTP (transactional) and OLAP (analytical) systems? Second, should these be part of the same system or separate systems?

Should they do it? I believe yes. Returning to what we discussed about what Agents enable you to do, maintaining these systems remains a challenge, but the barrier to deploying a Postgres service has indeed been significantly lowered.

I have another hypothesis: For a startup that began with an analytical database, is it better to add OLTP capabilities first, compared to an OLTP company later adding OLAP capabilities? If you want to build a new data stack, starting with analytics, gaining traction there, and then adding OLTP capabilities seems like a smarter strategic move from a business perspective. I can't strictly prove why from a business standpoint, but it appears to be the wiser path.

The current question is: Should you have a single API, a single interface, and one logical endpoint where people run transactions and analytics on what feels like a single database instance? Or should they be separate services? ClickHouse currently offers separate services, but they have a fast path to move data from Postgres into ClickHouse. What Databricks released this year is their LTAP engine, or rather, the LTA story for Lakebase. You can run transactions on Postgres, Lakebase, or Neon, and let the OLAP engine read that data directly, eliminating the need to duplicate data as required by ClickHouse or other systems. The idea of "two engines, one copy of data" isn't new. In the 2010s, a startup called Splice Machine ran HBase plus Spark SQL, but maintained only one copy of data stored in row format.

People have always wanted HTAP systems, these hybrid systems. The challenge has always been reconciling the fact that some data resides in row stores while other parts are in column stores. Major commercial vendors like Oracle and SQL Server maintain one copy in row format and one in column format—a "fraction mirror" approach—with internal mechanisms to synchronize them. However, these hybrid solutions haven't gained widespread adoption because the stakeholders for these two types of workloads within organizations always want the best possible tool for each job. The OLTP teams running applications and operations don't want a system that is mediocre for operations and mediocre for analytics; they want the best operational data system, which today is Postgres, regardless of pros and cons. The same applies to the analytics side. Therefore, this has always been the commercial and go-to-market challenge for hybrid systems.

However, I think Databricks might be positioned to succeed here because they already have market share on the analytics, machine learning, and data warehouse sides, and are now simply adding the OLTP piece. ClickHouse could potentially follow the same path. This delves deep into data system architecture details, but everyone has always wanted this capability. The challenges are often not just purely engineering problems; organizational factors or human reasons may prevent implementation.

Ratings for Vector, Graph, and GPU Databases

Matt: Let's look at other niche areas. Are you bullish on vector databases? Now that everyone can perform vector search, is this still a good business worth investing in? Can these companies survive?

Andy: When I was a professor, I would hear rumors about who was doing well and who was struggling, piecing together the entire data landscape through investors, former employees, students going out for internships, and candidates returning from interviews. But now, as an investor, I see everything. To my knowledge, Turbopuffer is currently thriving among vector database companies. They are extremely focused on vector search and offer cost-performance ratios far superior to everyone else.

So I don't think vector databases will disappear. They will evolve in two directions: either becoming general-purpose systems like Postgres or MySQL, serving as systems of record that store both raw tuples and corresponding embedding vectors; or becoming standalone systems similar to Elasticsearch, pulling a copy of data from the operational side. This allows them to comfortably exist as "add-ons." If you require extreme performance, in certain cases, you still need these specialized systems. But I have never seen predictions that "Postgres will be killed by vector databases" come true; quite the opposite.

Matt: What about graph databases? Neo4j has survived for 20 years. The AI era should have been the highlight moment for graph databases. What is happening now?

Andy: I made a bet with someone on Hacker News that by 2030, the graph database market would surpass the relational database market. If that happens, I will wear a T-shirt printed with "I Love Graph Databases," use it as my driver's license and university ID, and keep it on my website forever. It is now 2026, with four years remaining, and I am calm. This will not happen. Graph databases have always been a niche market. Research has shown that with certain optimizations in the engine—which ClickHouse has done and DuckDB has partially implemented—graph traversal is essentially self-joins on tables, which can be implemented very efficiently, easily outperforming Neo4j.

Saying "faster than Neo4j" is a bit like saying "I can run faster than someone in a wheelchair"; it feels like bullying. The SQL standard now supports property graph queries, with Oracle being a primary driver, allowing traversal to be performed server-side. So, regarding graph databases, I think it’s a bad idea. There is no real reason to use them.

Matt: What about GPU databases?

Andy: GPU databases involve pushing all query execution down to GPUs, whether over PCIe, on the same machine, or across different machines. The history of building accelerators for data systems dates back to the "database machines" of the 1970s, where people built specialized hardware to run sorting and query operators. By the early 1980s, this approach died out. By the time you designed and taped out new hardware, the next generation of GPUs from Intel or Motorola had already arrived, erasing any potential gains.

In the 2010s, a new wave of GPU database startups emerged. The biggest challenge at the time was that you had to fit the entire database into GPU memory; once data had to be transferred back over PCIe, performance degraded dramatically. Most of those companies fizzled out, and the survivors pivoted to visualization. Over the past year or so, NVIDIA has essentially acquired most of the struggling GPU database companies—Voltron, HeavyDB, etc.—and is now fully betting on this direction. However, whether the concept of GPU-native database systems will hold up in the long run remains to be seen.

Microsoft has publicly offered GPU-accelerated analytics services in the cloud; another major database company evaluated the economics of GPUs and decided it wasn’t worth it. A former student of mine is now a professor at the University of Wisconsin but took a leave of absence to join NVIDIA. They have a project called SiriusDB, which isn’t a new data system but rather a layer sitting between existing systems and GPUs. It supports running DuckDB queries on GPUs, as well as Doris, StarRocks, and DataFusion.

The difficulty lies in: How much conversion is needed between the data layout expected by ClickHouse and the layout desired by CUDA? How do you organize and share memory across different components? These questions remain open, but research interest is certainly high. NVIDIA is clearly pushing this because it drives more GPU sales. Currently, everyone is either compute-constrained or facing scarce memory availability; computing hardware is expensive and hard to obtain, with GPUs being the most expensive hardware of all. Saying that an entire database should run on GPUs—at least in the short term—I’m not sure that makes sense. But if the performance gains are significant enough, some studies suggest it might be worthwhile.

Matt: Are there any emerging categories, or specific niche opportunities within a category that people haven’t discussed much yet?

Andy: You can always build new data systems for new hardware, but my track record in this area is poor. We conducted extensive research on experimental hardware, only to see projects get cancelled or fall through. For example, Intel’s Optane persistent memory: we did a lot of research assuming DRAM was persistent (i.e., no data loss upon power failure), which would fundamentally change how data systems are built. But then Intel killed that product line. We also researched processing-in-memory (PIM) hardware, where CPU cores are integrated directly onto memory modules, allowing data systems to push query execution down to the DIMM. Our project, called OPM, was acquired by Qualcomm but was also discontinued last year.

There is always plenty of work to be done building data systems around new hardware. This is one of the things I want to figure out at ClickHouse: Do agentic workloads really differ significantly from human-driven or existing application workloads? If so, why and how? How would you modify data system development to better leverage them? This remains to be seen. Query optimization is always an interesting hard problem—it’s the most difficult part of data systems. Incremental materialized views are also a major challenge. But these aren’t unexplored areas; people have been working on them for decades. So for me, the agentic space is probably the most interesting and relevant direction right now. What changes will these workloads bring? How will system architectures evolve? I don’t know yet.

The Future of the Database Market

Matt: It feels like the database market is at a specific historical moment. In the 2010s, the evolution from SQL to NoSQL was an explosion of activity, followed by the rise of Databricks and Snowflake, and now ClickHouse. But it seems this momentum has slowed down slightly, perhaps even stagnating.

Andy: Maybe, but we’ve seen trends like this before. Relational databases were very active in the 1970s and 80s, then there was another wave in the 90s, after which the market solidified around the big vendors: Oracle, IBM, Teradata. If you look at databases solely from the perspective of these companies, it does seem stagnant for many years. But as you said, there was massive activity again in the 2000s and 2010s.

However, if you use AI as a reference point, everything else looks stagnant because computer science history has never seen anything like AI. It’s like giving a cheetah a bunch of cocaine and putting it in a Ferrari—the speed at which people are developing these things is insane. So by comparison, everything else seems slow, dead, or stagnant. But ultimately, I believe scale will be extremely important.

You mentioned cost and efficiency earlier—for example, squeezing the best performance out of given hardware and striving to reduce costs. That’s always an interesting challenge that can continue indefinitely. But in the end, I don’t think the nature of data itself will undergo such a radical transformation that we need to throw away everything known about databases and start over. Just as you wouldn’t invent a new arithmetic to replace 1+1=2. The relational model is the fundamental basis for representing data; you can only vary the implementation methods on top of it. So, making these systems more efficient definitely requires a lot of work, but I don’t think you’ll need to reinvent the wheel.

People always try to replace SQL, and I think that’s likely a doomed endeavor. However, SQL might become something like assembly language—just as no one hand-writes assembly anymore because text-to-SQL has become good enough.

You can imagine a world where I no longer need a general-purpose data system for every application, but instead vibe-code a completely specialized data system for just this one task. You’re already seeing hints of this approach today: code generation, or just-in-time compilation (JIT) for parts of queries. Some systems are already doing this—ClickHouse, Postgres, and Umbra all have features like this. But how to make this high degree of specialization sustainable might be a larger research question for the future.

Matt: So on one hand, it’s a somewhat stagnant industry, but on the other hand, this layer itself is more important than ever. This is precisely why your friend Larry Ellison became nearly the richest person in the world. Oracle is far more than just a database company; their headquarters building is shaped like a database, which speaks to the fundamental importance of databases.

Andy: Moreover, no one seriously tries to replace Linux. There are various niche operating systems, but no one is attempting a large-scale complete replacement of Linux. But databases are different; people always think they can do better than others, and new ideas keep popping up. So I wouldn’t say it’s stagnant. I’m just saying that in the short term, a lot of energy will focus on Postgres, and we can make improvements to it. But at the same time, I still believe these specialized engines will exist. I don’t know what the next workload will be. I’m not sure if agentic workloads are truly that different, because the things they model are similar to how applications were designed in the past. Is vector lookup fundamentally different from before? You need extra operators and extra indexes, but that’s about it. I don’t know what the next big thing is, but I believe there will be one.

Matt: What about the next five years? Does that mean companies like ClickHouse just need to keep getting better and more robust, since the fundamental demand is still there and we won’t see a new wave of startups?

Andy: When you say “better,” it actually has two meanings. One is whether existing queries can run faster? Yes, ClickHouse still has room for improvement, as do other systems. But the architecture ClickHouse is pursuing is similar at a high level to what Snowflake did in 2013 and what Databricks is essentially copying—everyone is doing roughly the same thing in analytics.

What really matters are the things surrounding the data system: user interfaces, developer experience, data ingestion, and interoperability with other systems. Part of Snowflake’s success stems from how well they handled these peripheral aspects. So, often what truly counts is the scaffolding built on top of the database management system itself. This isn’t research per se, but it’s something ClickHouse has been steadily improving over the years, helping expand the system’s applicability.

Original interview video link: https://www.youtube.com/watch?v=74X0DJOVEc8