MiniLake Launches Local Databricks API Emulator

MiniLake is a local Databricks API emulator with real SQL, real Delta Lake and real Spark job execution. Point databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at http://localhost:8000 — no account, no API key, no cloud bill.

`pip install minilake && minilake --port 8000` PyPI docker run -p 8000:8000 ghcr.io/dmux/minilake:latest GHCR Get started See it in code

Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime — it works air-gapped.

Drop-in for databricks-sdk (Python) Databricks CLI Terraform provider Asset Bundles PySpark + Delta MCP agents *30+***Databricks API groups emulated, from Unity Catalog to SCIM 67MCP tools for LLM agents, served on the same port *378+***tests driven by the real Databricks SDK, not mocks $0cloud spend — everything runs on your machine What you get

Not a mock server. A real engine behind a Databricks-shaped API.

Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers — so a green test locally means something.

Unity Catalog, for real

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native — not a regex rewrite.

Real execution

SQL that actually runs

Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.

Real execution

Real Delta Lake files

EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().

Real Delta

Jobs that really execute

spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.

Real Spark

Unity Catalog protocol for Spark

spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector — the same protocol Spark speaks to a real workspace.

Wire-compatible

A built-in SQL workspace

An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history — and an embedded JupyterLab.

Batteries included

Terraform & Asset Bundles

Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty. databricks bundle deploy and bundle run work end to end — jobs execute for real.

Wire-compatible

An MCP server for agents

67 tools, 8 resources and 4 prompts at /mcp — including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.

Real execution

Designed for CI

One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.

Test friendly

Workspace, DBFS, Files, Secrets

File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment — exactly like Databricks.

Real execution

Air-gapped by design

DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime — provable with docker run --network none.

Offline ready

Databricks-shaped errors

Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.

Wire-compatible Zero code changes

Change the host. That's the migration.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working — the same calls, the same response shapes, the same typed errors.

• Any token is accepted. No sign-up, no PAT, no OAuth dance to develop against.
• Types round-trip. STRING, LONG, DECIMAL(10,2), ARRAY<T>, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo.
• Physical truth wins. Declared column types that disagree with the files lose, so mismatches are visible instead of believed.
• Persistence when you want it. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.

quickstart.py

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType

# The only line that differs from production
w = WorkspaceClient(host="http://localhost:8000", token="dev")

w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
    name="orders", catalog_name="sales", schema_name="store",
    table_type=TableType.MANAGED,
    data_source_format="DELTA", storage_location="/data/sales/store/orders",
    columns=[
        ColumnInfo(name="id", type_text="BIGINT"),
        ColumnInfo(name="customer", type_text="STRING"),
        ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
    ],
)

wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)

result = w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array)   # [['Ana', '459.90']] — really stored

from pyspark.sql import SparkSession

UC = "http://localhost:8000"

spark = (
    SparkSession.builder
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    # Unity Catalog connector — three-part names resolve against MiniLake
    .config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.spark_catalog.uri", UC)
    .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.sales.uri", UC)
    .getOrCreate()
)

# Real Delta files: parquet + _delta_log/, written by real Spark
spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
     .write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")

spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
print(spark.table("sales.store.events").count())   # 3

# ...and the same bytes are readable through MiniLake's SQL API:
#   SELECT e.name, o.customer
#   FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id

provider "databricks" {
  host  = "http://localhost:8000"
  token = "dev"
}

resource "databricks_catalog" "sandbox" {
  name    = "sandbox"
  comment = "Local sandbox catalog"
}

resource "databricks_schema" "things" {
  catalog_name = databricks_catalog.sandbox.name
  name         = "things"
}

resource "databricks_sql_endpoint" "compute" {
  name         = "local-compute"
  cluster_size = "Small"
}

# terraform apply  -> creates real catalogs and real DuckDB databases
# terraform destroy -> removes them
# Resources backed by APIs MiniLake does not emulate fail loudly with 501,
# instead of silently pretending to succeed.

bundle:
  name: my_pipeline

targets:
  dev:
    mode: development
    workspace:
      host: http://localhost:8000

resources:
  jobs:
    daily_load:
      name: daily_load
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ../src/load.py

# $ databricks bundle deploy -t dev
# $ databricks bundle run daily_load -t dev
#
# The CLI syncs files through the Workspace API, the provider creates the job,
# and `bundle run` executes it on real Spark in a sibling container —
# returning the job's actual output.

# Is it up, and which services answered?
curl http://localhost:8000/_minilake/health

# Create a catalog through the Databricks API surface
curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
  -H 'Content-Type: application/json' \
  -d '{"name": "sales"}'

# Run SQL and get real rows back
curl -X POST http://localhost:8000/api/2.0/sql/statements \
  -H 'Content-Type: application/json' \
  -d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'

# Give the next test a clean workspace — no restart needed
curl -X POST http://localhost:8000/_minilake/reset

How it works

One process. One port. Real engines underneath.

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

For LLM agents

Give an agent a lakehouse it can't break.

A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps — and it costs nothing when the agent gets it wrong.

• setup_fixture()catalog + schema + warehouse in one idempotent call
• seed_table()table + rows, exactly the rows you asked for
• describe_catalog_tree()the whole hierarchy instead of dozens of calls
• run_python_script()stage, submit to real Spark, poll, return stdout

Plus 8 resources — including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB — and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.

MCP documentation All 67 tools register + use

# One line to register it with Claude Code
claude mcp add --transport http minilake http://localhost:8000/mcp

# Then just ask:
#   "Use minilake: create a sales catalog with a seeded
#    orders table, and tell me the total."

# The agent does this — three calls, no glue code:
#   setup_fixture({"catalog": "sales", "schema": "store"})
#   seed_table({"full_name": "sales.store.orders", ...})
#   run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})

# Enable it on the server side:
MINILAKE_MCP=1 minilake --port 8000

Coverage

What's emulated, and how honestly.

Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.

Straight talk

What MiniLake is not.

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally — not a shared, multi-tenant server.

No authentication

Any token is accepted and there is exactly one user. Never expose it beyond localhost.

No access control

The Permissions API is real CRUD that always allows everything — a passing test here says nothing about grants in a real workspace.

DuckDB, not Spark SQL

Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

No Spark on clusters

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers.

Single process, no HA

DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

MANAGED tables are invisible to Spark

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data — that trade-off is what keeps SQL fast.

Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.

Your lakehouse, on localhost, in one command.

Free and MIT-licensed. No account, no API key, no cloud bill — and nothing to uninstall from your cloud account when you're done.