Waymark 发布:基于 Postgres 的持久化工作流引擎
Waymark 是一个基于 Postgres 的持久化后台任务库,任务在服务器重启或崩溃后可恢复,无需额外基础设施。它采用“编译而非重放”的工作流执行模型,核心由 Rust 实现以并行调度节点,Python 为首个支持语言。
报道正文
本文目录2 个章节
waymark

waymark is a library to let you build durable background tasks that withstand server restarts, task crashes, and long-running jobs. It's built on Postgres, with no other infrastructure to deploy. Python is the first supported language; more are coming.
Documentation: waymark.sh
Getting started:
- Quickstart with Python - install, write a workflow, run it
- Why Waymark - the motivation and the workloads it's built for
Running Waymark:
- Configuration - every environment variable
- Webapp - the built-in view of your workflows and nodes
- Production Deployment - images, services, connections, scaling
Python:
- Workflows & Actions - the two primitives
- Control Flow - what a workflow body can contain, and its known issues
- Retries & Timeouts - per-call retry policies and timeouts
- Scheduled Workflows - cron and interval schedules
- AI Agents - wrapper around pydantic-ai for native agent declaration
Usage with Python
We ship all client and server binaries in one Python package. Install it via your package manager of choice:
uv add waymarkLet's say you need to send welcome emails to a batch of users, but only the active ones. You want to fetch them all, filter out inactive accounts, then fan out emails in parallel. Actions are the distributed work - plain async functions, sent to your workers:
from typing import Annotated
from waymark import Depends, action
@action
async def fetch_users(
user_ids: list[str],
db: Annotated[Database, Depends(get_db)],
) -> list[User]:
return await db.get_many(User, user_ids)
@action
async def send_email(
to: str,
subject: str,
emailer: Annotated[EmailClient, Depends(get_email_client)],
) -> EmailResult:
return await emailer.send(to=to, subject=subject)The workflow is the durable control flow that orchestrates them:
import asyncio
from waymark import Workflow, workflow
@workflow
class WelcomeEmailWorkflow(Workflow):
async def run(self, user_ids: list[str]) -> list[EmailResult]:
users = await fetch_users(user_ids)
active_users = [user for user in users if user.active]
results = await asyncio.gather(
*[
send_email(to=user.email, subject="Welcome")
for user in active_users
],
return_exceptions=False,
)
return resultsRun the node - the process that executes workflows and their actions - against your database:
export WAYMARK_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/waymark
export WAYMARK_HTTP_ENABLED=true # the webapp, on http://localhost:24119
uv run waymark-executorThen kick off the workflow from any Python process, and wait for its result:
async def welcome_users(user_ids: list[str]):
await WelcomeEmailWorkflow().run(user_ids)None of this executes inline in your webserver: the run is queued to Postgres and executed by the node.
Actions are the distributed work: network calls, database queries, anything that can fail and should be retried independently.
Workflows are the control flow: loops, conditionals, parallel branches. They orchestrate actions but don't do heavy lifting themselves.
What you can write
Workflows are plain async Python. A few of the things they can do:
- Retries and timeouts, per call. By default an action runs once, like a regular function call. Wrap a call in
self.run_action(...)to give it a retry policy and a timeout - see Retries & Timeouts.
from datetime import timedelta
from waymark import RetryPolicy
async def run(self, order_id: str) -> Receipt:
return await self.run_action(
charge_card(order_id),
retry=RetryPolicy(attempts=5, backoff_seconds=10),
timeout=timedelta(minutes=2),
)- Branches and loops.
if/elif/else,forandwhilecompile into the workflow program and are executed by the runtime, just like your actions.
async def run(self, order_ids: list[str]) -> list[str]:
shipped = []
for order_id in order_ids:
status = await fetch_status(order_id)
if status == "shipped":
shipped.append(order_id)
return shipped- Parallel fan-out and durable sleep.
asyncio.gatherruns actions in parallel;asyncio.sleeppauses the workflow durably, surviving restarts - for a second or for a day.
import asyncio
async def run(self, user_id: str) -> str:
profile, history = await asyncio.gather(
fetch_profile(user_id),
fetch_purchase_history(user_id),
return_exceptions=False,
)
# wait a day before following up
await asyncio.sleep(24 * 60 * 60)
return await send_recommendations(profile, history)- Schedules. Run a workflow on a cron expression or a fixed interval, with no extra infrastructure - see Scheduled Workflows.
await schedule_workflow(DataSyncWorkflow, schedule_name="hourly", schedule="0 * * * *")- A built-in webapp.
waymark-executorserves a webapp showing every workflow instance, each one's timeline of calls, and the load on your nodes.
(注:更多技术实现细节与完整 API 文档请查阅原项目 README)
作者发帖说明(HN):
Hey everyone! @mozgiii and I have been working on Waymark for the better part of the last year. Waymark’s an open source durable execution library with similar functionality to Temporal or DBOS. A few unique design decisions:
— Write jobs in normal code (right now Python but JS coming soon). Instead of running control flow as-is, which requires booting up an interpreter to replay past state, we’ll just compile your code directly. We walk your AST and map your logic to our stackless VM bytecode. We support variables, conditionals, await, sleep, helper functions, loops... basically what you’d expect in workflow logic.
— Since we’re compiling a language subset, we can proactively flag errors in your workflows locally instead of when they run in production.
— It only requires Postgres as the task broker. We’ve spent a lot of time tuning our design to play nicely with its strengths and limitations; our current benchmarks show around 1500 workflows/second but we still have some more optimization work to do here.
— Actions plug-and-play with FastAPI’s dependency injection syntax. So if you already have an ecosystem based on typehinting/DI, it should fit like a glove.
It allows for parallelization of workflows across one machine or a whole cluster. The bulk of the work per node is done by our central Rust core - which distributes workflows to properly saturate all cores. The idea is to be as batteries included as possible with the broker, webapp, and runtime.
What to use it for...