Waymark 发布:基于 Postgres 的持久化工作流引擎

原标题:Show HN: Waymark – A workflow engine that compiles jobs instead of replaying
58AI 研判分

Waymark 是一个基于 Postgres 的持久化后台任务库,任务在服务器重启或崩溃后可恢复,无需额外基础设施。它采用“编译而非重放”的工作流执行模型,核心由 Rust 实现以并行调度节点,Python 为首个支持语言。

报道正文

本文目录2 个章节

waymark

Waymark Logo

waymark is a library to let you build durable background tasks that withstand server restarts, task crashes, and long-running jobs. It's built on Postgres, with no other infrastructure to deploy. Python is the first supported language; more are coming.

Documentation: waymark.sh

Getting started:

Running Waymark:

Python:

Usage with Python

We ship all client and server binaries in one Python package. Install it via your package manager of choice:

BASH
uv add waymark

Let's say you need to send welcome emails to a batch of users, but only the active ones. You want to fetch them all, filter out inactive accounts, then fan out emails in parallel. Actions are the distributed work - plain async functions, sent to your workers:

PYTHON
from typing import Annotated

from waymark import Depends, action

@action
async def fetch_users(
    user_ids: list[str],
    db: Annotated[Database, Depends(get_db)],
) -> list[User]:
    return await db.get_many(User, user_ids)

@action
async def send_email(
    to: str,
    subject: str,
    emailer: Annotated[EmailClient, Depends(get_email_client)],
) -> EmailResult:
    return await emailer.send(to=to, subject=subject)

The workflow is the durable control flow that orchestrates them:

PYTHON
import asyncio

from waymark import Workflow, workflow

@workflow
class WelcomeEmailWorkflow(Workflow):
    async def run(self, user_ids: list[str]) -> list[EmailResult]:
        users = await fetch_users(user_ids)
        active_users = [user for user in users if user.active]

        results = await asyncio.gather(
            *[
                send_email(to=user.email, subject="Welcome")
                for user in active_users
            ],
            return_exceptions=False,
        )

        return results

Run the node - the process that executes workflows and their actions - against your database:

BASH
export WAYMARK_DATABASE_URL=postgresql://postgres:postgres@localhost:5432/waymark
export WAYMARK_HTTP_ENABLED=true  # the webapp, on http://localhost:24119
uv run waymark-executor

Then kick off the workflow from any Python process, and wait for its result:

PYTHON
async def welcome_users(user_ids: list[str]):
    await WelcomeEmailWorkflow().run(user_ids)

None of this executes inline in your webserver: the run is queued to Postgres and executed by the node.

Actions are the distributed work: network calls, database queries, anything that can fail and should be retried independently.

Workflows are the control flow: loops, conditionals, parallel branches. They orchestrate actions but don't do heavy lifting themselves.

What you can write

Workflows are plain async Python. A few of the things they can do:

  1. Retries and timeouts, per call. By default an action runs once, like a regular function call. Wrap a call in self.run_action(...) to give it a retry policy and a timeout - see Retries & Timeouts.
PYTHON
    from datetime import timedelta

    from waymark import RetryPolicy

    async def run(self, order_id: str) -> Receipt:
        return await self.run_action(
            charge_card(order_id),
            retry=RetryPolicy(attempts=5, backoff_seconds=10),
            timeout=timedelta(minutes=2),
        )
  1. Branches and loops. if/elif/else, for and while compile into the workflow program and are executed by the runtime, just like your actions.
PYTHON
    async def run(self, order_ids: list[str]) -> list[str]:
        shipped = []
        for order_id in order_ids:
            status = await fetch_status(order_id)
            if status == "shipped":
                shipped.append(order_id)
        return shipped
  1. Parallel fan-out and durable sleep. asyncio.gather runs actions in parallel; asyncio.sleep pauses the workflow durably, surviving restarts - for a second or for a day.
PYTHON
    import asyncio

    async def run(self, user_id: str) -> str:
        profile, history = await asyncio.gather(
            fetch_profile(user_id),
            fetch_purchase_history(user_id),
            return_exceptions=False,
        )

        # wait a day before following up
        await asyncio.sleep(24 * 60 * 60)
        return await send_recommendations(profile, history)
  1. Schedules. Run a workflow on a cron expression or a fixed interval, with no extra infrastructure - see Scheduled Workflows.
PYTHON
    await schedule_workflow(DataSyncWorkflow, schedule_name="hourly", schedule="0 * * * *")
  1. A built-in webapp. waymark-executor serves a webapp showing every workflow instance, each one's timeline of calls, and the load on your nodes.

(注:更多技术实现细节与完整 API 文档请查阅原项目 README)


作者发帖说明(HN):

Hey everyone! @mozgiii and I have been working on Waymark for the better part of the last year. Waymark’s an open source durable execution library with similar functionality to Temporal or DBOS. A few unique design decisions:

— Write jobs in normal code (right now Python but JS coming soon). Instead of running control flow as-is, which requires booting up an interpreter to replay past state, we’ll just compile your code directly. We walk your AST and map your logic to our stackless VM bytecode. We support variables, conditionals, await, sleep, helper functions, loops... basically what you’d expect in workflow logic.

— Since we’re compiling a language subset, we can proactively flag errors in your workflows locally instead of when they run in production.

— It only requires Postgres as the task broker. We’ve spent a lot of time tuning our design to play nicely with its strengths and limitations; our current benchmarks show around 1500 workflows/second but we still have some more optimization work to do here.

— Actions plug-and-play with FastAPI’s dependency injection syntax. So if you already have an ecosystem based on typehinting/DI, it should fit like a glove.

It allows for parallelization of workflows across one machine or a whole cluster. The bulk of the work per node is done by our central Rust core - which distributes workflows to properly saturate all cores. The idea is to be as batteries included as possible with the broker, webapp, and runtime.

What to use it for...