FastWorker
GitHub
Architecture

The brokerless control plane.

FastWorker ditches the external broker entirely. Here's how it coordinates distributed Python work with just two or three processes.

FastWorker control plane architecture FastAPI App Client.delay(...) Control Plane Router · Dispatcher Result Cache · GUI Worker Registry Subworker 1 Subworker 2 Subworker N TRANSPORT: NNG (pynng) • REQ/REP for task submission · DEALER/ROUTER for worker dispatch • PUB/SUB for heartbeats and discovery · Pure in-process Python • No external message broker — no Redis, no RabbitMQ, no Kafka

Three roles, one protocol

FastWorker has three participants. The client lives inside your application and submits tasks. The control plane is a standalone Python process that coordinates everything. Subworkers are optional Python processes that expand execution capacity.

All communication happens over NNG (via pynng), a lightweight messaging library. FastWorker uses REQ/REP for client submissions, DEALER/ROUTER for dispatch to subworkers, and PUB/SUB for worker registration and heartbeats. None of this requires an external broker — every NNG socket is just a TCP endpoint hosted by a Python process.

The control plane does the work

Unlike Celery's architecture — where the broker is passive and workers poll it — the FastWorker control plane is active. It maintains the worker registry, tracks load per subworker, dispatches tasks to the least-loaded node, holds the LRU result cache, and serves the web dashboard. It will also process tasks itself if subworkers are unavailable, which gives you a meaningful minimum viable deployment of one process.

Priority dispatch

Tasks are tagged with one of four priorities: critical, high, normal, low. The dispatcher drains higher priorities first and uses a load-balanced DEALER pattern to pick the least-busy subworker. No routing rules, no separate queues per priority, no broker configuration.

Result caching

When a task finishes, the control plane stores its return value in an LRU result cache (default 10,000 entries, 1-hour TTL). Clients can poll the result asynchronously or subscribe to completion callbacks. Expired entries are reaped on a one-minute tick. Cache size and TTL are configurable via --result-cache-size and --result-cache-ttl.

Failure modes

If a subworker dies, heartbeats stop arriving and the control plane removes it from the registry; in-flight tasks on that worker fail and can be retried. If the control plane itself dies, clients cannot submit and subworkers idle until it's restarted. FastWorker is designed for teams that prefer operational simplicity over multi-broker high availability — if you need the latter, the Celery comparison lays out the tradeoffs.

Why NNG instead of a broker?

A broker is great when you need durable queues, cross-language producers, or exactly-once delivery. Most Python applications don't. They need a task runner that lives alongside the app, scales horizontally, and doesn't turn into its own on-call rotation. NNG gives us the messaging primitives without the operational surface area.