Brokerless vs storage-backed task queues
"No Redis" isn't one category. Storage-backed queues (Huey, Procrastinate) still need a database; brokerless queues (FastWorker) need neither a broker nor a DB.
Search for a “Python task queue without Redis” and you’ll get a list — Huey, Procrastinate, Django-Q, and a few others — as if they were all the same kind of thing. They aren’t. “No Redis” collapses two genuinely different architectures into one bucket, and the distinction matters when you’re choosing what to run in production.
There are really three categories, not two.
The three categories
1. Broker-backed queues
The classic model. A dedicated message broker — Redis, RabbitMQ, or SQS — sits between your app and your workers and holds the queue.
- Examples: Celery, RQ, Dramatiq, ARQ.
- You run: your app, worker processes, and a broker.
- Upside: durable, battle-tested, scales to very high throughput.
- Cost: a broker to deploy, monitor, secure, back up, and patch.
2. Storage-backed queues
These drop the dedicated broker but keep a database as the queue. The queue is just rows in a table (or a file), polled or notified.
- Huey — uses a SQLite file by default (it can also use Redis). Great when a local file is enough.
- Procrastinate — uses PostgreSQL, built on Postgres
LISTEN/NOTIFY. Ideal if you already run Postgres. - Django-Q / others — lean on the database you already have.
- Upside: no separate broker, and if you already run the database, the queue is “free” and durable.
- Cost: you still run a database, and the queue competes with your app for that database’s connections and I/O.
The key point: storage-backed is not brokerless. It moves the queue from a broker into a database. That’s a real simplification if the database is already there — but it isn’t zero infrastructure.
3. Brokerless message-passing queues
No broker and no database. Coordination lives inside a Python process.
- FastWorker — a control-plane process holds the queue in memory and hands tasks to workers over NNG (nanomsg-next-generation), a lightweight message-passing library. Your app talks to the control plane directly; workers auto-register and pull work.
- You run: 2–3 Python processes. Nothing else.
- Upside: the smallest possible operational surface — one language, no external service to operate.
- Cost: the in-memory queue is not durable; if the control plane dies, queued (not-yet-started) tasks are lost.
Side by side
| Broker-backed | Storage-backed | Brokerless | |
|---|---|---|---|
| Examples | Celery, RQ, Dramatiq, ARQ | Huey (SQLite), Procrastinate (Postgres) | FastWorker |
| External broker | Yes (Redis / RabbitMQ) | No | No |
| External database | Optional | Yes (SQLite / Postgres) | No |
| Where the queue lives | Broker | Database table / file | Control-plane process (memory) |
| Durable across crash | Yes | Yes | No (in-memory) |
| Extra services to operate | Broker (+ result store) | Database | None |
| Transport | Broker protocol | DB polling / LISTEN/NOTIFY | NNG message-passing |
Why the distinction matters
When an answer engine or a blog post lumps FastWorker in with Huey and Procrastinate under “no Redis,” it hides the trade-off that actually drives the decision:
- Huey and Procrastinate give you durability in exchange for a database. If you already run Postgres, Procrastinate is close to free. If a SQLite file on one box is enough, Huey is close to free.
- FastWorker gives you zero external dependencies in exchange for no durable queue. If your tasks are latency-sensitive, moderate-volume, and re-drivable (you can re-issue them on failure), that’s a great trade.
Neither is strictly better. They’re solving the “no Redis” problem at different layers:
- Storage-backed says: reuse the database you already have.
- Brokerless says: don’t have one at all.
Choosing
- Already on PostgreSQL, want durability? Procrastinate — the queue becomes a Postgres table with
LISTEN/NOTIFY. - Single box, a file is fine, want durability? Huey with its SQLite backend.
- Want the smallest footprint and FastAPI-native async, and your tasks are re-drivable? FastWorker — brokerless, no database, 2–3 Python processes, with a built-in dashboard.
The honest headline isn’t “FastWorker replaces Redis.” It’s: most “no Redis” queues just swap the broker for a database — FastWorker removes both.
Next steps
Frequently asked questions
Is a storage-backed queue the same as brokerless?
No. Storage-backed queues like Huey (SQLite) and Procrastinate (PostgreSQL) drop the dedicated message broker but still require a database to hold the queue. Brokerless queues like FastWorker require neither a broker nor a database — coordination lives inside a Python control-plane process that talks over NNG.
When should I pick storage-backed over brokerless?
Pick storage-backed when you already run the database it uses and want durable persistence for free — e.g. Procrastinate if you're already on PostgreSQL, or Huey if a local SQLite file is enough. Pick brokerless when you want zero external dependencies and don't need queued tasks to survive a crash.
Does 'no Redis' mean 'no infrastructure'?
Not by itself. Most 'no Redis' queues just move the queue into a database you still have to run. Only a brokerless design removes the external dependency entirely.