Glossary
Task-queue terms, defined.
The vocabulary of brokerless task queues and FastWorker's architecture, in plain English.
- Brokerless task queue
- A task queue that distributes work without a standalone message broker (Redis, RabbitMQ, Kafka). FastWorker coordinates tasks through a control-plane process and direct NNG messaging instead.
- How brokerless works →
- Control plane
- FastWorker's coordinator process. It accepts submitted tasks, dispatches them to subworkers, holds the result cache and worker registry, and serves the management dashboard. It also processes tasks itself, so a minimal deployment is a single process.
- Architecture →
- Subworker
- A worker process that registers with the control plane and executes tasks. Subworkers discover the control plane automatically on startup; adding more scales throughput horizontally with no reconfiguration.
- Automatic discovery →
- NNG (nanomsg-next-gen)
- A lightweight messaging library that provides scalability protocols such as REQ/REP and DEALER/ROUTER. FastWorker uses NNG for direct Python-to-Python transport between the control plane and subworkers.
- NNG control-plane, explained →
- Priority levels
- FastWorker has four built-in priorities — critical, high, normal, and low. Priority is an argument on the task, and dispatch always prefers the highest-priority pending work on the least-loaded subworker.
- Priority & load balancing →
- Least-loaded dispatch
- The routing strategy the control plane uses to pick a subworker for the next task: the worker with the fewest in-flight tasks at the required priority wins, spreading load evenly without manual routing rules.
- Architecture →
- Result cache
- An in-memory store on the control plane that keeps recent task results so callers can fetch them by task id. It defaults to 10,000 entries with a one-hour TTL and is configurable.
- Result caching →
- Task callback
- A hook that fires when a task finishes, so your app is notified instead of polling. Callbacks can drive websockets, SSE streams, or any async handler in a FastAPI app.
- Task callbacks →
- OpenTelemetry (OTel)
- A vendor-neutral standard for traces and metrics. FastWorker can emit OTel spans for task submission, dispatch, and execution to any OTLP collector such as Jaeger or Honeycomb.
- Observability with OpenTelemetry →
- Result backend
- In broker-based queues like Celery, a separate datastore that holds task results. FastWorker has no separate result backend — results live in the control plane result cache.
- FastWorker vs Celery →
- Async client
- FastWorker's client for submitting tasks from asynchronous Python code. Because it is async-native, it can be awaited directly inside FastAPI endpoints without a thread-pool wrapper.
- FastAPI integration →
- Storage-backed queue
- A queue that persists jobs in an external datastore (a database or Redis) for durability. FastWorker is primarily in-memory and brokerless; the trade-offs are covered in the brokerless-vs-storage-backed guide.
- Brokerless vs storage-backed →