Filesystem As Broker: A Quick Guide
Using Redis or RabbitMQ as a Celery message broker can feel a bit over the top when you are just developing locally. In this blog post, I show you a simple alternative: the filesystem as a message broker.
Kombu is a messaging library that provides a high-level interface for the AMQ protocol (with RabbitMQ being one of the best-known implementations). Celery uses Kombu to send and receive messages, so any support for a particular broker comes down to Kombu and not Celery.
The message broker is the store which interacts as the transport between the producer and consumer of messages. Redis, RabbitMQ and Amazon SQS are among the most widely used message brokers.
Kombu also comes with support for File-system transport. The producer and consumer communicate via files. When the producer creates a new message, it writes the message content to a file in a particular directory. The consumer listens for new files in this directory and processes them as they are created.
Documentation on the file-system transport is a bit sparse, but the setup is straightforward. Use
filesystem:// as the
broker. The actual directory that is used for exchanging messages goes into the
from celery import Celery
app = Celery(
data_folder_outpoint to the same path
Celery will not create any directories, it is your responsibility to ensure that
A simple example
Clone the example GitHub repository and follow the installation instructions:
$ git clone https://github.com/bstiel/celery-filesystem-broker.git
$ python -m venv venv
$ pip install -r requirements.txt
Start the Celery worker:
# start celery worker
$ celery --app=worker.app worker --loglevel=INFO
Start the producer:
# start producer
$ python producer.py
The producer creates a
long_running_task task every two seconds. This gets transported to the
./.data/broker directory from which the worker picks it up for processing. The Celery logs should show something like this:
[2023-12-19 16:44:52,105: INFO/ForkPoolWorker-1] Task long_running_task[a31af3f0-78b8-499f-ae1b-a277c2319bbf] succeeded in 0.15355589999671793s: 501.67632
Hope you find this useful. Any questions? Please comment below 👇 or drop me an email email@example.com.
Last updated Dec 19, 2023
First published Jul 3, 2018
Did you find this article valuable?
Support Bjoern Stiel by becoming a sponsor. Any amount is appreciated!