Using Redis or RabbitMQ as a Celery message broker can feel a bit over the top when you are just developing locally. In this blog post, I show you a simple alternative: the filesystem as a message broker.
Kombu
Kombu is a messaging library that provides a high-level interface for the AMQ protocol (with RabbitMQ being one of the best-known implementations). Celery uses Kombu to send and receive messages, so any support for a particular broker comes down to Kombu and not Celery.
The message broker is the store which interacts as the transport between the producer and consumer of messages. Redis, RabbitMQ and Amazon SQS are among the most widely used message brokers.
Kombu also comes with support for File-system transport. The producer and consumer communicate via files. When the producer creates a new message, it writes the message content to a file in a particular directory. The consumer listens for new files in this directory and processes them as they are created.
Celery configuration
Documentation on the file-system transport is a bit sparse, but the setup is straightforward. Use filesystem://
as the broker
. The actual directory that is used for exchanging messages goes into the broker_transport_options
argument:
from celery import Celery
app = Celery(
__name__,
broker_url='filesystem://',
broker_transport_options={
'data_folder_in': './.data/broker',
'data_folder_out': './.data/broker/',
})
Note that:
data_folder_in
anddata_folder_out
point to the same pathCelery will not create any directories, it is your responsibility to ensure that
/.data/broker
exists
A simple example
Clone the example GitHub repository and follow the installation instructions:
$ git clone https://github.com/bstiel/celery-filesystem-broker.git
$ python -m venv venv
$ venv/bin/activate
$ pip install -r requirements.txt
Start the Celery worker:
# start celery worker
$ celery --app=worker.app worker --loglevel=INFO
Start the producer:
# start producer
$ python producer.py
The producer creates a long_running_task
task every two seconds. This gets transported to the ./.data/broker
directory from which the worker picks it up for processing. The Celery logs should show something like this:
[2023-12-19 16:44:52,105: INFO/ForkPoolWorker-1] Task long_running_task[a31af3f0-78b8-499f-ae1b-a277c2319bbf] succeeded in 0.15355589999671793s: 501.67632
Hope you find this useful. Any questions? Please comment below ๐ or drop me an email bjoern.stiel@celery.school.
Last updated Dec 19, 2023
First published Jul 3, 2018