Filesystem As Broker: A Quick Guide

Using Redis or RabbitMQ as a Celery message broker can feel a bit over the top when you are just developing locally. In this blog post, I show you a simple alternative: the filesystem as a message broker.

Kombu

Kombu is a messaging library that provides a high-level interface for the AMQ protocol (with RabbitMQ being one of the best-known implementations). Celery uses Kombu to send and receive messages, so any support for a particular broker comes down to Kombu and not Celery.

The message broker is the store which interacts as the transport between the producer and consumer of messages. Redis, RabbitMQ and Amazon SQS are among the most widely used message brokers.

Kombu also comes with support for File-system transport. The producer and consumer communicate via files. When the producer creates a new message, it writes the message content to a file in a particular directory. The consumer listens for new files in this directory and processes them as they are created.

Celery configuration

Documentation on the file-system transport is a bit sparse, but the setup is straightforward. Use filesystem:// as the broker. The actual directory that is used for exchanging messages goes into the broker_transport_options argument:

from celery import Celery

app = Celery(
    __name__,
    broker_url='filesystem://',
    broker_transport_options={
        'data_folder_in': './.data/broker',
        'data_folder_out': './.data/broker/',
    })

Note that:

• data_folder_in and data_folder_out point to the same path

• Celery will not create any directories, it is your responsibility to ensure that /.data/broker exists

A simple example

Clone the example GitHub repository and follow the installation instructions:

$ git clone https://github.com/bstiel/celery-filesystem-broker.git
$ python -m venv venv
$ venv/bin/activate
$ pip install -r requirements.txt

Start the Celery worker:

# start celery worker
$ celery --app=worker.app worker --loglevel=INFO

Start the producer:

# start producer
$ python producer.py

The producer creates a long_running_task task every two seconds. This gets transported to the ./.data/broker directory from which the worker picks it up for processing. The Celery logs should show something like this:

[2023-12-19 16:44:52,105: INFO/ForkPoolWorker-1] Task long_running_task[a31af3f0-78b8-499f-ae1b-a277c2319bbf] succeeded in 0.15355589999671793s: 501.67632

Hope you find this useful. Any questions? Please comment below 👇 or drop me an email bjoern.stiel@celery.school.

Last updated Dec 19, 2023
First published Jul 3, 2018