# Filesystem As Broker: A Quick Guide

Using Redis or RabbitMQ as a Celery message broker can feel a bit over the top when you are just developing locally. In this blog post, I show you a simple alternative: the filesystem as a message broker.

## **Kombu**

Kombu is a messaging library that provides a high-level interface for the AMQ protocol (with RabbitMQ being one of the best-known implementations). Celery uses Kombu to send and receive messages, so any support for a particular broker comes down to Kombu and not Celery.

The message broker is the store which interacts as the transport between the producer and consumer of messages. Redis, RabbitMQ and [Amazon SQS](https://celery.school/amazon-sqs-celery-broker) are among the most widely used message brokers.

Kombu also comes with support for File-system transport. The producer and consumer communicate via files. When the producer creates a new message, it writes the message content to a file in a particular directory. The consumer listens for new files in this directory and processes them as they are created.

## **Celery configuration**

Documentation on the file-system transport is a bit sparse, but the setup is straightforward. Use `filesystem://` as the `broker`. The actual directory that is used for exchanging messages goes into the `broker_transport_options` argument:

```python
from celery import Celery

app = Celery(
    __name__,
    broker_url='filesystem://',
    broker_transport_options={
        'data_folder_in': './.data/broker',
        'data_folder_out': './.data/broker/',
    })
```

Note that:

* `data_folder_in` and `data_folder_out` point to the same path
    
* Celery will not create any directories, it is your responsibility to ensure that `/.data/broker` exists
    

## **A simple example**

Clone the example [GitHub repository](https://github.com/bstiel/celery-filesystem-broker) and follow the installation instructions:

```bash
$ git clone https://github.com/bstiel/celery-filesystem-broker.git
$ python -m venv venv
$ venv/bin/activate
$ pip install -r requirements.txt
```

Start the Celery worker:

```bash
# start celery worker
$ celery --app=worker.app worker --loglevel=INFO
```

Start the producer:

```bash
# start producer
$ python producer.py
```

The producer creates a `long_running_task` task every two seconds. This gets transported to the `./.data/broker` directory from which the worker picks it up for processing. The Celery logs should show something like this:

```bash
[2023-12-19 16:44:52,105: INFO/ForkPoolWorker-1] Task long_running_task[a31af3f0-78b8-499f-ae1b-a277c2319bbf] succeeded in 0.15355589999671793s: 501.67632
```

Hope you find this useful. Any questions? Please comment below 👇 or drop me an email bjoern.stiel@celery.school.

*Last updated Dec 19, 2023  
First published Jul 3, 2018*