The greatest pain in my local Celery development setup is the overhead of running and debugging multiple things and processes. Most of my Celery apps require at least three components:

• Web application (eg Django, FastAPI or Flask)

• Celery worker(s)

• Message broker

I have tweaked my local development setup many times, to be able to develop and debug without things getting in my way. After years of using Docker, I have now moved towards Visual Studio Code. My new setup is more productive than anything I have worked with before.

tl;dr

Check out my example code on GitHub and watch my YouTube Video.

Visual Studio Code

Visual Studio Code (VS Code) is a free, open-source code editor developed by Microsoft. VS Code is lightweight and powerful at the same time, extendible, supports a wide range of programming languages and works cross-platform.

Right out of the gate, VS Code comes with a Python debugger and full support for breakpoints, step-through code and variable inspection 🙌.

launch.json

All of this is fully customisable via a project-specific JSON configuration file, launch.json, which is located inside the .vscode folder, starting at the root of your project.

What does launch.json do for you? With launch.json you can define one or multiple processes that are available to run, alone or together, via VS Code's Run command. With or without debugging.

In the Celery case, this allows you to define the web app, the Celery worker and the message broker in one launch.json configuration. With a single "Run" command you can launch your stack, set breakpoints, debug and inspect variables. To make this work, you need three ingredients:

• configurations

• compounds

• tasks

Configurations

A launch configuration defines the environment setup for running and debugging a single process and includes the command, arguments and environment variables.

For example, the Celery worker process is defined in one launch configuration, and the Django app process in another one. Each launch configuration goes into the configurations property, which can hold as many configurations (processes) as you need.

Here is an example of the Celery and Django launch configurations. To run them individually, select "Run" -> "Start Debugging" (or "Run Without Debugging") from the VS Code menu. Or click the "Run and Debug" icon in the sidebar. You can then select Celery or Django from the dropdown.

// launch.json
{
    "version": "0.2.0",
    "configurations": [
      {
            "name": "Celery",
            "type": "python",
            "request": "launch",
            "module": "celery",
            "console": "integratedTerminal",
            "args": [
                "--app",
                "api.worker.app",
                "worker",
                "--loglevel",
                "DEBUG",
                "--pool",
                "solo",
            ]
        },
        {
            "name": "Django",
            "type": "python",
            "request": "launch",
            "stopOnEntry": false,
            "python": "${workspaceRoot}/venv/bin/python",
            "program": "${workspaceRoot}/manage.py",
              "args": [
                  "runserver"
              ],
        }        
    ],
    "compounds": [
        {
            "name": "Celery and Django",
            "configurations": ["Celery", "Django"]
        }
    ]}

Compounds

You can now run and debug Django and Celery individually. To run both configurations simultaneously, you need compounds. A compound configuration allows you to run multiple launch configurations together.

Once set up, compounds are available via the Run/Debug dropdown menu to choose from, just like configurations.

"compounds": [
        {
            "name": "Celery and Django",
            "configurations": ["Celery", "Django"]
        }
]

Message Broker

You can now run and debug the Django and Celery app processes There is one last thing missing to make this work, and that is the message broker. Here are three options to run, say, Redis as a broker:

1. Install Redis on the development computer

2. Run Redis inside a Docker container

3. Don't use Redis and use the filesystem instead 😁

I don't like installing dependencies on my computer. Using the filesystem as a broker is a dependency-free alternative. However, I prefer my development stack to mirror my production stack as closely as possible. That leaves me with the dockerised Redis option.

tasks.json

The VS Code launch.json configurations are mainly for debugging. What I want is to automatically start Redis before running/debugging and to automatically shut it down after everything else.

The launch configuration provides two properties to support that:

• preLaunchTask: run the start-redis-container task when starting up

• postDebugTask: run the stop-redis-container task when shutting down

// launch.json
{
       "name": "Django",
       "type": "python",
       "request": "launch",
       "stopOnEntry": false,
       "python": "${workspaceRoot}/venv/bin/python",
       "program": "${workspaceRoot}/manage.py",
          "args": [
             "runserver"
          ],
       "preLaunchTask": "start-redis-container",
       "postDebugTask": "stop-redis-container"
}

In case you do wonder why I attach both tasks to the Django and not the Celery container, there is no reason. It simply does not matter in this case.

What are tasks?start-redis-container and stop-redis-containerare defined in thetasks.json file, inside the .vscode subfolder. Tasks are commands, in my case start-redis-container and stop-redis-container are defined as docker run and docker rm commands, respectively.

// tasks.json
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "start-redis-container",
            "type": "shell",
            "command": "docker",
            "args": [
              "run",
              "-d",
              "--name",
              "redis",
              "-p",
              "6379:6379",
              "redis:latest"
            ],
          },
          {
            "label": "stop-redis-container",
            "type": "shell",
            "command": "docker",
            "args": [
              "rm",
              "-f",
              "redis"
            ],
          }

    ]
}

This gives you a fully functioning, debuggable Celery/Django/Redis development stack for VS Code.

Make sure to check out my example code, including launch.json and tasks.json.

Do you have a question or a comment? I love to hear from you, comment below 👇 or drop me an email: bjoern.stiel@celery.school.