November, 22 2023 - I am working on an updated version of this article, sign up for my newsletter to get notified when it's out!

Officially, Celery no longer supports Windows since Celery version 4. And while Celery 3 does support Windows, it is not compatible with Celery 4. So if you have to resort to Windows for some (one) of your Celery tasks, you are stuck with a legacy Celery version across your infrastructure. Which is certainly not an acceptable situation.

However, even though Celery dropped Windows support, I'll show you two simple workarounds to make Celery 4 play nicely on Windows.

The Celery concurrency pool

The first strategy to make Celery 4 run on Windows has to do with the concurrency pool. In a nutshell, the concurrency pool implementation determines how the Celery worker executes tasks in parallel.

Celery defaults to the prefork implementation which spawns processes (and is limited to a handful of processes per CPU), whereas eventlet spawns thread-like things called greenlets (hundreds of them, without breaking a sweat). Celery comes with several concurrency pool types:

• prefork

• eventlet

• gevent

• solo

• threads

The Prefork pool is better suited for CPU-bound tasks while the eventlet pool works better if you're I/O bound. If you want to read more about the different pool options, I recommend checking out my article about Celery pool types.

Strategy 1: Celery on Windows with eventlet, gevent or solo

What makes Celery 4 incompatible with Windows is just the default prefork concurrency pool implementation. In other words, if your Celery-job-to-be-done copes well with eventlet, gevent or solo (solo is a blocking single-threaded execution pool), you can run Celery 4 on Windows with any of these execution pools.

Clone my GitHub repository, create a virtual environment and install the pip requirements:

C:\Developer\win>activate venv
(venv) C:\Developer\win>pip install -r requirements.txt

You can start the Celery worker with any of these pool arguments:

(venv) C:\Developer\win>celery worker --app=app.app --pool=eventlet --loglevel=INFO
(venv) C:\Developer\win>celery worker --app=app.app --pool=gevent --loglevel=INFO
(venv) C:\Developer\win>celery worker --app=app.app --pool=solo --loglevel=INFO

Open a new command line window to execute a task asynchronously and your Celery worker is back in Windows business:

C:\Developer\win>activate venv
(venv) C:\Developer\cwin>python app.py

Strategy 2: FORKED_BY_MULTIPROCESSING

If we dig a bit deeper, it turns out that the reason the default prefork concurrency pool implementation does no longer work on Windows, is because of the Celery billiard package. Billiard itself is a fork of the Python multiprocessing package with some fixes and improvements.

Billiard used to set the not-so-well-documented environment variable FORKED_BY_MULTIPROCESSING=1 by default. Celery in turn checks if FORKED_BY_MULTIPROCESSING is set to determine whether forking is disabled (it's an OS thing).

Since the billiard version Celery 4 depends on, billiard no longer sets FORKED_BY_MULTIPROCESSING which in turn causes the prefork pool to fail on Windows (have a look at the prefork source code and billiard change log).

To cut a long story short, you can work around the problem by setting a Windows environment variable. Go to: System Properties => Environment Variables => User or System variables => New...:

Variable name: FORKED_BY_MULTIPROCESSING
Variable value: 1

Open a new command prompt window to pick up the new environment variable. You can start the Celery worker without the pool argument:

C:\Developer\win>activate venv
(venv) C:\Developer\win>celery worker --app=app.app --loglevel=INFO

Open a new command line window to execute a task asynchronously and your Celery worker just works with the default prefork pool (which
is forked by multiprocessing):

C:\Developer\win>activate venv
(venv) C:\Developer\win>python app.py