Celery on Windows: What's the latest?
Celery dropped support for Windows a long time ago, somewhere around version 4. Celery 3 was the last version to support Windows, and back in 2018, when I published the first version of this article, it was a somewhat acceptable workaround.
Not these days. So if you still have the odd Celery task that needs to run on Windows, what are your options in 2024? Time for a reality check. Well, sort of, the reality is that Windows is unsupported and things might work or might not. Or they might work now and stop working later.
Before you do anything, make sure to pip-install pywin32
The prefork Problem
Celery never really stopped working on Windows. What stopped working was the prefork pool, which is Celery's default pool. The prefork pool is similar to a Python multiprocessing pool but based on the billiard package to cater better for Celery-specific needs.
To cut a long story short: the prefork pool is based on the idea of process forking (hence the name prefork pool and not multiprocessing pool). Windows does not support forking but spawning.
For many years, billiard had a switch deep down in its engine room that would make it either fork or spawn, depending on the operating system. That got removed, and that sealed the end of the prefork pool on Windows.
For some time, there was a hacky way around this, by setting the environment variable FORKED_BY_MULTIPROCESSING:
import os
from celery import Celery
os.environ.setdefault("FORKED_BY_MULTIPROCESSING", "1")
app = Celery(__name__, broker="redis://localhost:6379/0")
This no longer works and you do not get what you ask for. Celery creates a new SpawnPoolWorker for each incoming task, you can see incoming tasks being received but it fails to process them.
Option 1: Going solo
If the prefork pool is out of bounds, you can still choose another pool type. The solo pool is a simple, single-threaded execution pool. It is so simple, it's not even an execution pool. It simply executes incoming tasks in the same process and thread as the worker.
C:\Dev>celery --app=app.app worker --pool=solo
This still works on Windows and is my recommendation for a CPU-heavy workload. If you need more concurrency, you can create multiple solo workers.
Option 2: Threads
The threads in the threads pool type are managed directly by the operating system kernel. As long as Python's ThreadPoolExecutor supports Windows threads, this pool type will work on Windows (I probably shouldn't say that). The threads pool is my recommendation for I/O-bound tasks.
C:\Dev>celery --app=app.app worker --pool=threads --concurrency=8
Option 3: gevent
The gevent pool implements concurrency via greenlets. Greenlets are similar to coroutines in the asyncio world. While one task waits for its result, it yields to another task to do its thing.
Because the gevent package officially supports Windows, the gevent pool remains a suitable option for IO-bound task processing on Windows as long as you are aware of some gevent-specific complexities.
C:\Dev>celery --app=app.app worker --pool=gevent --concurrency=8
Conclusion
In this article, I've looked at three Celery pool options that still work on Windows in 2024:
solo
threads
gevent
Bear in mind that none of this is officially supported and that whatever works now might not be working in the future.
Last updated Dec 20, 2023
First published Aug 21, 2018
