Task Queues: A Celery Story

At my work, we’ve used Celery as the foundation of our task queueing/tracking system for years now, but we finally reached a point where the quirks of Celery are too much. Because of this, I had to spend two weeks at work researching different task queue systems, and have since implemented two options other than Celery that are about to enter production use. In the time doing this, I’ve gathered useful insights about several task queueing libraries and their uses.

I will introduce the use case for task queues, and explicitly point out when they’re not applicable. I will then give a basic overview of the Celery library and its use case, but will then cover where it fell short in our specific use case. I will then point out ways to deal with the particular failings of Celery, and focus on how we solved our issues (using RQ) and the differences between the Celery and RQ model.

The talk will assume a solid understanding of Python, though nothing more advanced than decorators will be covered.