What is Celery?

Celery is a distributed task queue that allows us to execute jobs in background. This article explains how to set up Celery with Django to perform a background task.

Advantages:

• Large or small, Celery makes scheduling such periodic tasks easy.
• You never want end users to have to wait unnecessarily for pages to load or actions to complete. If a long process is part of your application’s workflow, you can use Celery to execute that process in the background, as resources become available, so that your application can continue to respond to client requests.

Celery uses brokers to pass messages between a Django Project and the Celery workers. We will use Redis as the message broker.

Installation

Before diving into Celery, follow the below setup points

Create a new virtualenv ‘venv’ using following command:

To activate the environment use command:

Install django and create a django project ‘myproject’. Make sure to activate a virtualenv, create a requirements.txt file and run the migrations. Then fire up the server and navigate to http://localhost:8000/ in your browser. You should see the familiar “Congratulations on your first Django-powered page” text. When done, kill the sever.

Let’s install Celery:

Now we will integrate Celery into our Django project with the following steps:

Step 1:

Inside the myproject directory i.e beside your settings.py create a new file called celery.py and add the following code in that:

Let’s break down what happens in the first module, first we import absolute imports from the future, so that our celery.py module will not clash with the library:

Then we set the default DJANGO_SETTINGS_MODULE for the celery command-line program:

Specifying the settings here means the celery command line program will know where your Django project is. This statement must always appear before the app instance is created, which is what we do next:

This is your instance of the library, you can have many instances but there’s probably no reason for that when using Django.

We also add the Django settings module as a configuration source for Celery. This means that you don’t have to use multiple configuration files, and instead configure Celery directly from the Django settings.

You can pass the object directly here, but using a string is better since then the worker doesn’t have to serialize the object when using Windows or execv:

Next, a common practice for reusable apps is to define all tasks in a separate tasks.py module, and Celery does have a way to autodiscover these modules:

Step 2:

To ensure that the Celery app is loaded when Django starts, add the following code into the __init__.py file that sits next to your settings.py file:

Project layout should look like:

Step 3:

Celery uses “brokers” to pass messages between a Django Project and the Celery workers. In this article, we will use Redis as the message broker.

First, install Redis from the official download page and then turn to your terminal, in a new terminal window, fire up the server:

You can test that Redis is working properly by typing this into your terminal:

Redis should reply with PONG – try it!

Once Redis is up, add the following code to your settings.py file:

You also need to add Redis as a dependency in the Django Project:

Test that the Celery worker is ready to receive tasks:

Kill the process with CTRL-C. Now, test that the Celery task scheduler is ready for action:

That’s it! You can now use Celery with Django. For more information on setting up Celery with Django, please check out the official Celery documentation.