Django, Celery: Display the progress bar of the current executing task

-
I'm using Celery (http://www.celeryproject.org/) to do the users's long-running task in Django. It eases the users by not let them wait the process done. When a user call the action (the view to execute a certain task), Django immediately response back to the user and it's done. The  job which has to be done was sent to the message broker where it will be actually executed.

We all know all of the things mentioned above. Question is "How does the user know when the task will finish?"

Here is the solution:

- In your task code, continuously updating state of the task (over a for loop) , define a custom state names PROGRESS
- Writing an infinite loop of Ajax call in your template to check the state of the current process and update the progress bar.

I also make a complete and simple example to implement the above idea, call testcele project (and create celery app inside testcele project). The main (and only) functionality of testcele is that it let users create 1000 model objects by clicking a button in the template and they can see the progress of the task:

- Project's requirement:

Django==1.5.1
MySQL-python==1.2.4
django-celery==3.0.17

===============================

- In settings.py:

...
# BROKER_URL = 'message_broker://user:password@hostname:port/virtual_host'
# message_broker --> rabbitmq = amqp
BROKER_URL = 'amqp://new_user:1q2w3e@localhost:5672/myvhost'
# List of modules to import when celery starts.
CELERY_IMPORTS = ("testcele.tasks",)
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "mysql://mydb_user:mydb_password@localhost/celery"
...

===============================

- models.py:

from django.db import models

class MyModel(models.Model):
fn = models.CharField(max_length=255)
ln = models.CharField(max_length=255)
def __unicode__(self):
return "MyModel<%s, %s>" % (self.fn, self.ln)


===============================

- tasks.py (place it in the same place with settings.py, testcele/tasks.py):

from celery import task, current_task
from celery.result import AsyncResult
from time import sleep
from testcele.cele import models


NUM_OBJ_TO_CREATE = 1000

 # when this task is called, it will create 1000 objects in the database
@task()
def create_models():
for i in range(1, NUM_OBJ_TO_CREATE+1):
fn = 'Fn %s' % i
ln = 'Ln %s' % i
my_model = models.MyModel(fn=fn, ln=ln)
my_model.save()
process_percent = int(100 * float(i) / float(NUM_OBJ_TO_CREATE))

sleep(0.1)
current_task.update_state(state='PROGRESS',
meta={'process_percent': process_percent})


===============================

- views.py:

from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponse
from django.utils import simplejson as json
from django.views.decorators.csrf import csrf_exempt

from celery.result import AsyncResult

from testcele import tasks


def home(request):
if 'task_id' in request.session.keys() and request.session['task_id']:
task_id = request.session['task_id']
return render_to_response('home.html', locals(), context_instance=RequestContext(request))


@csrf_exempt
def do_task(request):
""" A view the call the task and write the task id to the session """
data = 'Fail'
if request.is_ajax():
job = tasks.create_models.delay()
request.session['task_id'] = job.id
data = job.id
else:
data = 'This is not an ajax request!'
json_data = json.dumps(data)

return HttpResponse(json_data, mimetype='application/json')


@csrf_exempt
def poll_state(request):
""" A view to report the progress to the user """
data = 'Fail'
if request.is_ajax():
if 'task_id' in request.POST.keys() and request.POST['task_id']:
task_id = request.POST['task_id']
task = AsyncResult(task_id)
data = task.result or task.state
else:
data = 'No task_id in the request'
else:
data = 'This is not an ajax request'

json_data = json.dumps(data)

return HttpResponse(json_data, mimetype='application/json')


===============================

- urls.py:

from django.conf.urls import patterns, include, url

from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    url(r'^$', 'testcele.cele.views.home', name='home'),
    url(r'^do_task$', 'testcele.cele.views.do_task', name='do_task'),
    url(r'^poll_state$', 'testcele.cele.views.poll_state', name='poll_state'),

    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)


===============================

- home.html, we will write a javascript script to continuously check the state of the current task:

<!DOCTYPE html>
<html>
<head>
<title>Celery testing</title>
<script src="{{ STATIC_URL }}js/jquery-1.10.1.min.js"></script>
<style>
.progress {
width:50%;
background:yellow;
}
.bar {
height:15px;
width:0%;
background:tomato;
text-align:right;
}
</style>
</head>
<body>
<h1>Create 1000 objects in 1 click!</h1>
<div id="container">
<div id="action">
<button id="do-task">Click here!</button>
</div>
<div class="progress_container">
<div class="current-task">
<h4>{% if task_id %} Task ID: {{ task_id }} {% endif %}</h4>
</div>
<div class="status"></div>
{% if task_id %}
<div class="progress">
<div class="bar"></div>
</div>
{% endif %}
</div>
</div>
{% if task_id %}
<script type="text/javascript">
jQuery(document).ready(function() {
// pole state of the current task
var PollState = function(task_id) {
jQuery.ajax({
url: "poll_state",
type: "POST",
data: "task_id=" + task_id,
}).done(function(task){
console.log(task);
if (task.process_percent) {
jQuery('.bar').css({'width': task.process_percent + '%'});
jQuery('.bar').html(task.process_percent + '%')
} else {
jQuery('.status').html(task);
};
// create the infinite loop of Ajax calls to check the state
// of the current task
PollState(task_id);
});
}
PollState('{{ task_id }}');
});
</script>
{% endif %}
<script type="text/javascript">
jQuery('#do-task').click( function() {
jQuery.ajax({
url: "do_task",
data: {},
success: function(){
jQuery.ajax({
url: "",
context: document.body,
success: function(s, x) {
jQuery(this).html(s);
}
});
}
})
});
</script>
</body>
</html>



And if you run it, you will see something like:




You can download the full project here.


* Notes:  

In this example, I use 1000 (objects) as the total jobs which have to be done, so that we can count explicitly the progress (percentage) that has finished when the task is being executed . In some cases such as connect to a remote AD server and create an account,.., we cannot measure how long it takes to finish a single action or calculate the completed percent of the job to update state of the task for the user. My solution for those scenarios is just enable the celery to track the STARTED state of the task and inform the user that state. 

 - To enable celery to track the STARTED state of the task, put the following line to settings.py: CELERY_TRACK_STARTED = True

 - In the tasks.py, do not define the custom state as in this example

 - Set data = task.state (not = task.result or task.state) in views.poll_state function

 - Check task.state == 'STARTED' instead of task.process_percent in the javascript 



References: