Celery and Sentry - Recording Errors

As part of improving our infrastructure for Conversocial we wanted to add django-sentry. This little app groups together similar errors and makes diagnosing problems far easier. It integrates with django seamlessly, but, it needs a bit of work to get celery sending errors to it.

This is not quite as simple as I first thought. After a bit of experimentation I found the following in our tasks.py worked:

# Hook up sentry to celery's logging
import logging 
from celery.signals import task_failure 
from sentry.client.handlers import SentryHandler

logger = logging.getLogger('task')
logger.addHandler(SentryHandler()) 
def process_failure_signal(exception, traceback, sender, task_id, 
                           signal, args, kwargs, einfo, **kw): 
  exc_info = (type(exception), exception, traceback) 
  logger.error( 
    'Celery job exception: %s(%s)' % (exception.__class__.__name__, exception), 
    exc_info=exc_info, 
    extra={ 
      'data': { 
        'task_id': task_id, 
        'sender': sender, 
        'args': args, 
        'kwargs': kwargs, 
      } 
    } 
  ) 
task_failure.connect(process_failure_signal) 

This is based on some code from the Celery user group. The main difference is that instead of adding the SentryHandler to the celery logger I define my own logger. I do this because I found numerous issues when trying to add it to the celery logger, including:

• Double-recording of the errors in Sentry
• The task ID appearing in the message in Sentry - eliminating Sentry's ability to group messages
• Celery's info/warning messages came through to Sentry - we use Splunk for checking our logs so wanted to get just the errors

Thought I'd share this nugget for anyone else who tries to get this working and hits problems :)

Discussion