======= Logging ======= .. module:: django.utils.log :synopsis: Logging tools for Django applications Overview ======== Django uses Python's builtin :mod:`logging` module to perform system logging. The usage of this module is discussed in detail in Python's own documentation. However, if you've never used Python's logging framework (or even if you have), here's a quick primer. The cast of players ------------------- A Python logging configuration consists of four parts: * :ref:`topic-logging-parts-loggers` * :ref:`topic-logging-parts-handlers` * :ref:`topic-logging-parts-filters` * :ref:`topic-logging-parts-formatters` .. _topic-logging-parts-loggers: Loggers ~~~~~~~ A *logger* is the entry point into the logging system. Each logger is a named bucket to which messages can be written for processing. A logger is configured to have a *log level*. This log level describes the severity of the messages that the logger will handle. Python defines the following log levels: * ``DEBUG``: Low level system information for debugging purposes * ``INFO``: General system information * ``WARNING``: Information describing a minor problem that has occurred. * ``ERROR``: Information describing a major problem that has occurred. * ``CRITICAL``: Information describing a critical problem that has occurred. Each message that is written to the logger is a *Log Record*. Each log record also has a *log level* indicating the severity of that specific message. A log record can also contain useful metadata that describes the event that is being logged. This can include details such as a stack trace or an error code. When a message is given to the logger, the log level of the message is compared to the log level of the logger. If the log level of the message meets or exceeds the log level of the logger itself, the message will undergo further processing. If it doesn't, the message will be ignored. Once a logger has determined that a message needs to be processed, it is passed to a *Handler*. .. _topic-logging-parts-handlers: Handlers ~~~~~~~~ The *handler* is the engine that determines what happens to each message in a logger. It describes a particular logging behavior, such as writing a message to the screen, to a file, or to a network socket. Like loggers, handlers also have a log level. If the log level of a log record doesn't meet or exceed the level of the handler, the handler will ignore the message. A logger can have multiple handlers, and each handler can have a different log level. In this way, it is possible to provide different forms of notification depending on the importance of a message. For example, you could install one handler that forwards ``ERROR`` and ``CRITICAL`` messages to a paging service, while a second handler logs all messages (including ``ERROR`` and ``CRITICAL`` messages) to a file for later analysis. .. _topic-logging-parts-filters: Filters ~~~~~~~ A *filter* is used to provide additional control over which log records are passed from logger to handler. By default, any log message that meets log level requirements will be handled. However, by installing a filter, you can place additional criteria on the logging process. For example, you could install a filter that only allows ``ERROR`` messages from a particular source to be emitted. Filters can also be used to modify the logging record prior to being emitted. For example, you could write a filter that downgrades ``ERROR`` log records to ``WARNING`` records if a particular set of criteria are met. Filters can be installed on loggers or on handlers; multiple filters can be used in a chain to perform multiple filtering actions. .. _topic-logging-parts-formatters: Formatters ~~~~~~~~~~ Ultimately, a log record needs to be rendered as text. *Formatters* describe the exact format of that text. A formatter usually consists of a Python formatting string containing :ref:`LogRecord attributes `; however, you can also write custom formatters to implement specific formatting behavior. .. _logging-how-to: How to use logging ================== Django provides a :ref:`default logging configuration `, that for example generates the messages that appear in the console when using the :djadmin:`runserver`. Make a basic logging call ------------------------- Python programmers will often use ``print()`` in their code as a quick and convenient debugging tool. Using the logging framework is only a little more effort than that, but it's much more elegant and flexible. As well as being useful for debugging, logging can also provide you with more - and better structured - information about the state and health of your application. To send a log message from within your code, you place a logging call into it. .. admonition:: Don't be tempted to use logging calls in ``settings.py`` The way that Django logging is configured as part of the ``setup()`` function means that logging calls placed in ``settings.py`` may not work as expected, because *logging will not be set up at that point*. To explore logging, use a view function as suggested in the example below. First, import the Python logging library, and then obtain a logger instance with :py:func:`logging.getLogger`. The ``getLogger()`` method must be provided with a name. A good option is to use ``__name__``, which will provide the name of the current Python module (see :ref:`naming-loggers` for use of explicit naming):: import logging logger = logging.getLogger(__name__) And then in a function, for example in a view, send a message to the logger:: def some_view(request): ... if some_risky_state: logger.warning('Platform is running at risk') When this code is executed, that message will be sent to the logger (and if you're using Django's default logging configuration, it will appear in the console). The ``WARNING`` level used in the example above is one of several :ref:`logging severity levels `: ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``, ``CRITICAL``. So, another example might be:: logger.critical('Payment system is not responding') The default logging configuration, which Django inherits from the Python logging module, prints all messages of level ``WARNING`` and higher to the console. Django's own defaults will *not* pass ``INFO`` or lower severity messages from applications other than Django itself to the console - that will need to be configured explicitly. .. _naming-loggers: Naming loggers -------------- The call to :func:`logging.getLogger()` obtains (creating, if necessary) an instance of a logger. The logger instance is identified by a name. This name is used to identify the logger for configuration purposes. By convention, the logger name is usually ``__name__``, the name of the Python module that contains the logger. This allows you to filter and handle logging calls on a per-module basis. However, if you have some other way of organizing your logging messages, you can provide any dot-separated name to identify your logger:: # Get an instance of a specific named logger logger = logging.getLogger('project.interesting.stuff') The dotted paths of logger names define a hierarchy. The ``project.interesting`` logger is considered to be a parent of the ``project.interesting.stuff`` logger; the ``project`` logger is a parent of the ``project.interesting`` logger. Why is the hierarchy important? Well, because loggers can be set to *propagate* their logging calls to their parents. In this way, you can define a single set of handlers at the root of a logger tree, and capture all logging calls in the subtree of loggers. A logger defined in the ``project`` namespace will catch all logging messages issued on the ``project.interesting`` and ``project.interesting.stuff`` loggers. This propagation can be controlled on a per-logger basis. If you don't want a particular logger to propagate to its parents, you can turn off this behavior. Making logging calls -------------------- The logger instance contains an entry method for each of the default log levels: * ``logger.debug()`` * ``logger.info()`` * ``logger.warning()`` * ``logger.error()`` * ``logger.critical()`` There are two other logging calls available: * ``logger.log()``: Manually emits a logging message with a specific log level. * ``logger.exception()``: Creates an ``ERROR`` level logging message wrapping the current exception stack frame. .. _configuring-logging: Configuring logging =================== It isn't enough to just put logging calls into your code. You also need to configure the loggers, handlers, filters, and formatters to ensure you can use the logging output. Python's logging library provides several techniques to configure logging, ranging from a programmatic interface to configuration files. By default, Django uses the :ref:`dictConfig format `. In order to configure logging, you use :setting:`LOGGING` to define a dictionary of logging settings. These settings describes the loggers, handlers, filters and formatters that you want in your logging setup, and the log levels and other properties that you want those components to have. By default, the :setting:`LOGGING` setting is merged with :ref:`Django's default logging configuration ` using the following scheme. If the ``disable_existing_loggers`` key in the :setting:`LOGGING` dictConfig is set to ``True`` (which is the ``dictConfig`` default if the key is missing) then all loggers from the default configuration will be disabled. Disabled loggers are not the same as removed; the logger will still exist, but will silently discard anything logged to it, not even propagating entries to a parent logger. Thus you should be very careful using ``'disable_existing_loggers': True``; it's probably not what you want. Instead, you can set ``disable_existing_loggers`` to ``False`` and redefine some or all of the default loggers; or you can set :setting:`LOGGING_CONFIG` to ``None`` and :ref:`handle logging config yourself `. Logging is configured as part of the general Django ``setup()`` function. Therefore, you can be certain that loggers are always ready for use in your project code. Examples -------- The full documentation for :ref:`dictConfig format ` is the best source of information about logging configuration dictionaries. However, to give you a taste of what is possible, here are several examples. To begin, here's a small configuration that will allow you to output all log messages to the console: .. code-block:: python :caption: settings.py import os LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'console': { 'class': 'logging.StreamHandler', }, }, 'root': { 'handlers': ['console'], 'level': 'WARNING', }, } This configures the parent ``root`` logger to send messages with the ``WARNING`` level and higher to the console handler. By adjusting the level to ``INFO`` or ``DEBUG`` you can display more messages. This may be useful during development. Next we can add more fine-grained logging. Here's an example of how to make the logging system print more messages from just the :ref:`django-logger` named logger: .. code-block:: python :caption: settings.py import os LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'console': { 'class': 'logging.StreamHandler', }, }, 'root': { 'handlers': ['console'], 'level': 'WARNING', }, 'loggers': { 'django': { 'handlers': ['console'], 'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'), 'propagate': False, }, }, } By default, this config sends messages from the ``django`` logger of level ``INFO`` or higher to the console. This is the same level as Django's default logging config, except that the default config only displays log records when ``DEBUG=True``. Django does not log many such ``INFO`` level messages. With this config, however, you can also set the environment variable ``DJANGO_LOG_LEVEL=DEBUG`` to see all of Django's debug logging which is very verbose as it includes all database queries. You don't have to log to the console. Here's a configuration which writes all logging from the :ref:`django-logger` named logger to a local file: .. code-block:: python :caption: settings.py LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'handlers': { 'file': { 'level': 'DEBUG', 'class': 'logging.FileHandler', 'filename': '/path/to/django/debug.log', }, }, 'loggers': { 'django': { 'handlers': ['file'], 'level': 'DEBUG', 'propagate': True, }, }, } If you use this example, be sure to change the ``'filename'`` path to a location that's writable by the user that's running the Django application. Finally, here's an example of a fairly complex logging setup: .. code-block:: python :caption: settings.py LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'verbose': { 'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}', 'style': '{', }, 'simple': { 'format': '{levelname} {message}', 'style': '{', }, }, 'filters': { 'special': { '()': 'project.logging.SpecialFilter', 'foo': 'bar', }, 'require_debug_true': { '()': 'django.utils.log.RequireDebugTrue', }, }, 'handlers': { 'console': { 'level': 'INFO', 'filters': ['require_debug_true'], 'class': 'logging.StreamHandler', 'formatter': 'simple' }, 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler', 'filters': ['special'] } }, 'loggers': { 'django': { 'handlers': ['console'], 'propagate': True, }, 'django.request': { 'handlers': ['mail_admins'], 'level': 'ERROR', 'propagate': False, }, 'myproject.custom': { 'handlers': ['console', 'mail_admins'], 'level': 'INFO', 'filters': ['special'] } } } This logging configuration does the following things: * Identifies the configuration as being in 'dictConfig version 1' format. At present, this is the only dictConfig format version. * Defines two formatters: * ``simple``, that outputs the log level name (e.g., ``DEBUG``) and the log message. The ``format`` string is a normal Python formatting string describing the details that are to be output on each logging line. The full list of detail that can be output can be found in :ref:`formatter-objects`. * ``verbose``, that outputs the log level name, the log message, plus the time, process, thread and module that generate the log message. * Defines two filters: * ``project.logging.SpecialFilter``, using the alias ``special``. If this filter required additional arguments, they can be provided as additional keys in the filter configuration dictionary. In this case, the argument ``foo`` will be given a value of ``bar`` when instantiating ``SpecialFilter``. * ``django.utils.log.RequireDebugTrue``, which passes on records when :setting:`DEBUG` is ``True``. * Defines two handlers: * ``console``, a :class:`~logging.StreamHandler`, which prints any ``INFO`` (or higher) message to ``sys.stderr``. This handler uses the ``simple`` output format. * ``mail_admins``, an :class:`AdminEmailHandler`, which emails any ``ERROR`` (or higher) message to the site :setting:`ADMINS`. This handler uses the ``special`` filter. * Configures three loggers: * ``django``, which passes all messages to the ``console`` handler. * ``django.request``, which passes all ``ERROR`` messages to the ``mail_admins`` handler. In addition, this logger is marked to *not* propagate messages. This means that log messages written to ``django.request`` will not be handled by the ``django`` logger. * ``myproject.custom``, which passes all messages at ``INFO`` or higher that also pass the ``special`` filter to two handlers -- the ``console``, and ``mail_admins``. This means that all ``INFO`` level messages (or higher) will be printed to the console; ``ERROR`` and ``CRITICAL`` messages will also be output via email. Custom logging configuration ---------------------------- If you don't want to use Python's dictConfig format to configure your logger, you can specify your own configuration scheme. The :setting:`LOGGING_CONFIG` setting defines the callable that will be used to configure Django's loggers. By default, it points at Python's :func:`logging.config.dictConfig()` function. However, if you want to use a different configuration process, you can use any other callable that takes a single argument. The contents of :setting:`LOGGING` will be provided as the value of that argument when logging is configured. .. _disabling-logging-configuration: Disabling logging configuration ------------------------------- If you don't want to configure logging at all (or you want to manually configure logging using your own approach), you can set :setting:`LOGGING_CONFIG` to ``None``. This will disable the configuration process for :ref:`Django's default logging `. Setting :setting:`LOGGING_CONFIG` to ``None`` only means that the automatic configuration process is disabled, not logging itself. If you disable the configuration process, Django will still make logging calls, falling back to whatever default logging behavior is defined. Here's an example that disables Django's logging configuration and then manually configures logging: .. code-block:: python :caption: settings.py LOGGING_CONFIG = None import logging.config logging.config.dictConfig(...) Note that the default configuration process only calls :setting:`LOGGING_CONFIG` once settings are fully-loaded. In contrast, manually configuring the logging in your settings file will load your logging config immediately. As such, your logging config must appear *after* any settings on which it depends. Django logging extension reference ================================== Django provides a number of utilities to handle the unique requirements of logging in a web server environment. Loggers ------- Django provides several built-in loggers. .. _django-logger: ``django`` ~~~~~~~~~~ The catch-all logger for messages in the ``django`` hierarchy. No messages are posted using this name but instead using one of the loggers below. .. _django-request-logger: ``django.request`` ~~~~~~~~~~~~~~~~~~ Log messages related to the handling of requests. 5XX responses are raised as ``ERROR`` messages; 4XX responses are raised as ``WARNING`` messages. Requests that are logged to the ``django.security`` logger aren't logged to ``django.request``. Messages to this logger have the following extra context: * ``status_code``: The HTTP response code associated with the request. * ``request``: The request object that generated the logging message. .. _django-server-logger: ``django.server`` ~~~~~~~~~~~~~~~~~ Log messages related to the handling of requests received by the server invoked by the :djadmin:`runserver` command. HTTP 5XX responses are logged as ``ERROR`` messages, 4XX responses are logged as ``WARNING`` messages, and everything else is logged as ``INFO``. Messages to this logger have the following extra context: * ``status_code``: The HTTP response code associated with the request. * ``request``: The request object that generated the logging message. .. _django-template-logger: ``django.template`` ~~~~~~~~~~~~~~~~~~~ Log messages related to the rendering of templates. * Missing context variables are logged as ``DEBUG`` messages. .. _django-db-logger: ``django.db.backends`` ~~~~~~~~~~~~~~~~~~~~~~ Messages relating to the interaction of code with the database. For example, every application-level SQL statement executed by a request is logged at the ``DEBUG`` level to this logger. Messages to this logger have the following extra context: * ``duration``: The time taken to execute the SQL statement. * ``sql``: The SQL statement that was executed. * ``params``: The parameters that were used in the SQL call. For performance reasons, SQL logging is only enabled when ``settings.DEBUG`` is set to ``True``, regardless of the logging level or handlers that are installed. This logging does not include framework-level initialization (e.g. ``SET TIMEZONE``) or transaction management queries (e.g. ``BEGIN``, ``COMMIT``, and ``ROLLBACK``). Turn on query logging in your database if you wish to view all database queries. .. _django-security-logger: ``django.security.*`` ~~~~~~~~~~~~~~~~~~~~~~ The security loggers will receive messages on any occurrence of :exc:`~django.core.exceptions.SuspiciousOperation` and other security-related errors. There is a sub-logger for each subtype of security error, including all ``SuspiciousOperation``\s. The level of the log event depends on where the exception is handled. Most occurrences are logged as a warning, while any ``SuspiciousOperation`` that reaches the WSGI handler will be logged as an error. For example, when an HTTP ``Host`` header is included in a request from a client that does not match :setting:`ALLOWED_HOSTS`, Django will return a 400 response, and an error message will be logged to the ``django.security.DisallowedHost`` logger. These log events will reach the ``django`` logger by default, which mails error events to admins when ``DEBUG=False``. Requests resulting in a 400 response due to a ``SuspiciousOperation`` will not be logged to the ``django.request`` logger, but only to the ``django.security`` logger. To silence a particular type of ``SuspiciousOperation``, you can override that specific logger following this example:: 'handlers': { 'null': { 'class': 'logging.NullHandler', }, }, 'loggers': { 'django.security.DisallowedHost': { 'handlers': ['null'], 'propagate': False, }, }, Other ``django.security`` loggers not based on ``SuspiciousOperation`` are: * ``django.security.csrf``: For :ref:`CSRF failures `. ``django.db.backends.schema`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Logs the SQL queries that are executed during schema changes to the database by the :doc:`migrations framework `. Note that it won't log the queries executed by :class:`~django.db.migrations.operations.RunPython`. Messages to this logger have ``params`` and ``sql`` in their extra context (but unlike ``django.db.backends``, not duration). The values have the same meaning as explained in :ref:`django-db-logger`. Handlers -------- Django provides one log handler in addition to those provided by the Python logging module. .. class:: AdminEmailHandler(include_html=False, email_backend=None, reporter_class=None) This handler sends an email to the site :setting:`ADMINS` for each log message it receives. If the log record contains a ``request`` attribute, the full details of the request will be included in the email. The email subject will include the phrase "internal IP" if the client's IP address is in the :setting:`INTERNAL_IPS` setting; if not, it will include "EXTERNAL IP". If the log record contains stack trace information, that stack trace will be included in the email. The ``include_html`` argument of ``AdminEmailHandler`` is used to control whether the traceback email includes an HTML attachment containing the full content of the debug Web page that would have been produced if :setting:`DEBUG` were ``True``. To set this value in your configuration, include it in the handler definition for ``django.utils.log.AdminEmailHandler``, like this:: 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler', 'include_html': True, } }, Be aware of the :ref:`security implications of logging ` when using the ``AdminEmailHandler``. By setting the ``email_backend`` argument of ``AdminEmailHandler``, the :ref:`email backend ` that is being used by the handler can be overridden, like this:: 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler', 'email_backend': 'django.core.mail.backends.filebased.EmailBackend', } }, By default, an instance of the email backend specified in :setting:`EMAIL_BACKEND` will be used. The ``reporter_class`` argument of ``AdminEmailHandler`` allows providing an ``django.views.debug.ExceptionReporter`` subclass to customize the traceback text sent in the email body. You provide a string import path to the class you wish to use, like this:: 'handlers': { 'mail_admins': { 'level': 'ERROR', 'class': 'django.utils.log.AdminEmailHandler', 'include_html': True, 'reporter_class': 'somepackage.error_reporter.CustomErrorReporter' } }, .. method:: send_mail(subject, message, *args, **kwargs) Sends emails to admin users. To customize this behavior, you can subclass the :class:`~django.utils.log.AdminEmailHandler` class and override this method. Filters ------- Django provides some log filters in addition to those provided by the Python logging module. .. class:: CallbackFilter(callback) This filter accepts a callback function (which should accept a single argument, the record to be logged), and calls it for each record that passes through the filter. Handling of that record will not proceed if the callback returns False. For instance, to filter out :exc:`~django.http.UnreadablePostError` (raised when a user cancels an upload) from the admin emails, you would create a filter function:: from django.http import UnreadablePostError def skip_unreadable_post(record): if record.exc_info: exc_type, exc_value = record.exc_info[:2] if isinstance(exc_value, UnreadablePostError): return False return True and then add it to your logging config:: 'filters': { 'skip_unreadable_posts': { '()': 'django.utils.log.CallbackFilter', 'callback': skip_unreadable_post, } }, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'filters': ['skip_unreadable_posts'], 'class': 'django.utils.log.AdminEmailHandler' } }, .. class:: RequireDebugFalse() This filter will only pass on records when settings.DEBUG is False. This filter is used as follows in the default :setting:`LOGGING` configuration to ensure that the :class:`AdminEmailHandler` only sends error emails to admins when :setting:`DEBUG` is ``False``:: 'filters': { 'require_debug_false': { '()': 'django.utils.log.RequireDebugFalse', } }, 'handlers': { 'mail_admins': { 'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler' } }, .. class:: RequireDebugTrue() This filter is similar to :class:`RequireDebugFalse`, except that records are passed only when :setting:`DEBUG` is ``True``. .. _default-logging-configuration: Django's default logging configuration ====================================== By default, Django configures the following logging: When :setting:`DEBUG` is ``True``: * The ``django`` logger sends messages in the ``django`` hierarchy (except ``django.server``) at the ``INFO`` level or higher to the console. When :setting:`DEBUG` is ``False``: * The ``django`` logger sends messages in the ``django`` hierarchy (except ``django.server``) with ``ERROR`` or ``CRITICAL`` level to :class:`AdminEmailHandler`. Independent of the value of :setting:`DEBUG`: * The :ref:`django-server-logger` logger sends messages at the ``INFO`` level or higher to the console. All loggers except :ref:`django-server-logger` propagate logging to their parents, up to the root ``django`` logger. The ``console`` and ``mail_admins`` handlers are attached to the root logger to provide the behavior described above. See also :ref:`Configuring logging ` to learn how you can complement or replace this default logging configuration defined in :source:`django/utils/log.py`. .. _logging-security-implications: Security implications ===================== The logging system handles potentially sensitive information. For example, the log record may contain information about a web request or a stack trace, while some of the data you collect in your own loggers may also have security implications. You need to be sure you know: * what information is collected * where it will subsequently be stored * how it will be transferred * who might have access to it. To help control the collection of sensitive information, you can explicitly designate certain sensitive information to be filtered out of error reports -- read more about how to :ref:`filter error reports `. ``AdminEmailHandler`` --------------------- The built-in :class:`AdminEmailHandler` deserves a mention in the context of security. If its ``include_html`` option is enabled, the email message it sends will contain a full traceback, with names and values of local variables at each level of the stack, plus the values of your Django settings (in other words, the same level of detail that is exposed in a web page when :setting:`DEBUG` is ``True``). It's generally not considered a good idea to send such potentially sensitive information over email. Consider instead using one of the many third-party services to which detailed logs can be sent to get the best of multiple worlds -- the rich information of full tracebacks, clear management of who is notified and has access to the information, and so on.