==============
Custom lookups
==============

.. module:: django.db.models.lookups
   :synopsis: Custom lookups

.. currentmodule:: django.db.models

(This documentation is candidate for complete rewrite, but contains
useful information of how to test the current implementation.)

This documentation constains instructions of how to create custom lookups
for model fields.

Django's ORM works using lookup paths when building query filters and other
query structures. For example in the query Book.filter(author__age__lte=30)
the author__age__lte is the lookup path.

The lookup path consist of three different part. First is the related lookups,
above part author refers to Book's related model Author. Second part of the
lookup path is the final field, above this is Author's field age. Finally the
lte part is commonly called just lookup (TODO: this nomenclature is confusing,
can we invent something better).

This documentation concentrates on writing custom lookups, that is custom
implementations for lte or any other lookup you wish to use.

Django will fetch a ``Lookup`` class from the final field using the field's
method get_lookup(lookup_name). This method can do three things:

  1. Return a Lookup class
  2. Raise a FieldError
  3. Return None

Above return None is only available during backwards compatibility period and
returning None will not be allowed in Django 1.9 or later. The interpretation
is to use the old way of lookup hadling inside the ORM.

The returned Lookup will be used to build the query.

The Lookup class
~~~~~~~~~~~~~~~~

The API is as follows:

.. attribute:: lookup_name

A string used by Django to distinguish different lookups.

.. method:: __init__(lhs, rhs)

The lhs and rhs are the field reference (reference to field age in the
author__age__lte=30 example), and rhs is the value (30 in the example).

.. attribute:: Lookup.lhs

The left hand side part of this lookup. You can assume it implements the
query part interface (TODO: write interface definition...).

.. method:: Lookup.as_sql(qn, connection)

This method is used to produce the query string of the Lookup. A typical
implementation is usually something like::

    def as_sql(self, qn, connection):
        lhs, params = self.process_lhs(qn, connection)
        rhs, rhs_params = self.process_rhs(qn, connection)
        params = lhs_params.extend(rhs_params)
        return '%s <OPERATOR> %s', (lhs, rhs), params

where the <OPERATOR> is some query operator. The qn is a callable that
can be used to convert strings to quoted variants (that is, colname to
"colname"). Note that the quotation is *not* safe against SQL injection.

In addition the qn implements method compile() which can be used to turn
anything with as_sql() method to query string. You should always call
qn.compile(part) instead of part.as_sql(qn, connection) so that 3rd party
backends have ability to customize the produced query string. More of this
later on.

The connection is the used connection.

.. method:: Lookup.process_lhs(qn, connection, lhs=None)

This method is used to convert the left hand side of the lookup into query
string. The left hand side can be a field reference or a nested lookup. The
lhs kwarg can be used to convert something else than self.lhs to query string.

.. method:: Lookup.process_rhs(qn, connection, rhs=None)

The process_rhs method is used to convert the right hand side into query string.
The rhs is the value given in the filter clause. It can be a raw value to
compare agains, a F() reference to another field or even a QuerySet.

In addition the Lookup class has some private methods - that is, implementing
just the above mentioned attributes and methods is not enough, instead you
should subclass Lookup.

The Extract class
~~~~~~~~~~~~~~~~~

An Extract is something that converts a value to another value in the query string.
For example you could have an Extract that procudes modulo 3 of the given value.
In SQL this would be something like "author"."age" % 3.

Extracts are used in nested lookups. The Extract class must implement the query
part interface. In addition the Extract should must lookup_name attribute.

A simple Lookup example
~~~~~~~~~~~~~~~~~~~~~~~

This is how to write a simple div3 lookup for IntegerField::

    from django.db.models import Lookup, IntegerField
    class Div3(Lookup):
        lookup_name = 'div3'

        def as_sql(self, qn, connection):
            lhs_sql, params = self.process_lhs(qn, connection)
            rhs_sql, rhs_params = self.process_rhs(qn, connection)
            params.extend(rhs_params)
            # We need doulbe-escaping for the %%%% operator.
            return '%s %%%% %s' % (lhs_sql, rhs_sql), params

    IntegerField.register_lookup(Div3)

Now all IntegerFields or subclasses of IntegerField will have
a div3 lookup. For example you could do Author.objects.filter(age__div3=2).
This query would return every author whose age % 3 == 2.

A simple nested lookup example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Here is how to write an Extract and a Lookup for IntegerField. The example
lookup can be used similarly as the above div3 lookup, and in addition it
support nesting lookups::

    class Div3Extract(Extract):
        lookup_name = 'div3'

        def as_sql(self, qn, connection):
            lhs, lhs_params = qn.compile(self.lhs)
            return '%s %%%% 3' % (lhs,), lhs_params

    IntegerField.register_lookup(Div3Extract)

Note that if you already added Div3 for IntegerField in the above
example, now Div3LookupWithExtract will override that lookup.

This lookup can be used like Div3 lookup, but in addition it supports
nesting, too. The default output type for Extracts is the same type as the
lhs' output_type. So, the Div3Extract supports all the same lookups as
IntegerField. For example Author.objects.filter(age__div3__in=[1, 2])
returns all authors for which age % 3 in (1, 2). 

A more complex nested lookup
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We will write a Year lookup that extracts year from date field. This
field will convert the output type of the field - the lhs (or "input")
field is DateField, but output is of type IntegerField.::

    from django.db.models import IntegerField, DateField
    from django.db.models.lookups import Extract

    class YearExtract(Extract):
        lookup_name = 'year'

        def as_sql(self, qn, connection):
            lhs_sql, params = qn.compile(self.lhs)
            # hmmh - this is internal API...
            return connection.ops.date_extract_sql('year', lhs_sql), params

        @property
        def output_type(self):
            return IntegerField()

    DateField.register_lookup(YearExtract)

Now you could write Author.objects.filter(birthdate__year=1981). This will
produce SQL like 'EXTRACT('year' from "author"."birthdate") = 1981'. The
produces SQL depends on used backend. In addtition you can use any lookup
defined for IntegerField, even div3 if you added that. So,
Authos.objects.filter(birthdate__year__div3=2) will return every author
with birthdate.year % 3 == 2.

We could go further and add an optimized implementation for exact lookups::

    from django.db.models.lookups import Lookup

    class YearExtractOptimized(YearExtract):
        def get_lookup(self, lookup):
            if lookup == 'exact':
                return YearExact
            return super(YearExtractOptimized, self).get_lookup()

    class YearExact(Lookup):
        def as_sql(self, qn, connection):
            # We will need to skip the extract part, and instead go
            # directly with the originating field, that is self.lhs.lhs
            lhs_sql, lhs_params = self.process_lhs(qn, connection, self.lhs.lhs)
            rhs_sql, rhs_params = self.process_rhs(qn, connection)
            # Note that we must be careful so that we have params in the
            # same order as we have the parts in the SQL.
            params = []
            params.extend(lhs_params)
            params.extend(rhs_params)
            params.extend(lhs_params)
            params.extend(rhs_params)
            # We use PostgreSQL specific SQL here. Note that we must do the
            # conversions in SQL instead of in Python to support F() references.
            return ("%(lhs)s >= (%(rhs)s || '-01-01')::date "
                    "AND %(lhs)s <= (%(rhs)s || '-12-31')::date" %
                    {'lhs': lhs_sql, 'rhs': rhs_sql}, params)

Note that we used PostgreSQL specific SQL above. What if we want to support
MySQL, too? This can be done by registering a different compiling implementation
for MySQL::

    from django.db.backends.utils import add_implementation
    @add_implementation(YearExact, 'mysql')
    def mysql_year_exact(node, qn, connection):
        lhs_sql, lhs_params = node.process_lhs(qn, connection, node.lhs.lhs)
        rhs_sql, rhs_params = node.process_rhs(qn, connection)
        params = []
        params.extend(lhs_params)
        params.extend(rhs_params)
        params.extend(lhs_params)
        params.extend(rhs_params)
        return ("%(lhs)s >= str_to_date(concat(%(rhs)s, '-01-01'), '%%%%Y-%%%%m-%%%%d') "
                "AND %(lhs)s <= str_to_date(concat(%(rhs)s, '-12-31'), '%%%%Y-%%%%m-%%%%d')" %
                {'lhs': lhs_sql, 'rhs': rhs_sql}, params)

Now, on MySQL instead of calling as_sql() of the YearExact Django will use the
above compile implementation.