337 lines
13 KiB
Plaintext
337 lines
13 KiB
Plaintext
==============
|
|
Custom lookups
|
|
==============
|
|
|
|
.. versionadded:: 1.7
|
|
|
|
.. module:: django.db.models.lookups
|
|
:synopsis: Custom lookups
|
|
|
|
.. currentmodule:: django.db.models
|
|
|
|
By default Django offers a wide variety of :ref:`built-in lookups
|
|
<field-lookups>` for filtering (for example, ``exact`` and ``icontains``). This
|
|
documentation explains how to write custom lookups and how to alter the working
|
|
of existing lookups.
|
|
|
|
A simple Lookup example
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Let's start with a simple custom lookup. We will write a custom lookup ``ne``
|
|
which works opposite to ``exact``. ``Author.objects.filter(name__ne='Jack')``
|
|
will translate to the SQL::
|
|
|
|
"author"."name" <> 'Jack'
|
|
|
|
This SQL is backend independent, so we don't need to worry about different
|
|
databases.
|
|
|
|
There are two steps to making this work. Firstly we need to implement the
|
|
lookup, then we need to tell Django about it. The implementation is quite
|
|
straightforward::
|
|
|
|
from django.db.models import Lookup
|
|
|
|
class NotEqual(Lookup):
|
|
lookup_name = 'ne'
|
|
|
|
def as_sql(self, qn, connection):
|
|
lhs, lhs_params = self.process_lhs(qn, connection)
|
|
rhs, rhs_params = self.process_rhs(qn, connection)
|
|
params = lhs_params + rhs_params
|
|
return '%s <> %s' % (lhs, rhs), params
|
|
|
|
To register the ``NotEqual`` lookup we will just need to call
|
|
``register_lookup`` on the field class we want the lookup to be available. In
|
|
this case, the lookup makes sense on all ``Field`` subclasses, so we register
|
|
it with ``Field`` directly::
|
|
|
|
from django.db.models.fields import Field
|
|
Field.register_lookup(NotEqual)
|
|
|
|
We can now use ``foo__ne`` for any field ``foo``. You will need to ensure that
|
|
this registration happens before you try to create any querysets using it. You
|
|
could place the implementation in a ``models.py`` file, or register the lookup
|
|
in the ``ready()`` method of an ``AppConfig``.
|
|
|
|
Taking a closer look at the implementation, the first required attribute is
|
|
``lookup_name``. This allows the ORM to understand how to interpret ``name__ne``
|
|
and use ``NotEqual`` to generate the SQL. By convention, these names are always
|
|
lowercase strings containing only letters, but the only hard requirement is
|
|
that it must not contain the string ``__``.
|
|
|
|
A ``Lookup`` works against two values, ``lhs`` and ``rhs``, standing for
|
|
left-hand side and right-hand side. The left-hand side is usually a field
|
|
reference, but it can be anything implementing the :ref:`query expression API
|
|
<query-expression>`. The right-hand is the value given by the user. In the
|
|
example ``Author.objects.filter(name__ne='Jack')``, the left-hand side is a
|
|
reference to the ``name`` field of the ``Author`` model, and ``'Jack'`` is the
|
|
right-hand side.
|
|
|
|
We call ``process_lhs`` and ``process_rhs`` to convert them into the values we
|
|
need for SQL. In the above example, ``process_lhs`` returns
|
|
``('"author"."name"', [])`` and ``process_rhs`` returns ``('"%s"', ['Jack'])``.
|
|
In this example there were no parameters for the left hand side, but this would
|
|
depend on the object we have, so we still need to include them in the
|
|
parameters we return.
|
|
|
|
Finally we combine the parts into a SQL expression with ``<>``, and supply all
|
|
the parameters for the query. We then return a tuple containing the generated
|
|
SQL string and the parameters.
|
|
|
|
A simple transformer example
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The custom lookup above is great, but in some cases you may want to be able to
|
|
chain lookups together. For example, let's suppose we are building an
|
|
application where we want to make use of the ``abs()`` operator.
|
|
We have an ``Experiment`` model which records a start value, end value and the
|
|
change (start - end). We would like to find all experiments where the change
|
|
was equal to a certain amount (``Experiment.objects.filter(change__abs=27)``),
|
|
or where it did not exceed a certain amount
|
|
(``Experiment.objects.filter(change__abs__lt=27)``).
|
|
|
|
.. note::
|
|
This example is somewhat contrived, but it demonstrates nicely the range of
|
|
functionality which is possible in a database backend independent manner,
|
|
and without duplicating functionality already in Django.
|
|
|
|
We will start by writing a ``AbsoluteValue`` transformer. This will use the SQL
|
|
function ``ABS()`` to transform the value before comparison::
|
|
|
|
from django.db.models import Transform
|
|
|
|
class AbsoluteValue(Transform):
|
|
lookup_name = 'abs'
|
|
|
|
def as_sql(self, qn, connection):
|
|
lhs, params = qn.compile(self.lhs)
|
|
return "ABS(%s)" % lhs, params
|
|
|
|
Next, lets register it for ``IntegerField``::
|
|
|
|
from django.db.models import IntegerField
|
|
IntegerField.register_lookup(AbsoluteValue)
|
|
|
|
We can now run the queries we had before.
|
|
``Experiment.objects.filter(change__abs=27)`` will generate the following SQL::
|
|
|
|
SELECT ... WHERE ABS("experiments"."change") = 27
|
|
|
|
By using ``Transform`` instead of ``Lookup`` it means we are able to chain
|
|
further lookups afterwards. So
|
|
``Experiment.objects.filter(change__abs__lt=27)`` will generate the following
|
|
SQL::
|
|
|
|
SELECT ... WHERE ABS("experiments"."change") < 27
|
|
|
|
Subclasses of ``Transform`` usually only operate on the left-hand side of the
|
|
expression. Further lookups will work on the transformed value. Note that in
|
|
this case where there is no other lookup specified, Django interprets
|
|
``change__abs=27`` as ``change__abs__exact=27``.
|
|
|
|
When looking for which lookups are allowable after the ``Transform`` has been
|
|
applied, Django uses the ``output_type`` attribute. We didn't need to specify
|
|
this here as it didn't change, but supposing we were applying ``AbsoluteValue``
|
|
to some field which represents a more complex type (for example a point
|
|
relative to an origin, or a complex number) then we may have wanted to specify
|
|
``output_type = FloatField``, which will ensure that further lookups like
|
|
``abs__lte`` behave as they would for a ``FloatField``.
|
|
|
|
Writing an efficient abs__lt lookup
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
When using the above written ``abs`` lookup, the SQL produced will not use
|
|
indexes efficiently in some cases. In particular, when we use
|
|
``change__abs__lt=27``, this is equivalent to ``change__gt=-27`` AND
|
|
``change__lt=27``. (For the ``lte`` case we could use the SQL ``BETWEEN``).
|
|
|
|
So we would like ``Experiment.objects.filter(change__abs__lt=27)`` to generate
|
|
the following SQL::
|
|
|
|
SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27
|
|
|
|
The implementation is::
|
|
|
|
from django.db.models import Lookup
|
|
|
|
class AbsoluteValueLessThan(Lookup):
|
|
lookup_name = 'lt'
|
|
|
|
def as_sql(self, qn, connection):
|
|
lhs, lhs_params = qn.compile(self.lhs.lhs)
|
|
rhs, rhs_params = self.process_rhs(qn, connection)
|
|
params = lhs_params + rhs_params + lhs_params + rhs_params
|
|
return '%s < %s AND %s > -%s' % (lhs, rhs, lhs, rhs), params
|
|
|
|
AbsoluteValue.register_lookup(AbsoluteValueLessThan)
|
|
|
|
There are a couple of notable things going on. First, ``AbsoluteValueLessThan``
|
|
isn't calling ``process_lhs()``. Instead it skips the transformation of the
|
|
``lhs`` done by ``AbsoluteValue`` and uses the original ``lhs``. That is, we
|
|
want to get ``27`` not ``ABS(27)``. Referring directly to ``self.lhs.lhs`` is
|
|
safe as ``AbsoluteValueLessThan`` can be accessed only from the
|
|
``AbsoluteValue`` lookup, that is the ``lhs`` is always an instance of
|
|
``AbsoluteValue``.
|
|
|
|
Notice also that as both sides are used multiple times in the query the params
|
|
need to contain ``lhs_params`` and ``rhs_params`` multiple times.
|
|
|
|
The final query does the inversion (``27`` to ``-27``) directly in the
|
|
database. The reason for doing this is that if the self.rhs is something else
|
|
than a plain integer value (for example an ``F()`` reference) we can't do the
|
|
transformations in Python.
|
|
|
|
.. note::
|
|
In fact, most lookups with ``__abs`` could be implemented as range queries
|
|
like this, and on most database backends it is likely to be more sensible to
|
|
do so as you can make use of the indexes. However with PostgreSQL you may
|
|
want to add an index on ``abs(change)`` which would allow these queries to
|
|
be very efficient.
|
|
|
|
Writing alternative implementations for existing lookups
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Sometimes different database vendors require different SQL for the same
|
|
operation. For this example we will rewrite a custom implementation for
|
|
MySQL for the NotEqual operator. Instead of ``<>`` we will be using ``!=``
|
|
operator. (Note that in reality almost all databases support both, including
|
|
all the official databases supported by Django).
|
|
|
|
We can change the behaviour on a specific backend by creating a subclass of
|
|
``NotEqual`` with a ``as_mysql`` method::
|
|
|
|
class MySQLNotEqual(NotEqual):
|
|
def as_mysql(self, qn, connection):
|
|
lhs, lhs_params = self.process_lhs(qn, connection)
|
|
rhs, rhs_params = self.process_rhs(qn, connection)
|
|
params = lhs_params + rhs_params
|
|
return '%s != %s' % (lhs, rhs), params
|
|
Field.register_lookup(MySQLNotExact)
|
|
|
|
We can then register it with ``Field``. It takes the place of the original
|
|
``NotEqual`` class as it has the same ``lookup_name``.
|
|
|
|
When compiling a query, Django first looks for ``as_%s % connection.vendor``
|
|
methods, and then falls back to ``as_sql``. The vendor names for the in-built
|
|
backends are ``sqlite``, ``postgresql``, ``oracle`` and ``mysql``.
|
|
|
|
.. _query-expression:
|
|
|
|
The Query Expression API
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
A lookup can assume that the lhs responds to the query expression API.
|
|
Currently direct field references, aggregates and ``Transform`` instances respond
|
|
to this API.
|
|
|
|
.. method:: as_sql(qn, connection)
|
|
|
|
Responsible for producing the query string and parameters for the
|
|
expression. The ``qn`` has a ``compile()`` method that can be used to
|
|
compile other expressions. The ``connection`` is the connection used to
|
|
execute the query.
|
|
|
|
Calling expression.as_sql() directly is usually incorrect - instead
|
|
qn.compile(expression) should be used. The qn.compile() method will take
|
|
care of calling vendor-specific methods of the expression.
|
|
|
|
.. method:: get_lookup(lookup_name)
|
|
|
|
The ``get_lookup()`` method is used to fetch lookups. By default the
|
|
lookup is fetched from the expression's output type in the same way
|
|
described in registering and fetching lookup documentation below.
|
|
It is possible to override this method to alter that behaviour.
|
|
|
|
.. method:: as_vendorname(qn, connection)
|
|
|
|
Works like ``as_sql()`` method. When an expression is compiled by
|
|
``qn.compile()``, Django will first try to call ``as_vendorname()``, where
|
|
vendorname is the vendor name of the backend used for executing the query.
|
|
The vendorname is one of ``postgresql``, ``oracle``, ``sqlite`` or
|
|
``mysql`` for Django's built-in backends.
|
|
|
|
.. attribute:: output_type
|
|
|
|
The ``output_type`` attribute is used by the ``get_lookup()`` method to check for
|
|
lookups. The output_type should be a field.
|
|
|
|
Note that this documentation lists only the public methods of the API.
|
|
|
|
Lookup reference
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
.. class:: Lookup
|
|
|
|
In addition to the attributes and methods below, lookups also support
|
|
``as_sql`` and ``as_vendorname`` from the query expression API.
|
|
|
|
.. attribute:: lhs
|
|
|
|
The ``lhs`` (left-hand side) of a lookup tells us what we are comparing the
|
|
rhs to. It is an object which implements the query expression API. This is
|
|
likely to be a field, an aggregate or a subclass of ``Transform``.
|
|
|
|
.. attribute:: rhs
|
|
|
|
The ``rhs`` (right-hand side) of a lookup is the value we are comparing the
|
|
left hand side to. It may be a plain value, or something which compiles
|
|
into SQL, for example an ``F()`` object or a ``Queryset``.
|
|
|
|
.. attribute:: lookup_name
|
|
|
|
This class level attribute is used when registering lookups. It determines
|
|
the name used in queries to trigger this lookup. For example, ``contains``
|
|
or ``exact``. This should not contain the string ``__``.
|
|
|
|
.. method:: process_lhs(qn, connection)
|
|
|
|
This returns a tuple of ``(lhs_string, lhs_params)``. In some cases you may
|
|
wish to compile ``lhs`` directly in your ``as_sql`` methods using
|
|
``qn.compile(self.lhs)``.
|
|
|
|
.. method:: process_rhs(qn, connection)
|
|
|
|
Behaves the same as ``process_lhs`` but acts on the right-hand side.
|
|
|
|
Transform reference
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. class:: Transform
|
|
|
|
In addition to implementing the query expression API Transforms have the
|
|
following methods and attributes.
|
|
|
|
.. attribute:: lhs
|
|
|
|
The ``lhs`` (left-hand-side) of a transform contains the value to be
|
|
transformed. The ``lhs`` implements the query expression API.
|
|
|
|
.. attribute:: lookup_name
|
|
|
|
This class level attribute is used when registering lookups. It determines
|
|
the name used in queries to trigger this lookup. For example, ``year``
|
|
or ``dayofweek``. This should not contain the string ``__``.
|
|
|
|
.. _lookup-registration-api:
|
|
|
|
Registering and fetching lookups
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The lookup registration API is explained below.
|
|
|
|
.. classmethod:: register_lookup(lookup)
|
|
|
|
Registers the Lookup or Transform for the class. For example
|
|
``DateField.register_lookup(YearExact)`` will register ``YearExact`` for
|
|
all ``DateFields`` in the project, but also for fields that are instances
|
|
of a subclass of ``DateField`` (for example ``DateTimeField``).
|
|
|
|
.. method:: get_lookup(lookup_name)
|
|
|
|
Django uses ``get_lookup(lookup_name)`` to fetch lookups or transforms.
|
|
The implementation of ``get_lookup()`` fetches lookups or transforms
|
|
registered for the current class based on their lookup_name attribute.
|
|
|
|
The lookup registration API is available for ``Transform`` and ``Field`` classes.
|