innov
/
django1
1
0
Fork 0
Code Issues Pull Requests 1 Projects Releases Wiki Activity

Reworked custom lookups docs. 

Mostly just formatting and rewording, but also replaced the example
using ``YearExtract`` to  use an example which is unlikely to ever be
possible directly in the ORM.
This commit is contained in:
Marc Tamlyn 2014-01-12 13:15:05 +00:00
parent 2509006506
commit f2dc4429a1
1 changed files with 188 additions and 146 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

334
docs/ref/models/custom_lookups.txt
View File

@ -2,37 +2,33 @@
Custom lookups Custom lookups
============== ==============
.. versionadded:: 1.7
.. module:: django.db.models.lookups .. module:: django.db.models.lookups
:synopsis: Custom lookups :synopsis: Custom lookups
.. currentmodule:: django.db.models .. currentmodule:: django.db.models
By default Django offers a wide variety of different lookups for filtering By default Django offers a wide variety of :ref:`built-in lookups
(for example, `exact` and `icontains`). This documentation explains how to <field-lookups>` for filtering (for example, ``exact`` and ``icontains``). This
write custom lookups and how to alter the working of existing lookups. In documentation explains how to write custom lookups and how to alter the working
addition how to transform field values is explained. fFor example how to of existing lookups.
extract the year from a DateField. By writing a custom `YearExtract`
transformer it is possible to filter on the transformed value, for example::
Author.objects.filter(birthdate__year__lte=1981)
Currently transformers are only available in filtering. So, it is not possible
to use it in other parts of the ORM, for example this will not work::
Author.objects.values_list('birthdate__year')
A simple Lookup example A simple Lookup example
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
Lets start with a simple custom lookup. We will write a custom lookup `ne` Let's start with a simple custom lookup. We will write a custom lookup ``ne``
which works opposite to `exact`. A `Author.objects.filter(name__ne='Jack')` which works opposite to ``exact``. ``Author.objects.filter(name__ne='Jack')``
will translate to:: will translate to the SQL::
"author"."name" <> 'Jack' "author"."name" <> 'Jack'
A custom lookup will need an implementation and Django needs to be told This SQL is backend independent, so we don't need to worry about different
the existence of the lookup. The implementation for this lookup will be databases.
simple to write::
There are two steps to making this work. Firstly we need to implement the
lookup, then we need to tell Django about it. The implementation is quite
straightforwards::
from django.db.models import Lookup from django.db.models import Lookup
@ -45,131 +41,165 @@ simple to write::
params = lhs_params + rhs_params params = lhs_params + rhs_params
return '%s <> %s' % (lhs, rhs), params return '%s <> %s' % (lhs, rhs), params
To register the `NotEqual` lookup we will just need to call register_lookup To register the ``NotEqual`` lookup we will just need to call
on the field class we want the lookup to be available:: ``register_lookup`` on the field class we want the lookup to be available. In
this case, the lookup makes sense on all ``Field`` subclasses, so we register
it with ``Field`` directly::
from django.db.models.fields import Field from django.db.models.fields import Field
Field.register_lookup(NotEqual) Field.register_lookup(NotEqual)
Now Field and all its subclasses have a NotEqual lookup. We can now use ``foo__ne`` for any field ``foo``. You will need to ensure that
this registration happens before you try to create any querysets using it. You
could place the implementation in a ``models.py`` file, or register the lookup
in the ``ready()`` method of an ``AppConfig``.
The first notable thing about `NotEqual` is the lookup_name. This name must Taking a closer look at the implementation, the first required attribute is
be supplied, and it is used by Django in the register_lookup() call so that ``lookup_name``. This allows the ORM to understand how to interpret ``name__ne``
Django knows to associate `ne` to the NotEqual implementation. and use ``NotEqual`` to generate the SQL. By convention, these names are always
` lowercase strings containing only letters, but the only hard requirement is
An Lookup works against two values, lhs and rhs. The abbreviations stand for that it must not contain the string ``__``.
left-hand side and right-hand side. The lhs is usually a field reference,
but it can be anything implementing the query expression API. The
rhs is the value given by the user. In the example `name__ne=Jack`, the
lhs is reference to Author's name field and Jack is the value.
The lhs and rhs are turned into values that are possible to use in SQL. A ``Lookup`` works against two values, ``lhs`` and ``rhs``, standing for
In the example above lhs is turned into "author"."name", [], and rhs is left-hand side and right-hand side. The left-hand side is usually a field
turned into "%s", ['Jack']. The lhs is just raw string without parameters reference, but it can be anything implementing the :ref:`query expression API
but the rhs is turned into a query parameter 'Jack'. <query-expression>`. The right-hand is the value given by the user. In the
example ``Author.objects.filter(name__ne='Jack')``, the left-hand side is a
reference to the ``name`` field of the ``Author`` model, and ``'Jack'`` is the
right-hand side.
Finally we combine the lhs and rhs by adding ` <> ` in between of them, We call ``process_lhs`` and ``process_rhs`` to convert them into the values we
and supply all the parameters for the query. need for SQL. In the above example, ``process_lhs`` returns
``('"author"."name"', [])`` and ``process_rhs`` returns ``('"%s"', ['Jack'])``.
In this example there were no parameters for the left hand side, but this would
depend on the object we have, so we still need to include them in the
parameters we return.
A Lookup needs to implement a limited part of query expression API. See Finally we combine the parts into a SQL expression with ``<>``, and supply all
the query expression API for details. the parameters for the query. We then return a tuple containing the generated
SQL string and the parameters.
A simple transformer example A simple transformer example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We will next write a simple transformer. The transformer will be called The custom lookup above is great, but in some cases you may want to be able to
`YearExtract`. It can be used to extract the year part from `DateField`. chain lookups together. For example, let's suppose we are building an
application where we want to make use of the ``abs()`` operator.
We have an ``Experiment`` model which records a start value, end value and the
change (start - end). We would like to find all experiments where the change
was equal to a certain amount (``Experiment.objects.filter(change__abs=27)``),
or where it did not exceede a certain amount
(``Experiment.objects.filter(change__abs__lt=27)``).
Lets start by writing the implementation:: .. note::
This example is somewhat contrived, but it demonstrates nicely the range of
functionality which is possible in a database backend independent manner,
and without duplicating functionality already in Django.
We will start by writing a ``AbsoluteValue`` transformer. This will use the SQL
function ``ABS()`` to transform the value before comparison::
from django.db.models import Extract from django.db.models import Extract
class YearExtract(Extract): class AbsoluteValue(Extract):
lookup_name = 'year' lookup_name = 'abs'
output_type = IntegerField()
def as_sql(self, qn, connection): def as_sql(self, qn, connection):
lhs, params = qn.compile(self.lhs) lhs, params = qn.compile(self.lhs)
return "EXTRACT(YEAR FROM %s)" % lhs, params return "ABS(%s)" % lhs, params
Next, lets register it for `DateField`:: Next, lets register it for ``IntegerField``::
from django.db.models import DateField from django.db.models import IntegerField
DateField.register_lookup(YearExtract) IntegerField.register_lookup(AbsoluteValue)
Now any DateField in your project will have `year` transformer. For example We can now run the queris we had before.
the following query:: ``Experiment.objects.filter(change__abs=27)`` will generate the following SQL::
Author.objects.filter(birthdate__year__lte=1981) SELECT ... WHERE ABS("experiments"."change") = 27
would translate to the following query on PostgreSQL:: By using ``Extract`` instead of ``Lookup`` it means we are able to chain
further lookups afterwards. So
``Experiment.objects.filter(change__abs__lt=27)`` will generate the following
SQL::
SELECT ... SELECT ... WHERE ABS("experiments"."change") < 27
FROM "author"
WHERE EXTRACT(YEAR FROM "author"."birthdate") <= 1981
An YearExtract class works only against self.lhs. Usually the lhs is Subclasses of ``Extract`` usually only operate on the left-hand side of the
transformed in some way. Further lookups and extracts work against the expression. Further lookups will work on the transformed value. Note that in
transformed value. this case where there is no other lookup specified, Django interprets
``change__abs=27`` as ``change__abs__exact=27``.
Note the definition of output_type in the `YearExtract`. The output_type is When looking for which lookups are allowable after the ``Extract`` has been
a field instance. It informs Django that the Extract class transformed the applied, Django uses the ``output_type`` attribute. We didn't need to specify
type of the value to an int. This is currently used only to check which this here as it didn't change, but supposing we were applying ``AbsoluteValue``
lookups the extract has. to some field which represents a more complex type (for example a point
relative to an origin, or a complex number) then we may have wanted to specify
``output_type = FloatField``, which will ensure that further lookups like
``abs__lte`` behave as they would for a ``FloatField``.
The used SQL in this example works on most databases. Check you database Writing an efficient abs__lt lookup
vendor's documentation to see if EXTRACT(year from date) is supported. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Writing an efficient year__exact lookup When using the above written ``abs`` lookup, the SQL produced will not use
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ indexes efficiently in some cases. In particular, when we use
``change__abs__lt=27``, this is equivalent to ``change__gt=-27`` AND
``change__lt=27``. (For the ``lte`` case we could use the SQL ``BETWEEN``).
When using the above written `year` lookup, the SQL produced will not use So we would like ``Experiment.objects.filter(change__abs__lt=27)`` to generate
indexes efficiently. We will fix that by writing a custom `exact` lookup the following SQL::
for YearExtract. For example if the user filters on
`birthdate__year__exact=1981`, then we want to produce the following SQL::
birthdate >= to_date('1981-01-01') AND birthdate <= to_date('1981-12-31') SELECT .. WHERE "experiments"."change" < 27 AND "experiments"."change" > -27
The implementation is:: The implementation is::
from django.db.models import Lookup from django.db.models import Lookup
class YearExact(Lookup): class AbsoluteValueLessThan(Lookup):
lookup_name = 'exact' lookup_name = 'lt'
def as_sql(self, qn, connection): def as_sql(self, qn, connection):
lhs, lhs_params = qn.compile(self.lhs.lhs) lhs, lhs_params = qn.compile(self.lhs.lhs)
rhs, rhs_params = self.process_rhs(qn, connection) rhs, rhs_params = self.process_rhs(qn, connection)
params = lhs_params + rhs_params + lhs_params + rhs_params params = lhs_params + rhs_params + lhs_params + rhs_params
return '%s >= to_date(%s || '-01-01') AND %s <= to_date(%s || '-12-31') % (lhs, rhs, lhs, rhs), params return '%s > %s AND %s < -%s % (lhs, rhs, lhs, rhs), params
YearExtract.register_lookup(YearExact) AbsoluteValue.register_lookup(AbsoluteValueLessThan)
There are a couple of notable things going on. First, `YearExact` isn't There are a couple of notable things going on. First, ``AbsoluteValueLessThan``
calling process_lhs(). Instead it skips and compiles directly the lhs used by isn't calling ``process_lhs()``. Instead it skips the transformation of the
self.lhs. The reason this is done is to skip `YearExtract` from adding the ``lhs`` done by ``AbsoluteValue`` and uses the original ``lhs``. That is, we
EXTRACT clause to the query. Referring directly to self.lhs.lhs is safe as want to get ``27`` not ``ABS(27)``. Referring directly to ``self.lhs.lhs`` is
`YearExact` can be accessed only from `year__exact` lookup, that is the lhs safe as ``AbsoluteValueLessThan`` can be accessed only from the
is always `YearExtract`. ``AbsoluteValue`` lookup, that is the ``lhs`` is always an instance of
``AbsoluteValue``.
Next, as both the lhs and rhs are used multiple times in the query the params Notice also that as both sides are used multiple times in the query the params
need to contain lhs_params and rhs_params multiple times. need to contain ``lhs_params`` and ``rhs_params`` multiple times.
The final query does string manipulation directly in the database. The reason The final query does the inversion (``27`` to ``-27``) directly in the
for doing this is that if the self.rhs is something else than a plain integer database. The reason for doing this is that if the self.rhs is something else
value (for exampel a `F()` reference) we can't do the transformations in than a plain integer value (for example an ``F()`` reference) we can't do the
Python. transformations in Python.
.. note::
In fact, most lookups with ``__abs`` could be implemented as range queries
like this, and on most database backend it is likely to be more sensible to
do so as you can make use of the indexes. However with PostgreSQL you may
want to add an index on ``abs(change)`` which would allow these queries to
be very efficient.
Writing alternative implemenatations for existing lookups Writing alternative implemenatations for existing lookups
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sometimes different database vendors require different SQL for the same Sometimes different database vendors require different SQL for the same
operation. For this example we will rewrite a custom implementation for operation. For this example we will rewrite a custom implementation for
MySQL for the NotEqual operator. Instead of `<>` we will be using `!=` MySQL for the NotEqual operator. Instead of ``<>`` we will be using ``!=``
operator. operator. (Note that in reality almost all databases support both, including
all the official databases supported by Django).
There are two ways to do this. The first is to write a subclass with a We can change the behaviour on a specific backend by creating a subclass of
as_mysql() method and registering the subclass over the original class:: ``NotEqual`` with a ``as_mysql`` method::
class MySQLNotEqual(NotEqual): class MySQLNotEqual(NotEqual):
def as_mysql(self, qn, connection): def as_mysql(self, qn, connection):
@ -179,80 +209,92 @@ as_mysql() method and registering the subclass over the original class::
return '%s != %s' % (lhs, rhs), params return '%s != %s' % (lhs, rhs), params
Field.register_lookup(MySQLNotExact) Field.register_lookup(MySQLNotExact)
The alternate is to monkey-patch the existing class in place:: We can then register it with ``Field``. It takes the place of the original
``NotEqual`` class as it has
def as_mysql(self, qn, connection): When compiling a query, Django first looks for ``as_%s % connection.vendor``
lhs, lhs_params = self.process_lhs(qn, connection) methods, and then falls back to ``as_sql``. The vendor names for the in-built
rhs, rhs_params = self.process_rhs(qn, connection) backends are ``sqlite``, ``postgresql``, ``oracle`` and ``mysql``.
params = lhs_params + rhs_params
return '%s != %s' % (lhs, rhs), params
NotEqual.as_mysql = as_mysql
The subclass way allows one to override methods of the lookup if needed. The .. note::
monkey-patch way allows writing different implementations for the same class If for some reason you need to change the lookup just for a specific query,
in different locations of the project. you can do that and reregister the original lookup afterwards. However you
need to be careful to ensure that your patch is in place until the queryset
is evaluated, not just created.
The way Django knows to call as_mysql() instead of as_sql() is as follows. .. _query-expression:
When qn.compile(notequal_instance) is called, Django first checks if there
is a method named 'as_%s' % connection.vendor. If that method doesn't exist,
the as_sql() will be called.
The vendor names for Django's in-built backends are 'sqlite', 'postgresql',
'oracle' and 'mysql'.
The Lookup API
~~~~~~~~~~~~~~
An lookup has attributes lhs and rhs. The lhs is something implementing the
query expression API and the rhs is either a plain value, or something that
needs to be compiled into SQL. Examples of SQL-compiled values include `F()`
references and usage of `QuerySets` as value.
A lookup needs to define lookup_name as a class level attribute. This is used
when registering lookups.
A lookup has three public methods. The as_sql(qn, connection) method needs
to produce a query string and parameters used by the query string. The qn has
a method compile() which can be used to compile self.lhs. However usually it
is better to call self.process_lhs(qn, connection) instead, which returns
query string and parameters for the lhs. Similary process_rhs(qn, connection)
returns query string and parameters for the rhs.
The Query Expression API The Query Expression API
~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~
A lookup can assume that the lhs responds to the query expression API. A lookup can assume that the lhs responds to the query expression API.
Currently direct field references, aggregates and `Extract` instances respond Currently direct field references, aggregates and ``Extract`` instances respond
to this API. to this API.
.. method:: as_sql(qn, connection) .. method:: as_sql(qn, connection)
Responsible for producing the query string and parameters for the expression. Responsible for producing the query string and parameters for the
The qn has a compile() method that can be used to compile other expressions. expression. The ``qn`` has a ``compile()`` method that can be used to
The connection is the connection used to execute the query. The compile other expressions. The ``connection`` is the connection used to
connection.vendor attribute can be used to return different query strings execute the query.
for different backends.
Calling expression.as_sql() directly is usually an error - instead Calling expression.as_sql() directly is usually incorrect - instead
qn.compile(expression) should be used. The qn.compile() method will take qn.compile(expression) should be used. The qn.compile() method will take
care of calling vendor-specific methods of the expression. care of calling vendor-specific methods of the expression.
.. method:: as_vendorname(qn, connection) .. method:: as_vendorname(qn, connection)
Works like as_sql() method. When an expression is compiled by qn.compile() Works like ``as_sql()`` method. When an expression is compiled by
Django will first try to call as_vendorname(), where vendorname is the vendor ``qn.compile()``, Django will first try to call ``as_vendorname()``, where
name of the backend used for executing the query. The vendorname is one of vendorname is the vendor name of the backend used for executing the query.
'postgresql', 'oracle', 'sqlite' or 'mysql' for Django's inbuilt backends. The vendorname is one of ``postgresql``, ``oracle``, ``sqlite`` or
``mysql`` for Django's built-in backends.
.. method:: get_lookup(lookup_name):: .. method:: get_lookup(lookup_name)
The get_lookup() method is used to fetch lookups. By default the lookup The ``get_lookup()`` method is used to fetch lookups. By default the lookup
is fetched from the expression's output type, but it is possible to override is fetched from the expression's output type, but it is possible to
this method to alter that behaviour. override this method to alter that behaviour.
.. attribute:: output_type .. attribute:: output_type
The output_type attribute is used by the get_lookup() method to check for The ``output_type`` attribute is used by the ``get_lookup()`` method to check for
lookups. The output_type should be a field instance. lookups. The output_type should be a field.
Note that this documentation lists only the public methods of the API. Note that this documentation lists only the public methods of the API.
Lookup reference
~~~~~~~~~~~~~~~~
.. class:: Lookup
In addition to the attributes and methods below, lookups also support
``as_sql`` and ``as_vendorname`` from the query expression API.
.. attribute:: lhs
The ``lhs`` (left-hand side) of a lookup tells us what we are comparing the
rhs to. It is an object which implements the query expression API. This is
likely to be a field, an aggregate or a subclass of ``Extract``.
.. attribute:: rhs
The ``rhs`` (right-hand side) of a lookup is the value we are comparing the
left hand side to. It may be a plain value, or something which compiles
into SQL, for example an ``F()`` object or a ``Queryset``.
.. attribute:: lookup_name
This class level attribute is used when registering lookups. It determines
the name used in queries to triger this lookup. For example, ``contains``
or ``exact``. This should not contain the string ``__``.
.. method:: process_lhs(qn, connection)
This returns a tuple of ``(lhs_string, lhs_params)``. In some cases you may
wish to compile ``lhs`` directly in your ``as_sql`` methods using
``qn.compile(self.lhs)``.
.. method:: process_rhs(qn, connection)
Behaves the same as ``process_lhs`` but acts on the right-hand side.