Fixed #22812 -- Refactored lookup API documentation.

Thanks Anssi and Tim for reviews.

docs/howto/custom-lookups.txt

@@ -1,20 +1,17 @@
 ==============
-Custom lookups
+Custom Lookups
 ==============
 
 .. versionadded:: 1.7
 
-.. module:: django.db.models.lookups
-   :synopsis: Custom lookups
-
 .. currentmodule:: django.db.models
 
-By default Django offers a wide variety of :ref:`built-in lookups
+Django offers a wide variety of :ref:`built-in lookups <field-lookups>` for
-<field-lookups>` for filtering (for example, ``exact`` and ``icontains``). This
+filtering (for example, ``exact`` and ``icontains``). This documentation
-documentation explains how to write custom lookups and how to alter the working
+explains how to write custom lookups and how to alter the working of existing
-of existing lookups.
+lookups. For the API references of lookups, see the :doc:`/ref/models/lookups`.
 
-A simple Lookup example
+A simple lookup example
 ~~~~~~~~~~~~~~~~~~~~~~~
 
 Let's start with a simple custom lookup. We will write a custom lookup ``ne``
@@ -95,14 +92,14 @@ A simple transformer example
 The custom lookup above is great, but in some cases you may want to be able to
 chain lookups together. For example, let's suppose we are building an
 application where we want to make use of the ``abs()`` operator.
-We have an ``Experiment`` model which records a start value, end value and the
+We have an ``Experiment`` model which records a start value, end value, and the
 change (start - end). We would like to find all experiments where the change
 was equal to a certain amount (``Experiment.objects.filter(change__abs=27)``),
 or where it did not exceed a certain amount
 (``Experiment.objects.filter(change__abs__lt=27)``).
 
 .. note::
-    This example is somewhat contrived, but it demonstrates nicely the range of
+    This example is somewhat contrived, but it nicely demonstrates the range of
     functionality which is possible in a database backend independent manner,
     and without duplicating functionality already in Django.
 
@@ -269,139 +266,3 @@ is not found, we look for a ``Transform`` and then the ``exact`` lookup on that
   ``myfield.get_lookup('mytransform')``, which will fail, so it will fall back
   to calling ``myfield.get_transform('mytransform')`` and then
   ``mytransform.get_lookup('exact')``.
-
-Lookups and transforms are registered using the same API - ``register_lookup``.
-
-.. _query-expression:
-
-The Query Expression API
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-A lookup can assume that the lhs responds to the query expression API.
-Currently direct field references, aggregates and ``Transform`` instances respond
-to this API.
-
-.. method:: as_sql(qn, connection)
-
-    Responsible for producing the query string and parameters for the
-    expression. The ``qn`` is a ``SQLCompiler`` object, which has a
-    ``compile()`` method that can be used to compile other expressions. The
-    ``connection`` is the connection used to execute the query.
-
-    Calling expression.as_sql() directly is usually incorrect - instead
-    ``qn.compile(expression)`` should be used. The ``qn.compile()`` method will
-    take care of calling vendor-specific methods of the expression.
-
-.. method:: as_vendorname(qn, connection)
-
-    Works like ``as_sql()`` method. When an expression is compiled by
-    ``qn.compile()``, Django will first try to call ``as_vendorname()``, where
-    vendorname is the vendor name of the backend used for executing the query.
-    The vendorname is one of ``postgresql``, ``oracle``, ``sqlite`` or
-    ``mysql`` for Django's built-in backends.
-
-.. method:: get_lookup(lookup_name)
-
-    The ``get_lookup()`` method is used to fetch lookups. By default the
-    lookup is fetched from the expression's output type in the same way
-    described in registering and fetching lookup documentation below.
-    It is possible to override this method to alter that behavior.
-
-.. method:: get_transform(lookup_name)
-
-    The ``get_transform()`` method is used when a transform is needed rather
-    than a lookup, or if a lookup is not found. This is a more complex
-    situation which is useful when there arbitrary possible lookups for a
-    field. Generally speaking, you will not need to override ``get_lookup()``
-    or ``get_transform()``, and can use ``register_lookup()`` instead.
-
-.. attribute:: output_field
-
-    The ``output_field`` attribute is used by the ``get_lookup()`` method to
-    check for lookups. The ``output_field`` should be a field.
-
-Note that this documentation lists only the public methods of the API.
-
-Lookup reference
-~~~~~~~~~~~~~~~~
-
-.. class:: Lookup
-
-    In addition to the attributes and methods below, lookups also support
-    ``as_sql`` and ``as_vendorname`` from the query expression API.
-
-.. attribute:: lhs
-
-    The ``lhs`` (left-hand side) of a lookup tells us what we are comparing the
-    rhs to. It is an object which implements the query expression API. This is
-    likely to be a field, an aggregate or a subclass of ``Transform``.
-
-.. attribute:: rhs
-
-    The ``rhs`` (right-hand side) of a lookup is the value we are comparing the
-    left hand side to. It may be a plain value, or something which compiles
-    into SQL, for example an ``F()`` object or a ``Queryset``.
-
-.. attribute:: lookup_name
-
-    This class level attribute is used when registering lookups. It determines
-    the name used in queries to trigger this lookup. For example, ``contains``
-    or ``exact``. This should not contain the string ``__``.
-
-.. method:: process_lhs(qn, connection)
-
-    This returns a tuple of ``(lhs_string, lhs_params)``. In some cases you may
-    wish to compile ``lhs`` directly in your ``as_sql`` methods using
-    ``qn.compile(self.lhs)``.
-
-.. method:: process_rhs(qn, connection)
-
-    Behaves the same as ``process_lhs`` but acts on the right-hand side.
-
-Transform reference
-~~~~~~~~~~~~~~~~~~~
-
-.. class:: Transform
-
-    In addition to implementing the query expression API Transforms have the
-    following methods and attributes.
-
-.. attribute:: lhs
-
-    The ``lhs`` (left-hand-side) of a transform contains the value to be
-    transformed. The ``lhs`` implements the query expression API.
-
-.. attribute:: lookup_name
-
-    This class level attribute is used when registering lookups. It determines
-    the name used in queries to trigger this lookup. For example, ``year``
-    or ``dayofweek``. This should not contain the string ``__``.
-
-.. _lookup-registration-api:
-
-Registering and fetching lookups
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The lookup registration API is explained below.
-
-.. classmethod:: register_lookup(lookup)
-
-    Registers the Lookup or Transform for the class. For example
-    ``DateField.register_lookup(YearExact)`` will register ``YearExact`` for
-    all ``DateFields`` in the project, but also for fields that are instances
-    of a subclass of ``DateField`` (for example ``DateTimeField``). You can
-    register a Lookup or a Transform using the same class method.
-
-.. method:: get_lookup(lookup_name)
-
-    Django uses ``get_lookup(lookup_name)`` to fetch lookups. The
-    implementation of ``get_lookup()`` looks for a subclass which is registered
-    for the current class with the correct ``lookup_name``.
-
-.. method:: get_transform(lookup_name)
-
-    Django uses ``get_transform(lookup_name)`` to fetch transforms. The
-    implementation of ``get_transform()`` looks for a subclass which is registered
-    for the current class with the correct ``transform_name``.
-
-The lookup registration API is available for ``Transform`` and ``Field`` classes.

docs/howto/custom-model-fields.txt

@@ -679,7 +679,7 @@ Django filter lookups: ``exact``, ``iexact``, ``contains``, ``icontains``,
 
 .. versionadded:: 1.7
 
-    If you are using :doc:`Custom lookups </ref/models/custom-lookups>` the
+    If you are using :doc:`Custom lookups </howto/custom-lookups>` the
     ``lookup_type`` can be any ``lookup_name`` used by the project's custom
     lookups.
 

docs/howto/index.txt

@@ -12,6 +12,7 @@ you quickly accomplish common tasks.
    auth-remote-user
    custom-management-commands
    custom-model-fields
+   custom-lookups
    custom-template-tags
    custom-file-storage
    deployment/index

docs/index.txt

@@ -67,7 +67,8 @@ manipulating the data of your Web application. Learn more about it below:
 * **QuerySets:**
   :doc:`Executing queries <topics/db/queries>` |
   :doc:`QuerySet method reference <ref/models/querysets>` |
-  :doc:`Query-related classes <ref/models/queries>`
+  :doc:`Query-related classes <ref/models/queries>` |
+  :doc:`Lookup expressions <ref/models/lookups>`
 
 * **Model instances:**
   :doc:`Instance methods <ref/models/instances>` |
@@ -85,7 +86,7 @@ manipulating the data of your Web application. Learn more about it below:
   :doc:`Aggregation <topics/db/aggregation>` |
   :doc:`Custom fields <howto/custom-model-fields>` |
   :doc:`Multiple databases <topics/db/multi-db>` |
-  :doc:`Custom lookups <ref/models/custom-lookups>`
+  :doc:`Custom lookups <howto/custom-lookups>`
 
 * **Other:**
   :doc:`Supported databases <ref/databases>` |

docs/ref/models/index.txt

@@ -14,4 +14,4 @@ Model API reference. For introductory material, see :doc:`/topics/db/models`.
    instances
    querysets
    queries
-   custom-lookups
+   lookups

docs/ref/models/lookups.txt

@@ -0,0 +1,207 @@
+====================
+Lookup API reference
+====================
+
+.. module:: django.db.models.lookups
+   :synopsis: Lookups API
+
+.. currentmodule:: django.db.models
+
+.. versionadded:: 1.7
+
+This document has the API references of lookups, the Django API for building
+the ``WHERE`` clause of a database query. To learn how to *use* lookups, see
+:doc:`/topics/db/queries`; to learn how to *create* new lookups, see
+:doc:`/howto/custom-lookups`.
+
+The lookup API has two components: a :class:`~lookups.RegisterLookupMixin` class
+that registers lookups, and the `Query Expression API <query-expression>`_, a
+set of methods that a class has to implement to be registrable as a lookup.
+
+Django has two base classes that follow the query expression API and from where
+all Django builtin lookups are derived:
+
+* :class:`Lookup`: to lookup a field (e.g. the ``exact`` of ``field_name__exact``)
+* :class:`Transform`: to transform a field
+
+A lookup expression consists of three parts:
+
+* Fields part (e.g. ``Book.objects.filter(author__best_friends__first_name...``);
+* Transforms part (may be omitted) (e.g. ``__lower__first3chars__reversed``);
+* A lookup (e.g. ``__icontains``) that, if omitted, defaults to ``__exact``.
+
+.. _lookup-registration-api:
+
+Registration API
+~~~~~~~~~~~~~~~~
+
+Django uses :class:`~lookups.RegisterLookupMixin` to give a class the interface to
+register lookups on itself. The two prominent examples are
+:class:`~django.db.models.Field`, the base class of all model fields, and
+``Aggregate``, the base class of all Django aggregates.
+
+.. class:: lookups.RegisterLookupMixin
+
+    A mixin that implements the lookup API on a class.
+
+    .. classmethod:: register_lookup(lookup)
+
+        Registers a new lookup in the class. For example
+        ``DateField.register_lookup(YearExact)`` will register ``YearExact``
+        lookup on ``DateField``. It overrides a lookup that already exists with
+        the same name.
+
+    .. method:: get_lookup(lookup_name)
+
+        Returns the :class:`Lookup` named ``lookup_name`` registered in the class.
+        The default implementation looks recursively on all parent classes
+        and checks if any has a registered lookup named ``lookup_name``, returning
+        the first match.
+
+    .. method:: get_transform(transform_name)
+
+        Returns a :class:`Transform` named ``transform_name``. The default
+        implementation looks recursively on all parent classes to check if any
+        has the registered transform named ``transform_name``, returning the first
+        match.
+
+For a class to be a lookup, it must follow the `Query Expression API
+<query-expression>`_. :class:`~Lookup` and :class:`~Transform` naturally
+follow this API.
+
+.. _query-expression:
+
+The Query Expression API
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The query expression API is a common set of methods that classes define to be
+usable in query expressions to translate themselves into SQL expressions. Direct
+field references, aggregates, and ``Transform`` are examples that follow this
+API. A class is said to follow the query expression API when it implements the
+following methods:
+
+.. method:: as_sql(self, qn, connection)
+
+    Responsible for producing the query string and parameters for the expression.
+    The ``qn`` is an ``SQLCompiler`` object, which has a ``compile()`` method
+    that can be used to compile other expressions. The ``connection`` is the
+    connection used to execute the query.
+
+    Calling ``expression.as_sql()`` is usually incorrect - instead
+    ``qn.compile(expression)`` should be used. The ``qn.compile()`` method will
+    take care of calling vendor-specific methods of the expression.
+
+.. method:: as_vendorname(self, qn, connection)
+
+    Works like ``as_sql()`` method. When an expression is compiled by
+    ``qn.compile()``, Django will first try to call ``as_vendorname()``, where
+    ``vendorname`` is the vendor name of the backend used for executing the
+    query. The ``vendorname`` is one of ``postgresql``, ``oracle``, ``sqlite``,
+    or ``mysql`` for Django's built-in backends.
+
+.. method:: get_lookup(lookup_name)
+
+    Must return the lookup named ``lookup_name``. For instance, by returning
+    ``self.output_field.get_lookup(lookup_name)``.
+
+.. method:: get_transform(transform_name)
+
+    Must return the lookup named ``transform_name``. For instance, by returning
+    ``self.output_field.get_transform(transform_name)``.
+
+.. attribute:: output_field
+
+    Defines the type of class returned by the ``get_lookup()`` method. It must
+    be a :class:`~django.db.models.Field` instance.
+
+Transform reference
+~~~~~~~~~~~~~~~~~~~
+
+.. class:: Transform
+
+    A ``Transform`` is a generic class to implement field transformations. A
+    prominent example is ``__year`` that transforms a ``DateField`` into a
+    ``IntegerField``.
+
+    The notation to use a ``Transform`` in an lookup expression is
+    ``<expression>__<transformation>`` (e.g. ``date__year``).
+
+    This class follows the `Query Expression API <query-expression>`_, which
+    implies that you can use ``<expression>__<transform1>__<transform2>``.
+
+    .. attribute:: lhs
+
+        The left-hand side - what is being transformed. It must follow the
+        `Query Expression API <query-expression>`_.
+
+    .. attribute:: lookup_name
+
+        The name of the lookup, used for identifying it on parsing query
+        expressions. It cannot contain the string ``"__"``.
+
+    .. attribute:: output_field
+
+        Defines the class this transformation outputs. It must be a
+        :class:`~django.db.models.Field` instance. By default is the same as
+        its ``lhs.output_field``.
+
+    .. method:: as_sql
+
+        To be overridden; raises :exc:`NotImplementedError`.
+
+    .. method:: get_lookup(lookup_name)
+
+        Same as :meth:`~lookups.RegisterLookupMixin.get_lookup()`.
+
+    .. method:: get_transform(transform_name)
+
+        Same as :meth:`~lookups.RegisterLookupMixin.get_transform()`.
+
+Lookup reference
+~~~~~~~~~~~~~~~~
+
+.. class:: Lookup
+
+    A ``Lookup`` is a generic class to implement lookups. A lookup is a query
+    expression with a left-hand side, :attr:`lhs`; a right-hand side,
+    :attr:`rhs`; and a ``lookup_name`` that is used to produce a boolean
+    comparison between ``lhs`` and ``rhs`` such as ``lhs in rhs`` or
+    ``lhs > rhs``.
+
+    The notation to use a lookup in an expression is
+    ``<lhs>__<lookup_name>=<rhs>``.
+
+    This class doesn't follow the `Query Expression API <query-expression>`_
+    since it has ``=<rhs>`` on its construction: lookups are always the end of
+    a lookup expression.
+
+    .. attribute:: lhs
+
+        The left-hand side - what is being looked up. The object must follow
+        the `Query Expression API <query-expression>`_.
+
+    .. attribute:: rhs
+
+        The right-hand side - what ``lhs`` is being compared against. It can be
+        a plain value, or something that compiles into SQL, typically an
+        ``F()`` object or a ``QuerySet``.
+
+    .. attribute:: lookup_name
+
+        The name of this lookup, used to identify it on parsing query
+        expressions. It cannot contain the string ``"__"``.
+
+    .. method:: process_lhs(qn, connection[, lhs=None])
+
+        Returns a tuple ``(lhs_string, lhs_params)``, as returned by
+        ``qn.compile(lhs)``. This method can be overridden to tune how the
+        ``lhs`` is processed.
+
+        ``qn`` is an ``SQLCompiler`` object, to be used like ``qn.compile(lhs)``
+        for compiling ``lhs``. The ``connection`` can be used for compiling
+        vendor specific SQL. If ``lhs`` is not ``None``, use it as the
+        processed ``lhs`` instead of ``self.lhs``.
+
+    .. method:: process_rhs(qn, connection)
+
+        Behaves the same way as :meth:`process_lhs`, for the right-hand side.

docs/ref/models/querysets.txt

@@ -2061,7 +2061,7 @@ For an introduction, see :ref:`models and database queries documentation
 <field-lookups-intro>`.
 
 Django's inbuilt lookups are listed below. It is also possible to write
-:doc:`custom lookups </ref/models/custom-lookups>` for model fields.
+:doc:`custom lookups </howto/custom-lookups>` for model fields.
 
 As a convenience when no lookup type is provided (like in
 ``Entry.objects.get(id=14)``) the lookup type is assumed to be :lookup:`exact`.

docs/releases/1.7.txt

@@ -277,7 +277,7 @@ to ``DateField`` it is possible to filter on the transformed value, for
 example ``qs.filter(author__birthdate__year__lte=1981)``.
 
 For more information about both custom lookups and transforms refer to
-:doc:`custom lookups </ref/models/custom-lookups>` documentation.
+:doc:`custom lookups </howto/custom-lookups>` documentation.
 
 Improvements to ``Form`` error handling
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~