Merge pull request #1544 from evildmp/ticket_20920_rebase

Fixed #20920 -- Consolidated F() and Q() documentation

docs/index.txt

@@ -65,7 +65,8 @@ manipulating the data of your Web application. Learn more about it below:
 
 * **QuerySets:**
   :doc:`Executing queries <topics/db/queries>` |
-  :doc:`QuerySet method reference <ref/models/querysets>`
+  :doc:`QuerySet method reference <ref/models/querysets>` |
+  :doc:`Query-related classes <ref/models/queries>`
 
 * **Model instances:**
   :doc:`Instance methods <ref/models/instances>` |

docs/ref/models/fields.txt

@@ -1062,11 +1062,12 @@ define the details of how the relation works.
     only allows the choice of related objects with a ``pub_date`` before the
     current date to be chosen.
 
-    Instead of a dictionary this can also be a :class:`~django.db.models.Q`
-    object for more :ref:`complex queries <complex-lookups-with-q>`. However,
-    if ``limit_choices_to`` is a :class:`~django.db.models.Q` object then it
-    will only have an effect on the choices available in the admin when the
-    field is not listed in ``raw_id_fields`` in the ``ModelAdmin`` for the model.
+    Instead of a dictionary this can also be a :class:`Q object
+    <django.db.models.Q>` for more :ref:`complex queries
+    <complex-lookups-with-q>`. However, if ``limit_choices_to`` is a :class:`Q
+    object <django.db.models.Q>` then it will only have an effect on the
+    choices available in the admin when the field is not listed in
+    ``raw_id_fields`` in the ``ModelAdmin`` for the model.
 
 .. attribute:: ForeignKey.related_name
 

docs/ref/models/index.txt

@@ -12,3 +12,4 @@ Model API reference. For introductory material, see :doc:`/topics/db/models`.
    options
    instances
    querysets
+   queries

docs/ref/models/instances.txt

@@ -342,6 +342,8 @@ only.
 
 Using ``update_fields`` will force an update similarly to ``force_update``.
 
+.. _ref-models-field-updates-using-f-expressions:
+
 Updating attributes based on existing fields
 --------------------------------------------
 
@@ -356,35 +358,21 @@ achieve this is to do something like::
 If the old ``number_sold`` value retrieved from the database was 10, then
 the value of 11 will be written back to the database.
 
-This sequence has a standard update problem in that it contains a race
-condition. If another thread of execution has already saved an updated value
-after the current thread retrieved the old value, the current thread will only
-save the old value plus one, rather than the new (current) value plus one.
-
-The process can be made robust and slightly faster by expressing the update
-relative to the original field value, rather than as an explicit assignment of
-a new value. Django provides :ref:`F() expressions <query-expressions>` for
-performing this kind of relative update. Using ``F()`` expressions, the
-previous example is expressed as::
+The process can be made robust, :ref:`avoiding a race condition
+<avoiding-race-conditions-using-f>`, as well as slightly faster by expressing
+the update relative to the original field value, rather than as an explicit
+assignment of a new value. Django provides :class:`F expressions
+<django.db.models.F>` for performing this kind of relative update. Using
+:class:`F expressions <django.db.models.F>`, the previous example is expressed
+as::
 
     >>> from django.db.models import F
     >>> product = Product.objects.get(name='Venezuelan Beaver Cheese')
     >>> product.number_sold = F('number_sold') + 1
     >>> product.save()
 
-This approach doesn't use the initial value from the database. Instead, it
-makes the database do the update based on whatever value is current at the time
-that the :meth:`~Model.save()` is executed.
-
-Once the object has been saved, you must reload the object in order to access
-the actual value that was applied to the updated field::
-
-    >>> product = Products.objects.get(pk=product.pk)
-    >>> print(product.number_sold)
-    42
-
-For more details, see the documentation on :ref:`F() expressions
-<query-expressions>` and their :ref:`use in update queries
+For more details, see the documentation on :class:`F expressions
+<django.db.models.F>` and their :ref:`use in update queries
 <topics-db-queries-update>`.
 
 Specifying which fields to save

docs/ref/models/queries.txt

@@ -0,0 +1,127 @@
+=====================
+Query-related classes
+=====================
+
+.. currentmodule:: django.db.models
+
+This document provides reference material for query-related tools not
+documented elsewhere.
+
+``F()`` expressions
+===================
+
+.. class:: F
+
+An ``F()`` object represents the value of a model field. It makes it possible
+to refer to model field values and perform database operations using them
+without actually having to pull them out of the database into Python memory.
+
+Instead, Django uses the ``F()`` object to generate a SQL expression that
+describes the required operation at the database level.
+
+This is easiest to understand though an example. Normally, one might do
+something like this::
+
+    # Tintin filed a news story!
+    reporter = Reporters.objects.get(name='Tintin')
+    reporter.stories_filed += 1
+    reporter.save()
+
+Here, we have pulled the value of ``reporter.stories_filed`` from the database
+into memory and manipulated it using familiar Python operators, and then saved
+the object back to the database. But instead we could also have done::
+
+    from django.db.models import F
+    reporter = Reporters.objects.get(name='Tintin')
+    reporter.stories_filed = F('stories_filed') + 1
+    reporter.save()
+
+Although ``reporter.stories_filed = F('stories_filed') + 1`` looks like a
+normal Python assignment of value to an instance attribute, in fact it's an SQL
+construct describing an operation on the database.
+
+When Django encounters an instance of ``F()``, it overrides the standard Python
+operators to create an encapsulated SQL expression; in this case, one which
+instructs the database to increment the database field represented by
+``reporter.stories_filed``.
+
+Whatever value is or was on ``reporter.stories_filed``, Python never gets to
+know about it - it is dealt with entirely by the database. All Python does,
+through Django's ``F()`` class, is create the SQL syntax to refer to the field
+and describe the operation.
+
+.. note::
+
+   In order to access the new value that has been saved in this way, the object
+   will need to be reloaded::
+
+       reporter = Reporters.objects.get(pk=reporter.pk)
+
+As well as being used in operations on single instances as above, ``F()`` can
+be used on ``QuerySets`` of object instances, with ``update()``. This reduces
+the two queries we were using above - the ``get()`` and the
+:meth:`~Model.save()` - to just one::
+
+    reporter = Reporters.objects.filter(name='Tintin')
+    reporter.update(stories_filed=F('stories_filed') + 1)
+
+We can also use :meth:`~django.db.models.query.QuerySet.update()` to increment
+the field value on multiple objects - which could be very much faster than
+pulling them all into Python from the database, looping over them, incrementing
+the field value of each one, and saving each one back to the database::
+
+    Reporter.objects.all().update(stories_filed=F('stories_filed) + 1)
+
+``F()`` therefore can offer performance advantages by:
+
+* getting the database, rather than Python, to do work
+* reducing the number of queries some operations require
+
+.. _avoiding-race-conditions-using-f:
+
+Avoiding race conditions using ``F()``
+--------------------------------------
+
+Another useful benefit of ``F()`` is that having the database - rather than
+Python - update a field's value avoids a *race condition*.
+
+If two Python threads execute the code in the first example above, one thread
+could retrieve, increment, and save a field's value after the other has
+retrieved it from the database. The value that the second thread saves will be
+based on the original value; the work of the first thread will simply be lost.
+
+If the database is responsible for updating the field, the process is more
+robust: it will only ever update the field based on the value of the field in
+the database when the :meth:`~Model.save()` or ``update()`` is executed, rather
+than based on its value when the instance was retrieved.
+
+Using ``F()`` in filters
+------------------------
+
+``F()`` is also very useful in ``QuerySet`` filters, where they make it
+possible to filter a set of objects against criteria based on their field
+values, rather than on Python values.
+
+This is documented in :ref:`using F() expressions in queries
+<using-f-expressions-in-filters>`
+
+Supported operations with ``F()``
+---------------------------------
+
+As well as addition, Django supports subtraction, multiplication, division,
+and modulo arithmetic with ``F()`` objects, using Python constants,
+variables, and even other ``F()`` objects.
+
+``Q()`` objects
+===============
+
+.. class:: Q
+
+A ``Q()`` object, like an :class:`~django.db.models.F` object, encapsulates a
+SQL expression in a Python object that can be used in database-related
+operations.
+
+In general, ``Q() objects`` make it possible to define and reuse conditions.
+This permits the :ref:`construction of complex database queries
+<complex-lookups-with-q>` using ``|`` (``OR``) and ``&`` (``AND``) operators;
+in particular, it is not otherwise possible to use ``OR`` in ``QuerySets``.

docs/releases/1.1-alpha-1.txt

@@ -46,15 +46,15 @@ Query expressions
 Queries can now refer to a another field on the query and can traverse
 relationships to refer to fields on related models. This is implemented in the
 new :class:`F` object; for full details, including examples, consult the
-:ref:`documentation for F expressions <query-expressions>`.
+:class:`F expressions documentation <django.db.models.F>`.
 
 Performance improvements
 ------------------------
 
 .. currentmodule:: django.test
 
-Tests written using Django's :doc:`testing framework </topics/testing/index>` now run
-dramatically faster (as much as 10 times faster in many cases).
+Tests written using Django's :doc:`testing framework </topics/testing/index>`
+now run dramatically faster (as much as 10 times faster in many cases).
 
 This was accomplished through the introduction of transaction-based tests: when
 using :class:`django.test.TestCase`, your tests will now be run in a transaction

docs/releases/1.1.txt

@@ -211,7 +211,7 @@ Query expressions
 Queries can now refer to a another field on the query and can traverse
 relationships to refer to fields on related models. This is implemented in the
 new :class:`~django.db.models.F` object; for full details, including examples,
-consult the :ref:`documentation for F expressions <query-expressions>`.
+consult the :class:`F expressions documentation <django.db.models.F>`.
 
 Model improvements
 ------------------

docs/releases/1.3.txt

@@ -324,7 +324,7 @@ requests. These include:
   to :meth:`~django.shortcuts.render_to_response()` providing a
   :class:`~django.template.RequestContext` by default.
 
-* Support for combining :ref:`F() expressions <query-expressions>`
+* Support for combining :class:`F expressions <django.db.models.F>`
   with timedelta values when retrieving or updating database values.
 
 .. _HTTPOnly: https://www.owasp.org/index.php/HTTPOnly

docs/releases/1.5-alpha-1.txt

@@ -562,11 +562,12 @@ Miscellaneous
   needs. The new default value is ``0666`` (octal) and the current umask value
   is first masked out.
 
-* The :ref:`F() expressions <query-expressions>` supported bitwise operators by
-  ``&`` and ``|``. These operators are now available using ``.bitand()`` and
-  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be consistent with
-  :ref:`Q() expressions <complex-lookups-with-q>` and ``QuerySet`` combining where
-  the operators are used as boolean AND and OR operators.
+* The :class:`F expressions <django.db.models.F>` supported bitwise operators
+  by ``&`` and ``|``. These operators are now available using ``.bitand()`` and
+  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be
+  consistent with :ref:`Q() expressions <complex-lookups-with-q>` and
+  ``QuerySet`` combining where the operators are used as boolean AND and OR
+  operators.
 
 * The :ttag:`csrf_token` template tag is no longer enclosed in a div. If you need
   HTML validation against pre-HTML5 Strict DTDs, you should add a div around it

docs/releases/1.5-beta-1.txt

@@ -601,13 +601,14 @@ Miscellaneous
   needs. The new default value is ``0666`` (octal) and the current umask value
   is first masked out.
 
-* The :ref:`F() expressions <query-expressions>` supported bitwise operators by
+* The :class:`F expressions <django.db.models.F>` supported bitwise operators by
   ``&`` and ``|``. These operators are now available using ``.bitand()`` and
-  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be consistent with
-  :ref:`Q() expressions <complex-lookups-with-q>` and ``QuerySet`` combining where
-  the operators are used as boolean AND and OR operators.
+  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be
+  consistent with :ref:`Q() expressions <complex-lookups-with-q>` and
+  ``QuerySet`` combining where the operators are used as boolean AND and OR
+  operators.
 
-* In a ``filter()`` call, when :ref:`F() expressions <query-expressions>`
+* In a ``filter()`` call, when :class:`F expressions <django.db.models.F>`
   contained lookups spanning multi-valued relations, they didn't always reuse
   the same relations as other lookups along the same chain. This was changed,
   and now F() expressions will always use the same relations as other lookups

docs/releases/1.5.txt

@@ -679,13 +679,14 @@ Miscellaneous
   needs. The new default value is ``0666`` (octal) and the current umask value
   is first masked out.
 
-* The :ref:`F() expressions <query-expressions>` supported bitwise operators by
+* The :class:`F expressions <django.db.models.F>` supported bitwise operators by
   ``&`` and ``|``. These operators are now available using ``.bitand()`` and
-  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be consistent with
-  :ref:`Q() expressions <complex-lookups-with-q>` and ``QuerySet`` combining where
-  the operators are used as boolean AND and OR operators.
+  ``.bitor()`` instead. The removal of ``&`` and ``|`` was done to be
+  consistent with :ref:`Q() expressions <complex-lookups-with-q>` and
+  ``QuerySet`` combining where the operators are used as boolean AND and OR
+  operators.
 
-* In a ``filter()`` call, when :ref:`F() expressions <query-expressions>`
+* In a ``filter()`` call, when :class:`F expressions <django.db.models.F>`
   contained lookups spanning multi-valued relations, they didn't always reuse
   the same relations as other lookups along the same chain. This was changed,
   and now F() expressions will always use the same relations as other lookups

docs/topics/db/optimization.txt

@@ -111,7 +111,7 @@ For instance:
 * At the most basic level, use :ref:`filter and exclude <queryset-api>` to do
   filtering in the database.
 
-* Use :ref:`F() object query expressions <query-expressions>` to do filtering
+* Use :class:`F expressions <django.db.models.F>` to do filtering
   against other fields within the same model.
 
 * Use :doc:`annotate to do aggregation in the database </topics/db/aggregation>`.

docs/topics/db/queries.txt

@@ -588,18 +588,16 @@ relation). Conditions in subsequent
 :meth:`~django.db.models.query.QuerySet.exclude` calls that refer to the same
 relation may end up filtering on different linked objects.
 
-.. _query-expressions:
+.. _using-f-expressions-in-filters:
 
 Filters can reference fields on the model
 -----------------------------------------
 
-.. class:: F
-
 In the examples given so far, we have constructed filters that compare
 the value of a model field with a constant. But what if you want to compare
 the value of a model field with another field on the same model?
 
-Django provides the :ref:`F() expressions <query-expressions>` to allow such
+Django provides :class:`F expressions <django.db.models.F>` to allow such
 comparisons. Instances of ``F()`` act as a reference to a model field within a
 query. These references can then be used in query filters to compare the values
 of two different fields on the same model instance.
@@ -779,15 +777,11 @@ being evaluated and therefore populate the cache::
 Complex lookups with Q objects
 ==============================
 
-.. class:: Q
-
 Keyword argument queries -- in :meth:`~django.db.models.query.QuerySet.filter`,
 etc. -- are "AND"ed together. If you need to execute more complex queries (for
-example, queries with ``OR`` statements), you can use ``Q`` objects.
+example, queries with ``OR`` statements), you can use :class:`Q objects <django.db.models.Q>`.
 
-.. comment: Link to Q does not work, since this documentation does not exist yet.
-
-A :class:`~django.db.models.Q` object (``django.db.models.Q``) is an object
+A :class:`Q object <django.db.models.Q>` (``django.db.models.Q``) is an object
 used to encapsulate a collection of keyword arguments. These keyword arguments
 are specified as in "Field lookups" above.
 
@@ -1019,10 +1013,10 @@ over them and call :meth:`~django.db.models.Model.save`::
     for item in my_queryset:
         item.save()
 
-Calls to update can also use :ref:`F() objects <query-expressions>` to update
-one field based on the value of another field in the model. This is especially
-useful for incrementing counters based upon their current value. For example, to
-increment the pingback count for every entry in the blog::
+Calls to update can also use :class:`F expressions <django.db.models.F>` to
+update one field based on the value of another field in the model. This is
+especially useful for incrementing counters based upon their current value. For
+example, to increment the pingback count for every entry in the blog::
 
     >>> Entry.objects.all().update(n_pingbacks=F('n_pingbacks') + 1)