Fixed #29563 -- Added result streaming for QuerySet.iterator() on SQLite.

django/db/backends/sqlite3/features.py

@@ -4,11 +4,10 @@ from django.utils.functional import cached_property
 
 
 class DatabaseFeatures(BaseDatabaseFeatures):
-    # SQLite cannot handle us only partially reading from a cursor's result set
+    # SQLite can read from a cursor since SQLite 3.6.5, subject to the caveat
-    # and then writing the same rows to the database in another cursor. This
+    # that statements within a connection aren't isolated from each other. See
-    # setting ensures we always read result sets fully into memory all in one
+    # https://sqlite.org/isolation.html.
-    # go.
+    can_use_chunked_reads = True
-    can_use_chunked_reads = False
     test_db_allows_multiple_connections = False
     supports_unspecified_pk = True
     supports_timezones = False

docs/ref/databases.txt

@@ -710,6 +710,19 @@ can use the "pyformat" parameter style, where placeholders in the query
 are given as ``'%(name)s'`` and the parameters are passed as a dictionary
 rather than a list. SQLite does not support this.
 
+.. _sqlite-isolation:
+
+Isolation when using ``QuerySet.iterator()``
+--------------------------------------------
+
+There are special considerations described in `Isolation In SQLite`_ when
+modifying a table while iterating over it using :meth:`.QuerySet.iterator`. If
+a row is added, changed, or deleted within the loop, then that row may or may
+not appear, or may appear twice, in subsequent results fetched from the
+iterator. Your code must handle this.
+
+.. _`Isolation in SQLite`: https://sqlite.org/isolation.html
+
 .. _oracle-notes:
 
 Oracle notes

docs/ref/models/querysets.txt

@@ -2178,10 +2178,15 @@ don't support server-side cursors.
 Without server-side cursors
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-MySQL and SQLite don't support streaming results, hence the Python database
+MySQL doesn't support streaming results, hence the Python database driver loads
-drivers load the entire result set into memory. The result set is then
+the entire result set into memory. The result set is then transformed into
-transformed into Python row objects by the database adapter using the
+Python row objects by the database adapter using the ``fetchmany()`` method
-``fetchmany()`` method defined in :pep:`249`.
+defined in :pep:`249`.
+
+SQLite can fetch results in batches using ``fetchmany()``, but since SQLite
+doesn't provide isolation between queries within a connection, be careful when
+writing to the table being iterated over. See :ref:`sqlite-isolation` for
+more information.
 
 The ``chunk_size`` parameter controls the size of batches Django retrieves from
 the database driver. Larger batches decrease the overhead of communicating with
@@ -2195,6 +2200,10 @@ psycopg mailing list <https://www.postgresql.org/message-id/4D2F2C71.8080805%40d
     between the number of rows transferred and the data discarded if the loop
     is exited early.
 
+.. versionchanged:: 2.2
+
+    Support for result streaming on SQLite was added.
+
 ``latest()``
 ~~~~~~~~~~~~
 

docs/releases/2.2.txt

@@ -126,7 +126,7 @@ CSRF
 Database backends
 ~~~~~~~~~~~~~~~~~
 
-* ...
+* Added result streaming for :meth:`.QuerySet.iterator` on SQLite.
 
 Email
 ~~~~~