The original problem was that queryset cloning was really expensive
when filtering with F() clauses. The __deepcopy__ went too deep copying
_meta attributes of the models used. To fix this the use of
__deepcopy__ in qs cloning was removed.
This commit results in some speed improvements across the djangobench
benchmark suite. Most query_* tests are 20-30% faster, save() is 50%
faster and finally complex filtering situations can see 2x to order
of magnitude improvments.
Thanks to Suor, Alex and lrekucki for valuable feedback.
The guarantee that no queries will be made when accessing results is
done by new EmptyWhere class which is used for query.where and having.
Thanks to Simon Charette for reviewing and valuable suggestions.
The added promotion logic is based on promoting any joins used in only
some of the childs of an OR clause unless the join existed before the
OR clause addition.
The ORM didn't reuse joins for direct foreign key traversals when using
chained filters. For example:
qs.filter(fk__somefield=1).filter(fk__somefield=2))
produced two joins.
As a bonus, reverse onetoone filters can now reuse joins correctly
The regression was caused by the join() method refactor in commit
68847135bc
Thanks for Simon Charette for spotting some issues with the first draft
of the patch.
This is a rather large refactoring. The "lookup traversal" code was
splitted out from the setup_joins. There is now names_to_path() method
which does the lookup traveling, the actual work of setup_joins() is
calling names_to_path() and then adding the joins found into the query.
As a side effect it was possible to remove the "process_extra"
functionality used by genric relations. This never worked for left
joins. Now the extra restriction is appended directly to the join
condition instead of the where clause.
To generate the extra condition we need to have the join field
available in the compiler. This has the side-effect that we need more
ugly code in Query.__getstate__ and __setstate__ as Field objects
aren't pickleable.
The join trimming code got a big change - now we trim all direct joins
and never trim reverse joins. This also fixes the problem in #10790
which was join trimming in null filter cases.
The dupe avoidance logic was removed as it doesn't seem to do anything,
it is complicated, and it has nearly zero documentation.
The removal of dupe_avoidance allowed for refactoring of both the
implementation and signature of Query.join(). This refactoring cascades
again to some other parts. The most significant of them is the changes
in qs.combine(), and compiler.select_related_descent().
The ORM generated a query with INNER JOIN instead of LEFT OUTER JOIN
in a somewhat complicated case. The main issue was that there was a
chain of nullable FK -> non-nullble FK, and the join promotion logic
didn't see the need to promote the non-nullable FK even if the
previous nullable FK was already promoted to LOUTER JOIN. This resulted
in a query like a LOUTER b INNER c, which incorrectly prunes results.
An ordering test had two problems related to dict.items() usage:
- It assumed the order of the dict was non-randomized
- It indexed to the dict.items() which is py3 incompatible.
I fixed the test by using dict['rank'] directly, where rank is the
column tested on the values queryset.
The qs.bulk_create() method did not work with large batches together
with SQLite3. This commit adds a way to split the bulk into smaller
batches. The default batch size is unlimited except for SQLite3 where
the batch size is limited to 999 SQL parameters per batch.
Thanks to everybody who participated in the discussions at Trac.
At least Oracle needs parentheses in negated where conditions, even if
there is only single condition negated. Fixed this by reverting to old
logic in that part of as_sql() and adding a comment about this.
I did not investigate why the parentheses are needed. The original
offending commit was bd283aa844.
Made sure the WhereNode.as_sql() handles various EmptyResultSet and
FullResultSet conditions correctly. Also, got rid of the FullResultSet
exception class. It is now represented by '', [] return value in the
as_sql() methods.
When order_by causes new joins to be added to the query, the joins must
be LEFT OUTER joins for nullable relations, otherwise the order_by
could cause the results to be altered. This commit fixes the logic to
only promote new joins, previously all joins in the order_by lookup
path were promoted.
Thanks to Bruno Desthuilliers for spotting this corner case.
Fixed#18248 -- proxy models were added to included_inherited_models
in sql.query.Query. The variable is meant to be used for multitable
inheritance only. This mistake caused problems in situations where
proxy model's query was reused.
Fixed#17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.
The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.
Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
QuerySet had previously some complex logic for dealing with nullable
fields in negated add_filter() calls. It seems the logic is leftover
from a time where the WhereNode wasn't as intelligent in handling
field__in=[] conditions.
Thanks to aaugustin for comments on the patch.
only consider some fields (PostgreSQL only).
For this, the ``distinct()`` QuerySet method now accepts an optional
list of model fields names and generates ``DISTINCT ON`` clauses on
these cases. Thanks Jeffrey Gelens and Anssi Kääriäinen for their work.
Fixes#6422.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@17244 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Replaced them with a new assertRaisesMessage method of a new SimpleTestCase, a lightweight subclass of unittest.TestCase. Both are also available for usage in user tests.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@16610 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Claude Paroz
Anssi Kääriäinen
Florian Apolloner
Jan Bednařík
Alex Gaynor
Alex Gaynor
Julien Phalip
Ramiro Morales
Adrian Holovaty
Russell Keith-Magee
Malcolm Tredinnick
Jannis Leidel
Ian Kelly
Carl Meyer