In the combination of .values().aggregate() the aggregate_select_mask
didn't include the aggregates added. This resulted in bogus query.
Thanks to Trac alias debanshuk for report.
There were a couple of places which used Query.join() directly. By
using setup_joins() in these places the code is more DRY, and in
addition there is no need to directly call field.get_joining_columns()
unless the field is the given join_field from get_path_info(). This
makes it easier to make sure a ForeignObject subclass generates joins
correctly in all cases.
The join used by select_related was incorrectly INNER when the query
had an ORed filter for nullable join that was trimmed away. Fixed this
by forcing the join type to LOUTER even when a join was trimmed away
in ORed queries.
The patch for #19385 caused a regression in certain generic relations
.exclude() filters if a subquery was needed. The fix contains a
refactoring to how Query.split_exclude() and Query.trim_start()
interact.
Thanks to Trac alias nferrari for the report.
The sql/query.py add_q method did a lot of where/having tree hacking to
get complex queries to work correctly. The logic was refactored so that
it should be simpler to understand. The new logic should also produce
leaner WHERE conditions.
The changes cascade somewhat, as some other parts of Django (like
add_filter() and WhereNode) expect boolean trees in certain format or
they fail to work. So to fix the add_q() one must fix utils/tree.py,
some things in add_filter(), WhereNode and so on.
This commit also fixed add_filter to see negate clauses up the path.
A query like .exclude(Q(reversefk__in=a_list)) didn't work similarly to
.filter(~Q(reversefk__in=a_list)). The reason for this is that only
the immediate parent negate clauses were seen by add_filter, and thus a
tree like AND: (NOT AND: (AND: condition)) will not be handled
correctly, as there is one intermediary AND node in the tree. The
example tree is generated by .exclude(~Q(reversefk__in=a_list)).
Still, aggregation lost connectors in OR cases, and F() objects and
aggregates in same filter clause caused GROUP BY problems on some
databases.
Fixed#17600, fixed#13198, fixed#17025, fixed#17000, fixed#11293.
Before there was need to have both .relabel_aliases() and .clone() for
many structs. Now there is only relabeled_clone() for those structs
where alias is the only mutable attribute.
The join promote=True was over-aggressive in select_related handling.
After that was removed, the only other user was query.combine(). That
use case is very easy to handle locally, so there is no more need for
the join(promote=True) flag.
Refs #19849.
The refactoring mainly concentrates on making sure the inner and outer
query agree about the split position. The split position is where the
multijoin happens, and thus the split position also determines the
columns used in the "WHERE col1 IN (SELECT col2 from ...)" condition.
This commit fixes a regression caused by #10790 and commit
69597e5bcc. The regression was caused
by wrong cols in the split position.
Previously it was possible to call clear_ordering without the
force_empty argument. The result was that the query was still ordered
by model's meta ordering if that was defined. By making the arg
mandatory it will be easier to spot possible errors caused by assuming
clear_ordering will remove all ordering.
Thanks to Dylan Klomparens for the suggestion. Refs #19720.
The original problem was that queryset cloning was really expensive
when filtering with F() clauses. The __deepcopy__ went too deep copying
_meta attributes of the models used. To fix this the use of
__deepcopy__ in qs cloning was removed.
This commit results in some speed improvements across the djangobench
benchmark suite. Most query_* tests are 20-30% faster, save() is 50%
faster and finally complex filtering situations can see 2x to order
of magnitude improvments.
Thanks to Suor, Alex and lrekucki for valuable feedback.
The guarantee that no queries will be made when accessing results is
done by new EmptyWhere class which is used for query.where and having.
Thanks to Simon Charette for reviewing and valuable suggestions.
The added promotion logic is based on promoting any joins used in only
some of the childs of an OR clause unless the join existed before the
OR clause addition.
The ORM didn't reuse joins for direct foreign key traversals when using
chained filters. For example:
qs.filter(fk__somefield=1).filter(fk__somefield=2))
produced two joins.
As a bonus, reverse onetoone filters can now reuse joins correctly
The regression was caused by the join() method refactor in commit
68847135bc
Thanks for Simon Charette for spotting some issues with the first draft
of the patch.
This is a rather large refactoring. The "lookup traversal" code was
splitted out from the setup_joins. There is now names_to_path() method
which does the lookup traveling, the actual work of setup_joins() is
calling names_to_path() and then adding the joins found into the query.
As a side effect it was possible to remove the "process_extra"
functionality used by genric relations. This never worked for left
joins. Now the extra restriction is appended directly to the join
condition instead of the where clause.
To generate the extra condition we need to have the join field
available in the compiler. This has the side-effect that we need more
ugly code in Query.__getstate__ and __setstate__ as Field objects
aren't pickleable.
The join trimming code got a big change - now we trim all direct joins
and never trim reverse joins. This also fixes the problem in #10790
which was join trimming in null filter cases.
The trim argument was used by split_exclude() only to trim the last
join from the given lookup. It is cleaner to just trim the last part
from the lookup in split_exclude() directly so that there is no need
to burden add_filter() with the logic needed for only split_exclude().
F() expressions reuse joins like any lookup in a .filter() call -
reuse multijoins generated in the same .filter() call else generate
new joins. Also, lookups can now reuse joins generated by F().
This change is backwards incompatible, but it is required to prevent
dict randomization from generating different queries depending on
.filter() kwarg ordering. The new way is also more consistent in how
joins are reused.
The dupe avoidance logic was removed as it doesn't seem to do anything,
it is complicated, and it has nearly zero documentation.
The removal of dupe_avoidance allowed for refactoring of both the
implementation and signature of Query.join(). This refactoring cascades
again to some other parts. The most significant of them is the changes
in qs.combine(), and compiler.select_related_descent().
The Query.select and Query.select_fields were collapsed into one list
because the attributes had to be always in sync. Now that they are in
one attribute it is impossible to edit them out of sync.
Similar collapse was done for Query.related_select_cols and
Query.related_select_fields.
There was a bug introduced in #18676 which caused fast-path deletes
implemented as "DELETE WHERE pk IN <subquery>" to fail if the SELECT
clause contained additional stuff (for example extra() and annotate()).
Thanks to Trac alias pressureman for spotting this regression.
In an ideal world, nothing except django.db.models.query should have to
import stuff from django.models.sql.*. A few things were needing to get
hold of sql.constants.LOOKUP_SEP, so this commit moves it up to
django.db.models.constants.LOOKUP_SEP.
There are still a couple of places (admin) poking into sql.* to get
QUERY_TERMS, which is unfortunate, but a slightly different issue and
harder to adjust.
The joins for nested nullable foreign keys were often created as INNER
when they should have been OUTER joins. The reason was that only the
first join in the chain was promoted correctly. There were also issues
with select_related etc.
The basic structure for this problem was:
A -[nullable]-> B -[nonnull]-> C
And the basic problem was that the A->B join was correctly LOUTER,
the B->C join not.
The major change taken in this patch is that now if we promote a join
A->B, we will automatically promote joins B->X for all X in the query.
Also, we now make sure there aren't ever join chains like:
a LOUTER b INNER c
If the a -> b needs to be LOUTER, then the INNER at the end of the
chain will cancel the LOUTER join and we have a broken query.
Sebastian reported this problem and did also major portions of the
patch.
The ORM generated a query with INNER JOIN instead of LEFT OUTER JOIN
in a somewhat complicated case. The main issue was that there was a
chain of nullable FK -> non-nullble FK, and the join promotion logic
didn't see the need to promote the non-nullable FK even if the
previous nullable FK was already promoted to LOUTER JOIN. This resulted
in a query like a LOUTER b INNER c, which incorrectly prunes results.
* Renamed smart_unicode to smart_text (but kept the old name under
Python 2 for backwards compatibility).
* Renamed smart_str to smart_bytes.
* Re-introduced smart_str as an alias for smart_text under Python 3
and smart_bytes under Python 2 (which is backwards compatible).
Thus smart_str always returns a str objects.
* Used the new smart_str in a few places where both Python 2 and 3
want a str.
This commit tackles a couple of issues. First, in certain cases there
were some mixups if field.attname or field.name should be deferred.
Field.attname is now always used.
Another issue tackled is a case where field is both deferred by
.only(), and selected by select_related. This case is now an error.
A lot of thanks to koniiiik (Michal Petrucha) for the patch, and
to Andrei Antoukh for review.
Fixed#18248 -- proxy models were added to included_inherited_models
in sql.query.Query. The variable is meant to be used for multitable
inheritance only. This mistake caused problems in situations where
proxy model's query was reused.
Fixed#17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.
The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.
Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
QuerySet had previously some complex logic for dealing with nullable
fields in negated add_filter() calls. It seems the logic is leftover
from a time where the WhereNode wasn't as intelligent in handling
field__in=[] conditions.
Thanks to aaugustin for comments on the patch.
only consider some fields (PostgreSQL only).
For this, the ``distinct()`` QuerySet method now accepts an optional
list of model fields names and generates ``DISTINCT ON`` clauses on
these cases. Thanks Jeffrey Gelens and Anssi Kääriäinen for their work.
Fixes#6422.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@17244 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This is the old Query.as_sql() method revived: it's like Query.__str__,
but the parameters aren't substituted into the placeholders. Thus, it's
a more accurate representation of the SQL the (default) backend will
see. Entirely internal.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@16655 bcc190cf-cafb-0310-a4f2-bffc1f526a37
A number of people worked on this patch over the years -- Hawkeye, Colin Grady,
KBS, sakyamuni, anih, jdemoor, and Issak Kelly. Thanks to them all, and
apologies if I missed anyone.
Special thanks to Dan Fairs for picking it up again at the end and seeing this
through to commit.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@16058 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Turns out that a lot more than just SELECT can return data, and this list is
very hard to define up front in a cross-database manner. So let's just assume
that anyone using raw() is at least halfway competant and can deal with
the error messages if they don't use a data-returning query.
Thanks to Christophe Pettus for the patch.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@15803 bcc190cf-cafb-0310-a4f2-bffc1f526a37
as that can lead to incorrect SQL being generated for the query. Thanks to mat
for the report and test, tobias for the fix, and Alex for review.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@12830 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This monster of a patch is the result of Alex Gaynor's 2009 Google Summer of Code project.
Congratulations to Alex for a job well done.
Big thanks also go to:
* Justin Bronn for keeping GIS in line with the changes,
* Karen Tracey and Jani Tiainen for their help testing Oracle support
* Brett Hoerner, Jon Loyens, and Craig Kimmerer for their feedback.
* Malcolm Treddinick for his guidance during the GSoC submission process.
* Simon Willison for driving the original design process
* Cal Henderson for complaining about ponies he wanted.
... and everyone else too numerous to mention that helped to bring this feature into fruition.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@11952 bcc190cf-cafb-0310-a4f2-bffc1f526a37
See `docs/topics/db/raw.txt` for details.
Thanks to seanoc for getting the ball rolling, and to Russ for wrapping things up.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@11921 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This enables querysets with an extra clause to be used in an __in filter; as a side effect, it also means that as_sql() now returns the correct result for any query with an extra clause.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10648 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Using select_related(...) across a nullable relation to a multi-table
model inheritance situation no longer excludes results. Thanks to AdamG
for a test demonstrating part of the problem.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10136 bcc190cf-cafb-0310-a4f2-bffc1f526a37
In extreme cases, some fields are expensive to load from the database
(e.g. GIS fields requiring conversion, or large text fields). This
commit adds defer() and only() methods to querysets that allow the
caller to specify which fields should not be loaded unless they are
accessed.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10090 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Large portions of this are needed for #5420, so I implemented it fully.
Thanks to Ryan Kelly for an initial patch to get this started.
Refs #5420.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10083 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This fixes a broad class of bugs involving filters that look for missing
related models and fields. Most of them don't seem to have been reported
(the added tests cover the root cause). The exception is that this has
also fixed#9868.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9979 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This includes a fairly large refactor of the update() query path (and
the initial portions of constructing the SQL for any query). The
previous code appears to have been only working more or less by accident
and was very fragile.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9967 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Some results were inadvertently being excluded if we were ordering across a
nullable relation which itself ordering by a non-nullable relation.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9916 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This involves a slight change to the interaction of annotate() and values() clauses that specify a list of columns. See the docs for details.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9888 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Many thanks to:
* Nicolas Lara, who worked on this feature during the 2008 Google Summer of Code.
* Alex Gaynor for his help debugging and fixing a number of issues.
* Malcolm Tredinnick for his invaluable review notes.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9792 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Many thanks to:
* Nicolas Lara, who worked on this feature during the 2008 Google Summer of Code.
* Alex Gaynor for his help debugging and fixing a number of issues.
* Justin Bronn for his help integrating with contrib.gis.
* Karen Tracey for her help with cross-platform testing.
* Ian Kelly for his help testing and fixing Oracle support.
* Malcolm Tredinnick for his invaluable review notes.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9742 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This extends previous functionality that allowed passing Query objects as the
rvals to filters. You can now pass QuerySets, which requires less poking at
opaque attributes. See the documentation of the "__in" lookup type for the
details.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9701 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Extricated the code that works directly with SQL columns (standard
"where" stuff) from the the code that takes SQL fragments and combines
it with lookup types and values. The latter portion is now more
generally reusable. Any existing code that was poking at Query.having
will now break in very visible ways (no subtle miscalculations, which is
a good thing).
This patch, en passant, removes the existing "having" test, since the
new implementation requires more setting up than previously. The
aggregates support (currently in a separate codebase) has tests for this
functionality that work as a replacement for the removed test.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9700 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Anssi Kääriäinen
Tim Graham
Alex Gaynor
Florian Hahn
Aymeric Augustin
Mike Fogel
Tai Lee
Malcolm Tredinnick
Alex Gaynor
Claude Paroz
Adrian Holovaty
Carl Meyer
Julien Phalip
Ramiro Morales
Karen Tracey
Russell Keith-Magee
Jannis Leidel
Luke Plant
Jacob Kaplan-Moss
Justin Bronn
Ian Kelly
Gary Wilson Jr