Commit Graph

3023 Commits

Author SHA1 Message Date
Simon Charette c6350d594c Refs #30158 -- Removed alias argument for Expression.get_group_by_cols().
Recent refactors allowed GROUP BY aliasing allowed for aliasing to be
entirely handled by the sql.Query.set_group_by and compiler layers.
2022-10-06 12:04:00 +02:00
Simon Charette b7b28c7c18 Refs #31150 -- Enabled implicit GROUP BY aliases.
This ensures implicit grouping from aggregate function annotations
groups by uncollapsed selected aliases if supported.

The feature is disabled on Oracle because it doesn't support it.
2022-10-06 11:51:32 +02:00
Simon Charette 3d734c09ff Refs #33992 -- Refactored subquery grouping logic.
This required moving the combined queries slicing logic to the compiler
in order to allow Query.exists() to be called at expression resolving
time.

It allowed for Query.exists() to be called at Exists() initialization
time and thus ensured that get_group_by_cols() was operating on the
terminal representation of the query that only has a single column
selected.
2022-10-06 11:38:03 +02:00
Simon Charette 04518e310d Refs #33308 -- Enabled explicit GROUP BY and ORDER BY aliases.
This ensures explicit grouping from using values() before annotating an
aggregate function groups by selected aliases if supported.

The GROUP BY feature is disabled on Oracle because it doesn't support it.
2022-10-06 09:34:31 +02:00
Simon Charette c58a8acd41 Fixed #33768 -- Fixed ordering compound queries by nulls_first/nulls_last on MySQL.
Columns of the left outer most select statement in a combined query
can be referenced by alias just like by index.

This removes combined query ordering by column index and avoids an
unnecessary usage of RawSQL which causes issues for backends that
specialize the treatment of null ordering.
2022-10-05 14:04:49 +02:00
Simon Charette a142edcc49 Avoided unnecessary call to .get_source_expressions().
The SQLCompiler._order_by_pairs() generator method yields instances of
OrderBy and not Expression.
2022-10-04 12:29:37 +02:00
David Sanders 4771a1694b Fixed #34012 -- Made QuerySet.order_by() apply transforms on related fields for models with Meta.ordering.
This makes QuerySet.order_by() no longer ignore trailing transforms for
models with Meta.ordering. As a consequence, FieldError is raised in
such cases for non-existent fields.

Thanks to Klaas van Schelven for the report and Mariusz Felisiak for the
review and advice.
2022-10-04 08:19:34 +02:00
Lily Foote 649b28eab6 Fixed #34070 -- Added subsecond support to Now() on SQLite and MySQL. 2022-10-03 12:13:27 +02:00
Mariusz Felisiak 5e0aa362d9
Fixed #33984 -- Reverted "Fixed #32980 -- Made models cache related managers."
This reverts 4f8c7fd9d9 and adds
two regression tests:
- test_related_manager_refresh(), and
- test_create_copy_with_m2m().

Thanks joeli for the report.
2022-09-30 18:18:33 +02:00
sarahboyce 6cc0f22a73 Fixed #32095 -- Made QuerySet.update_or_create() save only fields passed in defaults or with custom pre_save().
Thanks Florian Apolloner for the initial patch.
2022-09-30 06:26:19 +02:00
sarahboyce 1d77b931f7 Refs #32095 -- Added model's Options._non_pk_concrete_field_names property. 2022-09-29 20:32:08 +02:00
David Wobrock cff1f888e9
Fixed #33464 -- Resolved output_field for combined numeric expressions with MOD operator. 2022-09-27 20:41:10 +02:00
David Sanders 1674c70525
Fixed #34024 -- Fixed crash when aggregating querysets with Q objects annotations.
This reverts b64db05b9c.

It was reasonable to assume it was unnecessary code as there were
no failing tests upon its removal. This commit adds the necessary
regression tests for the failing condition identified in #34024
alongside the original tests added in the PR for which
WhereNode.is_summary was introduced.
2022-09-23 15:15:03 +01:00
Mariusz Felisiak ce6230aa97
Fixed #34015 -- Allowed filtering by transforms on relation fields. 2022-09-22 00:17:04 +02:00
Allen Jonathan David 10178197d5 Fixed #33966 -- Added support for using KeyTextTransform from lookup. 2022-09-16 05:36:57 +02:00
Simon Charette 3ba7f2e906
Refs #28333 -- Explicitly ordered outer qualify query on window filtering.
While most backends will propagate derived table ordering as long as
the outer query doesn't perform additional processing the SQL specs
doesn't explicitly state the ordering must be maintained.
2022-09-15 19:27:32 +02:00
David Sanders e14d08cd89 Fixed #33996 -- Fixed CheckConstraint validation on NULL values.
Bug in 667105877e.

Thanks James Beith for the report.
2022-09-13 12:48:31 +02:00
DevilsAutumn 32797e7fbf Fixed #33975 -- Fixed __in lookup when rhs is a queryset with annotate() and alias().
This fixes clearing selected fields.
2022-09-09 08:37:46 +02:00
Simon Charette 32536b1324 Fixed #33992 -- Fixed queryset crash when aggregating over a group containing Exists.
A more in-depth solution is likely to make sure that we always GROUP BY
selected annotations or revisit how we use Query.exists() in the Exists
expression but that requires extra work that isn't suitable for a
backport.

Regression in e5a92d400a.

Thanks Fernando Flores Villaça for the report.
2022-09-08 05:50:02 +02:00
Mariusz Felisiak 69fa2e8eb2
Refs #26780 -- Made prefetch_related() don't use window expressions fo sliced queries if not supported. 2022-09-06 05:54:35 +02:00
Allen Jonathan David cd1afd553f Fixed #29799 -- Allowed registering lookups per field instances.
Thanks Simon Charette and Mariusz Felisiak for reviews and mentoring
this Google Summer of Code 2022 project.
2022-09-02 10:02:24 +02:00
Simon Charette b3db6c8dcb Fixed #21204 -- Tracked field deferrals by field instead of models.
This ensures field deferral works properly when a model is involved
more than once in the same query with a distinct deferral mask.
2022-08-30 08:43:53 +02:00
Simon Charette 60613ef516 Merged nested if statements in select_related_descend(). 2022-08-30 06:59:55 +02:00
Mohamed Karam b92ffebb0c Fixed #33954 -- Prevented models.DecimalField from accepting NaN, Inf, and -Inf values. 2022-08-27 15:51:08 +02:00
David Wobrock 806e9e2d0d Fixed #33952 -- Reallowed creating reverse foreign key managers on unsaved instances.
Thanks Claude Paroz for the report.

Regression in 7ba6ebe914.
2022-08-27 15:03:48 +02:00
Simon Charette 0701bb8e1f Fixed #33928 -- Avoided unnecessary queries when cascade updating.
Models that use SET, SET_NULL, and SET_DEFAULT as on_delete handler
don't have to fetch objects for the sole purpose of passing them back to
a follow up UPDATE query filtered by the retrieved objects primary key.

This was achieved by flagging SET handlers as _lazy_ and having the
collector logic defer object collections until the last minute. This
should ensure that the rare cases where custom on_delete handlers are
defined remain uncalled when when dealing with an empty collection of
instances.

This reduces the number queries required to apply SET handlers from
2 to 1 where the remaining UPDATE use the same predicate as the non
removed SELECT query.

In a lot of ways this is similar to the fast-delete optimization that
was added in #18676 but for updates this time. The conditions only
happen to be simpler in this case because SET handlers are always
terminal. They never cascade to more deletes that can be combined.

Thanks Renan GEHAN for the report.
2022-08-27 14:25:55 +02:00
Simon Charette a9be1dc551 Refs #33928 -- Removed unnecessary attribute assignment on on-delete updates.
Model instances retrieved for bulk field update purposes are not exposed
to the outside world and thus are not required to be kept update to
date.
2022-08-27 14:25:50 +02:00
Jan Pieter Waagmeester 2480554dc4
Fixed typo in django/db/models/fields/files.py comment. 2022-08-25 09:41:33 +02:00
Mariusz Felisiak e9fd2b5724
Refs #26511 -- Fixed json.KeyTextTransform() on MySQL/MariaDB. 2022-08-18 21:02:29 +02:00
Simon Charette 242499f2dc Fixed #26780 -- Added prefetch_related() support for sliced queries.
This was made possible by window function filtering support added in
f387d024fc.
2022-08-15 10:25:20 +02:00
Simon Charette f387d024fc Refs #28333 -- Added partial support for filtering against window functions.
Adds support for joint predicates against window annotations through
subquery wrapping while maintaining errors for disjointed filter
attempts.

The "qualify" wording was used to refer to predicates against window
annotations as it's the name of a specialized Snowflake extension to
SQL that is to window functions what HAVING is to aggregates.

While not complete the implementation should cover most of the common
use cases for filtering against window functions without requiring
the complex subquery pushdown and predicate re-aliasing machinery to
deal with disjointed predicates against columns, aggregates, and window
functions.

A complete disjointed filtering implementation should likely be
deferred until proper QUALIFY support lands or the ORM gains a proper
subquery pushdown interface.
2022-08-15 08:26:26 +02:00
Simon Charette 8c3046daad Refs #28333 -- Moved SQLCompiler's forced column aliasing logic to get_select().
This extends query composability possibilities when dealing with
subqueries which is necessary to implement window function filtering.
2022-08-11 12:39:54 +02:00
Simon Charette 35911078fa Replaced Expression.replace_references() with .replace_expressions().
The latter allows for more generic use cases beyond the currently
limited ones constraints validation has.

Refs #28333, #30581.
2022-08-11 09:53:10 +02:00
Mariusz Felisiak 721706f4c4
Refs #33895 -- Fixed Case() crash with filters that match everything in When() on Oracle.
Follow up to 036bb2b74a.
2022-08-10 12:28:39 +02:00
David Wobrock 036bb2b74a Fixed #33895 -- Fixed Case() crash with filters that match everything in When().
Thanks shukryzablah for the report.
2022-08-09 07:55:31 +02:00
Mariusz Felisiak 63884829ac
Fixed #33902 -- Fixed Meta.constraints validation crash with F() expressions.
Thanks Adam Zahradník for the report.

Bug in 667105877e.
2022-08-09 06:08:48 +02:00
Mariusz Felisiak fd93db97c7
Fixed #33898 -- Fixed Window() expression crash with ArrayAgg().
Thanks Kia for the report.

Regression in e06dc4571e.
2022-08-06 17:59:31 +02:00
Nick Pope 9dff316be4 Refs #32948, Refs #32946 -- Used Q.create() internally for dynamic Q() objects.
Node.create() which has a compatible signature with Node.__init__()
takes in a single `children` argument rather than relying in unpacking
*args in Q.__init__() which calls Node.__init__().

In addition, we were often needing to unpack iterables into *args and
can instead pass a list direct to Node.create().
2022-07-27 10:06:24 +02:00
Nick Pope 845667f2d1 Refs #32948 -- Simplified and optimized Q._combine() and __invert__().
- Removed use of Q.deconstruct() in Q._combine().
- Simplified and optimized Q.__invert__() by taking a shallow copy and
  swapping the negated attribute only.
- Simplified construction in Q._combine().
- Simplified conditions in Q._combine() as Q.conditional = True the
  first isinstance() check is unnecessary.
- Removed copy.copy() branch in Q._combine().

Co-authored-by: Keryn Knight <keryn@kerynknight.com>
2022-07-27 10:06:24 +02:00
Nick Pope 19b866c254 Refs #32948 -- Added Node.__copy__().
This allows the copy.copy() usage in the Q._combine() method to finish
sooner, instead of having to fallback to using the __reduce_ex__(4)
method.

Thia also avoids having to fall into copy.copy() at in Q._combine(),
when combining a Q() with another Q().

Co-authored-by: Keryn Knight <keryn@kerynknight.com>
2022-07-27 10:06:24 +02:00
Nick Pope ed9eca8457 Refs #32948 -- Simplified WhereNode and Node.__deepcopy__()/add().
We can use copy() in Node.add() instead of create() as we don't need the
children to be cloned via [:] subscript in __init__().
2022-07-27 10:06:24 +02:00
Nick Pope ddf0002bb7 Refs #32948 -- Renamed Node._new_instance() to Node.create().
Node._new_instance() was added in
6dd2b5468f to work around Q.__init__()
having an incompatible signature with Node.__init__().

It was intended as a hook that could be overridden if subclasses needed
to change the behaviour of instantiation of their specialised form of
Node. In practice this doesn't ever seem to have been used for this
purpose and there are very few calls to Node._new_instance() with other
code, e.g. Node.__deepcopy__() calling Node and overriding __class__ as
required.

Rename this to Node.create() to make it a more "official" piece of
private API that we can use to simplify a lot of other areas internally.

The docstring and nearby comment have been reworded to read more
clearly.
2022-07-27 10:06:24 +02:00
Nick Pope 4f8bc75bc3 Removed obsolete docstring from WhereNode.clone().
Node.subtree_parents was removed in
d3f00bd570. That commit also added
Q.clone() which was identical to WhereNode.clone(), but lacked the
docstring. Q.clone() was later removed in
b454e2cbc9.
2022-07-27 07:55:09 +02:00
Nick Pope 769d7cce4a Used AND, OR, XOR constants instead of hard-coded values. 2022-07-27 07:55:09 +02:00
Mariusz Felisiak 66f30dbf25 Refs #27236 -- Reverted "Refs #27236 -- Refactored out DeprecationForHistoricalMigrationMixin."
This reverts commit 57793b4765.
2022-07-26 11:41:19 +02:00
Mariusz Felisiak f810325721
Refs #27236 -- Made cosmetic edits to Meta.index_together deprecation.
This should make it more straightforward to move forward when
deprecation ends.
2022-07-21 07:05:50 +02:00
David Wobrock a6385b382e
Fixed #27236 -- Deprecated Meta.index_together in favor of Meta.indexes.
This also deprecates AlterIndexTogether migration operation.
2022-07-12 09:04:31 +02:00
Stéphane "Twidi" Angel ccbf714ebe Fixed #33829 -- Made BaseConstraint.deconstruct() and equality handle violation_error_message.
Regression in 667105877e.
2022-07-08 08:17:42 +02:00
David Wobrock 57793b4765 Refs #27236 -- Refactored out DeprecationForHistoricalMigrationMixin. 2022-07-07 12:21:35 +02:00
Simon Charette 877c800f25 Refs CVE-2022-34265 -- Properly escaped Extract() and Trunc() parameters.
Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
2022-07-06 07:40:07 +02:00