Commit Graph

104 Commits

Author SHA1 Message Date
Simon Charette e5c844d6f2 Fixed #34551 -- Fixed QuerySet.aggregate() crash when referencing subqueries.
Regression in 59bea9efd2.

Refs #28477.

Thanks Denis Roldán and Mariusz for the test.
2023-05-23 06:25:58 +02:00
Simon Charette 2ee01747c3 Refs #34551 -- Fixed QuerySet.aggregate() crash on precending aggregation reference.
Regression in 1297c0d0d7.

Refs #31679.
2023-05-23 06:25:27 +02:00
Simon Charette 9daf8b4109 Fixed #34464 -- Fixed queryset aggregation over group by reference.
Regression in 59bea9efd2.

Refs #28477.

Thanks Ian Cubitt for the report.
2023-04-07 06:57:32 +02:00
Mariusz Felisiak 0e2649fdf4 Fixed #34255 -- Made PostgreSQL backend use client-side parameters binding with psycopg version 3.
Thanks Guillaume Andreu Sabater for the report.

Co-authored-by: Florian Apolloner <apollo13@users.noreply.github.com>
2023-01-17 08:24:08 +01:00
Simon Charette dd68af62b2 Fixed #34176 -- Fixed grouping by ambiguous aliases.
Regression in b7b28c7c18.

Refs #31377.

Thanks Shai Berger for the report and reviews.

test_aggregation_subquery_annotation_values_collision() has been
updated as queries that are explicitly grouped by a subquery should
always be grouped by it and not its outer columns even if its alias
collides with referenced table columns. This was not possible to
accomplish at the time 10866a10 landed because we didn't have compiler
level handling of colliding aliases.
2023-01-09 10:52:51 +01:00
Mariusz Felisiak 3b24a3fa33
Removed unnecessary commas in tests. 2022-12-21 11:41:29 +01:00
Simon Charette 1297c0d0d7 Fixed #31679 -- Delayed annotating aggregations.
By avoiding to annotate aggregations meant to be possibly pushed to an
outer query until their references are resolved it is possible to
aggregate over a query with the same alias.

Even if #34176 is a convoluted case to support, this refactor seems
worth it given the reduction in complexity it brings with regards to
annotation removal when performing a subquery pushdown.
2022-11-23 17:46:07 +01:00
Simon Charette 10037130c1
Refs #28477 -- Fixed handling aliased annotations on aggregation.
Just like when using .annotate(), the .alias() method will generate the
necessary JOINs to resolve the alias even if not selected.

Since these JOINs could be multi-valued non-selected aggregates must be
considered to require subquery wrapping as a GROUP BY is required to
combine duplicated tuples from the base table.

Regression in 59bea9efd2.
2022-11-14 05:45:33 +01:00
Simon Charette 59bea9efd2 Fixed #28477 -- Stripped unused annotations on aggregation.
Also avoid an unnecessary pushdown when aggregating over a query that doesn't
have aggregate annotations.
2022-11-09 13:22:14 +01:00
Gregor Gärtner f0c06f8ab7 Refs #33990 -- Renamed TransactionTestCase.assertQuerysetEqual() to assertQuerySetEqual().
Co-Authored-By: Michael Howitz <mh@gocept.com>
2022-10-08 08:07:38 +02:00
Mariusz Felisiak d795259ea9
Replaced assertQuerysetEqual() to assertSequenceEqual()/assertCountEqual() where appropriate.
Follow up to 3f7b327562.
2022-10-07 13:05:35 +02:00
Simon Charette b7b28c7c18 Refs #31150 -- Enabled implicit GROUP BY aliases.
This ensures implicit grouping from aggregate function annotations
groups by uncollapsed selected aliases if supported.

The feature is disabled on Oracle because it doesn't support it.
2022-10-06 11:51:32 +02:00
Simon Charette 3d734c09ff Refs #33992 -- Refactored subquery grouping logic.
This required moving the combined queries slicing logic to the compiler
in order to allow Query.exists() to be called at expression resolving
time.

It allowed for Query.exists() to be called at Exists() initialization
time and thus ensured that get_group_by_cols() was operating on the
terminal representation of the query that only has a single column
selected.
2022-10-06 11:38:03 +02:00
Simon Charette 32536b1324 Fixed #33992 -- Fixed queryset crash when aggregating over a group containing Exists.
A more in-depth solution is likely to make sure that we always GROUP BY
selected annotations or revisit how we use Query.exists() in the Exists
expression but that requires extra work that isn't suitable for a
backport.

Regression in e5a92d400a.

Thanks Fernando Flores Villaça for the report.
2022-09-08 05:50:02 +02:00
Mariusz Felisiak eb3699ea77
Fixed #33718 -- Dropped support for MySQL 5.7. 2022-07-08 13:30:12 +02:00
marcperrinoptel 4282fd468f Fixed #33655 -- Removed unnecessary constant from GROUP BY clause for QuerySet.exists(). 2022-04-26 06:19:18 +02:00
Mariusz Felisiak 93cae5cb2f Fixed CVE-2022-28346 -- Protected QuerySet.annotate(), aggregate(), and extra() against SQL injection in column aliases.
Thanks Splunk team: Preston Elder, Jacob Davis, Jacob Moore,
Matt Hanson, David Briggs, and a security researcher: Danylo Dmytriiev
(DDV_UA) for the report.
2022-04-11 08:59:33 +02:00
Luke Plant 40b8a6174f Fixed #33397 -- Corrected resolving output_field for DateField/DateTimeField/TimeField/DurationFields.
This includes refactoring of CombinedExpression._resolve_output_field()
so it no longer uses the behavior inherited from Expression of guessing
same output type if argument types match, and instead we explicitly
define the output type of all supported operations.

This also makes nonsensical operations involving dates
(e.g. date + date) raise a FieldError, and adds support for
automatically inferring output_field for cases such as:
* date - date
* date + duration
* date - duration
* time + duration
* time - time
2022-03-31 11:05:23 +02:00
Carlton Gibson bb61f0186d Refs #32365 -- Removed internal uses of utils.timezone.utc alias.
Remaining test case ensures that uses of the alias are mapped
canonically by the migration writer.
2022-03-24 06:29:50 +01:00
Nick Pope 847f46e9bf
Removed redundant QuerySet.all() calls in docs and tests.
Most QuerySet methods are mapped onto the Manager and, in general,
it isn't necessary to call .all() on the manager.
2022-02-22 10:29:38 +01:00
Mariusz Felisiak 7119f40c98 Refs #33476 -- Refactored code to strictly match 88 characters line length. 2022-02-07 20:37:05 +01:00
django-bot 9c19aff7c7 Refs #33476 -- Reformatted code with Black. 2022-02-07 20:37:05 +01:00
Mariusz Felisiak 71e7c8e737
Fixed #33468 -- Fixed QuerySet.aggregate() after annotate() crash on aggregates with default.
Thanks Adam Johnson for the report.
2022-01-31 11:33:24 +01:00
Simon Charette e5a92d400a Fixed #33282 -- Fixed a crash when OR'ing subquery and aggregation lookups.
As a QuerySet resolves to Query the outer column references grouping logic
should be defined on the latter and proxied from Subquery for the cases where
get_group_by_cols is called on unresolved expressions.

Thanks Antonio Terceiro for the report and initial patch.
2021-12-02 07:23:33 +01:00
David Wobrock ad36a198a1 Fixed #33141 -- Renamed Expression.empty_aggregate_value to empty_result_set_value. 2021-09-29 12:58:01 +02:00
David Wobrock 691486a5cf Fixed #33073 -- Fixed queryset crash with aggregation and empty/extra queryset annotation. 2021-09-01 20:59:16 +02:00
Tim Graham 022d29c934
Refs #10929 -- Allowed NowUTC SQL customization for third-party backends. 2021-08-24 08:28:03 +02:00
Nick Pope 501a8db465 Fixed #10929 -- Added default argument to aggregates.
Thanks to Simon Charette and Adam Johnson for the reviews.
2021-07-19 13:04:27 +02:00
Simon Charette 9f3cce172f Refs #26430 -- Re-introduced empty aggregation optimization.
The introduction of the Expression.empty_aggregate_value interface
allows the compilation stage to enable the EmptyResultSet optimization
if all the aggregates expressions implement it.

This also removes unnecessary RegrCount/Count.convert_value() methods.
Disabling the empty result set aggregation optimization when it wasn't
appropriate prevented None returned for a Count aggregation value.

Thanks Nick Pope for the review.
2021-07-02 07:25:42 +02:00
Simon Charette f3112fde98 Fixed #26430 -- Fixed coalesced aggregation of empty result sets.
Disable the EmptyResultSet optimization when performing aggregation as
it might interfere with coalescence.
2021-07-02 06:40:54 +02:00
Simon Charette 277eea8fcc Fixed #32478 -- Included nested columns referenced by subqueries in GROUP BY on aggregations.
Regression in fb3f034f1c.

Refs #31094, #31150.

Thanks Igor Pejic for the report.
2021-02-24 09:01:36 +01:00
Hasan Ramezani 275dd4ebba
Fixed #32178 -- Allowed database backends to skip tests and mark expected failures.
Co-authored-by: Tim Graham <timograham@gmail.com>
2020-12-10 18:00:57 +01:00
Ian Foote 8b040e3cbb Fixed #25534, Fixed #31639 -- Added support for transform references in expressions.
Thanks Mariusz Felisiak and Simon Charette for reviews.
2020-11-27 20:42:04 +01:00
Hasan Ramezani 3f7b327562 Fixed #31235 -- Made assertQuerysetEqual() compare querysets directly.
This also replaces assertQuerysetEqual() to
assertSequenceEqual()/assertCountEqual() where appropriate.

Co-authored-by: Peter Inglesby <peter.inglesby@gmail.com>
Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
2020-11-06 09:24:50 +01:00
Sicong 09e1ec71df
Fixed #32166 -- Removed redundant definition of Greatest in test_expression_on_aggregation. 2020-11-03 07:08:42 +01:00
Étienne Beaulé 509d9da26f Fixed #26390 -- Disabled grouping by Random().
Thanks to Tzu-ping Chung for the tests.
2020-10-21 20:54:53 +02:00
David Wobrock 058747b77a Fixed #31880 -- Made QuerySet.aggregate() raise FieldError when aggregating over aggregation aliases. 2020-09-29 09:04:25 +02:00
Ahmad A. Hussein 493b26bbfc Fixed #31888 -- Avoided module-level MySQL queries in tests. 2020-08-17 09:31:16 +02:00
Simon Charette 156a2138db Refs #30446 -- Removed unnecessary Value(..., output_field) in docs and tests. 2020-07-15 10:58:38 +02:00
Simon Charette 1e38f1191d Fixed #30446 -- Resolved Value.output_field for stdlib types.
This required implementing a limited form of dynamic dispatch to combine
expressions with numerical output. Refs #26355 should eventually provide
a better interface for that.
2020-07-15 10:58:29 +02:00
Hasan Ramezani 10866a10fe Fixed #31377 -- Disabled grouping by aliases on QuerySet.values()/values_list() when they collide with field names.
Regression in fb3f034f1c.

Thanks Holovashchenko Vadym for the report.
2020-03-25 09:13:46 +01:00
Mariusz Felisiak aee0bebc2f Refs #31331 -- Checked ONLY_FULL_GROUP_BY mode in AggregateTestCase.test_aggregation_subquery_annotation_multivalued(). 2020-03-03 11:25:37 +01:00
Mariusz Felisiak 7b8fa1653f Fixed #31150 -- Included subqueries that reference related fields in GROUP BY clauses.
Thanks Johannes Hoppe for the report.

Regression in fb3f034f1c.

Co-authored-by: Simon Charette <charette.s@gmail.com>
2020-03-03 11:25:37 +01:00
Nick Pope 335c9c94ac Simplified imports from django.db and django.contrib.gis.db. 2020-02-04 13:20:06 +01:00
Mariusz Felisiak 6b178a3e93
Fixed #31217 -- Made QuerySet.values()/values_list() group by not selected annotations with aggregations used in order_by().
Regression in 59b4e99dd0.

Thanks Jon Dufresne for the report and Simon Charette for the review.
2020-02-03 07:48:11 +01:00
Mariusz Felisiak 59b4e99dd0
Refs #31136 -- Made QuerySet.values()/values_list() group only by selected annotation.
Regression in 0f843fdd5b.
2020-01-15 09:32:42 +01:00
Mariusz Felisiak 0f843fdd5b
Fixed #31136 -- Disabled grouping by aliases on QuerySet.values()/values_list().
Regression in fb3f034f1c.

Thanks Sigurd Ljødal for the report.
2020-01-04 20:49:11 +01:00
Simon Charette 720de4d044 Fixed #31109 -- Disabled grouping by aliases on QuerySet.exists().
Clearing the SELECT clause in Query.has_results was orphaning GROUP BY
references to it.

Thanks Thierry Bastian for the report and Baptiste Mispelon for the
bisect.

Regression in fb3f034f1c.
2019-12-23 09:24:03 +01:00
Simon Charette 5a4d7285bd Fixed #31094 -- Included columns referenced by subqueries in GROUP BY on aggregations.
Thanks Johannes Hoppe for the report.

Regression in fb3f034f1c.

Co-authored-by: Mariusz Felisiak <felisiak.mariusz@gmail.com>
2019-12-19 10:04:56 +01:00
Simon Charette a0f34d8fef Refs #31094 -- Added test for not using aliases in GROUP BY clauses. 2019-12-19 09:12:29 +01:00