Having lookups group by subquery right-hand-sides is likely unnecessary
in the first place but relatively large amount of work would be needed
to achieve that such as making Lookup instances proper resolvable
expressions.
Regression in 3543129822.
Thanks James A. Munsch for the report.
Regression was introduced by fff5186 but was due a long standing issue.
AggregateQuery was abusing Query.subquery: bool by stashing its
compiled inner query's SQL for later use in its compiler which made
select_format checks for Query.subquery wrongly assume the provide
query was a subquery.
This patch prevents that from happening by using a dedicated
inner_query attribute which is compiled at a later time by
SQLAggregateCompiler.
Moving the inner query's compilation to SQLAggregateCompiler.compile
had the side effect of addressing a long standing issue with
aggregation subquery pushdown which prevented converters from being
run. This is now fixed as the aggregation_regress adjustments
demonstrate.
Refs #25367.
Thanks Eran Keydar for the report.
This required implementing a limited form of dynamic dispatch to combine
expressions with numerical output. Refs #26355 should eventually provide
a better interface for that.
The grouping caused an issue with database views as PostgreSQL's query planer
isn't smart enough to introspect primary keys through views. Django doesn't
support database views but documents that unmanaged models should be used to
query them.
Thanks powderflask for the detailed report and investigation.
On sqlite the SUM() of a decimal column doesn't have a NUMERIC type so
when comparing it to a string literal (which a Decimal gets converted to
in Django) it is not compared as expected.
When the query's model had a self-referential foreign key, the
compiler.get_group_by() code incorrectly used the self-referential
foreign key's column (for example parent_id) as GROUP BY clause
when it should have used the model's primary key column (id).
The query used a construct of qs.annotate().values().aggregate() where
the first annotate used an F-object reference and the values() and
aggregate() calls referenced that F-object.
Also made sure the inner query's select clause is as simple as possible,
and made sure .values().distinct().aggreate() works correctly.
Refactored compiler SELECT, GROUP BY and ORDER BY generation.
While there, also refactored select_related() implementation
(get_cached_row() and get_klass_info() are now gone!).
Made get_db_converters() method work on expressions instead of
internal_type. This allows the backend converters to target
specific expressions if need be.
Added query.context, this can be used to set per-query state.
Also changed the signature of database converters. They now accept
context as an argument.
Aggregation over subquery produced syntactically incorrect queries in
some cases as Django didn't ensure that source expressions of the
aggregation were present in the subquery.
Complete rework of translating data values from database
Deprecation of SubfieldBase, removal of resolve_columns and
convert_values in favour of a more general converter based approach and
public API Field.from_db_value(). Now works seamlessly with aggregation,
.values() and raw queries.
Thanks to akaariai in particular for extensive advice and inspiration,
also to shaib, manfre and timograham for their reviews.
Simon Charette
Christopher Wang
Simon Charette
Nick Pope
Mariusz Felisiak
Vojtech Bocek
Min ho Kim
Simon Charette
Ramiro Morales
Flávio Juvenal
Jon Dufresne
Mads Jensen
Claude Paroz
Josh Smeaton
za
Tim Graham
Hasan
Josh Soref
Dražen Odobašić
Renato Oliveira
Michael Tänzer
Anssi Kääriäinen
Anssi Kääriäinen
Loic Bistuer
Berker Peksag
Marc Tamlyn
Rodolfo Carvalho