The `remove()` and `clear()` methods of the related managers created by
`ForeignKey`, `GenericForeignKey`, and `ManyToManyField` suffered from a
number of issues. Some operations ran multiple data modifying queries without
wrapping them in a transaction, and some operations didn't respect default
filtering when it was present (i.e. when the default manager on the related
model implemented a custom `get_queryset()`).
Fixing the issues introduced some backward incompatible changes:
- The implementation of `remove()` for `ForeignKey` related managers changed
from a series of `Model.save()` calls to a single `QuerySet.update()` call.
The change means that `pre_save` and `post_save` signals aren't called anymore.
- The `remove()` and `clear()` methods for `GenericForeignKey` related
managers now perform bulk delete so `Model.delete()` isn't called anymore.
- The `remove()` and `clear()` methods for `ManyToManyField` related
managers perform nested queries when filtering is involved, which may
or may not be an issue depending on the database and the data itself.
Refs. #3871, #21174.
Thanks Anssi Kääriäinen and Tim Graham for the reviews.
This patch introduces the Prefetch object which allows customizing prefetch
operations.
This enables things like filtering prefetched relations, calling select_related
from a prefetched relation, or prefetching the same relation multiple times
with different querysets.
When a Prefetch instance specifies a to_attr argument, the result is stored
in a list rather than a QuerySet. This has the fortunate consequence of being
significantly faster. The preformance improvement is due to the fact that we
save the costly creation of a QuerySet instance.
Thanks @akaariai for the original patch and @bmispelon and @timgraham
for the reviews.
This commit introduced a new class JoinPromoter that can be used to
abstract away join promotion problems for complex filter conditions.
Query._add_q() and Query.combine() now use the new class.
Also, added a lot of comments about why join promotion is done the way
it is.
Thanks to Tim Graham for original report and testing the changes, and
for Loic Bistuer for review.
Thanks dan at dlo.me for the initial patch.
- Added __pow__ and __rpow__ to ExpressionNode
- Added oracle and mysql specific power expressions
- Added used-defined power function for sqlite
The typo could have consequences in exceptional cases, but I didn't
figure out a way to actually produce such a case, so not tests.
Report & patch by Michael Manfre.
select_related('foo').select_related('bar') is now equivalent to
select_related('foo', 'bar').
Also reworded docs to recommend select_related(*fields) over select_related()
Squashed commit of the following:
commit 63ddb271a44df389b2c302e421fc17b7f0529755
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 29 22:51:00 2013 +0200
Clarified interactions between atomic and exceptions.
commit 2899ec299228217c876ba3aa4024e523a41c8504
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 22 22:45:32 2013 +0200
Fixed TransactionManagementError in tests.
Previous commit introduced an additional check to prevent running
queries in transactions that will be rolled back, which triggered a few
failures in the tests. In practice using transaction.atomic instead of
the low-level savepoint APIs was enough to fix the problems.
commit 4a639b059ea80aeb78f7f160a7d4b9f609b9c238
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Sep 24 22:24:17 2013 +0200
Allowed nesting constraint_checks_disabled inside atomic.
Since MySQL handles transactions loosely, this isn't a problem.
commit 2a4ab1cb6e83391ff7e25d08479e230ca564bfef
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sat Sep 21 18:43:12 2013 +0200
Prevented running queries in transactions that will be rolled back.
This avoids a counter-intuitive behavior in an edge case on databases
with non-atomic transaction semantics.
It prevents using savepoint_rollback() inside an atomic block without
calling set_rollback(False) first, which is backwards-incompatible in
tests.
Refs #21134.
commit 8e3db393853c7ac64a445b66e57f3620a3fde7b0
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 22 22:14:17 2013 +0200
Replaced manual savepoints by atomic blocks.
This ensures the rollback flag is handled consistently in internal APIs.
Previously, if a database request spanned a related object manager, the
first manager encountered would cause a request to the router, and this
would bind all subsequent queries to the same database returned by the
router. Unfortunately, the first router query would be performed using
a read request to the router, resulting in bad routing information being
used if the subsequent query was actually a write.
This change defers the call to the router until the final query is acutally
made.
It includes a small *BACKWARDS INCOMPATIBILITY* on an edge case - see the
release notes for details.
Thanks to Paul Collins (@paulcollinsiii) for the excellent debugging
work and patch.
The use of OrderedDict (even an empty one) was surprisingly slow. By
initializing OrderedDict only when needed it is possible to save
non-trivial amount of computing time (Model.save() is around 30% faster
for example).
This commit targetted sql.Query only, there are likely other places
which could use similar optimizations.
exc_value might be None even though there's an exception, at least on
Python 2.6. Thanks Thomas Chaumeny for the report.
Fixed#21034.
Forward-port of a8624b2 from 1.6.x.
The option can be used to force pre 1.6 style SELECT on save behaviour.
This is needed in case the database returns zero updated rows even if
there is a matching row in the DB. One such case is PostgreSQL update
trigger that returns NULL.
Reviewed by Tim Graham.
Refs #16649
The __eq__ method now considers two instances without primary key value
equal only when they have same id(). The __hash__ method raises
TypeError for no primary key case.
Fixed#18864, fixed#18250
Thanks to Tim Graham for docs review.
In cases where the same connection (from model A to model B along the
same field) was needed multiple times in a select_related query, the
join setup code mistakenly reused an existing join.
Cleaned up the internal implementation of m2m fields by removing
related.py _get_fk_val(). The _get_fk_val() was doing the wrong thing
if asked for the foreign key value on foreign key to parent model's
primary key when child model had different primary key field.
If LEFT JOINs are required for correct results, then trimming the join
can lead to incorrect results. Consider case:
TBL A: ID | TBL B: ID A_ID
1 1 1
2
Now A.order_by('b__a') did use a join to B, and B's a_id column. This
was seen to contain the same value as A's id, and so the join was
trimmed. But this wasn't correct as the join is LEFT JOIN, and for row
A.id = 2 the B.a_id column is NULL.
Cleaned up the internal implementation of m2m fields by removing
related.py _get_fk_val(). The _get_fk_val() was doing the wrong thing
if asked for the foreign key value on foreign key to parent model's
primary key when child model had different primary key field.
If LEFT JOINs are required for correct results, then trimming the join
can lead to incorrect results. Consider case:
TBL A: ID | TBL B: ID A_ID
1 1 1
2
Now A.order_by('b__a') did use a join to B, and B's a_id column. This
was seen to contain the same value as A's id, and so the join was
trimmed. But this wasn't correct as the join is LEFT JOIN, and for row
A.id = 2 the B.a_id column is NULL.
The bug was already fixed by 01b9c3d519,
so only tests added.
At the same time promote_joins()'s uncoditional flag is gone, it isn't
needed for anything any more.
All Promise objects were passed to force_text() deep in ORM query code.
Not only does this make it difficult or impossible for developers to
prevent or alter this behaviour, but it is also wrong for non-text
fields.
This commit changes `Field.get_prep_value()` from a no-op to one that
resolved Promise objects. All subclasses now call super() method first
to ensure that they have a real value to work with.
The code that tests to see which LIKE expressions to use now runs
using non-error-wrapped cursor, so cx_Oracle exceptions need to be caught
rather than Django DatabaseErrors.
Thanks Trac user ludo for report and initial patch.
It has been possible to use models of wrong type in related field
lookups. For example pigs__in=[a_duck] has worked. Changes to
ForeignObject broke that.
It might be a good idea to restrict the model types usable in lookups.
This should be done intentionally, not accidentally and without any
consideration for deprecation path.
Additionally this patch solves the orthogonal problem that specialized
`QuerySet` like `ValuesQuerySet` didn't inherit from the current `QuerySet`
type. This wasn't an issue until now because we didn't officially support
custom `QuerySet` but it became necessary with the introduction of this new
feature.
Thanks aaugustin, akaariai, carljm, charettes, mjtamlyn, shaib and timgraham
for the reviews.
In the combination of .values().aggregate() the aggregate_select_mask
didn't include the aggregates added. This resulted in bogus query.
Thanks to Trac alias debanshuk for report.
In the intervening years, RelatedField has become less of a hack (though it still is one). Anyone who wants to can re-instate the comment, but please add more details.
Add support for Oracle, fix an issue with the repr of RawQuerySet,
add tests and documentations. Also added a 'supports_paramstyle_pyformat'
database feature, True by default, False for SQLite.
Thanks Donald Stufft for review of documentation.
This is useful:
- to force a rollback on the exit of an atomic block without having to
raise and catch an exception;
- to prevent a rollback after handling an exception manually.
Those methods were only used by `contrib.admin` internally and exclusively
related to `contrib.auth`. Since they were undocumented but used
in the wild the raised deprecation warning point to an also undocumented
alternative that lives in `contrib.auth`.
Also did some PEP8 and other cleanups in the affected modules.
There were a couple of places which used Query.join() directly. By
using setup_joins() in these places the code is more DRY, and in
addition there is no need to directly call field.get_joining_columns()
unless the field is the given join_field from get_path_info(). This
makes it easier to make sure a ForeignObject subclass generates joins
correctly in all cases.
The join used by select_related was incorrectly INNER when the query
had an ORed filter for nullable join that was trimmed away. Fixed this
by forcing the join type to LOUTER even when a join was trimmed away
in ORed queries.
This reverts commit 64041f0e6e.
lookup.tests.LookupTests.test_regex_non_string fails under Postgres.
We should also try to rewrite the test using an existing model.
This can be used to make Django's test suite significantly faster by
reducing the number of models for which content types and permissions
must be created and tables must be flushed in each non-transactional
test.
It's documented for Django contributors and committers but it's branded
as a private API to preserve our freedom to change it in the future.
Most of the credit goes to Anssi. He got the idea and did the research.
Fixed#20483.
This allows using flush on a subset of the tables without having to
manually cascade to all tables with foreign keys to the tables being
truncated, when they're known to be empty.
On databases where truncate is implemented with DELETE FROM, this
doesn't make a difference. The cascade is allowed, not mandatory.
The patch for #19385 caused a regression in certain generic relations
.exclude() filters if a subquery was needed. The fix contains a
refactoring to how Query.split_exclude() and Query.trim_start()
interact.
Thanks to Trac alias nferrari for the report.
Correctly calculate the ``aggregate_start`` offset from loaded fields,
if any are deferred, instead of ``self.query.select`` which includes all
fields on the model.
Also made some PEP 8 fixes.
The SubqueryConstraint defined relabeled_clone(), but that was never
called. Instead there is now clone() and relabel_aliases() methods for
SubqueryConstraint.
A related problem was that SubqueryConstraint didn't correctly use
quote_name_unless_alias() of the outer query. This resulted in failures
when running under PostgreSQL.
When an exception other than IntegrityError was raised, get_or_create
could fail and leave the database connection in an unusable state.
Thanks UloPe for the report.
Used formatted date instead of datetime object for the end of the
year range, as the datetime object loses fractions-of-seconds when
inserted into the db.
This is backward incompatible for custom form field/widgets that rely
on the hard-coded 'Hold down "Control", or "Command" on a Mac, to select
more than one.' sentence.
Application that use standard model form fields and widgets aren't
affected but need to start handling these help texts by themselves
before Django 1.8.
For more details, see the related release notes and deprecation timeline
sections added with this commit.
A regression caused by d5b93d3281 made .get() error
reporting recurse infinitely on certain rare conditions. Fixed this by
not trying to print the given lookup kwargs.
When a GenericRelation was defined on abstract model, queries on childs
of the abstract model didn't work. The problem was in the way fields and
in particular field.rel was copied from models to their children.
The regression was likely caused by #19385. Thanks to Gavin Wahl for
spotting the regression.
A decorator is easier to apply to CBVs. Backwards compatibility isn't an
issue here, except for people running on a recent clone of master.
Fixed a few minor problems in the transactions docs while I was there.
Queries can contain binary data undecodable with utf-8. In this
case, using the 'replace' errors mode when decoding seems like
an acceptable representation of the query.
Thanks Marcel Ryser for the report.
Previously, depending on the database backend or the cursor type,
you'd need to double the percent signs in the query before passing
it to cursor.execute. Now cursor.execute consistently need percent
doubling whenever params argument is not None (placeholder substitution
will happen).
Thanks Thomas Güttler for the report and Walter Doekes for his work
on the patch.
This reverts commit 2cd0edaa47.
This commit was the cause of a memory leak. See ticket for more details.
Thanks Anssi Kääriäinen for identifying the source of the bug.
Model.save() will use UPDATE - if not updated - INSERT instead of
SELECT - if found UPDATE else INSERT. This should save a query when
updating, but will cost a little when inserting model with PK set.
Also fixed#17341 -- made sure .save() commits transactions only after
the whole model has been saved. This wasn't the case in model
inheritance situations.
The save_base implementation was refactored into multiple methods.
A typical chain for inherited save is:
save_base()
_save_parents(self)
for each parent:
_save_parents(parent)
_save_table(parent)
_save_table(self)
Thanks Anssi for haggling until I implemented this.
This change alleviates the need for atomic_if_autocommit. When
autocommit is disabled for a database, atomic will simply create and
release savepoints, and not commit anything. This honors the contract of
not doing any transaction management.
This change also makes the hack to allow using atomic within the legacy
transaction management redundant.
None of the above will work with SQLite, because of a flaw in the design
of the sqlite3 library. This is a known limitation that cannot be lifted
without unacceptable side effects eg. triggering arbitrary commits.
The sql/query.py add_q method did a lot of where/having tree hacking to
get complex queries to work correctly. The logic was refactored so that
it should be simpler to understand. The new logic should also produce
leaner WHERE conditions.
The changes cascade somewhat, as some other parts of Django (like
add_filter() and WhereNode) expect boolean trees in certain format or
they fail to work. So to fix the add_q() one must fix utils/tree.py,
some things in add_filter(), WhereNode and so on.
This commit also fixed add_filter to see negate clauses up the path.
A query like .exclude(Q(reversefk__in=a_list)) didn't work similarly to
.filter(~Q(reversefk__in=a_list)). The reason for this is that only
the immediate parent negate clauses were seen by add_filter, and thus a
tree like AND: (NOT AND: (AND: condition)) will not be handled
correctly, as there is one intermediary AND node in the tree. The
example tree is generated by .exclude(~Q(reversefk__in=a_list)).
Still, aggregation lost connectors in OR cases, and F() objects and
aggregates in same filter clause caused GROUP BY problems on some
databases.
Fixed#17600, fixed#13198, fixed#17025, fixed#17000, fixed#11293.
Before there was need to have both .relabel_aliases() and .clone() for
many structs. Now there is only relabeled_clone() for those structs
where alias is the only mutable attribute.
Replaced them with per-database options, for proper multi-db support.
Also toned down the recommendation to tie transactions to HTTP requests.
Thanks Jeremy for sharing his experience.
Since "unless managed" now means "if database-level autocommit",
committing or rolling back doesn't have any effect.
Restored transactional integrity in a few places that relied on
automatically-started transactions with a transitory API.
For users who didn't activate autocommit in their database options, this
is backwards-incompatible in "non-managed" aka "auto" transaction state.
This state now uses database-level autocommit instead of ORM-level
autocommit.
Also removed the uses_autocommit feature which lost its purpose.
Autocommit cannot be manipulated independently from an open connection.
This commit introduces a minor change in behavior: entering transaction
management forces opening a databasse connection. This shouldn't be
backwards incompatible in any practical use case.
enter_transaction_management() was nearly always followed by managed().
In three places it wasn't, but they will all be refactored eventually.
The "forced" keyword argument avoids introducing behavior changes until
then.
This is mostly backwards-compatible, except, of course, for managed
itself. There's a minor difference in _enter_transaction_management:
the top self.transaction_state now contains the new 'managed' state
rather than the previous one. Django doesn't access
self.transaction_state in _enter_transaction_management.
* Grouped related methods together -- with banner comments :/
* Described which methods are intended to be implemented in backends.
* Added docstrings.
* Used the same order in all wrappers.
Squashed commit of the following:
commit 2181d833ed1a2e422494738dcef311164c4e097e
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 27 14:28:39 2013 +0100
Fixed#15901 -- Wrapped all PEP-249 exceptions.
commit 5476a5d93c19aa2f928c497d39ce6e33f52694e2
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 17:26:52 2013 +0100
Added PEP 3134 exception chaining.
Thanks Jacob Kaplan-Moss for the suggestion.
commit 9365fad0a650328002fb424457d675a273c95802
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 17:13:49 2013 +0100
Improved API for wrapping database errors.
Thanks Alex Gaynor for the proposal.
commit 1b463b765f2826f73a8d9266795cd5da4f8d5e9e
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 15:00:39 2013 +0100
Removed redundant exception wrapping.
This is now taken care of by the cursor wrapper.
commit 524bc7345a724bf526bdd2dd1bcf5ede67d6bb5c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 14:55:10 2013 +0100
Wrapped database exceptions in the base backend.
This covers the most common PEP-249 APIs:
- Connection APIs: close(), commit(), rollback(), cursor()
- Cursor APIs: callproc(), close(), execute(), executemany(),
fetchone(), fetchmany(), fetchall(), nextset().
Fixed#19920.
commit a66746bb5f0839f35543222787fce3b6a0d0a3ea
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 14:53:34 2013 +0100
Added a wrap_database_exception context manager and decorator.
It re-throws backend-specific exceptions using Django's common wrappers.
There were a couple of errors in ._dirty flag handling:
* It started as None, but was never reset to None.
* The _dirty flag was sometimes used to indicate if the connection
was inside transaction management, but this was not done
consistently. This also meant the flag had three separate values.
* The None value had a special meaning, causing for example inability
to commit() on new connection unless enter/leave tx management was
done.
* The _dirty was tracking "connection in transaction" state, but only
in managed transactions.
* Some tests never reset the transaction state of the used connection.
* And some additional less important changes.
This commit has some potential for regressions, but as the above list
shows, the current situation isn't perfect either.
Change patch from https://code.djangoproject.com/ticket/5568
to work on modern Django.
Add special case for MySQL which has different syntax for DROP INDEX.
Add unit tests for the new functionality.
When iteration over a queryset raised an exception, the result cache
remained initialized with an empty list, so subsequent iterations returned
an empty list instead of raising an exception
The regression was likely caused by the fix in #19606 which adjusted
Oracle's unicode detection, though it seems this would have been an
issue in some configurations even before.
Querying the reverse side of nullable to_field relation, where both
sides can contain null values resulted in incorrect results. The reason
was not detecting '' as NULL.
Refs #17541
There was a regression in case two models inherited the same parent,
and one contained a foreign key to other. When select_related travelled
the foreign key the other model reused the parent join made by the
first model. This was likely caused by Query.join_parent_model()
addition in commit 68985db482.
Thanks to Trac alias loic84 for report & tests.
The join promote=True was over-aggressive in select_related handling.
After that was removed, the only other user was query.combine(). That
use case is very easy to handle locally, so there is no more need for
the join(promote=True) flag.
Refs #19849.
This controls whether or not a database level cosntraint is created. This is useful in a few specialized circumstances, but in general should not be used!
The refactoring mainly concentrates on making sure the inner and outer
query agree about the split position. The split position is where the
multijoin happens, and thus the split position also determines the
columns used in the "WHERE col1 IN (SELECT col2 from ...)" condition.
This commit fixes a regression caused by #10790 and commit
69597e5bcc. The regression was caused
by wrong cols in the split position.
Django used to check the version of MySQL before handling the first
request, which required:
- opening a connection
- closing it, to avoid holding it idle until the first request.
This code isn't necessary any longer since Django dropped support for
some versions of MySQL, and other database backends don't implement a
similar dance. For consistency and maintenability, remove it.
Reverts 4423757c0c.
Closes#18135.
Thanks Carl Meyer for the review.
Squashed commit of the following:
commit 4f290bdb60
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 21:21:30 2013 +0100
Used '0:00' instead of 'UTC' which doesn't always exist in Oracle.
Thanks Ian Kelly for the suggestion.
commit 01b6366f3c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 13:38:43 2013 +0100
Made tzname a parameter of datetime_extract/trunc_sql.
This is required to work around a bug in Oracle.
commit 924a144ef8
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 14:47:44 2013 +0100
Added support for parameters in SELECT clauses.
commit b4351d2890
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 22:30:22 2013 +0100
Documented backwards incompatibilities in the two previous commits.
commit 91ef84713c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:42:31 2013 +0100
Used QuerySet.datetimes for the admin's date_hierarchy.
commit 0d0de288a5
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:29:38 2013 +0100
Used QuerySet.datetimes in date-based generic views.
commit 9c0859ff7c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:25 2013 +0100
Implemented QuerySet.datetimes on Oracle.
commit 68ab511a4f
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:14 2013 +0100
Implemented QuerySet.datetimes on MySQL.
commit 22d52681d3
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:42:29 2013 +0100
Implemented QuerySet.datetimes on SQLite.
commit f6800fd04c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:03 2013 +0100
Implemented QuerySet.datetimes on PostgreSQL.
commit 0c829c23f4
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:41:08 2013 +0100
Added datetime-handling infrastructure in the ORM layers.
commit 104d82a777
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 10:05:55 2013 +0100
Updated null_queries tests to avoid clashing with the __second lookup.
commit c01bbb3235
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 23:07:41 2013 +0100
Updated tests of .dates().
Replaced .dates() by .datetimes() for DateTimeFields.
Replaced dates with datetimes in the expected output for DateFields.
commit 50fb7a5246
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:40:09 2013 +0100
Updated and added tests for QuerySet.datetimes.
commit a8451a5004
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 22:34:46 2013 +0100
Documented the new time lookups and updated the date lookups.
commit 29413eab2b
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 16:15:49 2013 +0100
Documented QuerySet.datetimes and updated QuerySet.dates.
The reason was that the except clause needed to remove a connection
from the django.db.connections dict, but other parts of Django do not
expect this to happen. In addition the except clause was silently
swallowing the exception messages.
Refs #19707, special thanks to Carl Meyer for pointing out that this
approach should be taken.
value_annotation isn't very well defined. Before this change, setting it
to datetime.datetime could silently reverse the behavior of isnull
lookups. This commit doesn't have any consequences on the current code.
It's just a safeguard for future ORM hackers.
Previously it was possible to call clear_ordering without the
force_empty argument. The result was that the query was still ordered
by model's meta ordering if that was defined. By making the arg
mandatory it will be easier to spot possible errors caused by assuming
clear_ordering will remove all ordering.
Thanks to Dylan Klomparens for the suggestion. Refs #19720.
When a query had a complex where condition (a condition targeting more
than the base table) a subquery was used for deletion. However, the
query had default ordering from the model's meta and Oracle doesn't
work with ordered subqueries.
The regression was caused by fast-path deletion code introduced in
1cd6e04cd4 for fixing #18676.
Thanks to Dylan Klomparens for the report.
This undocumented method was used in an old version of the admin, is
totally untested and hails from 2008. Although it's listed in the
"public methods" section, as it's not documented or used I don't think
it needs a deprecation path.
If we think it's useful I'll write some tests/docs for it instead...
Thanks a lot to everybody participating in developing this feature.
The patch was developed by multiple people, at least Trac aliases
tonnzor, jimmysong, Fandekasp and slurms.
Stylistic changes added by committer.
The class wasn't used anywhere except in RelatedObject.bind(), which
wasn't used anywhere. The class had one method defined as
NotImplemented, yet the class wasn't subclassed anywhere. In short, the
class was dead code.
In a normal relational construct, if you're listening for an event
that signals a child was deleted, you dont expect that the parent
was deleted already.
This change ensures that post_delete signals are fired immediately
after objects are deleted in the graph.
The original problem was that queryset cloning was really expensive
when filtering with F() clauses. The __deepcopy__ went too deep copying
_meta attributes of the models used. To fix this the use of
__deepcopy__ in qs cloning was removed.
This commit results in some speed improvements across the djangobench
benchmark suite. Most query_* tests are 20-30% faster, save() is 50%
faster and finally complex filtering situations can see 2x to order
of magnitude improvments.
Thanks to Suor, Alex and lrekucki for valuable feedback.
The guarantee that no queries will be made when accessing results is
done by new EmptyWhere class which is used for query.where and having.
Thanks to Simon Charette for reviewing and valuable suggestions.
When &'ing or |'ing querysets, wrong values could be cached, and crashes
could happen.
Thanks Marc Tamlyn for figuring out the problem and writing the patch.
The added promotion logic is based on promoting any joins used in only
some of the childs of an OR clause unless the join existed before the
OR clause addition.
The ORM didn't reuse joins for direct foreign key traversals when using
chained filters. For example:
qs.filter(fk__somefield=1).filter(fk__somefield=2))
produced two joins.
As a bonus, reverse onetoone filters can now reuse joins correctly
The regression was caused by the join() method refactor in commit
68847135bc
Thanks for Simon Charette for spotting some issues with the first draft
of the patch.
This is necessary because get_model() checks are case insensitive, and if the swapable check isn't, the
swappable logic gets tied up in knots with models that are partially swapped out.
Thanks to chris@cogdon.org for the report and extensive analysis, and Preston for his work on the draft patch.
Cleaned up the implementation of base convert_values() a little, and
made sure it accepts None as a value for numeric fields.
There are no tests attached. The reason is that not all of the
convert_values() accept None as a value for numeric fields (for example
sqlite3.convert_values()).
The reason the base convert_values() needs to accept None is that this
situation might arise in custom compilers for 3rd party backends. It
is easy to keep the convert_values() working, so lets do that.
This is a rather large refactoring. The "lookup traversal" code was
splitted out from the setup_joins. There is now names_to_path() method
which does the lookup traveling, the actual work of setup_joins() is
calling names_to_path() and then adding the joins found into the query.
As a side effect it was possible to remove the "process_extra"
functionality used by genric relations. This never worked for left
joins. Now the extra restriction is appended directly to the join
condition instead of the where clause.
To generate the extra condition we need to have the join field
available in the compiler. This has the side-effect that we need more
ugly code in Query.__getstate__ and __setstate__ as Field objects
aren't pickleable.
The join trimming code got a big change - now we trim all direct joins
and never trim reverse joins. This also fixes the problem in #10790
which was join trimming in null filter cases.
The trim argument was used by split_exclude() only to trim the last
join from the given lookup. It is cleaner to just trim the last part
from the lookup in split_exclude() directly so that there is no need
to burden add_filter() with the logic needed for only split_exclude().
Connection creation was done in db backend ._cursor() call. This
included taking a new connection if needed, initializing the session
state for the new connection and finally creating the connection.
To allow easier modifying of these steps in subclasses (for example to
support connection pools) the _cursor() now calls get_new_connection()
and init_connection_state() if there isn't an existing connection. This
was done for all non-gis core backends. In addition the parameters used
for taking a connection are now created by get_connection_params().
We should also do the same for gis backends and encourage 3rd party
backends to use the same pattern. The pattern is not enforced in code,
and as the backends are private API this will not be required by
documentation either.
F() expressions reuse joins like any lookup in a .filter() call -
reuse multijoins generated in the same .filter() call else generate
new joins. Also, lookups can now reuse joins generated by F().
This change is backwards incompatible, but it is required to prevent
dict randomization from generating different queries depending on
.filter() kwarg ordering. The new way is also more consistent in how
joins are reused.
The select_related code got confused when it needed to travel a
reverse relation to a model which had different parent than the
originally travelled relation.
Thanks to Trac aliases shauncutts for report and ungenio for original
patch (committed patch is somewhat modified version of that).
The problem is the same as in #10888 which was reintroduced when
bulk_insert was added. Thanks to Jani Tiainen for report, patch and
also testing the final patch on Oracle GIS.
This is the start of a deprecation path for the depth kwarg on
select_related. Removing this will allow us to update select_related so
it chains properly and have an API similar to prefetch_related.
Thanks to Marc Tamlyn for spearheading and initial patch.
refs #16855
The dupe avoidance logic was removed as it doesn't seem to do anything,
it is complicated, and it has nearly zero documentation.
The removal of dupe_avoidance allowed for refactoring of both the
implementation and signature of Query.join(). This refactoring cascades
again to some other parts. The most significant of them is the changes
in qs.combine(), and compiler.select_related_descent().
There was a potential data-loss issue involved -- when clearing
instance's m2m assignments it was possible some other instance's
m2m data was deleted instead.
This commit also improved None handling for to_field cases.
The Query.select and Query.select_fields were collapsed into one list
because the attributes had to be always in sync. Now that they are in
one attribute it is impossible to edit them out of sync.
Similar collapse was done for Query.related_select_cols and
Query.related_select_fields.
There was a bug introduced in #18676 which caused fast-path deletes
implemented as "DELETE WHERE pk IN <subquery>" to fail if the SELECT
clause contained additional stuff (for example extra() and annotate()).
Thanks to Trac alias pressureman for spotting this regression.
Done for consistency with Q() expressions and QuerySet combining. This
will allow usage of '&' and '|' as boolean logical operators in the
future. Refs #16211.
RETURNING is an extension of the SQL standard, which is not implemented
the same by all databases. Allow DatabaseOperations.return_insert_id to
return a None to allow for other 3rd party backends with a different
implementation.
Objects can be fast-path deleted if there are no signals, and there are
no further cascades. If fast-path is taken, the objects do not need to
be loaded into memory before deletion.
Thanks to Jeremy Dunck, Simon Charette and Alex Gaynor for reviewing
the patch.
Thanks to the many people that contributed to the development and review of
this patch, including (but not limited to) Jacob Kaplan-Moss, Anssi
Kääriäinen, Ramiro Morales, Preston Holmes, Josh Ourisman, Thomas Sutton,
and Roger Barnes, as well as the many, many people who have contributed to
the design discussion around this ticket over many years.
Squashed commit of the following:
commit d84749a0f0
Merge: 531e7717c11b1a
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Wed Sep 26 18:37:04 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit 531e7715da
Merge: 29d1abb1f84b04
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Wed Sep 26 07:09:23 2012 +0800
Merged recent trunk changes.
commit 29d1abbe35
Merge: 8a527dd54c81a1
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Sep 24 07:49:46 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit 8a527dda13
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Sep 24 07:48:05 2012 +0800
Ensure sequences are reset correctly in the presence of swapped models.
commit e2b6e22f29
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 23 17:53:05 2012 +0800
Modifications to the handling and docs for auth forms.
commit 98aba856b5
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 23 15:28:57 2012 +0800
Improved error handling and docs for get_user_model()
commit 0229209c84
Merge: 6494bf98599f64
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 23 14:50:11 2012 +0800
Merged recent Django trunk changes.
commit 6494bf91f2
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Sep 17 21:38:44 2012 +0800
Improved validation of swappable model settings.
commit 5a04cde342
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Sep 17 07:15:14 2012 +0800
Removed some unused imports.
commit ffd535e413
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 20:31:28 2012 +0800
Corrected attribute access on for get_by_natural_key
commit 913e1ac84c
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 20:12:34 2012 +0800
Added test for proxy model safeguards on swappable models.
commit 280bf19e94
Merge: dbb3900935a863
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 18:16:49 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit dbb3900775
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 18:09:27 2012 +0800
Fixes for Python 3 compatibility.
commit dfd72131d8
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 15:54:30 2012 +0800
Added protection against proxying swapped models.
commit abcb027190
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 15:11:10 2012 +0800
Cleanup and documentation of AbstractUser base class.
commit a9491a8776
Merge: fd8bb4e08bcb4a
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 14:46:49 2012 +0800
Merge commit '08bcb4aec1ed154cefc631b8510ee13e9af0c19d' into t3011
commit fd8bb4e3e4
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 14:20:14 2012 +0800
Documentation improvements coming from community review.
commit b550a6d06d
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 13:52:47 2012 +0800
Refactored skipIfCustomUser into the contrib.auth tests.
commit 52a02f1110
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 13:46:10 2012 +0800
Refactored common 'get' pattern into manager method.
commit b441a6bbc7
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 16 13:41:33 2012 +0800
Added note about backwards incompatible change to admin login messages.
commit 08bcb4aec1
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Sep 15 18:30:33 2012 +0300
Splitted User to AbstractUser and User
commit d9f5e5addb
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Sep 15 18:30:02 2012 +0300
Reworked REQUIRED_FIELDS + create_user() interaction
commit 579f152e4a
Merge: 918497293e6733
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 15 20:18:37 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit 918497218c
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 15 20:18:19 2012 +0800
Deprecate AUTH_PROFILE_MODULE and get_profile().
commit 334cdfc1bb
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 15 20:00:12 2012 +0800
Added release notes for new swappable User feature.
commit 5d7bb22e8d
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 15 19:59:49 2012 +0800
Ensure swapped models can't be queried.
commit 57ac6e3d32
Merge: f2ec915abfba3b
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 15 14:31:54 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit f2ec915b20
Merge: 19526565e99a3d
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 9 08:29:51 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit 19526563b5
Merge: 2c5e833c4aa26a
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 9 08:22:26 2012 +0800
Merge recent changes from master.
commit 2c5e833a30
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 9 07:53:46 2012 +0800
Corrected admin_views tests following removal of the email fallback on admin logins.
commit 20d1892491
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sun Sep 9 01:00:37 2012 +0800
Added conditional skips for all tests dependent on the default User model
commit 40ea8b8882
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 8 23:47:02 2012 +0800
Added documentation for REQUIRED_FIELDS in custom auth.
commit e6aaf65970
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Sat Sep 8 23:20:02 2012 +0800
Added first draft of custom User docs.
Thanks to Greg Turner for the initial text.
commit 75118bd242
Author: Thomas Sutton <me@thomas-sutton.id.au>
Date: Mon Aug 20 11:17:26 2012 +0800
Admin app should not allow username discovery
The admin app login form should not allow users to discover the username
associated with an email address.
commit d088b3af58
Author: Thomas Sutton <me@thomas-sutton.id.au>
Date: Mon Aug 20 10:32:13 2012 +0800
Admin app login form should use swapped user model
commit 7e82e83d67
Merge: e29c01039aa890
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Fri Sep 7 23:45:03 2012 +0800
Merged master changes.
commit e29c010beb
Merge: 8e3fd7030bdf22
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Aug 20 13:12:57 2012 +0800
Merge remote-tracking branch 'django/master' into t3011
commit 8e3fd703d0
Merge: 507bb5026e0ba0
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Aug 20 13:09:09 2012 +0800
Merged recent changes from trunk.
commit 507bb50a92
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Jun 4 20:41:37 2012 +0800
Modified auth app so that login with alternate auth app is possible.
commit dabe362836
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Jun 4 20:10:51 2012 +0800
Modified auth management commands to handle custom user definitions.
commit 7cc0baf89d
Author: Russell Keith-Magee <russell@keith-magee.com>
Date: Mon Jun 4 14:17:28 2012 +0800
Added model Meta option for swappable models, and made auth.User a swappable model
Thanks to Jeremy Dunck for pointing out the problem with this change. If in a
single transaction, the master deletes a record and then get_or_creates a
similar record, under the new behavior the get_or_create would find the record
in the slave db and fail to re-create it, leaving the record nonexistent, which
violates the contract of get_or_create that the record should always exist
afterwards. We need to do everything against the master here in order to ensure
correctness.
This reverts commit 901af86550.
When doing deeper than one level select_related() + only queries(), the
code introduced in b6c356b7bb errored
incorrectly.
Thanks to mrmachine for report & test case.
In an ideal world, nothing except django.db.models.query should have to
import stuff from django.models.sql.*. A few things were needing to get
hold of sql.constants.LOOKUP_SEP, so this commit moves it up to
django.db.models.constants.LOOKUP_SEP.
There are still a couple of places (admin) poking into sql.* to get
QUERY_TERMS, which is unfortunate, but a slightly different issue and
harder to adjust.
Previous this used Python's builtin hash() function, which has never been guarnteed to be stable across implementations (CPython/Jython/etc.) or 32/64 bitness. However, this in practice it was stable. However, with the impending release of Python 3.3 hash randomizations is enabled by default, which would mean the index name changed between program invocations.
Unlike other model fields, the newly introduced (1.4)
GenericIPAddressField did not accept verbose_name and name as the
first positional arguments. This commit fixes it.
Thanks Dan McGee for the report and the patch.
The joins for nested nullable foreign keys were often created as INNER
when they should have been OUTER joins. The reason was that only the
first join in the chain was promoted correctly. There were also issues
with select_related etc.
The basic structure for this problem was:
A -[nullable]-> B -[nonnull]-> C
And the basic problem was that the A->B join was correctly LOUTER,
the B->C join not.
The major change taken in this patch is that now if we promote a join
A->B, we will automatically promote joins B->X for all X in the query.
Also, we now make sure there aren't ever join chains like:
a LOUTER b INNER c
If the a -> b needs to be LOUTER, then the INNER at the end of the
chain will cancel the LOUTER join and we have a broken query.
Sebastian reported this problem and did also major portions of the
patch.
The ORM generated a query with INNER JOIN instead of LEFT OUTER JOIN
in a somewhat complicated case. The main issue was that there was a
chain of nullable FK -> non-nullble FK, and the join promotion logic
didn't see the need to promote the non-nullable FK even if the
previous nullable FK was already promoted to LOUTER JOIN. This resulted
in a query like a LOUTER b INNER c, which incorrectly prunes results.
Deferred models now automatically update only the fields which are
loaded from the db (with .only() or .defer()). In addition, any field
set manually after the load is updated on save.
This patch removes an unconditional float(value) conversion from db
backend default convert_values() method. This can cause problems when
aggregating over character fields for example. In addition, Oracle
and SQLite already return the bare value from their convert_values().
In the long term the converting should be done by fields, and the
fields should then call database backend specific converters when
needed. The current setup is inflexible for 3rd party fields.
Thanks to Merlijn van Deen for the original patch.
* Renamed the __unicode__ methods
* Applied the python_2_unicode_compatible decorator
* Removed the StrAndUnicode mix-in that is superseded by
python_2_unicode_compatible
* Kept the __unicode__ methods in classes that specifically
test it under Python 2
* Renamed smart_unicode to smart_text (but kept the old name under
Python 2 for backwards compatibility).
* Renamed smart_str to smart_bytes.
* Re-introduced smart_str as an alias for smart_text under Python 3
and smart_bytes under Python 2 (which is backwards compatible).
Thus smart_str always returns a str objects.
* Used the new smart_str in a few places where both Python 2 and 3
want a str.
Previously, the flush was done before the test case execution and now
it is performed after it.
Other changes to the testing infrastructure include:
* TransactionTestCase now doesn't reset autoincrement sequences either
(previous behavior can achieved by using `reset_sequences`.)
With this, no implicit such reset is performed by any of the provided
TestCase classes.
* New ordering of test cases: All unittest tes cases are run first and
doctests are run at the end.
THse changes could be backward-incompatible with test cases that relied
on some kind of state being preserved between tests. Please read the
relevant sections of the release notes and testing documentation for
further details.
Thanks Andreas Pelme for the initial patch. Karen Tracey and Anssi
Kääriäinen for the feedback and Anssi for reviewing.
This also fixes#12408.
The qs.bulk_create() method did not work with large batches together
with SQLite3. This commit adds a way to split the bulk into smaller
batches. The default batch size is unlimited except for SQLite3 where
the batch size is limited to 999 SQL parameters per batch.
Thanks to everybody who participated in the discussions at Trac.
Cleared aggregations on add_date_select method so only distinct dates
are returned when dealing with a QuerySet that contained aggregations.
That would cause the query set to return repeated dates because it
would look for distinct (date kind, aggregation) pairs.
At least Oracle needs parentheses in negated where conditions, even if
there is only single condition negated. Fixed this by reverting to old
logic in that part of as_sql() and adding a comment about this.
I did not investigate why the parentheses are needed. The original
offending commit was bd283aa844.
When the postgresql_psycopg2 backend was used with DB-level autocommit
mode enabled, after entering transaction management and then leaving
it, the isolation level was never set back to autocommit mode.
Thanks brodie for report and working on this issue.
Made sure the WhereNode.as_sql() handles various EmptyResultSet and
FullResultSet conditions correctly. Also, got rid of the FullResultSet
exception class. It is now represented by '', [] return value in the
as_sql() methods.
This commit tackles a couple of issues. First, in certain cases there
were some mixups if field.attname or field.name should be deferred.
Field.attname is now always used.
Another issue tackled is a case where field is both deferred by
.only(), and selected by select_related. This case is now an error.
A lot of thanks to koniiiik (Michal Petrucha) for the patch, and
to Andrei Antoukh for review.
In addition, removed a possibly problematic .filter() call from
backends.test_query_encoding test. It is possible the .filter could
cause collation problems on MySQL, and as it wasn't absolutely needed
for the test it seemed better to get rid of the call.
Refs #18461.
This does remove the requirement to call features.confirm() method
before checking the properties.
Thanks cdestiger and Ramiro Morales for their work on the patch.
On MySQL when checking the server version, a new connection could be
created but never closed. This could result in open connections on
server startup.
Generic cleanup and dead code removal in deferred model field loading
and model.__reduce__().
Also fixed an issue where if an inherited model with a parent field
chain parent_ptr_id -> id would be deferred loaded, then accessing
the id field caused caused a database query, even if the id field's
value is already loaded in the parent_ptr_id field.
When order_by causes new joins to be added to the query, the joins must
be LEFT OUTER joins for nullable relations, otherwise the order_by
could cause the results to be altered. This commit fixes the logic to
only promote new joins, previously all joins in the order_by lookup
path were promoted.
Thanks to Bruno Desthuilliers for spotting this corner case.
This was recently fixed for one-to-one relations; this patch adds
support for foreign keys. Thanks kaiser.yann for the report and
the initial version of the patch.
Databases with update_can_self_select = False (MySQL for example)
generated non-necessary queries when saving a multitable inherited
model, and when the save resulted in update.
after the full file name is generated by the storage class.
Thanks Refefer for the report, carsongee for the patch, and
everyone else involved in the discussion.
Added the ability to update only part of the model's fields in
model.save() by introducing a new kwarg "update_fields". Thanks
to all the numerous reviewers and commenters in the ticket
Baptiste Mispelon
Andrew Godwin
maurycyp
Alex Hill
Loic Bistuer
Simon Charette
Christopher Medrela
Anssi Kääriäinen
Joel Bohman
Alex Gaynor
Claude Paroz
Florian Apolloner
Milton Mazzarri
coagulant
Tim Graham
Alasdair Nicol
Florian Hahn
Alasdair Nicol
Marc Tamlyn
Aymeric Augustin
Javed Khan
Russell Keith-Magee
Erik Romijn
Michael Manfre
Andrei Picus
Markus Holtermann
Gregor MacGregor
Садовский Николай
Eric Boersma
Chris Wilson
Carl Meyer
Dima Kurguzov
Harm Geerts
Alex Cucu
Curtis Maloney
Alex Couper
Tai Lee
Shai Berger
Julien Phalip
ersran9
Karol Sikora
Ramiro Morales
Andrew Clark
Gilberto Gonçalves
Axel Haustant
James Aylett
Dan Loewenherz
Tim Graham
Luke Plant
Donald Stufft
Selwin Ong
Erik Romijn
Shai Berger
Mike Fogel
Mike Fogel
Adrian Holovaty
Tobias McNulty
Alex Gaynor
Marti Raudsepp
Gavin Wahl
Daniele Procida
Honza Král
Tomasz Rybak
Marcin Biernat
Grzegorz Nosek
Tomek Paczkowski
Nick Sandford
Jani Tiainen
David Cramer
mpaolini
orblivion
Mike Johnson
Preston Holmes
Jan Bednařík
Adrien Lemaire
Malcolm Tredinnick
Mitar
Simon Meers
Dmitry Shevchenko
Andrei Antoukh
Brendan MacDonell
Nuno Maltez
Josh Smeaton
Jann Kleen
Michael Newman