Also, use Django templating for the dynamic generated JS code and use
more idiomatic coding techniques.
Thanks Matthew Tretter for the report and the patch.
I refactored RadioSelect and CheckboxSelectMultiple to
make them inherit from a base class, allowing them to share
the behavior of being able to iterate over their subwidgets.
Thanks to Matt McClanahan for the initial patch and to
Claude Paroz for the review.
Queries can contain binary data undecodable with utf-8. In this
case, using the 'replace' errors mode when decoding seems like
an acceptable representation of the query.
Thanks Marcel Ryser for the report.
This patch iproved two major parts in Django. First, the fields.related
was refactored. The main addition there was ForeignObject. Second, the
ORM now handles multicolumn joins in most cases, though there are still
cases that do not work correcly (split_exclude() for example).
In addition there were extesive changes to how GenericRelation works.
Before it was a fake m2m field, now it is a pure virtual fields and is
based on ForeignObject.
There is still much room for improvement. The related fields code is
still somewhat confusing, and how fields are represented in model._meta
should also be revisited.
This patch was written mostly by Jeremy Tillman with some final polish
by the committer.
Previously, depending on the database backend or the cursor type,
you'd need to double the percent signs in the query before passing
it to cursor.execute. Now cursor.execute consistently need percent
doubling whenever params argument is not None (placeholder substitution
will happen).
Thanks Thomas Güttler for the report and Walter Doekes for his work
on the patch.
This is provided as a new "validate_max" formset_factory option defaulting to
False, since the non-validating behavior of max_num is longstanding, and there
is certainly code relying on it. (In fact, even the Django admin relies on it
for the case where there are more existing inlines than the given max_num). It
may be that at some point we want to deprecate validate_max=False and
eventually remove the option, but this commit takes no steps in that direction.
This also fixes the DoS-prevention absolute_max enforcement so that it causes a
form validation error rather than an IndexError, and ensures that absolute_max
is always 1000 more than max_num, to prevent surprising changes in behavior
with max_num close to absolute_max.
Lastly, this commit fixes the previous inconsistency between a regular formset
and a model formset in the precedence of max_num and initial data. Previously
in a regular formset, if the provided initial data was longer than max_num, it
was truncated; in a model formset, all initial forms would be displayed
regardless of max_num. Now regular formsets are the same as model formsets; all
initial forms are displayed, even if more than max_num. (But if validate_max is
True, submitting these forms will result in a "too many forms" validation
error!) This combination of behaviors was chosen to keep the max_num validation
simple and consistent, and avoid silent data loss due to truncation of initial
data.
Thanks to Preston for discussion of the design choices.
The ticket dealt with a case where one query had .exclude() that
produced a subquery, the other query had a join to the same model that
was subqueried in the first query. This was already fixed in master, so
only test added.
This reverts commit 2cd0edaa47.
This commit was the cause of a memory leak. See ticket for more details.
Thanks Anssi Kääriäinen for identifying the source of the bug.
Before this change, the get_admin_log method would expect User model's
FK to be named `id`. When changing that FK name, admin/index.html
rendering would fail.
This includes:
* Changed the use of id for the use of pk property.
* Added a regression test that fails without the patch.
This commit refs #20088.
The original problem was that when filtering two levels up in
inheritance chain, Django optimized the join generation so that the
middle model was skipped. But then Django generated joins from top
to middle to bottom for SELECT clause, and thus there was one extra
join (top->middle->bottom + top -> bottom).
This case is fixed in master as the filtering optimization is gone.
This has the side effect that in some cases there is still extra join
if the SELECT clause doesn't contain anything from middle or bottom.
Model.save() will use UPDATE - if not updated - INSERT instead of
SELECT - if found UPDATE else INSERT. This should save a query when
updating, but will cost a little when inserting model with PK set.
Also fixed#17341 -- made sure .save() commits transactions only after
the whole model has been saved. This wasn't the case in model
inheritance situations.
The save_base implementation was refactored into multiple methods.
A typical chain for inherited save is:
save_base()
_save_parents(self)
for each parent:
_save_parents(parent)
_save_table(parent)
_save_table(self)
Thanks Anssi for haggling until I implemented this.
This change alleviates the need for atomic_if_autocommit. When
autocommit is disabled for a database, atomic will simply create and
release savepoints, and not commit anything. This honors the contract of
not doing any transaction management.
This change also makes the hack to allow using atomic within the legacy
transaction management redundant.
None of the above will work with SQLite, because of a flaw in the design
of the sqlite3 library. This is a known limitation that cannot be lifted
without unacceptable side effects eg. triggering arbitrary commits.
The sql/query.py add_q method did a lot of where/having tree hacking to
get complex queries to work correctly. The logic was refactored so that
it should be simpler to understand. The new logic should also produce
leaner WHERE conditions.
The changes cascade somewhat, as some other parts of Django (like
add_filter() and WhereNode) expect boolean trees in certain format or
they fail to work. So to fix the add_q() one must fix utils/tree.py,
some things in add_filter(), WhereNode and so on.
This commit also fixed add_filter to see negate clauses up the path.
A query like .exclude(Q(reversefk__in=a_list)) didn't work similarly to
.filter(~Q(reversefk__in=a_list)). The reason for this is that only
the immediate parent negate clauses were seen by add_filter, and thus a
tree like AND: (NOT AND: (AND: condition)) will not be handled
correctly, as there is one intermediary AND node in the tree. The
example tree is generated by .exclude(~Q(reversefk__in=a_list)).
Still, aggregation lost connectors in OR cases, and F() objects and
aggregates in same filter clause caused GROUP BY problems on some
databases.
Fixed#17600, fixed#13198, fixed#17025, fixed#17000, fixed#11293.
Before there was need to have both .relabel_aliases() and .clone() for
many structs. Now there is only relabeled_clone() for those structs
where alias is the only mutable attribute.
Replaced them with per-database options, for proper multi-db support.
Also toned down the recommendation to tie transactions to HTTP requests.
Thanks Jeremy for sharing his experience.
Since "unless managed" now means "if database-level autocommit",
committing or rolling back doesn't have any effect.
Restored transactional integrity in a few places that relied on
automatically-started transactions with a transitory API.
For users who didn't activate autocommit in their database options, this
is backwards-incompatible in "non-managed" aka "auto" transaction state.
This state now uses database-level autocommit instead of ORM-level
autocommit.
Also removed the uses_autocommit feature which lost its purpose.
Autocommit cannot be manipulated independently from an open connection.
This commit introduces a minor change in behavior: entering transaction
management forces opening a databasse connection. This shouldn't be
backwards incompatible in any practical use case.
enter_transaction_management() was nearly always followed by managed().
In three places it wasn't, but they will all be refactored eventually.
The "forced" keyword argument avoids introducing behavior changes until
then.
This is mostly backwards-compatible, except, of course, for managed
itself. There's a minor difference in _enter_transaction_management:
the top self.transaction_state now contains the new 'managed' state
rather than the previous one. Django doesn't access
self.transaction_state in _enter_transaction_management.
There were a couple of errors in ._dirty flag handling:
* It started as None, but was never reset to None.
* The _dirty flag was sometimes used to indicate if the connection
was inside transaction management, but this was not done
consistently. This also meant the flag had three separate values.
* The None value had a special meaning, causing for example inability
to commit() on new connection unless enter/leave tx management was
done.
* The _dirty was tracking "connection in transaction" state, but only
in managed transactions.
* Some tests never reset the transaction state of the used connection.
* And some additional less important changes.
This commit has some potential for regressions, but as the above list
shows, the current situation isn't perfect either.
Fixes#19584.
This implies stop storing file path command line arguments in envvars as
a security measure to start relying on with Popen's shell=False instead,
and addition of an 'utils' module.
Thanks kmichel_wgs for the report.
If Django was symlinked into site-packages the previous approach to discover
the tests subdirectory would fail. The revised version now always points to
the location of the source and not the import path.
The content of ifchanged template tag was rendered twice: first time, to
compare it with the previous value and the second time, to return the
rendered output.
Previously, the ifchanged node stored state on `self._last_seen`,
thereby giving undesired results when the node is reused by another
thread at the same time (e.g. globally caching a Template object).
Thanks to akaihola for the report and Diederik van der Boor and
Bas Peschier for the patch.
Changed the ip_address field for Comment to GenericIPAddressField. Added
instructions to the release notes on how to update the schema of existing
databases.
Change patch from https://code.djangoproject.com/ticket/5568
to work on modern Django.
Add special case for MySQL which has different syntax for DROP INDEX.
Add unit tests for the new functionality.
When iteration over a queryset raised an exception, the result cache
remained initialized with an empty list, so subsequent iterations returned
an empty list instead of raising an exception
Querying the reverse side of nullable to_field relation, where both
sides can contain null values resulted in incorrect results. The reason
was not detecting '' as NULL.
Refs #17541
There was a regression in case two models inherited the same parent,
and one contained a foreign key to other. When select_related travelled
the foreign key the other model reused the parent join made by the
first model. This was likely caused by Query.join_parent_model()
addition in commit 68985db482.
Thanks to Trac alias loic84 for report & tests.
The admin_widgets tests were issuing click() to the browser but
didn't wait for the effects of those clicks. This caused the resulting
request to be processed concurrently with the test case. When using
in-memory SQLite this caused weird failures.
Also added wait_page_loaded() to admin selenium tests for code
reuse.
Fixed#19856
The join promote=True was over-aggressive in select_related handling.
After that was removed, the only other user was query.combine(). That
use case is very easy to handle locally, so there is no more need for
the join(promote=True) flag.
Refs #19849.
This controls whether or not a database level cosntraint is created. This is useful in a few specialized circumstances, but in general should not be used!
The refactoring mainly concentrates on making sure the inner and outer
query agree about the split position. The split position is where the
multijoin happens, and thus the split position also determines the
columns used in the "WHERE col1 IN (SELECT col2 from ...)" condition.
This commit fixes a regression caused by #10790 and commit
69597e5bcc. The regression was caused
by wrong cols in the split position.
Django used to check the version of MySQL before handling the first
request, which required:
- opening a connection
- closing it, to avoid holding it idle until the first request.
This code isn't necessary any longer since Django dropped support for
some versions of MySQL, and other database backends don't implement a
similar dance. For consistency and maintenability, remove it.
Reverts 4423757c0c.
Closes#18135.
* Avoided calling BaseHttpResponse.close(). The test client take care of
that since acc5396e.
* Disconnected the request_finished signal when this method must be
called. The test client has a similar implementation since bacb097a.
Thanks Carl Meyer for the review.
Squashed commit of the following:
commit 4f290bdb60
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 21:21:30 2013 +0100
Used '0:00' instead of 'UTC' which doesn't always exist in Oracle.
Thanks Ian Kelly for the suggestion.
commit 01b6366f3c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 13:38:43 2013 +0100
Made tzname a parameter of datetime_extract/trunc_sql.
This is required to work around a bug in Oracle.
commit 924a144ef8
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 14:47:44 2013 +0100
Added support for parameters in SELECT clauses.
commit b4351d2890
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 22:30:22 2013 +0100
Documented backwards incompatibilities in the two previous commits.
commit 91ef84713c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:42:31 2013 +0100
Used QuerySet.datetimes for the admin's date_hierarchy.
commit 0d0de288a5
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:29:38 2013 +0100
Used QuerySet.datetimes in date-based generic views.
commit 9c0859ff7c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:25 2013 +0100
Implemented QuerySet.datetimes on Oracle.
commit 68ab511a4f
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:14 2013 +0100
Implemented QuerySet.datetimes on MySQL.
commit 22d52681d3
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:42:29 2013 +0100
Implemented QuerySet.datetimes on SQLite.
commit f6800fd04c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:03 2013 +0100
Implemented QuerySet.datetimes on PostgreSQL.
commit 0c829c23f4
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:41:08 2013 +0100
Added datetime-handling infrastructure in the ORM layers.
commit 104d82a777
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 10:05:55 2013 +0100
Updated null_queries tests to avoid clashing with the __second lookup.
commit c01bbb3235
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 23:07:41 2013 +0100
Updated tests of .dates().
Replaced .dates() by .datetimes() for DateTimeFields.
Replaced dates with datetimes in the expected output for DateFields.
commit 50fb7a5246
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:40:09 2013 +0100
Updated and added tests for QuerySet.datetimes.
commit a8451a5004
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 22:34:46 2013 +0100
Documented the new time lookups and updated the date lookups.
commit 29413eab2b
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 16:15:49 2013 +0100
Documented QuerySet.datetimes and updated QuerySet.dates.
For languages with non-breaking space as thousand separator,
standard space input should also be allowed, as few people know
how to enter non-breaking space on keyboards. Refs #17217.
Thanks Alexey Boriskin for the report and initial patch.
Wrap the Parser.compile_filter method call with a try/except and call the
newly added Parser.compile_filter_error(). Overwrite this method in the
DebugParser to throw the correct error.
Since this error was otherwise catched by the compile_function try/except
block the debugger highlighted the wrong line.
The reason was that the except clause needed to remove a connection
from the django.db.connections dict, but other parts of Django do not
expect this to happen. In addition the except clause was silently
swallowing the exception messages.
Refs #19707, special thanks to Carl Meyer for pointing out that this
approach should be taken.
When a query had a complex where condition (a condition targeting more
than the base table) a subquery was used for deletion. However, the
query had default ordering from the model's meta and Oracle doesn't
work with ordered subqueries.
The regression was caused by fast-path deletion code introduced in
1cd6e04cd4 for fixing #18676.
Thanks to Dylan Klomparens for the report.
Also, according to the comments on the ticket and its duplicates, added
tests execising saving an instance of a model with a GFK to:
* An unsaved object -- This actually doesn't generate the same failure
but another ORM-level exception. The test verifies it's the case.
* An instance of a model with a __nonzero__() method thant returns False
for it. This doesn't fail because that code path isn't executed.
* An instance of a model with a CharField PK and an empty value for it.
This doesn't fail.
They are handled independently now and the latter can be influenced by
the new BaseCommand.leave_locale_alone internal option.
Thanks chrischambers for the report, Claude, lpiatek, neaf and gabooo for
their work on a patch, originally on refs. #17379.
They are simply ignored now. This allows for a more correct behavior when
they are placed before translatable constructs on the same line.
Previously, the latter were wrongly ignored because the former were
preserved when converting template code to the internal Python-syntax
form later fed to xgettext but Python has no ``/* ... */``-style
comments.
Also, special comments directed to translators are now only taken in
account when they are located at the end of a line. e.g.::
{# Translators: ignored #}{% trans "Literal A" %}{# Translators: valid, associated with "Literal B" below #}
{% trans "Literal B" %}
Behavior of ``{% comment %}...{% endcomment %}``tags remains unchanged.
Thanks juneih at redpill-linpro dot com for the report and Claude for
his work on the issue.
The main cleanup was removal of non-necessary __unicode__ method. The
tests didn't break on py3 as the string representation was never used
in the tests.
Refs #17813. Thanks to Simon Charette for spotting this issue.
Thanks a lot to everybody participating in developing this feature.
The patch was developed by multiple people, at least Trac aliases
tonnzor, jimmysong, Fandekasp and slurms.
Stylistic changes added by committer.
It creates a `locale/django.pot` file once instead of one
`locale/<locale_code>/django.pot` file for every locale involved.
Thanks Michal Čihař for the report and patch.
In a normal relational construct, if you're listening for an event
that signals a child was deleted, you dont expect that the parent
was deleted already.
This change ensures that post_delete signals are fired immediately
after objects are deleted in the graph.
The original problem was that queryset cloning was really expensive
when filtering with F() clauses. The __deepcopy__ went too deep copying
_meta attributes of the models used. To fix this the use of
__deepcopy__ in qs cloning was removed.
This commit results in some speed improvements across the djangobench
benchmark suite. Most query_* tests are 20-30% faster, save() is 50%
faster and finally complex filtering situations can see 2x to order
of magnitude improvments.
Thanks to Suor, Alex and lrekucki for valuable feedback.
The tickets are either about different signature between qs.none() and
qs or problems with subclass types (either EmptyQS overrided the custom
qs class, or EmptyQS was overridden by another class - values() did
this).
Fixed#15959, fixed#17271, fixed#17712, fixed#19426
The guarantee that no queries will be made when accessing results is
done by new EmptyWhere class which is used for query.where and having.
Thanks to Simon Charette for reviewing and valuable suggestions.
When &'ing or |'ing querysets, wrong values could be cached, and crashes
could happen.
Thanks Marc Tamlyn for figuring out the problem and writing the patch.
* Used override_settings consistently -- changes to DEBUG could leak.
* Took advantage of assertRaisesRegexp.
* Fixed indentation -- some code was indented at 2 spaces.
An index on TextField results in a warning message when running tests
on MySQL or SQLite, and the test using the TextField was PostgreSQL
only in any case.
The added promotion logic is based on promoting any joins used in only
some of the childs of an OR clause unless the join existed before the
OR clause addition.
The ORM didn't reuse joins for direct foreign key traversals when using
chained filters. For example:
qs.filter(fk__somefield=1).filter(fk__somefield=2))
produced two joins.
As a bonus, reverse onetoone filters can now reuse joins correctly
The regression was caused by the join() method refactor in commit
68847135bc
Thanks for Simon Charette for spotting some issues with the first draft
of the patch.
This is necessary because get_model() checks are case insensitive, and if the swapable check isn't, the
swappable logic gets tied up in knots with models that are partially swapped out.
Thanks to chris@cogdon.org for the report and extensive analysis, and Preston for his work on the draft patch.
This is a rather large refactoring. The "lookup traversal" code was
splitted out from the setup_joins. There is now names_to_path() method
which does the lookup traveling, the actual work of setup_joins() is
calling names_to_path() and then adding the joins found into the query.
As a side effect it was possible to remove the "process_extra"
functionality used by genric relations. This never worked for left
joins. Now the extra restriction is appended directly to the join
condition instead of the where clause.
To generate the extra condition we need to have the join field
available in the compiler. This has the side-effect that we need more
ugly code in Query.__getstate__ and __setstate__ as Field objects
aren't pickleable.
The join trimming code got a big change - now we trim all direct joins
and never trim reverse joins. This also fixes the problem in #10790
which was join trimming in null filter cases.
F() expressions reuse joins like any lookup in a .filter() call -
reuse multijoins generated in the same .filter() call else generate
new joins. Also, lookups can now reuse joins generated by F().
This change is backwards incompatible, but it is required to prevent
dict randomization from generating different queries depending on
.filter() kwarg ordering. The new way is also more consistent in how
joins are reused.
Andrew Godwin
Carl Meyer
Matthew Tretter
Claude Paroz
Baptiste Mispelon
Preston Timmons
Baptiste Mispelon
Tobias McNulty
Simon Charette
Preston Timmons
Julien Phalip
Joe Friedl
Christoph Sieghart
Jacob Kaplan-Moss
Anssi Kääriäinen
Adam Wentz
Preston Holmes
Loic Bistuer
Aymeric Augustin
Marc Tamlyn
Andrew Gorcester
Carny Cheng
Matthew Wood
Paul Collins
konarkmodi
Juan Catalano
Deric Crago
Florian Apolloner
Ramiro Morales
Alex Gaynor
Igor Támara
Juan Pedro Fisanotti
Javier Mansilla
matiasb
Łukasz Langa
Florian Hahn
Christopher Medrela
Diederik van der Boor
Zbigniew Siciarz
Tomek Paczkowski
Erik Romijn
Bas Peschier
Wiktor Kolodziej
Honza Král
Tomasz Rybak
George Song
Marcin Biernat
Aleksandra Sendecka
Klaas van Schelven
Grzegorz Nosek
Konrad Hałas
Lennart Regebro
Vladimir A Filonov
Michael van Tellingen
Hiroki Kiyohara
Nick Sandford
Alexey Boriskin
Matt Robenolt
Tim Graham
Craig Blaszczyk
David Cramer
Loic Raucy
Anton Baklanov
Luke Plant
Ian Clelland
Russell Keith-Magee
Patryk Zawadzki
Andreas Hug
Marc Aymerich
Sebastián Magrí
Tom Insam
Tai Lee
Danilo Bargen
Edward Tjörnhammar
Chris Khoo
George Hickman
Jannis Leidel
Chris Beaven