Complete rework of translating data values from database
Deprecation of SubfieldBase, removal of resolve_columns and
convert_values in favour of a more general converter based approach and
public API Field.from_db_value(). Now works seamlessly with aggregation,
.values() and raw queries.
Thanks to akaariai in particular for extensive advice and inspiration,
also to shaib, manfre and timograham for their reviews.
In the MySQL backend, updated the can_introspect_small_integer feature
flag to True. In data_types_reverse, map FIELD_TYPE.SHORT to a
SmallIntegerField. Added test to verify introspecting SmallIntegerFields
and fixed existing tests influenced by this change.
Change database test settings from "TEST_"-prefixed entries in the
database settings dictionary to setting in a dictionary that is itself
an entry "TEST" in the database settings.
Refs #21775
Thanks Josh Smeaton for review.
This commit touchs various parts of the code base and test framework. Any
found usage of opening a cursor for the sake of initializing a connection
has been replaced with 'ensure_connection()'.
This is the result of Christopher Medrela's 2013 Summer of Code project.
Thanks also to Preston Holmes, Tim Graham, Anssi Kääriäinen, Florian
Apolloner, and Alex Gaynor for review notes along the way.
Also: Fixes#8579, fixes#3055, fixes#19844.
Thanks dan at dlo.me for the initial patch.
- Added __pow__ and __rpow__ to ExpressionNode
- Added oracle and mysql specific power expressions
- Added used-defined power function for sqlite
Squashed commit of the following:
commit 63ddb271a44df389b2c302e421fc17b7f0529755
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 29 22:51:00 2013 +0200
Clarified interactions between atomic and exceptions.
commit 2899ec299228217c876ba3aa4024e523a41c8504
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 22 22:45:32 2013 +0200
Fixed TransactionManagementError in tests.
Previous commit introduced an additional check to prevent running
queries in transactions that will be rolled back, which triggered a few
failures in the tests. In practice using transaction.atomic instead of
the low-level savepoint APIs was enough to fix the problems.
commit 4a639b059ea80aeb78f7f160a7d4b9f609b9c238
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Sep 24 22:24:17 2013 +0200
Allowed nesting constraint_checks_disabled inside atomic.
Since MySQL handles transactions loosely, this isn't a problem.
commit 2a4ab1cb6e83391ff7e25d08479e230ca564bfef
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sat Sep 21 18:43:12 2013 +0200
Prevented running queries in transactions that will be rolled back.
This avoids a counter-intuitive behavior in an edge case on databases
with non-atomic transaction semantics.
It prevents using savepoint_rollback() inside an atomic block without
calling set_rollback(False) first, which is backwards-incompatible in
tests.
Refs #21134.
commit 8e3db393853c7ac64a445b66e57f3620a3fde7b0
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Sep 22 22:14:17 2013 +0200
Replaced manual savepoints by atomic blocks.
This ensures the rollback flag is handled consistently in internal APIs.
This allows using flush on a subset of the tables without having to
manually cascade to all tables with foreign keys to the tables being
truncated, when they're known to be empty.
On databases where truncate is implemented with DELETE FROM, this
doesn't make a difference. The cascade is allowed, not mandatory.
Queries can contain binary data undecodable with utf-8. In this
case, using the 'replace' errors mode when decoding seems like
an acceptable representation of the query.
Thanks Marcel Ryser for the report.
Previously, depending on the database backend or the cursor type,
you'd need to double the percent signs in the query before passing
it to cursor.execute. Now cursor.execute consistently need percent
doubling whenever params argument is not None (placeholder substitution
will happen).
Thanks Thomas Güttler for the report and Walter Doekes for his work
on the patch.
* Grouped related methods together -- with banner comments :/
* Described which methods are intended to be implemented in backends.
* Added docstrings.
* Used the same order in all wrappers.
Squashed commit of the following:
commit 2181d833ed1a2e422494738dcef311164c4e097e
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 27 14:28:39 2013 +0100
Fixed#15901 -- Wrapped all PEP-249 exceptions.
commit 5476a5d93c19aa2f928c497d39ce6e33f52694e2
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 17:26:52 2013 +0100
Added PEP 3134 exception chaining.
Thanks Jacob Kaplan-Moss for the suggestion.
commit 9365fad0a650328002fb424457d675a273c95802
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 17:13:49 2013 +0100
Improved API for wrapping database errors.
Thanks Alex Gaynor for the proposal.
commit 1b463b765f2826f73a8d9266795cd5da4f8d5e9e
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 15:00:39 2013 +0100
Removed redundant exception wrapping.
This is now taken care of by the cursor wrapper.
commit 524bc7345a724bf526bdd2dd1bcf5ede67d6bb5c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 14:55:10 2013 +0100
Wrapped database exceptions in the base backend.
This covers the most common PEP-249 APIs:
- Connection APIs: close(), commit(), rollback(), cursor()
- Cursor APIs: callproc(), close(), execute(), executemany(),
fetchone(), fetchmany(), fetchall(), nextset().
Fixed#19920.
commit a66746bb5f0839f35543222787fce3b6a0d0a3ea
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Tue Feb 26 14:53:34 2013 +0100
Added a wrap_database_exception context manager and decorator.
It re-throws backend-specific exceptions using Django's common wrappers.
Change patch from https://code.djangoproject.com/ticket/5568
to work on modern Django.
Add special case for MySQL which has different syntax for DROP INDEX.
Add unit tests for the new functionality.
Django used to check the version of MySQL before handling the first
request, which required:
- opening a connection
- closing it, to avoid holding it idle until the first request.
This code isn't necessary any longer since Django dropped support for
some versions of MySQL, and other database backends don't implement a
similar dance. For consistency and maintenability, remove it.
Reverts 4423757c0c.
Closes#18135.
Thanks Carl Meyer for the review.
Squashed commit of the following:
commit 4f290bdb60
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 21:21:30 2013 +0100
Used '0:00' instead of 'UTC' which doesn't always exist in Oracle.
Thanks Ian Kelly for the suggestion.
commit 01b6366f3c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 13:38:43 2013 +0100
Made tzname a parameter of datetime_extract/trunc_sql.
This is required to work around a bug in Oracle.
commit 924a144ef8
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Wed Feb 13 14:47:44 2013 +0100
Added support for parameters in SELECT clauses.
commit b4351d2890
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 22:30:22 2013 +0100
Documented backwards incompatibilities in the two previous commits.
commit 91ef84713c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:42:31 2013 +0100
Used QuerySet.datetimes for the admin's date_hierarchy.
commit 0d0de288a5
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 09:29:38 2013 +0100
Used QuerySet.datetimes in date-based generic views.
commit 9c0859ff7c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:25 2013 +0100
Implemented QuerySet.datetimes on Oracle.
commit 68ab511a4f
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:14 2013 +0100
Implemented QuerySet.datetimes on MySQL.
commit 22d52681d3
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:42:29 2013 +0100
Implemented QuerySet.datetimes on SQLite.
commit f6800fd04c
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:43:03 2013 +0100
Implemented QuerySet.datetimes on PostgreSQL.
commit 0c829c23f4
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:41:08 2013 +0100
Added datetime-handling infrastructure in the ORM layers.
commit 104d82a777
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Mon Feb 11 10:05:55 2013 +0100
Updated null_queries tests to avoid clashing with the __second lookup.
commit c01bbb3235
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 23:07:41 2013 +0100
Updated tests of .dates().
Replaced .dates() by .datetimes() for DateTimeFields.
Replaced dates with datetimes in the expected output for DateFields.
commit 50fb7a5246
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 21:40:09 2013 +0100
Updated and added tests for QuerySet.datetimes.
commit a8451a5004
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 22:34:46 2013 +0100
Documented the new time lookups and updated the date lookups.
commit 29413eab2b
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date: Sun Feb 10 16:15:49 2013 +0100
Documented QuerySet.datetimes and updated QuerySet.dates.
Connection creation was done in db backend ._cursor() call. This
included taking a new connection if needed, initializing the session
state for the new connection and finally creating the connection.
To allow easier modifying of these steps in subclasses (for example to
support connection pools) the _cursor() now calls get_new_connection()
and init_connection_state() if there isn't an existing connection. This
was done for all non-gis core backends. In addition the parameters used
for taking a connection are now created by get_connection_params().
We should also do the same for gis backends and encourage 3rd party
backends to use the same pattern. The pattern is not enforced in code,
and as the backends are private API this will not be required by
documentation either.
Previously, the flush was done before the test case execution and now
it is performed after it.
Other changes to the testing infrastructure include:
* TransactionTestCase now doesn't reset autoincrement sequences either
(previous behavior can achieved by using `reset_sequences`.)
With this, no implicit such reset is performed by any of the provided
TestCase classes.
* New ordering of test cases: All unittest tes cases are run first and
doctests are run at the end.
THse changes could be backward-incompatible with test cases that relied
on some kind of state being preserved between tests. Please read the
relevant sections of the release notes and testing documentation for
further details.
Thanks Andreas Pelme for the initial patch. Karen Tracey and Anssi
Kääriäinen for the feedback and Anssi for reviewing.
This also fixes#12408.
This does remove the requirement to call features.confirm() method
before checking the properties.
Thanks cdestiger and Ramiro Morales for their work on the patch.
On MySQL when checking the server version, a new connection could be
created but never closed. This could result in open connections on
server startup.
Fixed#15933, #18082 -- the get_indexes() method introspection was
done inconsitently depending on the backend. For example SQLite
included all the columns in the table in the returned dictionary,
while MySQL introspected also multicolumn indexes.
All backends return now consistenly only single-column indexes.
Thanks to andi for the MySQL report, and ikelly for comments on
Oracle's get_indexes() changes.
Fixed#18218 -- previously Django's introspection table_names() and
get_table_list() methods did not sort the output consistently. This
resulted in random order of inspected models.
This commit also removed all external usages of get_table_list().
table_names() should be used instead.
Thanks to claudep for patch and report.
corner case (boolean fields under MySQL spatial backend) had a wider scope with
potentially unintended consequences affecting the main MySQL backend and the
required changes wouldn't be appropiate at this point of the 1.4 development
cycle.
Refs #15169.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@17603 bcc190cf-cafb-0310-a4f2-bffc1f526a37
MySQL provides the savepoint functionality starting with version 5.0.3
when using the MyISAM storage engine.
Thanks lamby for the report and patch.
Fixes#15507.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@17341 bcc190cf-cafb-0310-a4f2-bffc1f526a37
A number of people worked on this patch over the years -- Hawkeye, Colin Grady,
KBS, sakyamuni, anih, jdemoor, and Issak Kelly. Thanks to them all, and
apologies if I missed anyone.
Special thanks to Dan Fairs for picking it up again at the end and seeing this
through to commit.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@16058 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This monster of a patch is the result of Alex Gaynor's 2009 Google Summer of Code project.
Congratulations to Alex for a job well done.
Big thanks also go to:
* Justin Bronn for keeping GIS in line with the changes,
* Karen Tracey and Jani Tiainen for their help testing Oracle support
* Brett Hoerner, Jon Loyens, and Craig Kimmerer for their feedback.
* Malcolm Treddinick for his guidance during the GSoC submission process.
* Simon Willison for driving the original design process
* Cal Henderson for complaining about ponies he wanted.
... and everyone else too numerous to mention that helped to bring this feature into fruition.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@11952 bcc190cf-cafb-0310-a4f2-bffc1f526a37
We need to know the number of rows that are matched by an UPDATE query,
not just the number of rows that are changed. In the relatively unlikely
event that somebody was using Django's cursor proxy and relying on the
previous behaviour, well, that isn't the case any longer. We need to
this version.
Thanks to Daniel Tang for pointing out the solution here.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10532 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This is backwards-compatible but will likely break third-party database backends. Specific API changes are:
* BaseDatabaseWrapper.__init__() now takes a settings_dict instead of a settings module. It's called settings_dict to disambiguate, and for easy grepability. This should be a dictionary containing DATABASE_NAME, etc.
* BaseDatabaseWrapper has a settings_dict attribute instead of an options attribute. BaseDatabaseWrapper.options is now BaseDatabaseWrapper['DATABASE_OPTIONS']
* BaseDatabaseWrapper._cursor() no longer takes a settings argument.
* BaseDatabaseClient.__init__() now takes a connection argument (a DatabaseWrapper instance) instead of no arguments.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@10026 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Many thanks to:
* Nicolas Lara, who worked on this feature during the 2008 Google Summer of Code.
* Alex Gaynor for his help debugging and fixing a number of issues.
* Justin Bronn for his help integrating with contrib.gis.
* Karen Tracey for her help with cross-platform testing.
* Ian Kelly for his help testing and fixing Oracle support.
* Malcolm Tredinnick for his invaluable review notes.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9742 bcc190cf-cafb-0310-a4f2-bffc1f526a37
that happens with MySQL when a "GROUP BY" clause is included. This is a
backend-specific operation, so any other databases requiring similar
encouragement can have a function added to their own backend code.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@9637 bcc190cf-cafb-0310-a4f2-bffc1f526a37
convert it to an IntegrityError, which seems like the more natural case (and is
consistent with other backends). This makes exception handling in Django much
easier.
The solution is extensible for any other error codes we may wish to add going
forwards.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@8692 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This effectively reverses the change in [7798]. It was proving too difficult to
successfully manage all the side effects here and provide a satisfactory
solution for everybody. Many thanks to arne, Martin von Löwis and, particular,
Karen Tracey, for doing a lot of research and proto-patches here to establish what was possible and practical.
This is backwards incompatible if you were relying on the behaviour after
[7798]. The docs have been updated to indicate the solution.
Refs #2170, #7789, #8102.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@8319 bcc190cf-cafb-0310-a4f2-bffc1f526a37
loading/saving interactions with the databases into django.db.backend. This
helps external db backend writers and removes a bunch of database-specific
if-tests in django.db.models.fields.
Great work from Leo Soto.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@8131 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This is a backwards incompatible change if you were relying on 'exact' being
case-insensitive. For that, you should be using 'iexact'.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@7798 bcc190cf-cafb-0310-a4f2-bffc1f526a37
This patch simplifies a bunch of code for all backends and removes some
duplicate index creation for MySQL, in particular (versions 4.x and later).
Patch from Nis Jørgensen.
Fixed#5671, #5680, #7170, #7186.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@7790 bcc190cf-cafb-0310-a4f2-bffc1f526a37
database level. Also worked around the fact that MySQL (and maybe other
backends we don't know about) cannot select from the table they're updating.
Fixed#7095.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@7496 bcc190cf-cafb-0310-a4f2-bffc1f526a37
database backends differently now. Which meant the MySQLdb version check was
being swallowed. Changed the exception type to ensure this is percolated
correctly. Patch from Ramiro Morales.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@7358 bcc190cf-cafb-0310-a4f2-bffc1f526a37
references, for all tables, until after the table has been created. This means
that when using the InnoDB storage engine, true foreign key constraints are
created (inline "REFERENCES" are ignored by InnoDB, unfortunately).
Fully backwards compatible.
git-svn-id: http://code.djangoproject.com/svn/django/trunk@6650 bcc190cf-cafb-0310-a4f2-bffc1f526a37
Claude Paroz
Tim Graham
Thomas Chaumeny
Shai Berger
Marc Tamlyn
Jon Dufresne
zsoldosp
Mihail Milushev
Aymeric Augustin
Anubhav Joshi
Loic Bistuer
Andrew Godwin
Simon Charette
Russell Keith-Magee
Vajrasky Kok
Michael Manfre
Loic Bistuer
Christopher Medrela
coagulant
Florian Hahn
Alex Gaynor
Anssi Kääriäinen
Tomasz Rybak
Alex Gaynor
Ramiro Morales
Michael Newman
Jannis Leidel
Karen Tracey
Jacob Kaplan-Moss
Malcolm Tredinnick
Justin Bronn
Adrian Holovaty
Brian Rosner