Several issues resolved here, following from a report that a base_field
of GenericIpAddressField was failing.
We were using get_prep_value instead of get_db_prep_value in ArrayField
which was bypassing any extra modifications to the value being made in
the base field's get_db_prep_value. Changing this broke datetime
support, so the postgres backend has gained the relevant operation
methods to send dates/times/datetimes directly to the db backend instead
of casting them to strings. Similarly, a new database feature has been
added allowing the uuid to be passed directly to the backend, as we do
with timedeltas.
On the other side, psycopg2 expects an Inet() instance for IP address
fields, so we add a value_to_db_ipaddress method to wrap the strings on
postgres. We also have to manually add a database adapter to psycopg2,
as we do not wish to use the built in adapter which would turn
everything into Inet() instances.
Thanks to smclenithan for the report.
It was mainly for MySQL on Python 3, but now the current
recommended MySQL driver for Python 3 (mysqlclient) does support
binary fields, it is unneeded. Refs #20377.
Refactored compiler SELECT, GROUP BY and ORDER BY generation.
While there, also refactored select_related() implementation
(get_cached_row() and get_klass_info() are now gone!).
Made get_db_converters() method work on expressions instead of
internal_type. This allows the backend converters to target
specific expressions if need be.
Added query.context, this can be used to set per-query state.
Also changed the signature of database converters. They now accept
context as an argument.
Added functions and tests
Added docs and more tests
Added TextField converter to mysql backend
Aliased Value as V in example docs and tests
Removed unicode_compatible in example
Fixed console emulation in examples
Use INTERVAL DAY(9) TO SECOND(6) for Durationfield on Oracle rather than
storing as a NUMBER(19) of microseconds.
There are issues with cx_Oracle which require some extra data
manipulation in the database backend when constructing queries, but it
handles the conversion back to timedelta objects cleanly.
Thanks to Shai for the review.
A field for storing periods of time - modeled in Python by timedelta. It
is stored in the native interval data type on PostgreSQL and as a bigint
of microseconds on other backends.
Also includes significant changes to the internals of time related maths
in expressions, including the removal of DateModifierNode.
Thanks to Tim and Josh in particular for reviews.
The .dates() queries were implemented by using custom Query, QuerySet,
and Compiler classes. Instead implement them by using expressions and
database converters APIs.
Checked database destruction for ORA-29857, which happens when objects of
some types are left over in an old test database (notably, such objects
are created by the GIS backend). When this happens, we can resolve the
issue by dropping the test-user (if we are allowed to).
An alternative approach, to just switch the order of creation (so that,
if the test user exists and can be dropped, it is always dropped before
the tablespace) was considered; but since the user creation depends on
the tablespace, this would necessitate separating the dropping of the user
from its creation -- a change I am reluctant to make.
Thanks Josh Smeaton and Tim Graham for reviews.
because the roles (specifically RESOURCE) are deprecated.
Also added optional support for creating views in tests, and made an
introspection test fail (rather than skip) if a view cannot be created
due to lacking privileges.
Refs #18782
Thanks Tim Graham for review, and Josh Smeaton
Complete rework of translating data values from database
Deprecation of SubfieldBase, removal of resolve_columns and
convert_values in favour of a more general converter based approach and
public API Field.from_db_value(). Now works seamlessly with aggregation,
.values() and raw queries.
Thanks to akaariai in particular for extensive advice and inspiration,
also to shaib, manfre and timograham for their reviews.
In the MySQL backend, updated the can_introspect_small_integer feature
flag to True. In data_types_reverse, map FIELD_TYPE.SHORT to a
SmallIntegerField. Added test to verify introspecting SmallIntegerFields
and fixed existing tests influenced by this change.
Refs #23073 Workaround.
Refs #22738 Repeats the mysql "offense". When the issue is solved, the
Oracle special case should be made to play with the solution (that is,
Oracle should be fixed the same way that mysql and the 3rd-party backneds
are).
This is achieved by inserting a fake entry in connection.queries when
not releasing a savepoint (since Oracle doesn't support that operation.)
Also removed the can_release_savepoints feature that was recently added,
but is superseded by this solution.
Also fixes a slight issue in sqlite3.schema._remake_table where
default values where quoted with "column name" quoting rules.
Reference for quoting: http://www.sqlite.org/lang_expr.html
Thanks Shai Berger for the review. Refs #22424.
In django.db.backends.BaseDatabaseWrapper, pulled the creation of
cursors in the non-debug case into a separate method, in order to
make behavior more consistent when overriding the cursor creation
in derived classes.
Avoided introducing a new regex-based SQL splitter in the migrations
framework, before we're bound by backwards compatibility.
Adapted this change to the legacy "initial SQL data" feature, even
though it's already deprecated, in order to facilitate the transition
to migrations.
sqlparse becomes mandatory for RunSQL on some databases (all but
PostgreSQL). There's no API to provide a single statement and tell
Django not to attempt splitting. Since we have a more robust splitting
implementation, that seems like a good tradeoff. It's easier to add a
new keyword argument later if necessary than to remove one.
Many people contributed to both tickets, thank you all, and especially
Claude for the review.
Refs #22401.
Since all state is maintained on the connection at this time and none in
the atomic, it doesn't matter, but it could introduce some subtle bugs
if the implementation changed in the future.
Changed the migration autodetector to remove models last so that FK
and M2M fields will not be left as dangling references. Added a check
in the migration state renderer to error out in the presence of
dangling references instead of leaving them as strings. Fixed a bug
in the sqlite backend to handle the deletion of M2M fields with
"through" models properly (i.e., do nothing successfully).
Thanks to melinath for report, loic for tests and andrewgodwin and
charettes for assistance with architecture.
Changed the migration autodetector to remove models last so that FK
and M2M fields will not be left as dangling references. Added a check
in the migration state renderer to error out in the presence of
dangling references instead of leaving them as strings. Fixed a bug
in the sqlite backend to handle the deletion of M2M fields with
"through" models properly (i.e., do nothing successfully).
Thanks to melinath for report, loic for tests and andrewgodwin and
charettes for assistance with architecture.
This seems worthwhile in its own right, but also works around an Oracle
bug (in versions 10 -- 11.1) where the use of Unicode would reset the
date/time formats, causing ORA-01843 errors.
Thanks Trac users CarstenF for the report, jtiai for the initial patch,
and everyone who contributed to the discussion on the ticket.
Change database test settings from "TEST_"-prefixed entries in the
database settings dictionary to setting in a dictionary that is itself
an entry "TEST" in the database settings.
Refs #21775
Thanks Josh Smeaton for review.
This commit touchs various parts of the code base and test framework. Any
found usage of opening a cursor for the sake of initializing a connection
has been replaced with 'ensure_connection()'.
Broke InspectDBTestCase.test_field_types in two:
- a test_number_field_types, which now passes on Oracle too
- a test_field_types, for all non-numeric fields, which is still expected to fail
Also made some pep8 fixes in the tests file. Refs #19884
Thanks Tim Graham for review.
This is the result of Christopher Medrela's 2013 Summer of Code project.
Thanks also to Preston Holmes, Tim Graham, Anssi Kääriäinen, Florian
Apolloner, and Alex Gaynor for review notes along the way.
Also: Fixes#8579, fixes#3055, fixes#19844.
Allowed users to specify which lookups or transforms ("nested lookus")
are available for fields. The implementation is now class based.
Squashed commit of the following:
commit fa7a7195f1
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 18 10:53:24 2014 +0200
Added lookup registration API docs
commit eb1c8ce164
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Tue Jan 14 18:59:36 2014 +0200
Release notes and other minor docs changes
commit 11501c29c9
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Jan 12 20:53:03 2014 +0200
Forgot to add custom_lookups tests in prev commit
commit 83173b960e
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Jan 12 19:59:12 2014 +0200
Renamed Extract -> Transform
commit 3b18d9f3a1
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Jan 12 19:51:53 2014 +0200
Removed suggestion of temporary lookup registration from docs
commit 21d0c7631c
Merge: 2509006f2dc442
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Jan 12 09:38:23 2014 -0800
Merge pull request #2 from mjtamlyn/lookups_3
Reworked custom lookups docs.
commit f2dc4429a1
Author: Marc Tamlyn <marc.tamlyn@gmail.com>
Date: Sun Jan 12 13:15:05 2014 +0000
Reworked custom lookups docs.
Mostly just formatting and rewording, but also replaced the example
using ``YearExtract`` to use an example which is unlikely to ever be
possible directly in the ORM.
commit 2509006506
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Jan 12 13:19:13 2014 +0200
Removed unused import
commit 4fba5dfaa0
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 22:34:41 2014 +0200
Added docs to index
commit 6d53963f37
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 22:10:24 2014 +0200
Dead code removal
commit f9cc039007
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 19:00:43 2014 +0200
A new try for docs
commit 33aa18a6e3
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 14:57:12 2014 +0200
Renamed get_cols to get_group_by_cols
commit c7d5f8661b
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 14:45:53 2014 +0200
Altered query string customization for backends vendors
The new way is trying to call first method 'as_' + connection.vendor.
If that doesn't exist, then call as_sql().
Also altered how lookup registration is done. There is now
RegisterLookupMixin class that is used by Field, Extract and
sql.Aggregate. This allows one to register lookups for extracts and
aggregates in the same way lookup registration is done for fields.
commit 90e7004ec1
Merge: 66649fff7c2c0a
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 13:21:01 2014 +0200
Merge branch 'master' into lookups_3
commit 66649ff891
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Jan 11 13:16:01 2014 +0200
Some rewording in docs
commit 31b8faa627
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 29 15:52:29 2013 +0200
Cleanup based on review comments
commit 1016159f34
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 28 18:37:04 2013 +0200
Proof-of-concept fix for #16731
Implemented only for SQLite and PostgreSQL, and only for startswith
and istartswith lookups.
commit 193cd097ca
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 28 17:57:58 2013 +0200
Fixed#11722 -- iexact=F() produced invalid SQL
commit 08ed3c3b49
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 21 23:59:52 2013 +0200
Made Lookup and Extract available from django.db.models
commit b99c8d83c9
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 21 23:06:29 2013 +0200
Fixed review notes by Loic
commit 049eebc070
Merge: ed8fab7b80a835
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 21 22:53:10 2013 +0200
Merge branch 'master' into lookups_3
Conflicts:
django/db/models/fields/__init__.py
django/db/models/sql/compiler.py
django/db/models/sql/query.py
tests/null_queries/tests.py
commit ed8fab7fe8
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Dec 21 22:47:23 2013 +0200
Made Extracts aware of full lookup path
commit 27a57b7aed
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 1 21:10:11 2013 +0200
Removed debugger import
commit 074e0f5aca
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 1 21:02:16 2013 +0200
GIS lookup support added
commit 760e28e72b
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 1 20:04:31 2013 +0200
Removed usage of Constraint, used Lookup instead
commit eac4776684
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 1 02:22:30 2013 +0200
Minor cleanup of Lookup API
commit 2adf50428d
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sun Dec 1 02:14:19 2013 +0200
Added documentation, polished implementation
commit 32c04357a8
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Nov 30 23:10:15 2013 +0200
Avoid OrderedDict creation on lookup aggregate check
commit 7c8b3a32cc
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Sat Nov 30 23:04:34 2013 +0200
Implemented nested lookups
But there is no support of using lookups outside filtering yet.
commit 4d219d4cde
Author: Anssi Kääriäinen <akaariai@gmail.com>
Date: Wed Nov 27 22:07:30 2013 +0200
Initial implementation of custom lookups
Also ensured the transaction state is clean on Oracle while I was there.
This change cannot be backported to 1.6 because it's
backwards-incompatible for custom database backends.
When settings.DATABASES['default']['AUTOCOMMIT'] = False, the connection
wasn't in autocommit mode but Django pretended it was.
Thanks Anssi for analysing this issue.
Refs #17062.
Now that the refactorings are complete, it isn't particularly useful any
more, nor very well named. Let's keep the API as simple as possible.
Fixed#21689.
SQLite accepts the relevant standard SQL (although by default it doesn't
enforce the constraint), and the 'traditional' creation backend helper
generate it, so this allows us to:
- Maintain the status quo
- Improve readability of the SQL code generated for that backend.
Also, we will need this for when we fix Refs #14204.
Made it use 'AUTOINCREMENT' suffix for PK creation. This way it doeesn't
regress when compared with the 'traditional' DB backend creation
infrastructure.
Refs #10164.
Improved Andrew's hack to create temporary app caches to handle
migrations. Now the main app cache has a "master" flag set to True
(which is a non-default keyword argument, thus unlikely to be used by
mistake). Other app cache instances have "master" set to False.
The only sanctioned way to access the app cache is by importing
django.core.apps.app_cache.
If you were instanciating an app cache and relying on the Borg pattern,
you'll have to refactor your code.
Since the original ones in django.db.models.loading were kept only for
backwards compatibility, there's no need to recreate them. However, many
internals of Django still relied on them.
They were also imported in django.db.models. They never appear in the
documentation, except a quick mention of get_models and get_app in the
1.2 release notes to document an edge case in GIS. I don't think that
makes them a public API.
This commit doesn't change the overall amount of global state but
clarifies that it's tied to the app_cache object instead of hiding it
behind half a dozen functions.
Tim Graham
Aymeric Augustin
Josh Smeaton
Loic Bistuer
я котик пур-пур
Collin Anderson
Andriy Sokolovskiy
Adam Taylor
Markus Holtermann
Marc Tamlyn
Claude Paroz
Michał Modzelewski
Shai Berger
Anssi Kääriäinen
Daniel Pyrathon
Jon Dufresne
Thomas Chaumeny
Brad Walker
Mikhail Denisenko
Thomas Chaumeny
Sergey Fedoseev
zsoldosp
Florian Apolloner
Andrew Godwin
Alex Hill
Mihail Milushev
areski
Anubhav Joshi
Vlastimil Zíma
Aymeric Augustin
Greg Chapple
Alex Gaynor
Raphaël Barrois
Tim Martin
Andrew Gorcester
Simon Charette
Motiejus Jakštys
Chris Beaven
Daniel Hahler
Rodolfo Carvalho
Russell Keith-Magee
Baptiste Mispelon
Ben Davis
Vajrasky Kok
Michael Manfre
Ramiro Morales
Levi Gross
Loic Bistuer
Christopher Medrela
Joel Bohman