279 lines
7.1 KiB
ReStructuredText
279 lines
7.1 KiB
ReStructuredText
Cache: working with cross-testrun state
|
|
=======================================
|
|
|
|
.. versionadded:: 2.8
|
|
|
|
.. warning::
|
|
|
|
The functionality of this core plugin was previously distributed
|
|
as a third party plugin named ``pytest-cache``. The core plugin
|
|
is compatible regarding command line options and API usage except that you
|
|
can only store/receive data between test runs that is json-serializable.
|
|
|
|
|
|
Usage
|
|
---------
|
|
|
|
The plugin provides two command line options to rerun failures from the
|
|
last ``pytest`` invocation:
|
|
|
|
* ``--lf``, ``--last-failed`` - to only re-run the failures.
|
|
* ``--ff``, ``--failed-first`` - to run the failures first and then the rest of
|
|
the tests.
|
|
|
|
For cleanup (usually not needed), a ``--cache-clear`` option allows to remove
|
|
all cross-session cache contents ahead of a test run.
|
|
|
|
Other plugins may access the `config.cache`_ object to set/get
|
|
**json encodable** values between ``pytest`` invocations.
|
|
|
|
.. note::
|
|
|
|
This plugin is enabled by default, but can be disabled if needed: see
|
|
:ref:`cmdunregister` (the internal name for this plugin is
|
|
``cacheprovider``).
|
|
|
|
|
|
Rerunning only failures or failures first
|
|
-----------------------------------------------
|
|
|
|
First, let's create 50 test invocation of which only 2 fail::
|
|
|
|
# content of test_50.py
|
|
import pytest
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
pytest.fail("bad luck")
|
|
|
|
If you run this for the first time you will see two failures::
|
|
|
|
$ pytest -q
|
|
.................F.......F........................
|
|
======= FAILURES ========
|
|
_______ test_num[17] ________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______ test_num[25] ________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
2 failed, 48 passed in 0.12 seconds
|
|
|
|
If you then run it with ``--lf``::
|
|
|
|
$ pytest --lf
|
|
======= test session starts ========
|
|
platform linux -- Python 3.5.1, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
|
|
run-last-failure: rerun last 2 failures
|
|
rootdir: $REGENDOC_TMPDIR, inifile:
|
|
collected 50 items
|
|
|
|
test_50.py FF
|
|
|
|
======= FAILURES ========
|
|
_______ test_num[17] ________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______ test_num[25] ________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
======= 2 failed, 48 deselected in 0.12 seconds ========
|
|
|
|
You have run only the two failing test from the last run, while 48 tests have
|
|
not been run ("deselected").
|
|
|
|
Now, if you run with the ``--ff`` option, all tests will be run but the first
|
|
previous failures will be executed first (as can be seen from the series
|
|
of ``FF`` and dots)::
|
|
|
|
$ pytest --ff
|
|
======= test session starts ========
|
|
platform linux -- Python 3.5.1, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
|
|
run-last-failure: rerun last 2 failures first
|
|
rootdir: $REGENDOC_TMPDIR, inifile:
|
|
collected 50 items
|
|
|
|
test_50.py FF................................................
|
|
|
|
======= FAILURES ========
|
|
_______ test_num[17] ________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______ test_num[25] ________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
======= 2 failed, 48 passed in 0.12 seconds ========
|
|
|
|
.. _`config.cache`:
|
|
|
|
The new config.cache object
|
|
--------------------------------
|
|
|
|
.. regendoc:wipe
|
|
|
|
Plugins or conftest.py support code can get a cached value using the
|
|
pytest ``config`` object. Here is a basic example plugin which
|
|
implements a :ref:`fixture` which re-uses previously created state
|
|
across pytest invocations::
|
|
|
|
# content of test_caching.py
|
|
import pytest
|
|
import time
|
|
|
|
@pytest.fixture
|
|
def mydata(request):
|
|
val = request.config.cache.get("example/value", None)
|
|
if val is None:
|
|
time.sleep(9*0.6) # expensive computation :)
|
|
val = 42
|
|
request.config.cache.set("example/value", val)
|
|
return val
|
|
|
|
def test_function(mydata):
|
|
assert mydata == 23
|
|
|
|
If you run this command once, it will take a while because
|
|
of the sleep::
|
|
|
|
$ pytest -q
|
|
F
|
|
======= FAILURES ========
|
|
_______ test_function ________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
1 failed in 0.12 seconds
|
|
|
|
If you run it a second time the value will be retrieved from
|
|
the cache and this will be quick::
|
|
|
|
$ pytest -q
|
|
F
|
|
======= FAILURES ========
|
|
_______ test_function ________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
1 failed in 0.12 seconds
|
|
|
|
See the `cache-api`_ for more details.
|
|
|
|
|
|
Inspecting Cache content
|
|
-------------------------------
|
|
|
|
You can always peek at the content of the cache using the
|
|
``--cache-show`` command line option::
|
|
|
|
$ py.test --cache-show
|
|
======= test session starts ========
|
|
platform linux -- Python 3.5.1, pytest-2.9.2, py-1.4.31, pluggy-0.3.1
|
|
rootdir: $REGENDOC_TMPDIR, inifile:
|
|
collected 1 items
|
|
|
|
test_caching.py F
|
|
|
|
======= FAILURES ========
|
|
_______ test_function ________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
======= 1 failed in 0.12 seconds ========
|
|
|
|
Clearing Cache content
|
|
-------------------------------
|
|
|
|
You can instruct pytest to clear all cache files and values
|
|
by adding the ``--cache-clear`` option like this::
|
|
|
|
pytest --cache-clear
|
|
|
|
This is recommended for invocations from Continous Integration
|
|
servers where isolation and correctness is more important
|
|
than speed.
|
|
|
|
|
|
.. _`cache-api`:
|
|
|
|
config.cache API
|
|
------------------
|
|
|
|
The ``config.cache`` object allows other plugins,
|
|
including ``conftest.py`` files,
|
|
to safely and flexibly store and retrieve values across
|
|
test runs because the ``config`` object is available
|
|
in many places.
|
|
|
|
Under the hood, the cache plugin uses the simple
|
|
dumps/loads API of the json stdlib module
|
|
|
|
.. currentmodule:: _pytest.cacheprovider
|
|
|
|
.. automethod:: Cache.get
|
|
.. automethod:: Cache.set
|
|
.. automethod:: Cache.makedir
|