282 lines
8.2 KiB
ReStructuredText
282 lines
8.2 KiB
ReStructuredText
cache: working with cross-testrun state
|
|
=======================================
|
|
|
|
.. versionadded:: 2.8
|
|
|
|
.. warning::
|
|
|
|
The functionality of this core plugin was previosuly distributed
|
|
as a third party plugin named ``pytest-cache``. The core plugin
|
|
is compatible regarding command line options and API usage except that you
|
|
can only store/receive data between test runs that is json-serializable.
|
|
|
|
If for any reason you want to disable this plugin, you can do so by
|
|
adding ``-p no:cacheprovider`` to the command-line, or more permanently
|
|
to your ``pytest.ini`` file:
|
|
|
|
.. code-block:: ini
|
|
|
|
[pytest]
|
|
addopts = -p no:cacheprovider
|
|
|
|
|
|
Usage
|
|
---------
|
|
|
|
The plugin provides two command line options to rerun failures from the
|
|
last ``py.test`` invocation:
|
|
|
|
* ``--lf`` (last failures) - to only re-run the failures.
|
|
* ``--ff`` (failures first) - to run the failures first and then the rest of
|
|
the tests.
|
|
|
|
For cleanup (usually not needed), a ``--cache-clear`` option allows to remove
|
|
all cross-session cache contents ahead of a test run.
|
|
|
|
Other plugins may access the `config.cache`_ object to set/get
|
|
**json encodable** values between ``py.test`` invocations.
|
|
|
|
|
|
Rerunning only failures or failures first
|
|
-----------------------------------------------
|
|
|
|
First, let's create 50 test invocation of which only 2 fail::
|
|
|
|
# content of test_50.py
|
|
import pytest
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
pytest.fail("bad luck")
|
|
|
|
If you run this for the first time you will see two failures::
|
|
|
|
$ py.test -q
|
|
.................F.......F........................
|
|
================================= FAILURES =================================
|
|
_______________________________ test_num[17] _______________________________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______________________________ test_num[25] _______________________________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
2 failed, 48 passed in 0.04 seconds
|
|
|
|
If you then run it with ``--lf``::
|
|
|
|
$ py.test --lf
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.6, pytest-2.7.3.dev428+ng79d22bf.d20150916, py-1.4.30, pluggy-0.3.0
|
|
run-last-failure: rerun last 2 failures
|
|
rootdir: /tmp/doc-exec-94, inifile:
|
|
collected 50 items
|
|
|
|
test_50.py FF
|
|
|
|
================================= FAILURES =================================
|
|
_______________________________ test_num[17] _______________________________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______________________________ test_num[25] _______________________________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
================= 2 failed, 48 deselected in 0.01 seconds ==================
|
|
|
|
You have run only the two failing test from the last run, while 48 tests have
|
|
not been run ("deselected").
|
|
|
|
Now, if you run with the ``--ff`` option, all tests will be run but the first
|
|
previous failures will be executed first (as can be seen from the series
|
|
of ``FF`` and dots)::
|
|
|
|
$ py.test --ff
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.6, pytest-2.7.3.dev428+ng79d22bf.d20150916, py-1.4.30, pluggy-0.3.0
|
|
run-last-failure: rerun last 2 failures first
|
|
rootdir: /tmp/doc-exec-94, inifile:
|
|
collected 50 items
|
|
|
|
test_50.py FF................................................
|
|
|
|
================================= FAILURES =================================
|
|
_______________________________ test_num[17] _______________________________
|
|
|
|
i = 17
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
_______________________________ test_num[25] _______________________________
|
|
|
|
i = 25
|
|
|
|
@pytest.mark.parametrize("i", range(50))
|
|
def test_num(i):
|
|
if i in (17, 25):
|
|
> pytest.fail("bad luck")
|
|
E Failed: bad luck
|
|
|
|
test_50.py:6: Failed
|
|
=================== 2 failed, 48 passed in 0.03 seconds ====================
|
|
|
|
.. _`config.cache`:
|
|
|
|
The new config.cache object
|
|
--------------------------------
|
|
|
|
.. regendoc:wipe
|
|
|
|
Plugins or conftest.py support code can get a cached value using the
|
|
pytest ``config`` object. Here is a basic example plugin which
|
|
implements a :ref:`fixture` which re-uses previously created state
|
|
across py.test invocations::
|
|
|
|
# content of test_caching.py
|
|
import pytest
|
|
import time
|
|
|
|
@pytest.fixture
|
|
def mydata(request):
|
|
val = request.config.cache.get("example/value", None)
|
|
if val is None:
|
|
time.sleep(9*0.6) # expensive computation :)
|
|
val = 42
|
|
request.config.cache.set("example/value", val)
|
|
return val
|
|
|
|
def test_function(mydata):
|
|
assert mydata == 23
|
|
|
|
If you run this command once, it will take a while because
|
|
of the sleep::
|
|
|
|
$ py.test -q
|
|
F
|
|
================================= FAILURES =================================
|
|
______________________________ test_function _______________________________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
1 failed in 5.41 seconds
|
|
|
|
If you run it a second time the value will be retrieved from
|
|
the cache and this will be quick::
|
|
|
|
$ py.test -q
|
|
F
|
|
================================= FAILURES =================================
|
|
______________________________ test_function _______________________________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
1 failed in 0.01 seconds
|
|
|
|
See the `cache-api`_ for more details.
|
|
|
|
|
|
Inspecting Cache content
|
|
-------------------------------
|
|
|
|
You can always peek at the content of the cache using the
|
|
``--cache-clear`` command line option::
|
|
|
|
$ py.test --cache-clear
|
|
=========================== test session starts ============================
|
|
platform linux2 -- Python 2.7.6, pytest-2.7.3.dev428+ng79d22bf.d20150916, py-1.4.30, pluggy-0.3.0
|
|
rootdir: /tmp/doc-exec-94, inifile:
|
|
collected 1 items
|
|
|
|
test_caching.py F
|
|
|
|
================================= FAILURES =================================
|
|
______________________________ test_function _______________________________
|
|
|
|
mydata = 42
|
|
|
|
def test_function(mydata):
|
|
> assert mydata == 23
|
|
E assert 42 == 23
|
|
|
|
test_caching.py:14: AssertionError
|
|
========================= 1 failed in 5.41 seconds =========================
|
|
|
|
Clearing Cache content
|
|
-------------------------------
|
|
|
|
You can instruct pytest to clear all cache files and values
|
|
by adding the ``--cache-clear`` option like this::
|
|
|
|
py.test --cache-clear
|
|
|
|
This is recommended for invocations from Continous Integration
|
|
servers where isolation and correctness is more important
|
|
than speed.
|
|
|
|
|
|
.. _`cache-api`:
|
|
|
|
config.cache API
|
|
========================================
|
|
|
|
The `config.cache`` object allows other plugins,
|
|
including ``conftest.py`` files,
|
|
to safely and flexibly store and retrieve values across
|
|
test runs because the ``config`` object is available
|
|
in many places.
|
|
|
|
Under the hood, the cache plugin uses the simple
|
|
dumps/loads API of the json stdlib module
|
|
|
|
.. currentmodule:: _pytest.cacheprovider
|
|
|
|
.. automethod:: Cache.get
|
|
.. automethod:: Cache.set
|
|
.. automethod:: Cache.makedir
|