Go to file
Ran Benita 50114d4731 python: fix quadratic behavior in collection of items using xunit fixtures
Since commit 0f918b1a9d pytest uses auto-generated autouse
pytest fixtures for the xunit fixtures
{setup,teardown}_{module,class,method,function}. All of these fixtures
were given the same name.

Unfortunately, pytest fixture lookup for a name works by grabbing all of
the fixtures globally declared with a name and filtering to only those
which match the specific node. So each xunit-using item iterates over a
list (of fixturedefs) of a size of all previous same-xunit-using items,
i.e. quadratic.

Fixing this properly to use a better data structure is likely to take
some effort, but we can avoid the immediate problem by just using
a different name for each item's autouse fixture, so it only matches
itself.

A benchmark is added to demonstrate the issue. It is still way too slow
after the fix and possibly still quadratic, but for a different reason
which is another matter.

Running --collect-only, before (snipped):

         202533232 function calls (201902604 primitive calls) in 86.379 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   85.688   85.688 main.py:320(pytest_collection)
        1    0.000    0.000   85.688   85.688 main.py:567(perform_collect)
80557/556    0.021    0.000   85.050    0.153 {method 'extend' of 'list' objects}
85001/15001  0.166    0.000   85.045    0.006 main.py:785(genitems)
    10002    0.050    0.000   84.219    0.008 runner.py:455(collect_one_node)
    10002    0.049    0.000   83.763    0.008 runner.py:340(pytest_make_collect_report)
    10002    0.079    0.000   83.668    0.008 runner.py:298(from_call)
    10002    0.019    0.000   83.541    0.008 runner.py:341(<lambda>)
     5001    0.184    0.000   81.922    0.016 python.py:412(collect)
     5000    0.020    0.000   81.072    0.016 python.py:842(collect)
    30003    0.118    0.000   78.478    0.003 python.py:218(pytest_pycollect_makeitem)
    30000    0.190    0.000   77.957    0.003 python.py:450(_genfunctions)
    40001    0.081    0.000   76.664    0.002 nodes.py:183(from_parent)
    30000    0.087    0.000   76.629    0.003 python.py:1595(from_parent)
    40002    0.092    0.000   76.583    0.002 nodes.py:102(_create)
    30000    0.305    0.000   76.404    0.003 python.py:1533(__init__)
    15000    0.132    0.000   74.765    0.005 fixtures.py:1439(getfixtureinfo)
    15000    0.165    0.000   73.664    0.005 fixtures.py:1492(getfixtureclosure)
    15000    0.044    0.000   57.584    0.004 fixtures.py:1653(getfixturedefs)
    30000   18.840    0.001   57.540    0.002 fixtures.py:1668(_matchfactories)
 37507500   31.352    0.000   38.700    0.000 nodes.py:76(ischildnode)
    15000   10.464    0.001   15.806    0.001 fixtures.py:1479(_getautousenames)
112930587/112910019   7.333    0.000    7.339    0.000 {built-in method builtins.len}

After:

         51890333 function calls (51259706 primitive calls) in 27.306 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   26.783   26.783 main.py:320(pytest_collection)
        1    0.000    0.000   26.783   26.783 main.py:567(perform_collect)
80557/556    0.020    0.000   26.108    0.047 {method 'extend' of 'list' objects}
85001/15001  0.151    0.000   26.103    0.002 main.py:785(genitems)
    10002    0.047    0.000   25.324    0.003 runner.py:455(collect_one_node)
    10002    0.045    0.000   24.888    0.002 runner.py:340(pytest_make_collect_report)
    10002    0.069    0.000   24.805    0.002 runner.py:298(from_call)
    10002    0.017    0.000   24.690    0.002 runner.py:341(<lambda>)
     5001    0.168    0.000   23.150    0.005 python.py:412(collect)
     5000    0.019    0.000   22.223    0.004 python.py:858(collect)
    30003    0.101    0.000   19.818    0.001 python.py:218(pytest_pycollect_makeitem)
    30000    0.161    0.000   19.368    0.001 python.py:450(_genfunctions)
    30000    0.302    0.000   18.236    0.001 python.py:1611(from_parent)
    40001    0.084    0.000   18.051    0.000 nodes.py:183(from_parent)
    40002    0.116    0.000   17.967    0.000 nodes.py:102(_create)
    30000    0.308    0.000   17.770    0.001 python.py:1549(__init__)
    15000    0.117    0.000   16.111    0.001 fixtures.py:1439(getfixtureinfo)
    15000    0.134    0.000   15.135    0.001 fixtures.py:1492(getfixtureclosure)
    15000    9.320    0.001   14.738    0.001 fixtures.py:1479(_getautousenames)
2020-10-23 22:36:23 +03:00
.github ci: decrease job timeout from 6 hours to 30 minutes 2020-10-21 10:17:25 +03:00
bench python: fix quadratic behavior in collection of items using xunit fixtures 2020-10-23 22:36:23 +03:00
changelog cacheprovider: fix some files in packages getting lost from --lf 2020-10-19 19:02:43 +03:00
doc/en doc: Remove unused imports in examples (#7924) 2020-10-23 17:31:17 +03:00
extra Replace some usages of py.path.local 2020-08-06 18:46:17 +03:00
scripts release-on-comment: add "Closes <release issue number>" to release PR 2020-10-10 19:01:41 +03:00
src python: fix quadratic behavior in collection of items using xunit fixtures 2020-10-23 22:36:23 +03:00
testing Merge pull request #7910 from pytest-dev/dependabot/pip/testing/plugins_integration/pytest-trio-0.7.0 2020-10-23 16:52:54 +03:00
.coveragerc Don't use NotImplementedError in `@overload`s 2020-08-14 13:54:46 +03:00
.gitattributes Add codecov support to AppVeyor and remove coveralls 2018-09-05 19:50:06 -03:00
.gitblameignore Add .gitblameignore (#6848) 2020-03-04 05:32:30 +01:00
.gitignore Add pyproject.toml support (#7247) 2020-06-08 10:03:10 -03:00
.pre-commit-config.yaml Update mypy 0.782 -> 0.790 2020-10-17 19:25:45 +03:00
.readthedocs.yml Add readthedocs config file to use pip for installation 2020-08-01 14:21:36 -03:00
AUTHORS Add example for registering multiple custom mark (#7886) 2020-10-14 15:17:50 +03:00
CHANGELOG.rst Point to stable docs instead of latest 2020-07-07 07:45:27 -03:00
CITATION Fix linting issues 2018-07-14 16:35:33 +01:00
CODE_OF_CONDUCT.md remove myself from the list of coc handlers (#6857) 2020-03-03 19:08:59 -03:00
CONTRIBUTING.rst ask for commit after changelog and authors file edit (#7878) 2020-10-09 16:10:54 +02:00
LICENSE Update copyright year to 2020 2020-01-04 08:46:58 -03:00
OPENCOLLECTIVE.rst doc: fix typos in OPENCOLLECTIVE.rst [ci skip] 2019-07-10 20:15:22 +02:00
README.rst Drop support for EOL Python 3.5 2020-10-19 10:02:36 +03:00
RELEASING.rst RELEASING: clarify where to push the tag 2020-09-19 21:35:17 +03:00
TIDELIFT.rst Add asottile as well 2020-05-08 08:57:22 -07:00
codecov.yml Adjust codecov: only patch statuses 2020-06-02 08:17:43 -03:00
pyproject.toml Drop support for EOL Python 3.5 2020-10-19 10:02:36 +03:00
setup.cfg Update mypy 0.782 -> 0.790 2020-10-17 19:25:45 +03:00
setup.py Configure setuptools_scm using pyproject.toml 2020-07-31 15:46:02 -03:00
tox.ini Put smoke test deps in requirements.txt for Dependabot (#7806) 2020-10-10 18:51:35 +03:00

README.rst

.. image:: https://docs.pytest.org/en/stable/_static/pytest1.png
   :target: https://docs.pytest.org/en/stable/
   :align: center
   :alt: pytest


------

.. image:: https://img.shields.io/pypi/v/pytest.svg
    :target: https://pypi.org/project/pytest/

.. image:: https://img.shields.io/conda/vn/conda-forge/pytest.svg
    :target: https://anaconda.org/conda-forge/pytest

.. image:: https://img.shields.io/pypi/pyversions/pytest.svg
    :target: https://pypi.org/project/pytest/

.. image:: https://codecov.io/gh/pytest-dev/pytest/branch/master/graph/badge.svg
    :target: https://codecov.io/gh/pytest-dev/pytest
    :alt: Code coverage Status

.. image:: https://travis-ci.org/pytest-dev/pytest.svg?branch=master
    :target: https://travis-ci.org/pytest-dev/pytest

.. image:: https://github.com/pytest-dev/pytest/workflows/main/badge.svg
    :target: https://github.com/pytest-dev/pytest/actions?query=workflow%3Amain

.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/psf/black

.. image:: https://www.codetriage.com/pytest-dev/pytest/badges/users.svg
    :target: https://www.codetriage.com/pytest-dev/pytest

.. image:: https://readthedocs.org/projects/pytest/badge/?version=latest
    :target: https://pytest.readthedocs.io/en/latest/?badge=latest
    :alt: Documentation Status

The ``pytest`` framework makes it easy to write small tests, yet
scales to support complex functional testing for applications and libraries.

An example of a simple test:

.. code-block:: python

    # content of test_sample.py
    def inc(x):
        return x + 1


    def test_answer():
        assert inc(3) == 5


To execute it::

    $ pytest
    ============================= test session starts =============================
    collected 1 items

    test_sample.py F

    ================================== FAILURES ===================================
    _________________________________ test_answer _________________________________

        def test_answer():
    >       assert inc(3) == 5
    E       assert 4 == 5
    E        +  where 4 = inc(3)

    test_sample.py:5: AssertionError
    ========================== 1 failed in 0.04 seconds ===========================


Due to ``pytest``'s detailed assertion introspection, only plain ``assert`` statements are used. See `getting-started <https://docs.pytest.org/en/stable/getting-started.html#our-first-test-run>`_ for more examples.


Features
--------

- Detailed info on failing `assert statements <https://docs.pytest.org/en/stable/assert.html>`_ (no need to remember ``self.assert*`` names)

- `Auto-discovery
  <https://docs.pytest.org/en/stable/goodpractices.html#python-test-discovery>`_
  of test modules and functions

- `Modular fixtures <https://docs.pytest.org/en/stable/fixture.html>`_ for
  managing small or parametrized long-lived test resources

- Can run `unittest <https://docs.pytest.org/en/stable/unittest.html>`_ (or trial),
  `nose <https://docs.pytest.org/en/stable/nose.html>`_ test suites out of the box

- Python 3.6+ and PyPy3

- Rich plugin architecture, with over 850+ `external plugins <http://plugincompat.herokuapp.com>`_ and thriving community


Documentation
-------------

For full documentation, including installation, tutorials and PDF documents, please see https://docs.pytest.org/en/stable/.


Bugs/Requests
-------------

Please use the `GitHub issue tracker <https://github.com/pytest-dev/pytest/issues>`_ to submit bugs or request features.


Changelog
---------

Consult the `Changelog <https://docs.pytest.org/en/stable/changelog.html>`__ page for fixes and enhancements of each version.


Support pytest
--------------

`Open Collective`_ is an online funding platform for open and transparent communities.
It provides tools to raise money and share your finances in full transparency.

It is the platform of choice for individuals and companies that want to make one-time or
monthly donations directly to the project.

See more details in the `pytest collective`_.

.. _Open Collective: https://opencollective.com
.. _pytest collective: https://opencollective.com/pytest


pytest for enterprise
---------------------

Available as part of the Tidelift Subscription.

The maintainers of pytest and thousands of other packages are working with Tidelift to deliver commercial support and
maintenance for the open source dependencies you use to build your applications.
Save time, reduce risk, and improve code health, while paying the maintainers of the exact dependencies you use.

`Learn more. <https://tidelift.com/subscription/pkg/pypi-pytest?utm_source=pypi-pytest&utm_medium=referral&utm_campaign=enterprise&utm_term=repo>`_

Security
^^^^^^^^

pytest has never been associated with a security vulnerability, but in any case, to report a
security vulnerability please use the `Tidelift security contact <https://tidelift.com/security>`_.
Tidelift will coordinate the fix and disclosure.


License
-------

Copyright Holger Krekel and others, 2004-2020.

Distributed under the terms of the `MIT`_ license, pytest is free and open source software.

.. _`MIT`: https://github.com/pytest-dev/pytest/blob/master/LICENSE