Merge remote-tracking branch 'upstream/master' into merge-master-into-features

This commit is contained in:
Bruno Oliveira 2019-05-16 20:10:44 -03:00
commit 041ea3704b
15 changed files with 263 additions and 146 deletions

View File

@ -5,7 +5,6 @@ Here's a quick checklist that should be present in PRs.
(please delete this text from the final description, this is just a guideline)
-->
- [ ] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/master/changelog/README.rst) for details.
- [ ] Target the `master` branch for bug fixes, documentation updates and trivial changes.
- [ ] Target the `features` branch for new features and removals/deprecations.
- [ ] Include documentation when adding new features.
@ -13,4 +12,5 @@ Here's a quick checklist that should be present in PRs.
Unless your change is trivial or a small documentation fix (e.g., a typo or reword of a small section) please:
- [ ] Create a new changelog file in the `changelog` folder, with a name like `<ISSUE NUMBER>.<TYPE>.rst`. See [changelog/README.rst](https://github.com/pytest-dev/pytest/blob/master/changelog/README.rst) for details.
- [ ] Add yourself to `AUTHORS` in alphabetical order;

View File

@ -86,6 +86,7 @@ Endre Galaczi
Eric Hunsberger
Eric Siegerman
Erik M. Bray
Evan Kepner
Fabien Zarifian
Fabio Zadrozny
Feng Ma

View File

@ -5,7 +5,6 @@ trigger:
variables:
PYTEST_ADDOPTS: "--junitxml=build/test-results/$(tox.env).xml -vv"
python.needs_vc: False
python.exe: "python"
COVERAGE_FILE: "$(Build.Repository.LocalPath)/.coverage"
COVERAGE_PROCESS_START: "$(Build.Repository.LocalPath)/.coveragerc"
PYTEST_COVERAGE: '0'
@ -42,15 +41,13 @@ jobs:
# Also seen with py27-nobyte (using xdist), and py27-xdist.
# But no exception with py27-pexpect,py27-twisted,py27-numpy.
PYTEST_COVERAGE: '1'
pypy:
python.version: 'pypy2'
tox.env: 'pypy'
python.exe: 'pypy'
# NOTE: pypy3 fails to install pip currently due to an internal error.
# -- pypy2 and pypy3 are disabled for now: #5279 --
# pypy:
# python.version: 'pypy2'
# tox.env: 'pypy'
# pypy3:
# python.version: 'pypy3'
# tox.env: 'pypy3'
# python.exe: 'pypy3'
py34-xdist:
python.version: '3.4'
tox.env: 'py34-xdist'
@ -94,12 +91,12 @@ jobs:
condition: eq(variables['python.needs_vc'], True)
displayName: 'Install VC for py27'
- script: $(python.exe) -m pip install --upgrade pip && $(python.exe) -m pip install tox
- script: python -m pip install --upgrade pip && python -m pip install tox
displayName: 'Install tox'
- script: |
call scripts/setup-coverage-vars.bat || goto :eof
$(python.exe) -m tox -e $(tox.env)
python -m tox -e $(tox.env)
displayName: 'Run tests'
- task: PublishTestResults@2
@ -112,6 +109,5 @@ jobs:
displayName: 'Report and upload coverage'
condition: eq(variables['PYTEST_COVERAGE'], '1')
env:
PYTHON: $(python.exe)
CODECOV_TOKEN: $(CODECOV_TOKEN)
PYTEST_CODECOV_NAME: $(tox.env)

1
changelog/5250.doc.rst Normal file
View File

@ -0,0 +1 @@
Expand docs on use of ``setenv`` and ``delenv`` with ``monkeypatch``.

View File

@ -0,0 +1 @@
Handle internal error due to a lone surrogate unicode character not being representable in Jython.

View File

@ -0,0 +1 @@
Ensure that ``sys.stdout.mode`` does not include ``'b'`` as it is a text stream.

View File

@ -4,7 +4,9 @@
Working with custom markers
=================================================
Here are some example using the :ref:`mark` mechanism.
Here are some examples using the :ref:`mark` mechanism.
.. _`mark run`:
Marking test functions and selecting them for a run
----------------------------------------------------

View File

@ -4,6 +4,117 @@ Historical Notes
This page lists features or behavior from previous versions of pytest which have changed over the years. They are
kept here as a historical note so users looking at old code can find documentation related to them.
.. _marker-revamp:
Marker revamp and iteration
---------------------------
.. versionchanged:: 3.6
pytest's marker implementation traditionally worked by simply updating the ``__dict__`` attribute of functions to cumulatively add markers. As a result, markers would unintentionally be passed along class hierarchies in surprising ways. Further, the API for retrieving them was inconsistent, as markers from parameterization would be stored differently than markers applied using the ``@pytest.mark`` decorator and markers added via ``node.add_marker``.
This state of things made it technically next to impossible to use data from markers correctly without having a deep understanding of the internals, leading to subtle and hard to understand bugs in more advanced usages.
Depending on how a marker got declared/changed one would get either a ``MarkerInfo`` which might contain markers from sibling classes,
``MarkDecorators`` when marks came from parameterization or from a ``node.add_marker`` call, discarding prior marks. Also ``MarkerInfo`` acts like a single mark, when it in fact represents a merged view on multiple marks with the same name.
On top of that markers were not accessible in the same way for modules, classes, and functions/methods.
In fact, markers were only accessible in functions, even if they were declared on classes/modules.
A new API to access markers has been introduced in pytest 3.6 in order to solve the problems with
the initial design, providing the :func:`_pytest.nodes.Node.iter_markers` method to iterate over
markers in a consistent manner and reworking the internals, which solved a great deal of problems
with the initial design.
.. _update marker code:
Updating code
~~~~~~~~~~~~~
The old ``Node.get_marker(name)`` function is considered deprecated because it returns an internal ``MarkerInfo`` object
which contains the merged name, ``*args`` and ``**kwargs`` of all the markers which apply to that node.
In general there are two scenarios on how markers should be handled:
1. Marks overwrite each other. Order matters but you only want to think of your mark as a single item. E.g.
``log_level('info')`` at a module level can be overwritten by ``log_level('debug')`` for a specific test.
In this case, use ``Node.get_closest_marker(name)``:
.. code-block:: python
# replace this:
marker = item.get_marker("log_level")
if marker:
level = marker.args[0]
# by this:
marker = item.get_closest_marker("log_level")
if marker:
level = marker.args[0]
2. Marks compose in an additive manner. E.g. ``skipif(condition)`` marks mean you just want to evaluate all of them,
order doesn't even matter. You probably want to think of your marks as a set here.
In this case iterate over each mark and handle their ``*args`` and ``**kwargs`` individually.
.. code-block:: python
# replace this
skipif = item.get_marker("skipif")
if skipif:
for condition in skipif.args:
# eval condition
...
# by this:
for skipif in item.iter_markers("skipif"):
condition = skipif.args[0]
# eval condition
If you are unsure or have any questions, please consider opening
`an issue <https://github.com/pytest-dev/pytest/issues>`_.
Related issues
~~~~~~~~~~~~~~
Here is a non-exhaustive list of issues fixed by the new implementation:
* Marks don't pick up nested classes (`#199 <https://github.com/pytest-dev/pytest/issues/199>`_).
* Markers stain on all related classes (`#568 <https://github.com/pytest-dev/pytest/issues/568>`_).
* Combining marks - args and kwargs calculation (`#2897 <https://github.com/pytest-dev/pytest/issues/2897>`_).
* ``request.node.get_marker('name')`` returns ``None`` for markers applied in classes (`#902 <https://github.com/pytest-dev/pytest/issues/902>`_).
* Marks applied in parametrize are stored as markdecorator (`#2400 <https://github.com/pytest-dev/pytest/issues/2400>`_).
* Fix marker interaction in a backward incompatible way (`#1670 <https://github.com/pytest-dev/pytest/issues/1670>`_).
* Refactor marks to get rid of the current "marks transfer" mechanism (`#2363 <https://github.com/pytest-dev/pytest/issues/2363>`_).
* Introduce FunctionDefinition node, use it in generate_tests (`#2522 <https://github.com/pytest-dev/pytest/issues/2522>`_).
* Remove named marker attributes and collect markers in items (`#891 <https://github.com/pytest-dev/pytest/issues/891>`_).
* skipif mark from parametrize hides module level skipif mark (`#1540 <https://github.com/pytest-dev/pytest/issues/1540>`_).
* skipif + parametrize not skipping tests (`#1296 <https://github.com/pytest-dev/pytest/issues/1296>`_).
* Marker transfer incompatible with inheritance (`#535 <https://github.com/pytest-dev/pytest/issues/535>`_).
More details can be found in the `original PR <https://github.com/pytest-dev/pytest/pull/3317>`_.
.. note::
in a future major relase of pytest we will introduce class based markers,
at which point markers will no longer be limited to instances of :py:class:`Mark`.
cache plugin integrated into the core
-------------------------------------

View File

@ -15,8 +15,10 @@ some builtin markers, for example:
to the same test function.
It's easy to create custom markers or to apply markers
to whole test classes or modules. See :ref:`mark examples` for examples
which also serve as documentation.
to whole test classes or modules. Those markers can be used by plugins, and also
are commonly used to :ref:`select tests <mark run>` on the command-line with the ``-m`` option.
See :ref:`mark examples` for examples which also serve as documentation.
.. note::
@ -24,142 +26,53 @@ which also serve as documentation.
:ref:`fixtures <fixtures>`.
.. _unknown-marks:
Registering marks
-----------------
Raising errors on unknown marks
-------------------------------
Unknown marks applied with the ``@pytest.mark.name_of_the_mark`` decorator
will always emit a warning, in order to avoid silently doing something
surprising due to mis-typed names. You can disable the warning for custom
marks by registering them in ``pytest.ini`` like this:
You can register custom marks in your ``pytest.ini`` file like this:
.. code-block:: ini
[pytest]
markers =
slow
slow: marks tests as slow (deselect with '-m "not slow"')
serial
Note that everything after the ``:`` is an optional description.
Alternatively, you can register new markers programatically in a
:ref:`pytest_configure <initialization-hooks>` hook:
.. code-block:: python
def pytest_configure(config):
config.addinivalue_line(
"markers", "env(name): mark test to run only on named environment"
)
Registered marks appear in pytest's help text and do not emit warnings (see the next section). It
is recommended that third-party plugins always :ref:`register their markers <registering-markers>`.
.. _unknown-marks:
Raising errors on unknown marks
-------------------------------
Unregistered marks applied with the ``@pytest.mark.name_of_the_mark`` decorator
will always emit a warning in order to avoid silently doing something
surprising due to mis-typed names. As described in the previous section, you can disable
the warning for custom marks by registering them in your ``pytest.ini`` file or
using a custom ``pytest_configure`` hook.
When the ``--strict-markers`` command-line flag is passed, any unknown marks applied
with the ``@pytest.mark.name_of_the_mark`` decorator will trigger an error.
Marks added by pytest or by a plugin instead of the decorator will not trigger
this error. To enforce validation of markers, add ``--strict-markers`` to ``addopts``:
with the ``@pytest.mark.name_of_the_mark`` decorator will trigger an error. You can
enforce this validation in your project by adding ``--strict-markers`` to ``addopts``:
.. code-block:: ini
[pytest]
addopts = --strict-markers
markers =
slow
slow: marks tests as slow (deselect with '-m "not slow"')
serial
Third-party plugins should always :ref:`register their markers <registering-markers>`
so that they appear in pytest's help text and do not emit warnings.
.. _marker-revamp:
Marker revamp and iteration
---------------------------
pytest's marker implementation traditionally worked by simply updating the ``__dict__`` attribute of functions to cumulatively add markers. As a result, markers would unintentionally be passed along class hierarchies in surprising ways. Further, the API for retrieving them was inconsistent, as markers from parameterization would be stored differently than markers applied using the ``@pytest.mark`` decorator and markers added via ``node.add_marker``.
This state of things made it technically next to impossible to use data from markers correctly without having a deep understanding of the internals, leading to subtle and hard to understand bugs in more advanced usages.
Depending on how a marker got declared/changed one would get either a ``MarkerInfo`` which might contain markers from sibling classes,
``MarkDecorators`` when marks came from parameterization or from a ``node.add_marker`` call, discarding prior marks. Also ``MarkerInfo`` acts like a single mark, when it in fact represents a merged view on multiple marks with the same name.
On top of that markers were not accessible the same way for modules, classes, and functions/methods.
In fact, markers were only accessible in functions, even if they were declared on classes/modules.
A new API to access markers has been introduced in pytest 3.6 in order to solve the problems with the initial design, providing :func:`_pytest.nodes.Node.iter_markers` method to iterate over markers in a consistent manner and reworking the internals, which solved great deal of problems with the initial design.
.. _update marker code:
Updating code
~~~~~~~~~~~~~
The old ``Node.get_marker(name)`` function is considered deprecated because it returns an internal ``MarkerInfo`` object
which contains the merged name, ``*args`` and ``**kwargs`` of all the markers which apply to that node.
In general there are two scenarios on how markers should be handled:
1. Marks overwrite each other. Order matters but you only want to think of your mark as a single item. E.g.
``log_level('info')`` at a module level can be overwritten by ``log_level('debug')`` for a specific test.
In this case, use ``Node.get_closest_marker(name)``:
.. code-block:: python
# replace this:
marker = item.get_marker("log_level")
if marker:
level = marker.args[0]
# by this:
marker = item.get_closest_marker("log_level")
if marker:
level = marker.args[0]
2. Marks compose in an additive manner. E.g. ``skipif(condition)`` marks mean you just want to evaluate all of them,
order doesn't even matter. You probably want to think of your marks as a set here.
In this case iterate over each mark and handle their ``*args`` and ``**kwargs`` individually.
.. code-block:: python
# replace this
skipif = item.get_marker("skipif")
if skipif:
for condition in skipif.args:
# eval condition
...
# by this:
for skipif in item.iter_markers("skipif"):
condition = skipif.args[0]
# eval condition
If you are unsure or have any questions, please consider opening
`an issue <https://github.com/pytest-dev/pytest/issues>`_.
Related issues
~~~~~~~~~~~~~~
Here is a non-exhaustive list of issues fixed by the new implementation:
* Marks don't pick up nested classes (`#199 <https://github.com/pytest-dev/pytest/issues/199>`_).
* Markers stain on all related classes (`#568 <https://github.com/pytest-dev/pytest/issues/568>`_).
* Combining marks - args and kwargs calculation (`#2897 <https://github.com/pytest-dev/pytest/issues/2897>`_).
* ``request.node.get_marker('name')`` returns ``None`` for markers applied in classes (`#902 <https://github.com/pytest-dev/pytest/issues/902>`_).
* Marks applied in parametrize are stored as markdecorator (`#2400 <https://github.com/pytest-dev/pytest/issues/2400>`_).
* Fix marker interaction in a backward incompatible way (`#1670 <https://github.com/pytest-dev/pytest/issues/1670>`_).
* Refactor marks to get rid of the current "marks transfer" mechanism (`#2363 <https://github.com/pytest-dev/pytest/issues/2363>`_).
* Introduce FunctionDefinition node, use it in generate_tests (`#2522 <https://github.com/pytest-dev/pytest/issues/2522>`_).
* Remove named marker attributes and collect markers in items (`#891 <https://github.com/pytest-dev/pytest/issues/891>`_).
* skipif mark from parametrize hides module level skipif mark (`#1540 <https://github.com/pytest-dev/pytest/issues/1540>`_).
* skipif + parametrize not skipping tests (`#1296 <https://github.com/pytest-dev/pytest/issues/1296>`_).
* Marker transfer incompatible with inheritance (`#535 <https://github.com/pytest-dev/pytest/issues/535>`_).
More details can be found in the `original PR <https://github.com/pytest-dev/pytest/pull/3317>`_.
.. note::
in a future major relase of pytest we will introduce class based markers,
at which point markers will no longer be limited to instances of :py:class:`Mark`.

View File

@ -16,7 +16,7 @@ and a discussion of its motivation.
Simple example: monkeypatching functions
---------------------------------------------------
----------------------------------------
If you want to pretend that ``os.expanduser`` returns a certain
directory, you can use the :py:meth:`monkeypatch.setattr` method to
@ -38,8 +38,8 @@ Here our test function monkeypatches ``os.path.expanduser`` and
then calls into a function that calls it. After the test function
finishes the ``os.path.expanduser`` modification will be undone.
example: preventing "requests" from remote operations
------------------------------------------------------
Global patch example: preventing "requests" from remote operations
------------------------------------------------------------------
If you want to prevent the "requests" library from performing http
requests in all your tests, you can do::
@ -81,6 +81,80 @@ so that any attempts within tests to create http requests will fail.
See issue `#3290 <https://github.com/pytest-dev/pytest/issues/3290>`_ for details.
Monkeypatching environment variables
------------------------------------
If you are working with environment variables you often need to safely change the values
or delete them from the system for testing purposes. ``Monkeypatch`` provides a mechanism
to do this using the ``setenv`` and ``delenv`` method. Our example code to test:
.. code-block:: python
# contents of our original code file e.g. code.py
import os
def get_os_user_lower():
"""Simple retrieval function.
Returns lowercase USER or raises EnvironmentError."""
username = os.getenv("USER")
if username is None:
raise EnvironmentError("USER environment is not set.")
return username.lower()
There are two potential paths. First, the ``USER`` environment variable is set to a
value. Second, the ``USER`` environment variable does not exist. Using ``monkeypatch``
both paths can be safely tested without impacting the running environment:
.. code-block:: python
# contents of our test file e.g. test_code.py
import pytest
def test_upper_to_lower(monkeypatch):
"""Set the USER env var to assert the behavior."""
monkeypatch.setenv("USER", "TestingUser")
assert get_os_user_lower() == "testinguser"
def test_raise_exception(monkeypatch):
"""Remove the USER env var and assert EnvironmentError is raised."""
monkeypatch.delenv("USER", raising=False)
with pytest.raises(EnvironmentError):
_ = get_os_user_lower()
This behavior can be moved into ``fixture`` structures and shared across tests:
.. code-block:: python
import pytest
@pytest.fixture
def mock_env_user(monkeypatch):
monkeypatch.setenv("USER", "TestingUser")
@pytest.fixture
def mock_env_missing(monkeypatch):
monkeypatch.delenv("USER", raising=False)
# Notice the tests reference the fixtures for mocks
def test_upper_to_lower(mock_env_user):
assert get_os_user_lower() == "testinguser"
def test_raise_exception(mock_env_missing):
with pytest.raises(EnvironmentError):
_ = get_os_user_lower()
.. currentmodule:: _pytest.monkeypatch
API Reference

View File

@ -581,6 +581,8 @@ Bootstrapping hooks called for plugins registered early enough (internal and set
.. autofunction:: pytest_cmdline_parse
.. autofunction:: pytest_cmdline_main
.. _`initialization-hooks`:
Initialization hooks
~~~~~~~~~~~~~~~~~~~~

View File

@ -6,11 +6,11 @@ if "%PYTEST_COVERAGE%" == "1" (
) else (
echo CODECOV_TOKEN NOT defined
)
%PYTHON% -m pip install codecov
%PYTHON% -m coverage combine
%PYTHON% -m coverage xml
%PYTHON% -m coverage report -m
scripts\retry %PYTHON% -m codecov --required -X gcov pycov search -f coverage.xml --name %PYTEST_CODECOV_NAME%
python -m pip install codecov
python -m coverage combine
python -m coverage xml
python -m coverage report -m
scripts\retry python -m codecov --required -X gcov pycov search -f coverage.xml --name %PYTEST_CODECOV_NAME%
) else (
echo Skipping coverage upload, PYTEST_COVERAGE=%PYTEST_COVERAGE%
)

View File

@ -447,6 +447,10 @@ class EncodedFile(object):
"""Ensure that file.name is a string."""
return repr(self.buffer)
@property
def mode(self):
return self.buffer.mode.replace("b", "")
def __getattr__(self, name):
return getattr(object.__getattribute__(self, "buffer"), name)

View File

@ -998,7 +998,15 @@ def _get_line_with_reprcrash_message(config, rep, termwidth):
# u'😄' will result in a High Surrogate (U+D83D) character, which is
# rendered as u'<27>'; in this case we just strip that character out as it
# serves no purpose being rendered
msg = msg.rstrip(u"\uD83D")
try:
surrogate = six.unichr(0xD83D)
msg = msg.rstrip(surrogate)
except ValueError: # pragma: no cover
# Jython cannot represent this lone surrogate at all (#5256):
# ValueError: unichr() arg is a lone surrogate in range
# (0xD800, 0xDFFF) (Jython UTF-16 encoding)
# ignore this case as it shouldn't appear in the string anyway
pass
msg += ellipsis
line += sep + msg
return line

View File

@ -1051,6 +1051,9 @@ class TestFDCapture(object):
cap.done()
pytest.raises(AttributeError, cap.suspend)
def test_capfd_sys_stdout_mode(self, capfd):
assert "b" not in sys.stdout.mode
@contextlib.contextmanager
def saved_fd(fd):