Previously, skipif/xfail marks were evaluated using a `MarkEvaluator`
class. I found this class very difficult to understand.
Instead of `MarkEvaluator`, rewrite using straight functions which are
hopefully easier to follow.
I tried to keep the semantics exactly as before, except improving a few
error messages.
There are state transitions start/done/suspend/resume and two additional
operations snap/writeorg.
Previously it was not well defined in what order they can be called, and
which operations are idempotent.
Formalize this and enforce using assert checks with informative error
messages if they fail (rather than random AttributeErrors).
`TerminalWriter`, imported recently from `py`, contains its own
incomplete wcwidth (`char_with`/`get_line_width`) implementation. The
`TerminalReporter` also needs this, but uses the external `wcwidth`
package.
This commit brings the `TerminalWriter` implementation up-to-par with
`wcwidth`, moves to implementation to a new file `_pytest._io.wcwidth`
which is used everywhere, and removes the dependency.
The differences compared to the `wcwidth` package are:
- Normalizes the string before counting.
- Uses Python's `unicodedata` instead of vendored Unicode tables. This
means the data corresponds to the Python's version Unicode version
instead of the `wcwidth`'s package version.
- Apply some optimizations.
_pytest.timing is an indirection to 'time' functions, which pytest production
code should use instead of 'time' directly.
'mock_timing' is a new fixture which then mocks those functions, allowing us
to write time-reliable tests which run instantly and are not flaky.
This was triggered by recent flaky junitxml tests on Windows related to timing
issues.
The `FDCapture`/`FDCaptureBinary` classes, used by `capfd`/`capfdbinary`
fixtures and the `--capture=fd` option (set by default), redirect FDs
1/2 (stdout/stderr) to a temporary file. To do this, they need to save
the old file by duplicating the FD before redirecting it, to be restored
once finished.
Previously, if this duplicating (`os.dup()`) failed, most likely due to
that FD being invalid, the FD redirection would silently not be done. The
FD capturing also performs python-level redirection (monkeypatching
`sys.stdout`/`sys.stderr`) which would still be done, but direct writes
to the FDs would fail.
This is not great. If pytest is run with `--capture=fd`, or a test is
using `capfd`, it expects writes to the FD to work and be captured,
regardless of external circumstances.
So, instead of disabling FD capturing, keep the redirection to a
temporary file, just don't restore it after closing, because there is
nothing to restore to.
Currently, a bad logging call, e.g.
logger.info('oops', 'first', 2)
triggers the default logging handling, which is printing an error to
stderr but otherwise continuing.
For regular programs this behavior makes sense, a bad log message
shouldn't take down the program. But during tests, it is better not to
skip over such mistakes, but propagate them to the user.
Previously, a LoggingCaptureHandler was instantiated for each test's
setup/call/teardown which turns out to be expensive.
Instead, only keep one instance and reset it between runs.
The tests came via c629f6b18 and c61ff31ffa.
The fixes from there are kind of obsoleted by 4cd08f9 (moving to importlib),
but it makes sense to keep them as integration tests in general.
The default message is often hard to read:
E _pytest.config.ConftestImportFailure: (local('D:\\projects\\pytest\\.tmp\\root\\foo\\conftest.py'), (<class 'RuntimeError'>, RuntimeError('some error',), <traceback object at 0x000001CCC3E39348>))
Using a shorter message is better:
E _pytest.config.ConftestImportFailure: RuntimeError: some error (from D:\projects\pytest\.tmp\root\foo\conftest.py)
And we don't really lose any information due to exception chaining.
Only filter with known failures, and explicitly keep paths of passed
arguments.
This also displays the "run-last-failure" status before collected files,
and does not update the cache with "--collect-only".
Fixes https://github.com/pytest-dev/pytest/issues/6968.
The previous commit made this possible, so utilize it.
Since legacy.py becomes pretty bare, I inlined it into __init__.py. I'm
not sure it's really "legacy" anyway!
Using a simple 50000 items benchmark with `--collect-only -k nomatch`:
Before (two commits ago):
======================== 50000 deselected in 10.31s =====================
19129345 function calls (18275596 primitive calls) in 10.634 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.001 0.001 2.270 2.270 __init__.py:149(pytest_collection_modifyitems)
1 0.036 0.036 2.270 2.270 __init__.py:104(deselect_by_keyword)
50000 0.055 0.000 2.226 0.000 legacy.py:87(matchkeyword)
After:
======================== 50000 deselected in 9.37s =========================
18029363 function calls (17175972 primitive calls) in 9.701 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.394 1.394 __init__.py:239(pytest_collection_modifyitems)
1 0.057 0.057 1.393 1.393 __init__.py:162(deselect_by_keyword)
The matching itself can be optimized more but that's a different story.
In current pytest, the same expression is matched against all items. But
it is re-parsed for every match.
Add support for "compiling" an expression and reusing the result. Errors
may only occur during compilation.
This is done by parsing the expression into a Python `ast.Expression`,
then `compile()`ing it into a code object. Evaluation is then done using
`eval()`.
Note: historically we used to use `eval` directly on the user input --
this is not the case here, the expression is entirely under our control
according to our grammar, we just JIT-compile it to Python as a
(completely safe) optimization.
New errors:
testing/test_setupplan.py:104:15: E741 ambiguous variable name 'l'
testing/test_setupplan.py:107:15: E741 ambiguous variable name 'l'
extra/get_issues.py:48:29: E741 ambiguous variable name 'l'
testing/test_error_diffs.py:270:32: E741 ambiguous variable name 'l'
Not so sure about it but easier to just fix.
But more importantly, is a large amount of typing-related issues there
were fixed which necessitated noqa's which can now be removed.
The `-k '-expr'` syntax is an old alias to `-k 'not expr'`. It's also
not a very convenient to have syntax that start with `-` on the CLI.
Deprecate it and suggest replacing with `not`.
---
The `-k 'expr:'` syntax discards all items until the first match and
keeps all subsequent, e.g. `-k foo` with
test_bar
test_foo
test_baz
results in `test_foo`, `test_baz`. That's a bit weird, so deprecate it
without a replacement. If someone complains we can reconsider or devise
a better alternative.
Running `pytest | head -1` and similar causes an annoying error to be
printed to stderr:
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
(or possibly even a propagating exception in older/other Python versions).
The standard UNIX behavior is to handle the EPIPE silently. To
recommended method to do this in Python is described here:
https://docs.python.org/3/library/signal.html#note-on-sigpipe
It is not appropriate to apply this recommendation to `pytest.main()`,
which is used programmatically for in-process runs. Hence, change
pytest's entrypoint to a new `pytest.console_main()` function, to be
used exclusively by pytest's CLI, and add the SIGPIPE code there.
Fixes#4375.
- replace "tests with warnings" with just warnings: they a) might not
come from a test in the first place, and b) a single test might have
multiple warnings.
- fix usage of (unused) argument to `collapsed_location_report`
Co-authored-by: Ran Benita <ran@unusedvar.com>
Move {Passthrough,CaptureIO} to capture module, and rename Passthrough
-> Tee to match the existing terminology.
Co-authored-by: Ran Benita <ran@unusedvar.com>
Also delay calling tearDown() when --pdb is given, so users still have
access to the instance variables (which are usually cleaned up during tearDown())
when debugging.
Fix#6947
Instead of trying to handle unittest-async functions in pytest_pyfunc_call,
let the unittest framework handle them instead.
This lets us remove the hack in pytest_pyfunc_call, with the upside that
we should support any unittest-async based framework.
Also included 'asynctest' as test dependency for py37-twisted, and renamed
'twisted' to 'unittestextras' to better reflect that we install 'twisted' and
'asynctest' now.
This also fixes the problem of cleanUp functions not being properly called
for async functions.
Fix#7110Fix#6924
Previously, the expressions given to the `-m` and `-k` options were
evaluated with `eval`. This causes a few issues:
- Python keywords cannot be used.
- Constants like numbers, None, True, False are not handled correctly.
- Various syntax like numeric operators and `X if Y else Z` is supported
unintentionally.
- `eval()` is somewhat dangerous for arbitrary input.
- Can fail in many ways so requires `except Exception`.
The format we want to support is quite simple, so change to a custom
parser. This fixes the issues above, and gives us full control of the
format, so can be documented comprehensively and even be extended in the
future if we wish.
The code used an O(n^2) loop. Replace list with set to make it O(n).
For backward compatibility the filesystem cache still remains a list.
On this test:
import pytest
@pytest.mark.parametrize("x", range(5000))
def test_foo(x): pass
run with `pytest --collect-only`:
Before: 0m1.251s
After: 0m0.921s
Remove this message which was shown in the warning summary of pytest's
own testsuite:
testing/test_cacheprovider.py::TestNewAPI::test_cache_writefail_cachfile_silent
testing/test_cacheprovider.py:40: PytestCacheWarning: could not create cache path /tmp/pytest-of-ran/pytest-2/test_cache_writefail_cachfile_silent0/.pytest_cache/v/test/broken
cache.set("test/broken", [])
Currently this test issues a warning which is displayed in the warning
summary (of pytest's own test suite):
testing/acceptance_test.py::TestGeneralUsage::test_early_skip
/tmp/pytest-of-ran/pytest-396/test_early_skip0/conftest.py:2: PytestDeprecationWarning: The pytest_collect_directory hook is not working.
Please use collect_ignore in conftests or pytest_collection_modifyitems.
def pytest_collect_directory():
I think the filter was meant to be `ignore` in the first place, and not
`always` which is not a valid action AFAIK.
This function is exposed and kept alive for the oejskit plugin which is
abandoned and no longer works with recent plugins, so let's prepare to
completely remove it.
The `_pytest._code._reprcompare` that was referred to previously doesn't
exist -- it was moved to other places but wasn't updated. This regressed
in f423ce9c01. Now we don't want it
anymore, so keep the status quo by explicitly removing them.
* tests: fix TypeError with test_mark_closest
It fails when trying to run it actually:
> TypeError: test_has_inherited() takes 0 positional arguments but 1 was given
* Fix testing/test_collection.py::TestCollector::test_getparent
TestDurations tests the `--durations=N` functionality which reports N
slowest tests, with durations <= 0.005s not shown by default.
The test relies on real time.sleep() (in addition to the code which uses
time.perf_counter()) which makes it flaky and inconsistent between
platforms.
Instead of trying to tweak it more, make it use fake time instead. The
way it is done is a little hacky but seems to work.
This hook has some functionality to provide explicit markup for the test
status. It seemed unused and wasn't tested, so I was tempted to remove
it, but I found that the pytest-rerunfailures plugin uses it, so
document it and add a test instead.
Use `testdir.syspathinsert()` with multiprocessing tests:
- test_chained_exceptions_no_reprcrash
- test_exception_handling_no_traceback
This only works currently because `_importtestmodule` changes `sys.path`
as a side-effect.
It appears to be only required on Windows though - likely due to the
multiprocessing method used there.
The dynamic scope feature added in 10bf6aac76
necessitated some wrangling of arguments in pytest.fixture(). In
particular, it deprecated positional arguments in favor of keyword-only
arguments, while keeping backward compatibility.
The way it did this avoided some code duplication but ended up being
quite hard to follow and to annotate with types.
Replace it with some straightforward code, which is not very DRY but is
simple and easy to remove when the time comes.
Co-authored-by: Sylvain MARIE <sylvain.marie@se.com>
Co-authored-by: Ran Benita <ran@unusedvar.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
Previously, writing to sys.stdout/stderr in text-mode (e.g.
`print('foo')`) while a `capsysbinary` fixture is active, would crash
with:
/usr/lib/python3.7/contextlib.py:119: in __exit__
next(self.gen)
E TypeError: write() argument must be str, not bytes
This is due to some confusion in the types. The relevant functions are
`snap()` and `writeorg()`. The function `snap()` returns what was
captured, and the return type should be `bytes` for the binary captures
and `str` for the regular ones. The `snap()` return value is eventually
passed to `writeorg()` to be written to the original file, so it's input
type should correspond to `snap()`. But this was incorrect for
`SysCaptureBinary`, which handled it like `str`.
To fix this, be explicit in the `snap()` and `writeorg()`
implementations, also of the other Capture types.
We can't add type annotations yet, because the current inheritance
scheme breaks Liskov Substitution and mypy would complain. To be
refactored later.
Fixes: https://github.com/pytest-dev/pytest/issues/6871
Co-authored-by: Ran Benita (some modifications & commit message)
I tried to understand what the `safe_text_dupfile()` function and
`EncodedFile` class do. Outside tests, `EncodedFile` is only used by
`safe_text_dupfile`, and `safe_text_dupfile` is only used by
`FDCaptureBinary.__init__()`. I then started to eliminate always-true
conditions based on the single call site, and in the end nothing was
left except of a couple workarounds that are still needed.
The commit which added the checks for os.dup a15afb5e48
suggests it was done for Jython. But pytest doesn't support Jython
anymore (Jython is Python 2 only).
Furthermore, it looks like the faulthandler plugin (bundled in pytest
and enabled by default) uses os.dup() unprotected and there have not
been any complaints.
So seems better to just remove these checks, and only add if someone
with a legitimate use case complains.
This started from fixing the test, where `"xdist_strict True"` was used
as a single argument, although you typically would see `["xdist_strict",
"True"]`.
Improves the error message to mention the option that caused the error.
pytest has several instances where plugins set their own attributes on
objects they receive in hooks, like nodes and config. Since plugins are
detached from these object's definition by design, this causes a problem
for type checking because these attributes are not defined and mypy
complains.
Fix this by giving these objects a "store" which can be used by plugins
in a type-safe manner.
Currently this mechanism is private. We can consider exposing it at a
later point.
Passing in a tuple crashes in `_prepareconfig`:
def test_invoke_with_tuple(self):
> pytest.main(("-h",))
src/_pytest/config/__init__.py:82: in main
config = _prepareconfig(args, plugins)
src/_pytest/config/__init__.py:229: in _prepareconfig
return pluginmanager.hook.pytest_cmdline_parse(
…
src/_pytest/helpconfig.py:98: in pytest_cmdline_parse
config = outcome.get_result() # type: Config
src/_pytest/config/__init__.py:808: in pytest_cmdline_parse
self.parse(args)
src/_pytest/config/__init__.py:1017: in parse
self._preparse(args, addopts=addopts)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def _preparse(self, args: List[str], addopts: bool = True) -> None:
…
if addopts:
ini_addopts = self.getini("addopts")
if ini_addopts:
> args[:] = self._validate_args(ini_addopts, "via addopts config") + args
E TypeError: can only concatenate list (not "tuple") to list
addopts = True
args = ('-h',)
env_addopts = ''
ini_addopts = ['-rfEX', …]
src/_pytest/config/__init__.py:956: TypeError: can only concatenate list (not "tuple") to list
Might be worth handling (converting it to a list for example), but it
was documented to be a list to begin with when removing support for
strings (a7e401656).
Ran Benita
Gleb Nikonorov
David Diaz Barquero
Ram Rachum
Andrew
Bruno Oliveira
Ronny Pfannschmidt
Zac Hatfield-Dodds
Zac-HD
ibriquem
piotrhm
Prashant Anand
Fabio Zadrozny
Tor Colvin
Katarzyna Król
Anthony Sottile
Florian Dahlitz
Claire Cecil
Daniel Hahler
Daniel Hahler
Felix Nieuwenhuizen
Pavel Karateev
Katarzyna
Keri Volans
Simon K
smarie
earonesty
Ronny Pfannschmidt