Merge pull request #4749 from blueyed/merge-master-into-features

Merge master into features

.travis.yml

@@ -23,6 +23,11 @@ env:
     - TOXENV=py37-pluggymaster PYTEST_NO_COVERAGE=1
     - TOXENV=py37-freeze PYTEST_NO_COVERAGE=1
 
+matrix:
+  allow_failures:
+    - python: '3.8-dev'
+      env: TOXENV=py38
+
 jobs:
   include:
     # Coverage tracking is slow with pypy, skip it.
@@ -35,6 +40,8 @@ jobs:
       python: '3.5'
     - env: TOXENV=py36
       python: '3.6'
+    - env: TOXENV=py38
+      python: '3.8-dev'
     - env: TOXENV=py37
     - &test-macos
       language: generic

AUTHORS

@@ -27,6 +27,7 @@ Anthony Shaw
 Anthony Sottile
 Anton Lodder
 Antony Lee
+Arel Cordero
 Armin Rigo
 Aron Coyle
 Aron Curzon
@@ -173,6 +174,7 @@ Nathaniel Waisbrot
 Ned Batchelder
 Neven Mundar
 Nicholas Devenish
+Nicholas Murphy
 Niclas Olofsson
 Nicolas Delaby
 Oleg Pidsadnyi

CHANGELOG.rst

@@ -24,7 +24,7 @@ pytest 4.2.0 (2019-01-30)
 Features
 --------
 
-- `#3094 <https://github.com/pytest-dev/pytest/issues/3094>`_: `Class xunit-style <https://docs.pytest.org/en/latest/xunit_setup.html>`__ functions and methods
+- `#3094 <https://github.com/pytest-dev/pytest/issues/3094>`_: `Classic xunit-style <https://docs.pytest.org/en/latest/xunit_setup.html>`__ functions and methods
   now obey the scope of *autouse* fixtures.
 
   This fixes a number of surprising issues like ``setup_method`` being called before session-scoped
@@ -96,6 +96,9 @@ Trivial/Internal Changes
 - `#4657 <https://github.com/pytest-dev/pytest/issues/4657>`_: Copy saferepr from pylib
 
 
+- `#4668 <https://github.com/pytest-dev/pytest/issues/4668>`_: The verbose word for expected failures in the teststatus report changes from ``xfail`` to ``XFAIL`` to be consistent with other test outcomes.
+
+
 pytest 4.1.1 (2019-01-12)
 =========================
 

changelog/2895.bugfix.rst

@@ -0,0 +1 @@
+The ``pytest_report_collectionfinish`` hook now is also called with ``--collect-only``.

changelog/3899.bugfix.rst

@@ -0,0 +1 @@
+Do not raise ``UsageError`` when an imported package has a ``pytest_plugins.py`` child module.

changelog/3899.doc.rst

@@ -0,0 +1 @@
+Add note to ``plugins.rst`` that ``pytest_plugins`` should not be used as a name for a user module containing plugins.

changelog/4324.doc.rst

@@ -0,0 +1 @@
+Document how to use ``raises`` and ``does_not_raise`` to write parametrized tests with conditional raises.

changelog/4592.bugfix.rst

@@ -0,0 +1 @@
+Fix handling of ``collect_ignore`` via parent ``conftest.py``.

changelog/4700.bugfix.rst

@@ -0,0 +1,2 @@
+Fix regression where ``setUpClass`` would always be called in subclasses even if all tests
+were skipped by a ``unittest.skip()`` decorator applied in the subclass.

changelog/4709.doc.rst

@@ -0,0 +1,2 @@
+Document how to customize test failure messages when using
+``pytest.warns``.

changelog/4739.bugfix.rst

@@ -0,0 +1 @@
+Fix ``parametrize(... ids=<function>)`` when the function returns non-strings.

doc/en/example/parametrize.rst

@@ -565,3 +565,50 @@ As the result:
 - The test ``test_eval[1+7-8]`` passed, but the name is autogenerated and confusing.
 - The test ``test_eval[basic_2+4]`` passed.
 - The test ``test_eval[basic_6*9]`` was expected to fail and did fail.
+
+.. _`parametrizing_conditional_raising`:
+
+Parametrizing conditional raising
+--------------------------------------------------------------------
+
+Use :func:`pytest.raises` with the
+:ref:`pytest.mark.parametrize ref` decorator to write parametrized tests
+in which some tests raise exceptions and others do not.
+
+It is helpful to define a no-op context manager ``does_not_raise`` to serve
+as a complement to ``raises``. For example::
+
+    from contextlib import contextmanager
+    import pytest
+
+    @contextmanager
+    def does_not_raise():
+        yield
+
+
+    @pytest.mark.parametrize('example_input,expectation', [
+        (3, does_not_raise()),
+        (2, does_not_raise()),
+        (1, does_not_raise()),
+        (0, pytest.raises(ZeroDivisionError)),
+    ])
+    def test_division(example_input, expectation):
+        """Test how much I know division."""
+        with expectation:
+            assert (6 / example_input) is not None
+
+In the example above, the first three test cases should run unexceptionally,
+while the fourth should raise ``ZeroDivisionError``.
+
+If you're only supporting Python 3.7+, you can simply use ``nullcontext``
+to define ``does_not_raise``::
+
+    from contextlib import nullcontext as does_not_raise
+
+Or, if you're supporting Python 3.3+ you can use::
+
+    from contextlib import ExitStack as does_not_raise
+
+Or, if desired, you can ``pip install contextlib2`` and use::
+
+    from contextlib2 import ExitStack as does_not_raise

doc/en/plugins.rst

@@ -84,6 +84,11 @@ will be loaded as well.
     :ref:`full explanation <requiring plugins in non-root conftests>`
     in the Writing plugins section.
 
+.. note::
+   The name ``pytest_plugins`` is reserved and should not be used as a
+   name for a custom plugin module.
+
+
 .. _`findpluginname`:
 
 Finding out which plugins are active

doc/en/warnings.rst

@@ -233,7 +233,7 @@ You can also use it as a contextmanager::
 .. _warns:
 
 Asserting warnings with the warns function
------------------------------------------------
+------------------------------------------
 
 .. versionadded:: 2.8
 
@@ -291,7 +291,7 @@ Alternatively, you can examine raised warnings in detail using the
 .. _recwarn:
 
 Recording warnings
-------------------------
+------------------
 
 You can record raised warnings either using ``pytest.warns`` or with
 the ``recwarn`` fixture.
@@ -329,6 +329,26 @@ warnings, or index into it to get a particular recorded warning.
 
 Full API: :class:`WarningsRecorder`.
 
+.. _custom_failure_messages:
+
+Custom failure messages
+-----------------------
+
+Recording warnings provides an opportunity to produce custom test
+failure messages for when no warnings are issued or other conditions
+are met.
+
+.. code-block:: python
+
+    def test():
+        with pytest.warns(Warning) as record:
+            f()
+            if not record:
+                pytest.fail("Expected a warning!")
+
+If no warnings are issued when calling ``f``, then ``not record`` will
+evaluate to ``True``.  You can then call ``pytest.fail`` with a
+custom error message.
 
 .. _internal-warnings:
 

pyproject.toml

@@ -5,6 +5,7 @@ requires = [
   "setuptools-scm",
   "wheel",
 ]
+build-backend = "setuptools.build_meta"
 
 [tool.towncrier]
 package = "pytest"

src/_pytest/config/__init__.py

@@ -559,8 +559,8 @@ def _get_plugin_specs_as_list(specs):
     which case it is returned as a list. Specs can also be `None` in which case an
     empty list is returned.
     """
-    if specs is not None:
-        if isinstance(specs, str):
+    if specs is not None and not isinstance(specs, types.ModuleType):
+        if isinstance(specs, six.string_types):
             specs = specs.split(",") if specs else []
         if not isinstance(specs, (list, tuple)):
             raise UsageError(

src/_pytest/logging.py

@@ -370,6 +370,8 @@ def get_actual_log_level(config, *setting_names):
         )
 
 
+# run after terminalreporter/capturemanager are configured
+@pytest.hookimpl(trylast=True)
 def pytest_configure(config):
     config.pluginmanager.register(LoggingPlugin(config), "logging-plugin")
 
@@ -388,8 +390,6 @@ class LoggingPlugin(object):
 
         # enable verbose output automatically if live logging is enabled
         if self._log_cli_enabled() and not config.getoption("verbose"):
-            # sanity check: terminal reporter should not have been loaded at this point
-            assert self._config.pluginmanager.get_plugin("terminalreporter") is None
             config.option.verbose = 1
 
         self.print_logs = get_option_ini(config, "log_print")
@@ -420,6 +420,54 @@ class LoggingPlugin(object):
 
         self.log_cli_handler = None
 
+        self.live_logs_context = lambda: dummy_context_manager()
+        # Note that the lambda for the live_logs_context is needed because
+        # live_logs_context can otherwise not be entered multiple times due
+        # to limitations of contextlib.contextmanager.
+
+        if self._log_cli_enabled():
+            self._setup_cli_logging()
+
+    def _setup_cli_logging(self):
+        config = self._config
+        terminal_reporter = config.pluginmanager.get_plugin("terminalreporter")
+        if terminal_reporter is None:
+            # terminal reporter is disabled e.g. by pytest-xdist.
+            return
+
+        # FIXME don't set verbosity level and derived attributes of
+        # terminalwriter directly
+        terminal_reporter.verbosity = config.option.verbose
+        terminal_reporter.showheader = terminal_reporter.verbosity >= 0
+        terminal_reporter.showfspath = terminal_reporter.verbosity >= 0
+        terminal_reporter.showlongtestinfo = terminal_reporter.verbosity > 0
+
+        capture_manager = config.pluginmanager.get_plugin("capturemanager")
+        # if capturemanager plugin is disabled, live logging still works.
+        log_cli_handler = _LiveLoggingStreamHandler(terminal_reporter, capture_manager)
+        log_cli_format = get_option_ini(config, "log_cli_format", "log_format")
+        log_cli_date_format = get_option_ini(
+            config, "log_cli_date_format", "log_date_format"
+        )
+        if (
+            config.option.color != "no"
+            and ColoredLevelFormatter.LEVELNAME_FMT_REGEX.search(log_cli_format)
+        ):
+            log_cli_formatter = ColoredLevelFormatter(
+                create_terminal_writer(config),
+                log_cli_format,
+                datefmt=log_cli_date_format,
+            )
+        else:
+            log_cli_formatter = logging.Formatter(
+                log_cli_format, datefmt=log_cli_date_format
+            )
+        log_cli_level = get_actual_log_level(config, "log_cli_level", "log_level")
+        self.log_cli_handler = log_cli_handler
+        self.live_logs_context = lambda: catching_logs(
+            log_cli_handler, formatter=log_cli_formatter, level=log_cli_level
+        )
+
     def _log_cli_enabled(self):
         """Return True if log_cli should be considered enabled, either explicitly
         or because --log-cli-level was given in the command-line.
@@ -430,10 +478,6 @@ class LoggingPlugin(object):
 
     @pytest.hookimpl(hookwrapper=True, tryfirst=True)
     def pytest_collection(self):
-        # This has to be called before the first log message is logged,
-        # so we can access the terminal reporter plugin.
-        self._setup_cli_logging()
-
         with self.live_logs_context():
             if self.log_cli_handler:
                 self.log_cli_handler.set_when("collection")
@@ -513,7 +557,6 @@ class LoggingPlugin(object):
 
     @pytest.hookimpl(hookwrapper=True, tryfirst=True)
     def pytest_sessionstart(self):
-        self._setup_cli_logging()
         with self.live_logs_context():
             if self.log_cli_handler:
                 self.log_cli_handler.set_when("sessionstart")
@@ -533,46 +576,6 @@ class LoggingPlugin(object):
             else:
                 yield  # run all the tests
 
-    def _setup_cli_logging(self):
-        """Sets up the handler and logger for the Live Logs feature, if enabled."""
-        terminal_reporter = self._config.pluginmanager.get_plugin("terminalreporter")
-        if self._log_cli_enabled() and terminal_reporter is not None:
-            capture_manager = self._config.pluginmanager.get_plugin("capturemanager")
-            log_cli_handler = _LiveLoggingStreamHandler(
-                terminal_reporter, capture_manager
-            )
-            log_cli_format = get_option_ini(
-                self._config, "log_cli_format", "log_format"
-            )
-            log_cli_date_format = get_option_ini(
-                self._config, "log_cli_date_format", "log_date_format"
-            )
-            if (
-                self._config.option.color != "no"
-                and ColoredLevelFormatter.LEVELNAME_FMT_REGEX.search(log_cli_format)
-            ):
-                log_cli_formatter = ColoredLevelFormatter(
-                    create_terminal_writer(self._config),
-                    log_cli_format,
-                    datefmt=log_cli_date_format,
-                )
-            else:
-                log_cli_formatter = logging.Formatter(
-                    log_cli_format, datefmt=log_cli_date_format
-                )
-            log_cli_level = get_actual_log_level(
-                self._config, "log_cli_level", "log_level"
-            )
-            self.log_cli_handler = log_cli_handler
-            self.live_logs_context = lambda: catching_logs(
-                log_cli_handler, formatter=log_cli_formatter, level=log_cli_level
-            )
-        else:
-            self.live_logs_context = lambda: dummy_context_manager()
-        # Note that the lambda for the live_logs_context is needed because
-        # live_logs_context can otherwise not be entered multiple times due
-        # to limitations of contextlib.contextmanager
-
 
 class _LiveLoggingStreamHandler(logging.StreamHandler):
     """

src/_pytest/main.py

@@ -607,6 +607,7 @@ class Session(nodes.FSCollector):
                 yield y
 
     def _collectfile(self, path, handle_dupes=True):
+        assert path.isfile()
         ihook = self.gethookproxy(path)
         if not self.isinitpath(path):
             if ihook.pytest_ignore_collect(path=path, config=self.config):

src/_pytest/python.py

@@ -599,6 +599,7 @@ class Package(Module):
         return proxy
 
     def _collectfile(self, path, handle_dupes=True):
+        assert path.isfile()
         ihook = self.gethookproxy(path)
         if not self.isinitpath(path):
             if ihook.pytest_ignore_collect(path=path, config=self.config):
@@ -642,9 +643,10 @@ class Package(Module):
             ):
                 continue
 
-            if path.isdir() and path.join("__init__.py").check(file=1):
+            if path.isdir():
+                if path.join("__init__.py").check(file=1):
                     pkg_prefixes.add(path)
-
+            else:
                 for x in self._collectfile(path):
                     yield x
 
@@ -1144,9 +1146,10 @@ def _find_parametrized_scope(argnames, arg2fixturedefs, indirect):
 
 def _idval(val, argname, idx, idfn, item, config):
     if idfn:
-        s = None
         try:
-            s = idfn(val)
+            generated_id = idfn(val)
+            if generated_id is not None:
+                val = generated_id
         except Exception as e:
             # See issue https://github.com/pytest-dev/pytest/issues/2169
             msg = "{}: error raised while trying to determine id of parameter '{}' at position {}\n"
@@ -1154,10 +1157,7 @@ def _idval(val, argname, idx, idfn, item, config):
             # we only append the exception type and message because on Python 2 reraise does nothing
             msg += "  {}: {}\n".format(type(e).__name__, e)
             six.raise_from(ValueError(msg), e)
-        if s:
-            return ascii_escaped(s)
-
-    if config:
+    elif config:
         hook_id = config.hook.pytest_make_parametrize_id(
             config=config, val=val, argname=argname
         )

src/_pytest/python_api.py

@@ -621,6 +621,14 @@ def raises(expected_exception, *args, **kwargs):
            ...
            >>> assert exc_info.type is ValueError
 
+    **Using with** ``pytest.mark.parametrize``
+
+    When using :ref:`pytest.mark.parametrize ref`
+    it is possible to parametrize tests such that
+    some runs raise an exception and others do not.
+
+    See :ref:`parametrizing_conditional_raising` for an example.
+
     **Legacy form**
 
     It is possible to specify a callable by passing a to-be-called lambda::

src/_pytest/terminal.py

@@ -574,19 +574,20 @@ class TerminalReporter(object):
         return lines
 
     def pytest_collection_finish(self, session):
-        if self.config.option.collectonly:
+        if self.config.getoption("collectonly"):
             self._printcollecteditems(session.items)
-            if self.stats.get("failed"):
-                self._tw.sep("!", "collection failures")
-                for rep in self.stats.get("failed"):
-                    rep.toterminal(self._tw)
-                return 1
-            return 0
+
         lines = self.config.hook.pytest_report_collectionfinish(
             config=self.config, startdir=self.startdir, items=session.items
         )
         self._write_report_lines_from_hooks(lines)
 
+        if self.config.getoption("collectonly"):
+            if self.stats.get("failed"):
+                self._tw.sep("!", "collection failures")
+                for rep in self.stats.get("failed"):
+                    rep.toterminal(self._tw)
+
     def _printcollecteditems(self, items):
         # to print out items and their parent collectors
         # we take care to leave out Instances aka ()

src/_pytest/unittest.py

@@ -87,6 +87,9 @@ def _make_xunit_fixture(obj, setup_name, teardown_name, scope, pass_self):
 
     @pytest.fixture(scope=scope, autouse=True)
     def fixture(self, request):
+        if getattr(self, "__unittest_skip__", None):
+            reason = self.__unittest_skip_why__
+            pytest.skip(reason)
         if setup is not None:
             if pass_self:
                 setup(self, request.function)

testing/acceptance_test.py

@@ -969,6 +969,20 @@ def test_import_plugin_unicode_name(testdir):
     assert r.ret == 0
 
 
+def test_pytest_plugins_as_module(testdir):
+    """Do not raise an error if pytest_plugins attribute is a module (#3899)"""
+    testdir.makepyfile(
+        **{
+            "__init__.py": "",
+            "pytest_plugins.py": "",
+            "conftest.py": "from . import pytest_plugins",
+            "test_foo.py": "def test(): pass",
+        }
+    )
+    result = testdir.runpytest()
+    result.stdout.fnmatch_lines("* 1 passed in *")
+
+
 def test_deferred_hook_checking(testdir):
     """
     Check hooks as late as possible (#1821).

testing/example_scripts/unittest/test_setup_skip.py

@@ -0,0 +1,13 @@
+"""Skipping an entire subclass with unittest.skip() should *not* call setUp from a base class."""
+import unittest
+
+
+class Base(unittest.TestCase):
+    def setUp(self):
+        assert 0
+
+
+@unittest.skip("skip all tests")
+class Test(Base):
+    def test_foo(self):
+        assert 0

testing/example_scripts/unittest/test_setup_skip_class.py

@@ -0,0 +1,14 @@
+"""Skipping an entire subclass with unittest.skip() should *not* call setUpClass from a base class."""
+import unittest
+
+
+class Base(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        assert 0
+
+
+@unittest.skip("skip all tests")
+class Test(Base):
+    def test_foo(self):
+        assert 0

testing/example_scripts/unittest/test_setup_skip_module.py

@@ -0,0 +1,12 @@
+"""setUpModule is always called, even if all tests in the module are skipped"""
+import unittest
+
+
+def setUpModule():
+    assert 0
+
+
+@unittest.skip("skip all tests")
+class Base(unittest.TestCase):
+    def test(self):
+        assert 0

testing/python/metafunc.py

@@ -418,6 +418,21 @@ class TestMetafunc(object):
             ]
         )
 
+    def test_parametrize_ids_returns_non_string(self, testdir):
+        testdir.makepyfile(
+            """\
+            import pytest
+
+            def ids(d):
+                return d
+
+            @pytest.mark.parametrize("arg", ({1: 2}, {3, 4}), ids=ids)
+            def test(arg):
+                assert arg
+            """
+        )
+        assert testdir.runpytest().ret == 0
+
     def test_idmaker_with_ids(self):
         from _pytest.python import idmaker
 

testing/python/raises.py

@@ -94,6 +94,54 @@ class TestRaises(object):
         result = testdir.runpytest()
         result.stdout.fnmatch_lines(["*3 passed*"])
 
+    def test_does_not_raise(self, testdir):
+        testdir.makepyfile(
+            """
+            from contextlib import contextmanager
+            import pytest
+
+            @contextmanager
+            def does_not_raise():
+                yield
+
+            @pytest.mark.parametrize('example_input,expectation', [
+                (3, does_not_raise()),
+                (2, does_not_raise()),
+                (1, does_not_raise()),
+                (0, pytest.raises(ZeroDivisionError)),
+            ])
+            def test_division(example_input, expectation):
+                '''Test how much I know division.'''
+                with expectation:
+                    assert (6 / example_input) is not None
+        """
+        )
+        result = testdir.runpytest()
+        result.stdout.fnmatch_lines(["*4 passed*"])
+
+    def test_does_not_raise_does_raise(self, testdir):
+        testdir.makepyfile(
+            """
+            from contextlib import contextmanager
+            import pytest
+
+            @contextmanager
+            def does_not_raise():
+                yield
+
+            @pytest.mark.parametrize('example_input,expectation', [
+                (0, does_not_raise()),
+                (1, pytest.raises(ZeroDivisionError)),
+            ])
+            def test_division(example_input, expectation):
+                '''Test how much I know division.'''
+                with expectation:
+                    assert (6 / example_input) is not None
+        """
+        )
+        result = testdir.runpytest()
+        result.stdout.fnmatch_lines(["*2 failed*"])
+
     def test_noclass(self):
         with pytest.raises(TypeError):
             pytest.raises("wrong", lambda: None)

testing/test_assertrewrite.py

@@ -68,38 +68,16 @@ def getmsg(f, extra_ns=None, must_pass=False):
             pytest.fail("function didn't raise at all")
 
 
-def adjust_body_for_new_docstring_in_module_node(m):
-    """Module docstrings in 3.8 are part of Module node.
-    This was briefly in 3.7 as well but got reverted in beta 5.
-
-    It's not in the body so we remove it so the following body items have
-    the same indexes on all Python versions:
-
-    TODO:
-
-    We have a complicated sys.version_info if in here to ease testing on
-    various Python 3.7 versions, but we should remove the 3.7 check after
-    3.7 is released as stable to make this check more straightforward.
-    """
-    if sys.version_info < (3, 8) and not (
-        (3, 7) <= sys.version_info <= (3, 7, 0, "beta", 4)
-    ):
-        assert len(m.body) > 1
-        assert isinstance(m.body[0], ast.Expr)
-        assert isinstance(m.body[0].value, ast.Str)
-        del m.body[0]
-
-
 class TestAssertionRewrite(object):
     def test_place_initial_imports(self):
         s = """'Doc string'\nother = stuff"""
         m = rewrite(s)
-        adjust_body_for_new_docstring_in_module_node(m)
-        for imp in m.body[0:2]:
+        assert isinstance(m.body[0], ast.Expr)
+        for imp in m.body[1:3]:
             assert isinstance(imp, ast.Import)
             assert imp.lineno == 2
             assert imp.col_offset == 0
-        assert isinstance(m.body[2], ast.Assign)
+        assert isinstance(m.body[3], ast.Assign)
         s = """from __future__ import division\nother_stuff"""
         m = rewrite(s)
         assert isinstance(m.body[0], ast.ImportFrom)
@@ -110,24 +88,24 @@ class TestAssertionRewrite(object):
         assert isinstance(m.body[3], ast.Expr)
         s = """'doc string'\nfrom __future__ import division"""
         m = rewrite(s)
-        adjust_body_for_new_docstring_in_module_node(m)
-        assert isinstance(m.body[0], ast.ImportFrom)
-        for imp in m.body[1:3]:
+        assert isinstance(m.body[0], ast.Expr)
+        assert isinstance(m.body[1], ast.ImportFrom)
+        for imp in m.body[2:4]:
             assert isinstance(imp, ast.Import)
             assert imp.lineno == 2
             assert imp.col_offset == 0
         s = """'doc string'\nfrom __future__ import division\nother"""
         m = rewrite(s)
-        adjust_body_for_new_docstring_in_module_node(m)
-        assert isinstance(m.body[0], ast.ImportFrom)
-        for imp in m.body[1:3]:
+        assert isinstance(m.body[0], ast.Expr)
+        assert isinstance(m.body[1], ast.ImportFrom)
+        for imp in m.body[2:4]:
             assert isinstance(imp, ast.Import)
             assert imp.lineno == 3
             assert imp.col_offset == 0
-        assert isinstance(m.body[3], ast.Expr)
+        assert isinstance(m.body[4], ast.Expr)
         s = """from . import relative\nother_stuff"""
         m = rewrite(s)
-        for imp in m.body[0:2]:
+        for imp in m.body[:2]:
             assert isinstance(imp, ast.Import)
             assert imp.lineno == 1
             assert imp.col_offset == 0
@@ -136,9 +114,8 @@ class TestAssertionRewrite(object):
     def test_dont_rewrite(self):
         s = """'PYTEST_DONT_REWRITE'\nassert 14"""
         m = rewrite(s)
-        adjust_body_for_new_docstring_in_module_node(m)
-        assert len(m.body) == 1
-        assert m.body[0].msg is None
+        assert len(m.body) == 2
+        assert m.body[1].msg is None
 
     def test_dont_rewrite_plugin(self, testdir):
         contents = {

testing/test_collection.py

@@ -1144,3 +1144,16 @@ def test_collect_symlink_out_of_tree(testdir):
         ]
     )
     assert result.ret == 0
+
+
+def test_collectignore_via_conftest(testdir, monkeypatch):
+    """collect_ignore in parent conftest skips importing child (issue #4592)."""
+    tests = testdir.mkpydir("tests")
+    tests.ensure("conftest.py").write("collect_ignore = ['ignore_me']")
+
+    ignore_me = tests.mkdir("ignore_me")
+    ignore_me.ensure("__init__.py")
+    ignore_me.ensure("conftest.py").write("assert 0, 'should_not_be_called'")
+
+    result = testdir.runpytest()
+    assert result.ret == EXIT_NOTESTSCOLLECTED

testing/test_terminal.py

@@ -649,7 +649,10 @@ class TestTerminalFunctional(object):
         assert "===" not in s
         assert "passed" not in s
 
-    def test_report_collectionfinish_hook(self, testdir):
+    @pytest.mark.parametrize(
+        "params", [(), ("--collect-only",)], ids=["no-params", "collect-only"]
+    )
+    def test_report_collectionfinish_hook(self, testdir, params):
         testdir.makeconftest(
             """
             def pytest_report_collectionfinish(config, startdir, items):
@@ -664,7 +667,7 @@ class TestTerminalFunctional(object):
                 pass
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*params)
         result.stdout.fnmatch_lines(["collected 3 items", "hello from hook: 3 items"])
 
 

testing/test_unittest.py

@@ -1026,3 +1026,18 @@ def test_error_message_with_parametrized_fixtures(testdir):
             "*Function type: TestCaseFunction",
         ]
     )
+
+
+@pytest.mark.parametrize(
+    "test_name, expected_outcome",
+    [
+        ("test_setup_skip.py", "1 skipped"),
+        ("test_setup_skip_class.py", "1 skipped"),
+        ("test_setup_skip_module.py", "1 error"),
+    ],
+)
+def test_setup_inheritance_skipping(testdir, test_name, expected_outcome):
+    """Issue #4700"""
+    testdir.copy_example("unittest/{}".format(test_name))
+    result = testdir.runpytest()
+    result.stdout.fnmatch_lines("* {} in *".format(expected_outcome))

tox.ini

@@ -1,5 +1,6 @@
 [tox]
-minversion = 2.0
+isolated_build = True
+minversion = 3.3
 distshare = {homedir}/.tox/distshare
 # make sure to update environment list in travis.yml and appveyor.yml
 envlist =
@@ -9,6 +10,7 @@ envlist =
     py35
     py36
     py37
+    py38
     pypy
     {py27,py37}-{pexpect,xdist,trial,numpy,pluggymaster}
     py27-nobyte
@@ -165,7 +167,9 @@ commands =
 
 [testenv:py37-freeze]
 changedir = testing/freeze
+# Disable PEP 517 with pip, which does not work with PyInstaller currently.
 deps =
+    --no-use-pep517
     pyinstaller
 commands =
     {envpython} create_executable.py