Merge remote-tracking branch 'upstream/features' into merge-features-into-master

AUTHORS

@@ -46,6 +46,7 @@ Christian Boelsen
 Christian Theunert
 Christian Tismer
 Christopher Gilling
+CrazyMerlyn
 Cyrus Maden
 Dhiren Serai
 Daniel Grana
@@ -212,10 +213,12 @@ Vasily Kuznetsov
 Victor Maryama
 Victor Uriarte
 Vidar T. Fauske
+Virgil Dupras
 Vitaly Lashmanov
 Vlad Dragos
 Wil Cooley
 William Lee
+Wim Glenn
 Wouter van Ackooy
 Xuan Luong
 Xuecong Liao

CHANGELOG.rst

@@ -18,6 +18,90 @@ with advance notice in the **Deprecations** section of releases.
 
 .. towncrier release notes start
 
+pytest 3.8.0 (2018-09-05)
+=========================
+
+Deprecations and Removals
+-------------------------
+
+- `#2452 <https://github.com/pytest-dev/pytest/issues/2452>`_: ``Config.warn`` has been deprecated, it should be replaced by calls to the standard ``warnings.warn``.
+
+  ``Node.warn`` now supports two signatures:
+
+  * ``node.warn(PytestWarning("some message"))``: is now the recommended way to call this function. The warning
+    instance must be a ``PytestWarning`` or subclass  instance.
+
+  * ``node.warn("CI", "some message")``: this code/message form is now deprecated and should be converted to
+    the warning instance form above.
+
+  ``RemovedInPytest4Warning`` and ``PytestExperimentalApiWarning`` are now part of the public API and should be accessed
+  using ``pytest.RemovedInPytest4Warning`` and ``pytest.PytestExperimentalApiWarning``.
+
+
+- `#3936 <https://github.com/pytest-dev/pytest/issues/3936>`_: ``@pytest.mark.filterwarnings`` second parameter is no longer regex-escaped,
+  making it possible to actually use regular expressions to check the warning message.
+
+  **Note**: regex-escaping the match string was an implementation oversight that might break test suites which depend
+  on the old behavior.
+
+
+
+Features
+--------
+
+- `#2452 <https://github.com/pytest-dev/pytest/issues/2452>`_: Internal pytest warnings are now issued using the standard ``warnings`` module, making it possible to use
+  the standard warnings filters to manage those warnings. This introduces ``PytestWarning``,
+  ``PytestDeprecationWarning`` and ``RemovedInPytest4Warning`` warning types as part of the public API.
+
+  Consult `the documentation <https://docs.pytest.org/en/latest/warnings.html#internal-pytest-warnings>`_ for more info.
+
+
+- `#2908 <https://github.com/pytest-dev/pytest/issues/2908>`_: ``DeprecationWarning`` and ``PendingDeprecationWarning`` are now shown by default if no other warning filter is
+  configured. This makes pytest more compliant with
+  `PEP-0506 <https://www.python.org/dev/peps/pep-0565/#recommended-filter-settings-for-test-runners>`_. See
+  `the docs <https://docs.pytest.org/en/latest/warnings.html#deprecationwarning-and-pendingdeprecationwarning>`_ for
+  more info.
+
+
+- `#3784 <https://github.com/pytest-dev/pytest/issues/3784>`_: Add option to disable plugin auto-loading.
+
+
+- `#3829 <https://github.com/pytest-dev/pytest/issues/3829>`_: Added the ``count`` option to ``console_output_style`` to enable displaying the progress as a count instead of a percentage.
+
+
+- `#3837 <https://github.com/pytest-dev/pytest/issues/3837>`_: Added support for 'xfailed' and 'xpassed' outcomes to the ``pytester.RunResult.assert_outcomes`` signature.
+
+
+
+Bug Fixes
+---------
+
+- `#3911 <https://github.com/pytest-dev/pytest/issues/3911>`_: Terminal writer now takes into account unicode character width when writing out progress.
+
+
+- `#3913 <https://github.com/pytest-dev/pytest/issues/3913>`_: Pytest now returns with correct exit code (EXIT_USAGEERROR, 4) when called with unknown arguments.
+
+
+- `#3918 <https://github.com/pytest-dev/pytest/issues/3918>`_: Improve performance of assertion rewriting.
+
+
+
+Improved Documentation
+----------------------
+
+- `#3566 <https://github.com/pytest-dev/pytest/issues/3566>`_: Added a blurb in usage.rst for the usage of -r flag which is used to show an extra test summary info.
+
+
+- `#3907 <https://github.com/pytest-dev/pytest/issues/3907>`_: Corrected type of the exceptions collection passed to ``xfail``: ``raises`` argument accepts a ``tuple`` instead of ``list``.
+
+
+
+Trivial/Internal Changes
+------------------------
+
+- `#3853 <https://github.com/pytest-dev/pytest/issues/3853>`_: Removed ``"run all (no recorded failures)"`` message printed with ``--failed-first`` and ``--last-failed`` when there are no failed tests.
+
+
 pytest 3.7.4 (2018-08-29)
 =========================
 

changelog/3251.feture.rst

@@ -0,0 +1 @@
+Warnings are now captured and displayed during test collection.

changelog/3566.doc.rst

@@ -1 +0,0 @@
-Added a blurb in usage.rst for the usage of -r flag which is used to show an extra test summary info.

changelog/3907.doc.rst

@@ -1 +0,0 @@
-Corrected type of the exceptions collection passed to ``xfail``: ``raises`` argument accepts a ``tuple`` instead of ``list``.

changelog/3911.bugfix.rst

@@ -1 +0,0 @@
-Terminal writer now takes into account unicode character width when writing out progress.

changelog/3918.bugfix.rst

@@ -1 +0,0 @@
-Improve performance of assertion rewriting.

doc/en/announce/index.rst

@@ -6,6 +6,7 @@ Release announcements
    :maxdepth: 2
 
 
+   release-3.8.0
    release-3.7.4
    release-3.7.3
    release-3.7.2

doc/en/announce/release-3.8.0.rst

@@ -0,0 +1,38 @@
+pytest-3.8.0
+=======================================
+
+The pytest team is proud to announce the 3.8.0 release!
+
+pytest is a mature Python testing tool with more than a 2000 tests
+against itself, passing on many different interpreters and platforms.
+
+This release contains a number of bugs fixes and improvements, so users are encouraged
+to take a look at the CHANGELOG:
+
+    https://docs.pytest.org/en/latest/changelog.html
+
+For complete documentation, please visit:
+
+    https://docs.pytest.org/en/latest/
+
+As usual, you can upgrade from pypi via:
+
+    pip install -U pytest
+
+Thanks to all who contributed to this release, among them:
+
+* Anthony Sottile
+* Bruno Oliveira
+* CrazyMerlyn
+* Daniel Hahler
+* Fabio Zadrozny
+* Jeffrey Rackauckas
+* Ronny Pfannschmidt
+* Virgil Dupras
+* dhirensr
+* hoefling
+* wim glenn
+
+
+Happy testing,
+The Pytest Development Team

doc/en/example/reportingdemo.rst

@@ -613,9 +613,9 @@ get on the terminal - we are working on that)::
 
     failure_demo.py:261: AssertionError
     ============================= warnings summary =============================
-    <undetermined location>
-      Metafunc.addcall is deprecated and scheduled to be removed in pytest 4.0.
-      Please use Metafunc.parametrize instead.
+    $REGENDOC_TMPDIR/assertion/failure_demo.py:24: RemovedInPytest4Warning: Metafunc.addcall is deprecated and scheduled to be removed in pytest 4.0.
+    Please use Metafunc.parametrize instead.
+      metafunc.addcall(funcargs=dict(param1=3, param2=6))
 
     -- Docs: https://docs.pytest.org/en/latest/warnings.html
     ================== 42 failed, 1 warnings in 0.12 seconds ===================

doc/en/reference.rst

@@ -611,6 +611,8 @@ Session related reporting hooks:
 .. autofunction:: pytest_terminal_summary
 .. autofunction:: pytest_fixture_setup
 .. autofunction:: pytest_fixture_post_finalizer
+.. autofunction:: pytest_logwarning
+.. autofunction:: pytest_warning_captured
 
 And here is the central hook for reporting about
 test execution:
@@ -866,6 +868,11 @@ Contains comma-separated list of modules that should be loaded as plugins:
 
     export PYTEST_PLUGINS=mymodule.plugin,xdist
 
+PYTEST_DISABLE_PLUGIN_AUTOLOAD
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When set, disables plugin auto-loading through setuptools entrypoints. Only explicitly specified plugins will be
+loaded.
 
 PYTEST_CURRENT_TEST
 ~~~~~~~~~~~~~~~~~~~
@@ -935,6 +942,7 @@ passed multiple times. The expected format is ``name=value``. For example::
 
    * ``classic``: classic pytest output.
    * ``progress``: like classic pytest output, but with a progress indicator.
+   * ``count``: like progress, but shows progress as the number of tests completed instead of a percent.
 
    The default is ``progress``, but you can fallback to ``classic`` if you prefer or
    the new mode is causing unexpected problems:

doc/en/usage.rst

@@ -153,15 +153,12 @@ making it easy in large test suites to get a clear picture of all failures, skip
 Example::
 
     $ pytest -ra
-    ======================== test session starts ========================
-    ...
-    ====================== short test summary info ======================
-    FAIL summary\test_foo.py::test_1
-    SKIP [1] summary\test_foo.py:12: not supported in this platform
-    XPASS summary\test_bar.py::test_4 flaky
-
-    ===== 1 failed, 1 passed, 1 skipped, 1 xpassed in 0.08 seconds ======
+    =========================== test session starts ============================
+    platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
+    rootdir: $REGENDOC_TMPDIR, inifile:
+    collected 0 items
 
+    ======================= no tests ran in 0.12 seconds =======================
 
 The ``-r`` options accepts a number of characters after it, with ``a`` used above meaning "all except passes".
 
@@ -179,8 +176,12 @@ Here is the full list of available characters that can be used:
 More than one character can be used, so for example to only see failed and skipped tests, you can execute::
 
     $ pytest -rfs
+    =========================== test session starts ============================
+    platform linux -- Python 3.x.y, pytest-3.x.y, py-1.x.y, pluggy-0.x.y
+    rootdir: $REGENDOC_TMPDIR, inifile:
+    collected 0 items
 
-
+    ======================= no tests ran in 0.12 seconds =======================
 
 .. _pdb-option:
 

doc/en/warnings.rst

@@ -29,15 +29,12 @@ Running pytest now produces this output::
     test_show_warnings.py .                                              [100%]
 
     ============================= warnings summary =============================
-    test_show_warnings.py::test_one
-      $REGENDOC_TMPDIR/test_show_warnings.py:4: UserWarning: api v1, should use functions from v2
-        warnings.warn(UserWarning("api v1, should use functions from v2"))
+    $REGENDOC_TMPDIR/test_show_warnings.py:4: UserWarning: api v1, should use functions from v2
+      warnings.warn(UserWarning("api v1, should use functions from v2"))
 
     -- Docs: https://docs.pytest.org/en/latest/warnings.html
     =================== 1 passed, 1 warnings in 0.12 seconds ===================
 
-Pytest by default catches all warnings except for ``DeprecationWarning`` and ``PendingDeprecationWarning``.
-
 The ``-W`` flag can be passed to control which warnings will be displayed or even turn
 them into errors::
 
@@ -78,6 +75,53 @@ Both ``-W`` command-line option and ``filterwarnings`` ini option are based on P
 `-W option`_ and `warnings.simplefilter`_, so please refer to those sections in the Python
 documentation for other examples and advanced usage.
 
+Disabling warning summary
+-------------------------
+
+Although not recommended, you can use the ``--disable-warnings`` command-line option to suppress the
+warning summary entirely from the test run output.
+
+Disabling warning capture entirely
+----------------------------------
+
+This plugin is enabled by default but can be disabled entirely in your ``pytest.ini`` file with:
+
+    .. code-block:: ini
+
+        [pytest]
+        addopts = -p no:warnings
+
+Or passing ``-p no:warnings`` in the command-line. This might be useful if your test suites handles warnings
+using an external system.
+
+
+.. _`deprecation-warnings`:
+
+DeprecationWarning and PendingDeprecationWarning
+------------------------------------------------
+
+.. versionadded:: 3.8
+
+By default pytest will display ``DeprecationWarning`` and ``PendingDeprecationWarning`` if no other warning filters
+are configured.
+
+To disable showing ``DeprecationWarning`` and ``PendingDeprecationWarning`` warnings, you might define any warnings
+filter either in the command-line or in the ini file, or you can use:
+
+.. code-block:: ini
+
+    [pytest]
+    filterwarnings =
+        ignore::DeprecationWarning
+        ignore::PendingDeprecationWarning
+
+.. note::
+    This makes pytest more compliant with `PEP-0506 <https://www.python.org/dev/peps/pep-0565/#recommended-filter-settings-for-test-runners>`_ which suggests that those warnings should
+    be shown by default by test runners, but pytest doesn't follow ``PEP-0506`` completely because resetting all
+    warning filters like suggested in the PEP will break existing test suites that configure warning filters themselves
+    by calling ``warnings.simplefilter`` (see issue `#2430 <https://github.com/pytest-dev/pytest/issues/2430>`_
+    for an example of that).
+
 
 .. _`filterwarnings`:
 
@@ -144,18 +188,6 @@ decorator or to all tests in a module by setting the ``pytestmark`` variable:
 .. _`pytest-warnings`: https://github.com/fschulze/pytest-warnings
 
 
-Disabling warning capture
--------------------------
-
-This feature is enabled by default but can be disabled entirely in your ``pytest.ini`` file with:
-
-    .. code-block:: ini
-
-        [pytest]
-        addopts = -p no:warnings
-
-Or passing ``-p no:warnings`` in the command-line.
-
 .. _`asserting warnings`:
 
 .. _assertwarnings:
@@ -296,3 +328,51 @@ You can also use it as a contextmanager::
     def test_global():
         with pytest.deprecated_call():
             myobject.deprecated_method()
+
+
+Internal pytest warnings
+------------------------
+
+.. versionadded:: 3.8
+
+pytest may generate its own warnings in some situations, such as improper usage or deprecated features.
+
+For example, pytest will emit a warning if it encounters a class that matches :confval:`python_classes` but also
+defines an ``__init__`` constructor, as this prevents the class from being instantiated:
+
+.. code-block:: python
+
+    # content of test_pytest_warnings.py
+    class Test:
+        def __init__(self):
+            pass
+
+        def test_foo(self):
+            assert 1 == 1
+
+::
+
+    $ pytest test_pytest_warnings.py -q
+
+    ============================= warnings summary =============================
+    $REGENDOC_TMPDIR/test_pytest_warnings.py:1: PytestWarning: cannot collect test class 'Test' because it has a __init__ constructor
+      class Test:
+
+    -- Docs: https://docs.pytest.org/en/latest/warnings.html
+    1 warnings in 0.12 seconds
+
+These warnings might be filtered using the same builtin mechanisms used to filter other types of warnings.
+
+Following our :ref:`backwards-compatibility`, deprecated features will be kept *at least* two minor releases. After that,
+they will changed so they by default raise errors instead of just warnings, so users can adapt to it on their own time
+if not having done so until now. In a later release the deprecated feature will be removed completely.
+
+The following warning types ares used by pytest and are part of the public API:
+
+.. autoclass:: pytest.PytestWarning
+
+.. autoclass:: pytest.PytestDeprecationWarning
+
+.. autoclass:: pytest.RemovedInPytest4Warning
+
+.. autoclass:: pytest.PytestExperimentalApiWarning

doc/en/writing_plugins.rst

@@ -418,9 +418,8 @@ additionally it is possible to copy examples for a example folder before running
     test_example.py ..                                                   [100%]
 
     ============================= warnings summary =============================
-    test_example.py::test_plugin
-      $REGENDOC_TMPDIR/test_example.py:4: PytestExerimentalApiWarning: testdir.copy_example is an experimental api that may change over time
-        testdir.copy_example("test_example.py")
+    $REGENDOC_TMPDIR/test_example.py:4: PytestExperimentalApiWarning: testdir.copy_example is an experimental api that may change over time
+      testdir.copy_example("test_example.py")
 
     -- Docs: https://docs.pytest.org/en/latest/warnings.html
     =================== 2 passed, 1 warnings in 0.12 seconds ===================

src/_pytest/assertion/rewrite.py

@@ -266,8 +266,12 @@ class AssertionRewritingHook(object):
         self._marked_for_rewrite_cache.clear()
 
     def _warn_already_imported(self, name):
-        self.config.warn(
-            "P1", "Module already imported so cannot be rewritten: %s" % name
+        from _pytest.warning_types import PytestWarning
+        from _pytest.warnings import _issue_config_warning
+
+        _issue_config_warning(
+            PytestWarning("Module already imported so cannot be rewritten: %s" % name),
+            self.config,
         )
 
     def load_module(self, name):
@@ -803,13 +807,17 @@ class AssertionRewriter(ast.NodeVisitor):
         the expression is false.
 
         """
-        if isinstance(assert_.test, ast.Tuple) and self.config is not None:
-            fslocation = (self.module_path, assert_.lineno)
-            self.config.warn(
-                "R1",
-                "assertion is always true, perhaps " "remove parentheses?",
-                fslocation=fslocation,
+        if isinstance(assert_.test, ast.Tuple) and len(assert_.test.elts) >= 1:
+            from _pytest.warning_types import PytestWarning
+            import warnings
+
+            warnings.warn_explicit(
+                PytestWarning("assertion is always true, perhaps remove parentheses?"),
+                category=None,
+                filename=str(self.module_path),
+                lineno=assert_.lineno,
             )
+
         self.statements = []
         self.variables = []
         self.variable_counter = itertools.count()

src/_pytest/cacheprovider.py

@@ -33,7 +33,7 @@ See [the docs](https://docs.pytest.org/en/latest/cache.html) for more informatio
 @attr.s
 class Cache(object):
     _cachedir = attr.ib(repr=False)
-    _warn = attr.ib(repr=False)
+    _config = attr.ib(repr=False)
 
     @classmethod
     def for_config(cls, config):
@@ -41,14 +41,19 @@ class Cache(object):
         if config.getoption("cacheclear") and cachedir.exists():
             shutil.rmtree(str(cachedir))
             cachedir.mkdir()
-        return cls(cachedir, config.warn)
+        return cls(cachedir, config)
 
     @staticmethod
     def cache_dir_from_config(config):
         return paths.resolve_from_str(config.getini("cache_dir"), config.rootdir)
 
     def warn(self, fmt, **args):
-        self._warn(code="I9", message=fmt.format(**args) if args else fmt)
+        from _pytest.warnings import _issue_config_warning
+        from _pytest.warning_types import PytestWarning
+
+        _issue_config_warning(
+            PytestWarning(fmt.format(**args) if args else fmt), self._config
+        )
 
     def makedir(self, name):
         """ return a directory path object with the given name.  If the
@@ -134,15 +139,12 @@ class LFPlugin(object):
     def pytest_report_collectionfinish(self):
         if self.active and self.config.getoption("verbose") >= 0:
             if not self._previously_failed_count:
-                mode = "run {} (no recorded failures)".format(
-                    self._no_failures_behavior
-                )
-            else:
-                noun = "failure" if self._previously_failed_count == 1 else "failures"
-                suffix = " first" if self.config.getoption("failedfirst") else ""
-                mode = "rerun previous {count} {noun}{suffix}".format(
-                    count=self._previously_failed_count, suffix=suffix, noun=noun
-                )
+                return None
+            noun = "failure" if self._previously_failed_count == 1 else "failures"
+            suffix = " first" if self.config.getoption("failedfirst") else ""
+            mode = "rerun previous {count} {noun}{suffix}".format(
+                count=self._previously_failed_count, suffix=suffix, noun=noun
+            )
             return "run-last-failure: %s" % mode
 
     def pytest_runtest_logreport(self, report):

src/_pytest/config/__init__.py

@@ -178,7 +178,9 @@ def _prepareconfig(args=None, plugins=None):
                 else:
                     pluginmanager.register(plugin)
         if warning:
-            config.warn("C1", warning)
+            from _pytest.warnings import _issue_config_warning
+
+            _issue_config_warning(warning, config=config)
         return pluginmanager.hook.pytest_cmdline_parse(
             pluginmanager=pluginmanager, args=args
         )
@@ -419,7 +421,12 @@ class PytestPluginManager(PluginManager):
                         PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST
                     )
 
-                    warnings.warn(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST)
+                    warnings.warn_explicit(
+                        PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST,
+                        category=None,
+                        filename=str(conftestpath),
+                        lineno=0,
+                    )
             except Exception:
                 raise ConftestImportFailure(conftestpath, sys.exc_info())
 
@@ -604,7 +611,29 @@ class Config(object):
             fin()
 
     def warn(self, code, message, fslocation=None, nodeid=None):
-        """ generate a warning for this test session. """
+        """
+        .. deprecated:: 3.8
+
+            Use :py:func:`warnings.warn` or :py:func:`warnings.warn_explicit` directly instead.
+
+        Generate a warning for this test session.
+        """
+        from _pytest.warning_types import RemovedInPytest4Warning
+
+        if isinstance(fslocation, (tuple, list)) and len(fslocation) > 2:
+            filename, lineno = fslocation[:2]
+        else:
+            filename = "unknown file"
+            lineno = 0
+        msg = "config.warn has been deprecated, use warnings.warn instead"
+        if nodeid:
+            msg = "{}: {}".format(nodeid, msg)
+        warnings.warn_explicit(
+            RemovedInPytest4Warning(msg),
+            category=None,
+            filename=filename,
+            lineno=lineno,
+        )
         self.hook.pytest_logwarning.call_historic(
             kwargs=dict(
                 code=code, message=message, fslocation=fslocation, nodeid=nodeid
@@ -669,8 +698,8 @@ class Config(object):
         r = determine_setup(
             ns.inifilename,
             ns.file_or_dir + unknown_args,
-            warnfunc=self.warn,
             rootdir_cmd_arg=ns.rootdir or None,
+            config=self,
         )
         self.rootdir, self.inifile, self.inicfg = r
         self._parser.extra_info["rootdir"] = self.rootdir
@@ -708,6 +737,10 @@ class Config(object):
 
         self.pluginmanager.rewrite_hook = hook
 
+        if os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
+            # We don't autoload from setuptools entry points, no need to continue.
+            return
+
         # 'RECORD' available for plugins installed normally (pip install)
         # 'SOURCES.txt' available for plugins installed in dev mode (pip install -e)
         # for installed plugins 'SOURCES.txt' returns an empty list, and vice-versa
@@ -733,7 +766,10 @@ class Config(object):
         self._checkversion()
         self._consider_importhook(args)
         self.pluginmanager.consider_preparse(args)
-        self.pluginmanager.load_setuptools_entrypoints("pytest11")
+        if not os.environ.get("PYTEST_DISABLE_PLUGIN_AUTOLOAD"):
+            # Don't autoload from setuptools entry point. Only explicitly specified
+            # plugins are going to be loaded.
+            self.pluginmanager.load_setuptools_entrypoints("pytest11")
         self.pluginmanager.consider_env()
         self.known_args_namespace = ns = self._parser.parse_known_args(
             args, namespace=copy.copy(self.option)

src/_pytest/config/argparsing.py

@@ -2,8 +2,13 @@ import six
 import warnings
 import argparse
 
+from gettext import gettext as _
+import sys as _sys
+
 import py
 
+from ..main import EXIT_USAGEERROR
+
 FILE_OR_DIR = "file_or_dir"
 
 
@@ -329,6 +334,16 @@ class MyOptionParser(argparse.ArgumentParser):
         # an usage error to provide more contextual information to the user
         self.extra_info = extra_info
 
+    def error(self, message):
+        """error(message: string)
+
+        Prints a usage message incorporating the message to stderr and
+        exits.
+        Overrides the method in parent class to change exit code"""
+        self.print_usage(_sys.stderr)
+        args = {"prog": self.prog, "message": message}
+        self.exit(EXIT_USAGEERROR, _("%(prog)s: error: %(message)s\n") % args)
+
     def parse_args(self, args=None, namespace=None):
         """allow splitting of positional arguments"""
         args, argv = self.parse_known_args(args, namespace)

src/_pytest/config/findpaths.py

@@ -10,15 +10,12 @@ def exists(path, ignore=EnvironmentError):
         return False
 
 
-def getcfg(args, warnfunc=None):
+def getcfg(args, config=None):
     """
     Search the list of arguments for a valid ini-file for pytest,
     and return a tuple of (rootdir, inifile, cfg-dict).
 
-    note: warnfunc is an optional function used to warn
-        about ini-files that use deprecated features.
-        This parameter should be removed when pytest
-        adopts standard deprecation warnings (#1804).
+    note: config is optional and used only to issue warnings explicitly (#2891).
     """
     from _pytest.deprecated import CFG_PYTEST_SECTION
 
@@ -34,9 +31,15 @@ def getcfg(args, warnfunc=None):
                 if exists(p):
                     iniconfig = py.iniconfig.IniConfig(p)
                     if "pytest" in iniconfig.sections:
-                        if inibasename == "setup.cfg" and warnfunc:
-                            warnfunc(
-                                "C1", CFG_PYTEST_SECTION.format(filename=inibasename)
+                        if inibasename == "setup.cfg" and config is not None:
+                            from _pytest.warnings import _issue_config_warning
+                            from _pytest.warning_types import RemovedInPytest4Warning
+
+                            _issue_config_warning(
+                                RemovedInPytest4Warning(
+                                    CFG_PYTEST_SECTION.format(filename=inibasename)
+                                ),
+                                config=config,
                             )
                         return base, p, iniconfig["pytest"]
                     if (
@@ -95,7 +98,7 @@ def get_dirs_from_args(args):
     return [get_dir_from_path(path) for path in possible_paths if path.exists()]
 
 
-def determine_setup(inifile, args, warnfunc=None, rootdir_cmd_arg=None):
+def determine_setup(inifile, args, rootdir_cmd_arg=None, config=None):
     dirs = get_dirs_from_args(args)
     if inifile:
         iniconfig = py.iniconfig.IniConfig(inifile)
@@ -105,23 +108,30 @@ def determine_setup(inifile, args, warnfunc=None, rootdir_cmd_arg=None):
         for section in sections:
             try:
                 inicfg = iniconfig[section]
-                if is_cfg_file and section == "pytest" and warnfunc:
+                if is_cfg_file and section == "pytest" and config is not None:
                     from _pytest.deprecated import CFG_PYTEST_SECTION
+                    from _pytest.warning_types import RemovedInPytest4Warning
+                    from _pytest.warnings import _issue_config_warning
 
-                    warnfunc("C1", CFG_PYTEST_SECTION.format(filename=str(inifile)))
+                    _issue_config_warning(
+                        RemovedInPytest4Warning(
+                            CFG_PYTEST_SECTION.format(filename=str(inifile))
+                        ),
+                        config,
+                    )
                 break
             except KeyError:
                 inicfg = None
         rootdir = get_common_ancestor(dirs)
     else:
         ancestor = get_common_ancestor(dirs)
-        rootdir, inifile, inicfg = getcfg([ancestor], warnfunc=warnfunc)
+        rootdir, inifile, inicfg = getcfg([ancestor], config=config)
         if rootdir is None:
             for rootdir in ancestor.parts(reverse=True):
                 if rootdir.join("setup.py").exists():
                     break
             else:
-                rootdir, inifile, inicfg = getcfg(dirs, warnfunc=warnfunc)
+                rootdir, inifile, inicfg = getcfg(dirs, config=config)
                 if rootdir is None:
                     rootdir = get_common_ancestor([py.path.local(), ancestor])
                     is_fs_root = os.path.splitdrive(str(rootdir))[1] == "/"

src/_pytest/deprecated.py

@@ -7,14 +7,16 @@ be removed when the time comes.
 """
 from __future__ import absolute_import, division, print_function
 
+from _pytest.warning_types import RemovedInPytest4Warning
 
-class RemovedInPytest4Warning(DeprecationWarning):
-    """warning class for features removed in pytest 4.0"""
+MAIN_STR_ARGS = RemovedInPytest4Warning(
+    "passing a string to pytest.main() is deprecated, "
+    "pass a list of arguments instead."
+)
 
-
-MAIN_STR_ARGS = "passing a string to pytest.main() is deprecated, " "pass a list of arguments instead."
-
-YIELD_TESTS = "yield tests are deprecated, and scheduled to be removed in pytest 4.0"
+YIELD_TESTS = RemovedInPytest4Warning(
+    "yield tests are deprecated, and scheduled to be removed in pytest 4.0"
+)
 
 FUNCARG_PREFIX = (
     '{name}: declaring fixtures using "pytest_funcarg__" prefix is deprecated '
@@ -23,7 +25,7 @@ FUNCARG_PREFIX = (
 )
 
 FIXTURE_FUNCTION_CALL = (
-    "Fixture {name} called directly. Fixtures are not meant to be called directly, "
+    'Fixture "{name}" called directly. Fixtures are not meant to be called directly, '
     "are created automatically when test functions request them as parameters. "
     "See https://docs.pytest.org/en/latest/fixture.html for more information."
 )
@@ -32,7 +34,7 @@ CFG_PYTEST_SECTION = (
     "[pytest] section in {filename} files is deprecated, use [tool:pytest] instead."
 )
 
-GETFUNCARGVALUE = "use of getfuncargvalue is deprecated, use getfixturevalue"
+GETFUNCARGVALUE = "getfuncargvalue is deprecated, use getfixturevalue"
 
 RESULT_LOG = (
     "--result-log is deprecated and scheduled for removal in pytest 4.0.\n"
@@ -51,7 +53,11 @@ MARK_PARAMETERSET_UNPACKING = RemovedInPytest4Warning(
     "For more details, see: https://docs.pytest.org/en/latest/parametrize.html"
 )
 
-RECORD_XML_PROPERTY = (
+NODE_WARN = RemovedInPytest4Warning(
+    "Node.warn(code, message) form has been deprecated, use Node.warn(warning_instance) instead."
+)
+
+RECORD_XML_PROPERTY = RemovedInPytest4Warning(
     'Fixture renamed from "record_xml_property" to "record_property" as user '
     "properties are now available to all reporters.\n"
     '"record_xml_property" is now deprecated.'
@@ -61,7 +67,7 @@ COLLECTOR_MAKEITEM = RemovedInPytest4Warning(
     "pycollector makeitem was removed " "as it is an accidentially leaked internal api"
 )
 
-METAFUNC_ADD_CALL = (
+METAFUNC_ADD_CALL = RemovedInPytest4Warning(
     "Metafunc.addcall is deprecated and scheduled to be removed in pytest 4.0.\n"
     "Please use Metafunc.parametrize instead."
 )

src/_pytest/experiments.py

@@ -1,13 +0,0 @@
-class PytestExerimentalApiWarning(FutureWarning):
-    "warning category used to denote experiments in pytest"
-
-    @classmethod
-    def simple(cls, apiname):
-        return cls(
-            "{apiname} is an experimental api that may change over time".format(
-                apiname=apiname
-            )
-        )
-
-
-PYTESTER_COPY_EXAMPLE = PytestExerimentalApiWarning.simple("testdir.copy_example")

src/_pytest/fixtures.py

@@ -1257,6 +1257,8 @@ class FixtureManager(object):
         items[:] = reorder_items(items)
 
     def parsefactories(self, node_or_obj, nodeid=NOTSET, unittest=False):
+        from _pytest import deprecated
+
         if nodeid is not NOTSET:
             holderobj = node_or_obj
         else:
@@ -1279,10 +1281,15 @@ class FixtureManager(object):
                 if not callable(obj):
                     continue
                 marker = defaultfuncargprefixmarker
-                from _pytest import deprecated
 
-                self.config.warn(
-                    "C1", deprecated.FUNCARG_PREFIX.format(name=name), nodeid=nodeid
+                filename, lineno = getfslineno(obj)
+                warnings.warn_explicit(
+                    RemovedInPytest4Warning(
+                        deprecated.FUNCARG_PREFIX.format(name=name)
+                    ),
+                    category=None,
+                    filename=str(filename),
+                    lineno=lineno + 1,
                 )
                 name = name[len(self._argprefix) :]
             elif not isinstance(marker, FixtureFunctionMarker):

src/_pytest/helpconfig.py

@@ -156,6 +156,7 @@ def showhelp(config):
     vars = [
         ("PYTEST_ADDOPTS", "extra command line options"),
         ("PYTEST_PLUGINS", "comma-separated plugins to load during startup"),
+        ("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "set to disable plugin auto-loading"),
         ("PYTEST_DEBUG", "set to enable debug tracing of pytest's internals"),
     ]
     for name, help in vars:

src/_pytest/hookspec.py

@@ -526,7 +526,17 @@ def pytest_terminal_summary(terminalreporter, exitstatus):
 
 @hookspec(historic=True)
 def pytest_logwarning(message, code, nodeid, fslocation):
-    """ process a warning specified by a message, a code string,
+    """
+    .. deprecated:: 3.8
+
+        This hook is will stop working in a future release.
+
+        pytest no longer triggers this hook, but the
+        terminal writer still implements it to display warnings issued by
+        :meth:`_pytest.config.Config.warn` and :meth:`_pytest.nodes.Node.warn`. Calling those functions will be
+        an error in future releases.
+
+    process a warning specified by a message, a code string,
     a nodeid and fslocation (both of which may be None
     if the warning is not tied to a particular node/location).
 
@@ -535,6 +545,27 @@ def pytest_logwarning(message, code, nodeid, fslocation):
     """
 
 
+@hookspec(historic=True)
+def pytest_warning_captured(warning_message, when, item):
+    """
+    Process a warning captured by the internal pytest warnings plugin.
+
+    :param warnings.WarningMessage warning_message:
+        The captured warning. This is the same object produced by :py:func:`warnings.catch_warnings`, and contains
+        the same attributes as the parameters of :py:func:`warnings.showwarning`.
+
+    :param str when:
+        Indicates when the warning was captured. Possible values:
+
+        * ``"config"``: during pytest configuration/initialization stage.
+        * ``"collect"``: during test collection.
+        * ``"runtest"``: during test execution.
+
+    :param pytest.Item|None item:
+        The item being executed if ``when`` is ``"runtest"``, otherwise ``None``.
+    """
+
+
 # -------------------------------------------------------------------------
 # doctest hooks
 # -------------------------------------------------------------------------

src/_pytest/junitxml.py

@@ -258,12 +258,11 @@ def record_property(request):
 
 
 @pytest.fixture
-def record_xml_property(record_property):
+def record_xml_property(record_property, request):
     """(Deprecated) use record_property."""
-    import warnings
     from _pytest import deprecated
 
-    warnings.warn(deprecated.RECORD_XML_PROPERTY, DeprecationWarning, stacklevel=2)
+    request.node.warn(deprecated.RECORD_XML_PROPERTY)
 
     return record_property
 
@@ -274,9 +273,9 @@ def record_xml_attribute(request):
     The fixture is callable with ``(name, value)``, with value being
     automatically xml-encoded
     """
-    request.node.warn(
-        code="C3", message="record_xml_attribute is an experimental feature"
-    )
+    from _pytest.warning_types import PytestWarning
+
+    request.node.warn(PytestWarning("record_xml_attribute is an experimental feature"))
     xml = getattr(request.config, "_xml", None)
     if xml is not None:
         node_reporter = xml.node_reporter(request.node.nodeid)

src/_pytest/mark/structures.py

@@ -65,7 +65,7 @@ class ParameterSet(namedtuple("ParameterSet", "values, marks, id")):
         return cls(values, marks, id_)
 
     @classmethod
-    def extract_from(cls, parameterset, legacy_force_tuple=False):
+    def extract_from(cls, parameterset, belonging_definition, legacy_force_tuple=False):
         """
         :param parameterset:
             a legacy style parameterset that may or may not be a tuple,
@@ -75,6 +75,7 @@ class ParameterSet(namedtuple("ParameterSet", "values, marks, id")):
             enforce tuple wrapping so single argument tuple values
             don't get decomposed and break tests
 
+        :param belonging_definition: the item that we will be extracting the parameters from.
         """
 
         if isinstance(parameterset, cls):
@@ -93,20 +94,24 @@ class ParameterSet(namedtuple("ParameterSet", "values, marks, id")):
         if legacy_force_tuple:
             argval = (argval,)
 
-        if newmarks:
-            warnings.warn(MARK_PARAMETERSET_UNPACKING)
+        if newmarks and belonging_definition is not None:
+            belonging_definition.warn(MARK_PARAMETERSET_UNPACKING)
 
         return cls(argval, marks=newmarks, id=None)
 
     @classmethod
-    def _for_parametrize(cls, argnames, argvalues, func, config):
+    def _for_parametrize(cls, argnames, argvalues, func, config, function_definition):
         if not isinstance(argnames, (tuple, list)):
             argnames = [x.strip() for x in argnames.split(",") if x.strip()]
             force_tuple = len(argnames) == 1
         else:
             force_tuple = False
         parameters = [
-            ParameterSet.extract_from(x, legacy_force_tuple=force_tuple)
+            ParameterSet.extract_from(
+                x,
+                legacy_force_tuple=force_tuple,
+                belonging_definition=function_definition,
+            )
             for x in argvalues
         ]
         del argvalues

src/_pytest/nodes.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import, division, print_function
 import os
+import warnings
 
 import six
 import py
@@ -7,6 +8,7 @@ import attr
 
 import _pytest
 import _pytest._code
+from _pytest.compat import getfslineno
 
 from _pytest.mark.structures import NodeKeywords, MarkInfo
 
@@ -134,19 +136,98 @@ class Node(object):
     def __repr__(self):
         return "<%s %r>" % (self.__class__.__name__, getattr(self, "name", None))
 
-    def warn(self, code, message):
-        """ generate a warning with the given code and message for this
-        item. """
+    def warn(self, _code_or_warning=None, message=None, code=None):
+        """Issue a warning for this item.
+
+        Warnings will be displayed after the test session, unless explicitly suppressed.
+
+        This can be called in two forms:
+
+        **Warning instance**
+
+        This was introduced in pytest 3.8 and uses the standard warning mechanism to issue warnings.
+
+        .. code-block:: python
+
+            node.warn(PytestWarning("some message"))
+
+        The warning instance must be a subclass of :class:`pytest.PytestWarning`.
+
+        **code/message (deprecated)**
+
+        This form was used in pytest prior to 3.8 and is considered deprecated. Using this form will emit another
+        warning about the deprecation:
+
+        .. code-block:: python
+
+            node.warn("CI", "some message")
+
+        :param Union[Warning,str] _code_or_warning:
+            warning instance or warning code (legacy). This parameter receives an underscore for backward
+            compatibility with the legacy code/message form, and will be replaced for something
+            more usual when the legacy form is removed.
+
+        :param Union[str,None] message: message to display when called in the legacy form.
+        :param str code: code for the warning, in legacy form when using keyword arguments.
+        :return:
+        """
+        if message is None:
+            if _code_or_warning is None:
+                raise ValueError("code_or_warning must be given")
+            self._std_warn(_code_or_warning)
+        else:
+            if _code_or_warning and code:
+                raise ValueError(
+                    "code_or_warning and code cannot both be passed to this function"
+                )
+            code = _code_or_warning or code
+            self._legacy_warn(code, message)
+
+    def _legacy_warn(self, code, message):
+        """
+        .. deprecated:: 3.8
+
+            Use :meth:`Node.std_warn <_pytest.nodes.Node.std_warn>` instead.
+
+        Generate a warning with the given code and message for this item.
+        """
+        from _pytest.deprecated import NODE_WARN
+
+        self._std_warn(NODE_WARN)
+
         assert isinstance(code, str)
-        fslocation = getattr(self, "location", None)
-        if fslocation is None:
-            fslocation = getattr(self, "fspath", None)
+        fslocation = get_fslocation_from_item(self)
         self.ihook.pytest_logwarning.call_historic(
             kwargs=dict(
                 code=code, message=message, nodeid=self.nodeid, fslocation=fslocation
             )
         )
 
+    def _std_warn(self, warning):
+        """Issue a warning for this item.
+
+        Warnings will be displayed after the test session, unless explicitly suppressed
+
+        :param Warning warning: the warning instance to issue. Must be a subclass of PytestWarning.
+
+        :raise ValueError: if ``warning`` instance is not a subclass of PytestWarning.
+        """
+        from _pytest.warning_types import PytestWarning
+
+        if not isinstance(warning, PytestWarning):
+            raise ValueError(
+                "warning must be an instance of PytestWarning or subclass, got {!r}".format(
+                    warning
+                )
+            )
+        path, lineno = get_fslocation_from_item(self)
+        warnings.warn_explicit(
+            warning,
+            category=None,
+            filename=str(path),
+            lineno=lineno + 1 if lineno is not None else None,
+        )
+
     # methods for ordering nodes
     @property
     def nodeid(self):
@@ -310,6 +391,24 @@ class Node(object):
     repr_failure = _repr_failure_py
 
 
+def get_fslocation_from_item(item):
+    """Tries to extract the actual location from an item, depending on available attributes:
+
+    * "fslocation": a pair (path, lineno)
+    * "obj": a Python object that the item wraps.
+    * "fspath": just a path
+
+    :rtype: a tuple of (str|LocalPath, int) with filename and line number.
+    """
+    result = getattr(item, "location", None)
+    if result is not None:
+        return result[:2]
+    obj = getattr(item, "obj", None)
+    if obj is not None:
+        return getfslineno(obj)
+    return getattr(item, "fspath", "unknown location"), -1
+
+
 class Collector(Node):
     """ Collector instances create children through collect()
         and thus iteratively build a tree.

src/_pytest/pytester.py

@@ -126,7 +126,7 @@ class LsofFdLeakChecker(object):
             error.append(error[0])
             error.append("*** function %s:%s: %s " % item.location)
             error.append("See issue #2366")
-            item.warn("", "\n".join(error))
+            item.warn(pytest.PytestWarning("\n".join(error)))
 
 
 # XXX copied from execnet's conftest.py - needs to be merged
@@ -407,7 +407,9 @@ class RunResult(object):
                     return d
         raise ValueError("Pytest terminal report not found")
 
-    def assert_outcomes(self, passed=0, skipped=0, failed=0, error=0):
+    def assert_outcomes(
+        self, passed=0, skipped=0, failed=0, error=0, xpassed=0, xfailed=0
+    ):
         """Assert that the specified outcomes appear with the respective
         numbers (0 means it didn't occur) in the text output from a test run.
 
@@ -418,10 +420,18 @@ class RunResult(object):
             "skipped": d.get("skipped", 0),
             "failed": d.get("failed", 0),
             "error": d.get("error", 0),
+            "xpassed": d.get("xpassed", 0),
+            "xfailed": d.get("xfailed", 0),
         }
-        assert obtained == dict(
-            passed=passed, skipped=skipped, failed=failed, error=error
-        )
+        expected = {
+            "passed": passed,
+            "skipped": skipped,
+            "failed": failed,
+            "error": error,
+            "xpassed": xpassed,
+            "xfailed": xfailed,
+        }
+        assert obtained == expected
 
 
 class CwdSnapshot(object):
@@ -515,7 +525,6 @@ class Testdir(object):
 
     def make_hook_recorder(self, pluginmanager):
         """Create a new :py:class:`HookRecorder` for a PluginManager."""
-        assert not hasattr(pluginmanager, "reprec")
         pluginmanager.reprec = reprec = HookRecorder(pluginmanager)
         self.request.addfinalizer(reprec.finish_recording)
         return reprec
@@ -633,10 +642,10 @@ class Testdir(object):
         return p
 
     def copy_example(self, name=None):
-        from . import experiments
         import warnings
+        from _pytest.warning_types import PYTESTER_COPY_EXAMPLE
 
-        warnings.warn(experiments.PYTESTER_COPY_EXAMPLE, stacklevel=2)
+        warnings.warn(PYTESTER_COPY_EXAMPLE, stacklevel=2)
         example_dir = self.request.config.getini("pytester_example_dir")
         if example_dir is None:
             raise ValueError("pytester_example_dir is unset, can't copy examples")

src/_pytest/python.py

@@ -44,7 +44,7 @@ from _pytest.mark.structures import (
     get_unpacked_marks,
     normalize_mark_list,
 )
-
+from _pytest.warning_types import RemovedInPytest4Warning, PytestWarning
 
 # relative paths that we use to filter traceback entries from appearing to the user;
 # see filter_traceback
@@ -239,9 +239,14 @@ def pytest_pycollect_makeitem(collector, name, obj):
         # or a funtools.wrapped.
         # We musn't if it's been wrapped with mock.patch (python 2 only)
         if not (isfunction(obj) or isfunction(get_real_func(obj))):
-            collector.warn(
-                code="C2",
-                message="cannot collect %r because it is not a function." % name,
+            filename, lineno = getfslineno(obj)
+            warnings.warn_explicit(
+                message=PytestWarning(
+                    "cannot collect %r because it is not a function." % name
+                ),
+                category=None,
+                filename=str(filename),
+                lineno=lineno + 1,
             )
         elif getattr(obj, "__test__", True):
             if is_generator(obj):
@@ -349,11 +354,6 @@ class PyCollector(PyobjMixin, nodes.Collector):
             if isinstance(obj, staticmethod):
                 # static methods need to be unwrapped
                 obj = safe_getattr(obj, "__func__", False)
-                if obj is False:
-                    # Python 2.6 wraps in a different way that we won't try to handle
-                    msg = "cannot collect static method %r because it is not a function"
-                    self.warn(code="C2", message=msg % name)
-                    return False
             return (
                 safe_getattr(obj, "__call__", False)
                 and fixtures.getfixturemarker(obj) is None
@@ -662,16 +662,18 @@ class Class(PyCollector):
             return []
         if hasinit(self.obj):
             self.warn(
-                "C1",
-                "cannot collect test class %r because it has a "
-                "__init__ constructor" % self.obj.__name__,
+                PytestWarning(
+                    "cannot collect test class %r because it has a "
+                    "__init__ constructor" % self.obj.__name__
+                )
             )
             return []
         elif hasnew(self.obj):
             self.warn(
-                "C1",
-                "cannot collect test class %r because it has a "
-                "__new__ constructor" % self.obj.__name__,
+                PytestWarning(
+                    "cannot collect test class %r because it has a "
+                    "__new__ constructor" % self.obj.__name__
+                )
             )
             return []
         return [self._getcustomclass("Instance")(name="()", parent=self)]
@@ -799,7 +801,7 @@ class Generator(FunctionMixin, PyCollector):
                 )
             seen[name] = True
             values.append(self.Function(name, self, args=args, callobj=call))
-        self.warn("C1", deprecated.YIELD_TESTS)
+        self.warn(deprecated.YIELD_TESTS)
         return values
 
     def getcallargs(self, obj):
@@ -966,7 +968,11 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
         from _pytest.mark import ParameterSet
 
         argnames, parameters = ParameterSet._for_parametrize(
-            argnames, argvalues, self.function, self.config
+            argnames,
+            argvalues,
+            self.function,
+            self.config,
+            function_definition=self.definition,
         )
         del argvalues
 
@@ -977,7 +983,7 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
 
         arg_values_types = self._resolve_arg_value_types(argnames, indirect)
 
-        ids = self._resolve_arg_ids(argnames, ids, parameters)
+        ids = self._resolve_arg_ids(argnames, ids, parameters, item=self.definition)
 
         scopenum = scope2index(scope, descr="call to {}".format(self.parametrize))
 
@@ -1000,13 +1006,14 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
                 newcalls.append(newcallspec)
         self._calls = newcalls
 
-    def _resolve_arg_ids(self, argnames, ids, parameters):
+    def _resolve_arg_ids(self, argnames, ids, parameters, item):
         """Resolves the actual ids for the given argnames, based on the ``ids`` parameter given
         to ``parametrize``.
 
         :param List[str] argnames: list of argument names passed to ``parametrize()``.
         :param ids: the ids parameter of the parametrized call (see docs).
         :param List[ParameterSet] parameters: the list of parameter values, same size as ``argnames``.
+        :param Item item: the item that generated this parametrized call.
         :rtype: List[str]
         :return: the list of ids for each argname given
         """
@@ -1027,7 +1034,7 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
                     raise ValueError(
                         msg % (saferepr(id_value), type(id_value).__name__)
                     )
-        ids = idmaker(argnames, parameters, idfn, ids, self.config)
+        ids = idmaker(argnames, parameters, idfn, ids, self.config, item=item)
         return ids
 
     def _resolve_arg_value_types(self, argnames, indirect):
@@ -1100,10 +1107,8 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
         :arg param: a parameter which will be exposed to a later fixture function
             invocation through the ``request.param`` attribute.
         """
-        if self.config:
-            self.config.warn(
-                "C1", message=deprecated.METAFUNC_ADD_CALL, fslocation=None
-            )
+        warnings.warn(deprecated.METAFUNC_ADD_CALL, stacklevel=2)
+
         assert funcargs is None or isinstance(funcargs, dict)
         if funcargs is not None:
             for name in funcargs:
@@ -1153,21 +1158,20 @@ def _find_parametrized_scope(argnames, arg2fixturedefs, indirect):
     return "function"
 
 
-def _idval(val, argname, idx, idfn, config=None):
+def _idval(val, argname, idx, idfn, item, config):
     if idfn:
         s = None
         try:
             s = idfn(val)
-        except Exception:
+        except Exception as e:
             # See issue https://github.com/pytest-dev/pytest/issues/2169
-            import warnings
-
             msg = (
-                "Raised while trying to determine id of parameter %s at position %d."
-                % (argname, idx)
+                "While trying to determine id of parameter {} at position "
+                "{} the following exception was raised:\n".format(argname, idx)
             )
-            msg += "\nUpdate your code as this will raise an error in pytest-4.0."
-            warnings.warn(msg, DeprecationWarning)
+            msg += "  {}: {}\n".format(type(e).__name__, e)
+            msg += "This warning will be an error error in pytest-4.0."
+            item.warn(RemovedInPytest4Warning(msg))
         if s:
             return ascii_escaped(s)
 
@@ -1191,12 +1195,12 @@ def _idval(val, argname, idx, idfn, config=None):
     return str(argname) + str(idx)
 
 
-def _idvalset(idx, parameterset, argnames, idfn, ids, config=None):
+def _idvalset(idx, parameterset, argnames, idfn, ids, item, config):
     if parameterset.id is not None:
         return parameterset.id
     if ids is None or (idx >= len(ids) or ids[idx] is None):
         this_id = [
-            _idval(val, argname, idx, idfn, config)
+            _idval(val, argname, idx, idfn, item=item, config=config)
             for val, argname in zip(parameterset.values, argnames)
         ]
         return "-".join(this_id)
@@ -1204,9 +1208,9 @@ def _idvalset(idx, parameterset, argnames, idfn, ids, config=None):
         return ascii_escaped(ids[idx])
 
 
-def idmaker(argnames, parametersets, idfn=None, ids=None, config=None):
+def idmaker(argnames, parametersets, idfn=None, ids=None, config=None, item=None):
     ids = [
-        _idvalset(valindex, parameterset, argnames, idfn, ids, config)
+        _idvalset(valindex, parameterset, argnames, idfn, ids, config=config, item=item)
         for valindex, parameterset in enumerate(parametersets)
     ]
     if len(set(ids)) != len(ids):

src/_pytest/resultlog.py

@@ -31,8 +31,10 @@ def pytest_configure(config):
         config.pluginmanager.register(config._resultlog)
 
         from _pytest.deprecated import RESULT_LOG
+        from _pytest.warning_types import RemovedInPytest4Warning
+        from _pytest.warnings import _issue_config_warning
 
-        config.warn("C1", RESULT_LOG)
+        _issue_config_warning(RemovedInPytest4Warning(RESULT_LOG), config)
 
 
 def pytest_unconfigure(config):

src/_pytest/terminal.py

@@ -9,6 +9,7 @@ import platform
 import sys
 import time
 
+import attr
 import pluggy
 import py
 import six
@@ -184,23 +185,23 @@ def pytest_report_teststatus(report):
     return report.outcome, letter, report.outcome.upper()
 
 
+@attr.s
 class WarningReport(object):
     """
-    Simple structure to hold warnings information captured by ``pytest_logwarning``.
+    Simple structure to hold warnings information captured by ``pytest_logwarning`` and ``pytest_warning_captured``.
+
+    :ivar str message: user friendly message about the warning
+    :ivar str|None nodeid: node id that generated the warning (see ``get_location``).
+    :ivar tuple|py.path.local fslocation:
+        file system location of the source of the warning (see ``get_location``).
+
+    :ivar bool legacy: if this warning report was generated from the deprecated ``pytest_logwarning`` hook.
     """
 
-    def __init__(self, code, message, nodeid=None, fslocation=None):
-        """
-        :param code: unused
-        :param str message: user friendly message about the warning
-        :param str|None nodeid: node id that generated the warning (see ``get_location``).
-        :param tuple|py.path.local fslocation:
-            file system location of the source of the warning (see ``get_location``).
-        """
-        self.code = code
-        self.message = message
-        self.nodeid = nodeid
-        self.fslocation = fslocation
+    message = attr.ib()
+    nodeid = attr.ib(default=None)
+    fslocation = attr.ib(default=None)
+    legacy = attr.ib(default=False)
 
     def get_location(self, config):
         """
@@ -213,6 +214,8 @@ class WarningReport(object):
             if isinstance(self.fslocation, tuple) and len(self.fslocation) >= 2:
                 filename, linenum = self.fslocation[:2]
                 relpath = py.path.local(filename).relto(config.invocation_dir)
+                if not relpath:
+                    relpath = str(filename)
                 return "%s:%s" % (relpath, linenum)
             else:
                 return str(self.fslocation)
@@ -254,7 +257,7 @@ class TerminalReporter(object):
         # do not show progress if we are showing fixture setup/teardown
         if self.config.getoption("setupshow"):
             return False
-        return self.config.getini("console_output_style") == "progress"
+        return self.config.getini("console_output_style") in ("progress", "count")
 
     def hasopt(self, char):
         char = {"xfailed": "x", "skipped": "s"}.get(char, char)
@@ -327,13 +330,27 @@ class TerminalReporter(object):
             self.write_line("INTERNALERROR> " + line)
         return 1
 
-    def pytest_logwarning(self, code, fslocation, message, nodeid):
+    def pytest_logwarning(self, fslocation, message, nodeid):
         warnings = self.stats.setdefault("warnings", [])
         warning = WarningReport(
-            code=code, fslocation=fslocation, message=message, nodeid=nodeid
+            fslocation=fslocation, message=message, nodeid=nodeid, legacy=True
         )
         warnings.append(warning)
 
+    def pytest_warning_captured(self, warning_message, item):
+        # from _pytest.nodes import get_fslocation_from_item
+        from _pytest.warnings import warning_record_to_str
+
+        warnings = self.stats.setdefault("warnings", [])
+        fslocation = warning_message.filename, warning_message.lineno
+        message = warning_record_to_str(warning_message)
+
+        nodeid = item.nodeid if item is not None else ""
+        warning_report = WarningReport(
+            fslocation=fslocation, message=message, nodeid=nodeid
+        )
+        warnings.append(warning_report)
+
     def pytest_plugin_registered(self, plugin):
         if self.config.option.traceconfig:
             msg = "PLUGIN registered: %s" % (plugin,)
@@ -404,6 +421,12 @@ class TerminalReporter(object):
                 self.currentfspath = -2
 
     def pytest_runtest_logfinish(self, nodeid):
+        if self.config.getini("console_output_style") == "count":
+            num_tests = self._session.testscollected
+            progress_length = len(" [{}/{}]".format(str(num_tests), str(num_tests)))
+        else:
+            progress_length = len(" [100%]")
+
         if self.verbosity <= 0 and self._show_progress_info:
             self._progress_nodeids_reported.add(nodeid)
             last_item = (
@@ -413,21 +436,27 @@ class TerminalReporter(object):
                 self._write_progress_information_filling_space()
             else:
                 w = self._width_of_current_line
-                past_edge = w + self._PROGRESS_LENGTH + 1 >= self._screen_width
+                past_edge = w + progress_length + 1 >= self._screen_width
                 if past_edge:
                     msg = self._get_progress_information_message()
                     self._tw.write(msg + "\n", cyan=True)
 
-    _PROGRESS_LENGTH = len(" [100%]")
-
     def _get_progress_information_message(self):
         if self.config.getoption("capture") == "no":
             return ""
         collected = self._session.testscollected
-        if collected:
-            progress = len(self._progress_nodeids_reported) * 100 // collected
-            return " [{:3d}%]".format(progress)
-        return " [100%]"
+        if self.config.getini("console_output_style") == "count":
+            if collected:
+                progress = self._progress_nodeids_reported
+                counter_format = "{{:{}d}}".format(len(str(collected)))
+                format_string = " [{}/{{}}]".format(counter_format)
+                return format_string.format(len(progress), collected)
+            return " [ {} / {} ]".format(collected, collected)
+        else:
+            if collected:
+                progress = len(self._progress_nodeids_reported) * 100 // collected
+                return " [{:3d}%]".format(progress)
+            return " [100%]"
 
     def _write_progress_information_filling_space(self):
         msg = self._get_progress_information_message()
@@ -691,11 +720,20 @@ class TerminalReporter(object):
 
             self.write_sep("=", "warnings summary", yellow=True, bold=False)
             for location, warning_records in grouped:
-                self._tw.line(str(location) if location else "<undetermined location>")
+                # legacy warnings show their location explicitly, while standard warnings look better without
+                # it because the location is already formatted into the message
+                warning_records = list(warning_records)
+                is_legacy = warning_records[0].legacy
+                if location and is_legacy:
+                    self._tw.line(str(location))
                 for w in warning_records:
-                    lines = w.message.splitlines()
-                    indented = "\n".join("  " + x for x in lines)
-                    self._tw.line(indented)
+                    if is_legacy:
+                        lines = w.message.splitlines()
+                        indented = "\n".join("  " + x for x in lines)
+                        message = indented.rstrip()
+                    else:
+                        message = w.message.rstrip()
+                    self._tw.line(message)
                 self._tw.line()
             self._tw.line("-- Docs: https://docs.pytest.org/en/latest/warnings.html")
 

src/_pytest/warning_types.py

@@ -0,0 +1,42 @@
+class PytestWarning(UserWarning):
+    """
+    Bases: :class:`UserWarning`.
+
+    Base class for all warnings emitted by pytest.
+    """
+
+
+class PytestDeprecationWarning(PytestWarning, DeprecationWarning):
+    """
+    Bases: :class:`pytest.PytestWarning`, :class:`DeprecationWarning`.
+
+    Warning class for features that will be removed in a future version.
+    """
+
+
+class RemovedInPytest4Warning(PytestDeprecationWarning):
+    """
+    Bases: :class:`pytest.PytestDeprecationWarning`.
+
+    Warning class for features scheduled to be removed in pytest 4.0.
+    """
+
+
+class PytestExperimentalApiWarning(PytestWarning, FutureWarning):
+    """
+    Bases: :class:`pytest.PytestWarning`, :class:`FutureWarning`.
+
+    Warning category used to denote experiments in pytest. Use sparingly as the API might change or even be
+    removed completely in future version
+    """
+
+    @classmethod
+    def simple(cls, apiname):
+        return cls(
+            "{apiname} is an experimental api that may change over time".format(
+                apiname=apiname
+            )
+        )
+
+
+PYTESTER_COPY_EXAMPLE = PytestExperimentalApiWarning.simple("testdir.copy_example")

src/_pytest/warnings.py

@@ -1,5 +1,6 @@
 from __future__ import absolute_import, division, print_function
 
+import sys
 import warnings
 from contextlib import contextmanager
 
@@ -58,62 +59,114 @@ def pytest_configure(config):
 
 
 @contextmanager
-def catch_warnings_for_item(item):
+def catch_warnings_for_item(config, ihook, when, item):
     """
-    catches the warnings generated during setup/call/teardown execution
-    of the given item and after it is done posts them as warnings to this
-    item.
+    Context manager that catches warnings generated in the contained execution block.
+
+    ``item`` can be None if we are not in the context of an item execution.
+
+    Each warning captured triggers the ``pytest_warning_captured`` hook.
     """
-    args = item.config.getoption("pythonwarnings") or []
-    inifilters = item.config.getini("filterwarnings")
+    args = config.getoption("pythonwarnings") or []
+    inifilters = config.getini("filterwarnings")
     with warnings.catch_warnings(record=True) as log:
+        filters_configured = args or inifilters or sys.warnoptions
+
         for arg in args:
             warnings._setoption(arg)
 
         for arg in inifilters:
             _setoption(warnings, arg)
 
-        for mark in item.iter_markers(name="filterwarnings"):
-            for arg in mark.args:
-                warnings._setoption(arg)
+        if item is not None:
+            for mark in item.iter_markers(name="filterwarnings"):
+                for arg in mark.args:
+                    _setoption(warnings, arg)
+                    filters_configured = True
+
+        if not filters_configured:
+            # if user is not explicitly configuring warning filters, show deprecation warnings by default (#2908)
+            warnings.filterwarnings("always", category=DeprecationWarning)
+            warnings.filterwarnings("always", category=PendingDeprecationWarning)
 
         yield
 
-        for warning in log:
-            warn_msg = warning.message
-            unicode_warning = False
-
-            if compat._PY2 and any(
-                isinstance(m, compat.UNICODE_TYPES) for m in warn_msg.args
-            ):
-                new_args = []
-                for m in warn_msg.args:
-                    new_args.append(
-                        compat.ascii_escaped(m)
-                        if isinstance(m, compat.UNICODE_TYPES)
-                        else m
-                    )
-                unicode_warning = list(warn_msg.args) != new_args
-                warn_msg.args = new_args
-
-            msg = warnings.formatwarning(
-                warn_msg,
-                warning.category,
-                warning.filename,
-                warning.lineno,
-                warning.line,
+        for warning_message in log:
+            ihook.pytest_warning_captured.call_historic(
+                kwargs=dict(warning_message=warning_message, when=when, item=item)
             )
-            item.warn("unused", msg)
 
-            if unicode_warning:
-                warnings.warn(
-                    "Warning is using unicode non convertible to ascii, "
-                    "converting to a safe representation:\n  %s" % msg,
-                    UnicodeWarning,
-                )
+
+def warning_record_to_str(warning_message):
+    """Convert a warnings.WarningMessage to a string, taking in account a lot of unicode shenaningans in Python 2.
+
+    When Python 2 support is dropped this function can be greatly simplified.
+    """
+    warn_msg = warning_message.message
+    unicode_warning = False
+    if compat._PY2 and any(isinstance(m, compat.UNICODE_TYPES) for m in warn_msg.args):
+        new_args = []
+        for m in warn_msg.args:
+            new_args.append(
+                compat.ascii_escaped(m) if isinstance(m, compat.UNICODE_TYPES) else m
+            )
+        unicode_warning = list(warn_msg.args) != new_args
+        warn_msg.args = new_args
+
+    msg = warnings.formatwarning(
+        warn_msg,
+        warning_message.category,
+        warning_message.filename,
+        warning_message.lineno,
+        warning_message.line,
+    )
+    if unicode_warning:
+        warnings.warn(
+            "Warning is using unicode non convertible to ascii, "
+            "converting to a safe representation:\n  %s" % msg,
+            UnicodeWarning,
+        )
+    return msg
+
+
+@pytest.hookimpl(hookwrapper=True, tryfirst=True)
+def pytest_runtest_protocol(item):
+    with catch_warnings_for_item(
+        config=item.config, ihook=item.ihook, when="runtest", item=item
+    ):
+        yield
+
+
+@pytest.hookimpl(hookwrapper=True, tryfirst=True)
+def pytest_collection(session):
+    config = session.config
+    with catch_warnings_for_item(
+        config=config, ihook=config.hook, when="collect", item=None
+    ):
+        yield
 
 
 @pytest.hookimpl(hookwrapper=True)
-def pytest_runtest_protocol(item):
-    with catch_warnings_for_item(item):
+def pytest_terminal_summary(terminalreporter):
+    config = terminalreporter.config
+    with catch_warnings_for_item(
+        config=config, ihook=config.hook, when="config", item=None
+    ):
         yield
+
+
+def _issue_config_warning(warning, config):
+    """
+    This function should be used instead of calling ``warnings.warn`` directly when we are in the "configure" stage:
+    at this point the actual options might not have been set, so we manually trigger the pytest_warning_captured
+    hook so we can display this warnings in the terminal. This is a hack until we can sort out #2891.
+
+    :param warning: the warning instance.
+    :param config:
+    """
+    with warnings.catch_warnings(record=True) as records:
+        warnings.simplefilter("always", type(warning))
+        warnings.warn(warning, stacklevel=2)
+    config.hook.pytest_warning_captured.call_historic(
+        kwargs=dict(warning_message=records[0], when="config", item=None)
+    )

src/pytest.py

@@ -19,45 +19,54 @@ from _pytest.main import Session
 from _pytest.nodes import Item, Collector, File
 from _pytest.fixtures import fillfixtures as _fillfuncargs
 from _pytest.python import Package, Module, Class, Instance, Function, Generator
-
 from _pytest.python_api import approx, raises
+from _pytest.warning_types import (
+    PytestWarning,
+    PytestDeprecationWarning,
+    RemovedInPytest4Warning,
+    PytestExperimentalApiWarning,
+)
 
 set_trace = __pytestPDB.set_trace
 
 __all__ = [
-    "main",
-    "UsageError",
-    "cmdline",
-    "hookspec",
-    "hookimpl",
     "__version__",
-    "register_assert_rewrite",
-    "freeze_includes",
-    "set_trace",
-    "warns",
-    "deprecated_call",
-    "fixture",
-    "yield_fixture",
-    "fail",
-    "skip",
-    "xfail",
-    "importorskip",
-    "exit",
-    "mark",
-    "param",
-    "approx",
     "_fillfuncargs",
-    "Item",
-    "File",
-    "Collector",
-    "Package",
-    "Session",
-    "Module",
+    "approx",
     "Class",
-    "Instance",
+    "cmdline",
+    "Collector",
+    "deprecated_call",
+    "exit",
+    "fail",
+    "File",
+    "fixture",
+    "freeze_includes",
     "Function",
     "Generator",
+    "hookimpl",
+    "hookspec",
+    "importorskip",
+    "Instance",
+    "Item",
+    "main",
+    "mark",
+    "Module",
+    "Package",
+    "param",
+    "PytestDeprecationWarning",
+    "PytestExperimentalApiWarning",
+    "PytestWarning",
     "raises",
+    "register_assert_rewrite",
+    "RemovedInPytest4Warning",
+    "Session",
+    "set_trace",
+    "skip",
+    "UsageError",
+    "warns",
+    "xfail",
+    "yield_fixture",
 ]
 
 if __name__ == "__main__":

testing/acceptance_test.py

@@ -1061,3 +1061,8 @@ def test_fixture_mock_integration(testdir):
     p = testdir.copy_example("acceptance/fixture_mock_integration.py")
     result = testdir.runpytest(p)
     result.stdout.fnmatch_lines("*1 passed*")
+
+
+def test_usage_error_code(testdir):
+    result = testdir.runpytest("-unknown-option-")
+    assert result.ret == EXIT_USAGEERROR

testing/deprecated_test.py

@@ -1,9 +1,11 @@
 from __future__ import absolute_import, division, print_function
 
+import os
 
 import pytest
 
 
+@pytest.mark.filterwarnings("default")
 def test_yield_tests_deprecation(testdir):
     testdir.makepyfile(
         """
@@ -17,16 +19,18 @@ def test_yield_tests_deprecation(testdir):
                 yield func1, 1, 1
     """
     )
-    result = testdir.runpytest("-ra")
+    result = testdir.runpytest()
     result.stdout.fnmatch_lines(
         [
-            "*yield tests are deprecated, and scheduled to be removed in pytest 4.0*",
+            "*test_yield_tests_deprecation.py:3:*yield tests are deprecated*",
+            "*test_yield_tests_deprecation.py:6:*yield tests are deprecated*",
             "*2 passed*",
         ]
     )
     assert result.stdout.str().count("yield tests are deprecated") == 2
 
 
+@pytest.mark.filterwarnings("default")
 def test_funcarg_prefix_deprecation(testdir):
     testdir.makepyfile(
         """
@@ -41,16 +45,15 @@ def test_funcarg_prefix_deprecation(testdir):
     result.stdout.fnmatch_lines(
         [
             (
-                "*pytest_funcarg__value: "
-                'declaring fixtures using "pytest_funcarg__" prefix is deprecated '
-                "and scheduled to be removed in pytest 4.0.  "
-                "Please remove the prefix and use the @pytest.fixture decorator instead."
+                "*test_funcarg_prefix_deprecation.py:1: *pytest_funcarg__value: "
+                'declaring fixtures using "pytest_funcarg__" prefix is deprecated*'
             ),
             "*1 passed*",
         ]
     )
 
 
+@pytest.mark.filterwarnings("default")
 def test_pytest_setup_cfg_deprecated(testdir):
     testdir.makefile(
         ".cfg",
@@ -65,6 +68,7 @@ def test_pytest_setup_cfg_deprecated(testdir):
     )
 
 
+@pytest.mark.filterwarnings("default")
 def test_pytest_custom_cfg_deprecated(testdir):
     testdir.makefile(
         ".cfg",
@@ -79,15 +83,15 @@ def test_pytest_custom_cfg_deprecated(testdir):
     )
 
 
-def test_str_args_deprecated(tmpdir, testdir):
+def test_str_args_deprecated(tmpdir):
     """Deprecate passing strings to pytest.main(). Scheduled for removal in pytest-4.0."""
     from _pytest.main import EXIT_NOTESTSCOLLECTED
 
     warnings = []
 
     class Collect(object):
-        def pytest_logwarning(self, message):
-            warnings.append(message)
+        def pytest_warning_captured(self, warning_message):
+            warnings.append(str(warning_message.message))
 
     ret = pytest.main("%s -x" % tmpdir, plugins=[Collect()])
     msg = (
@@ -102,6 +106,7 @@ def test_getfuncargvalue_is_deprecated(request):
     pytest.deprecated_call(request.getfuncargvalue, "tmpdir")
 
 
+@pytest.mark.filterwarnings("default")
 def test_resultlog_is_deprecated(testdir):
     result = testdir.runpytest("--help")
     result.stdout.fnmatch_lines(["*DEPRECATED path for machine-readable result log*"])
@@ -197,8 +202,11 @@ def test_pytest_plugins_in_non_top_level_conftest_deprecated(testdir):
     )
     res = testdir.runpytest_subprocess()
     assert res.ret == 0
-    res.stderr.fnmatch_lines(
-        "*" + str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
+    msg = str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
+    res.stdout.fnmatch_lines(
+        "*subdirectory{sep}conftest.py:0: RemovedInPytest4Warning: {msg}*".format(
+            sep=os.sep, msg=msg
+        )
     )
 
 
@@ -227,8 +235,11 @@ def test_pytest_plugins_in_non_top_level_conftest_deprecated_no_top_level_confte
 
     res = testdir.runpytest_subprocess()
     assert res.ret == 0
-    res.stderr.fnmatch_lines(
-        "*" + str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
+    msg = str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
+    res.stdout.fnmatch_lines(
+        "*subdirectory{sep}conftest.py:0: RemovedInPytest4Warning: {msg}*".format(
+            sep=os.sep, msg=msg
+        )
     )
 
 
@@ -261,10 +272,8 @@ def test_pytest_plugins_in_non_top_level_conftest_deprecated_no_false_positives(
     )
     res = testdir.runpytest_subprocess()
     assert res.ret == 0
-    assert (
-        str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
-        not in res.stderr.str()
-    )
+    msg = str(PYTEST_PLUGINS_FROM_NON_TOP_LEVEL_CONFTEST).splitlines()[0]
+    assert msg not in res.stdout.str()
 
 
 def test_call_fixture_function_deprecated():
@@ -276,3 +285,23 @@ def test_call_fixture_function_deprecated():
 
     with pytest.deprecated_call():
         assert fix() == 1
+
+
+def test_pycollector_makeitem_is_deprecated():
+    from _pytest.python import PyCollector
+    from _pytest.warning_types import RemovedInPytest4Warning
+
+    class PyCollectorMock(PyCollector):
+        """evil hack"""
+
+        def __init__(self):
+            self.called = False
+
+        def _makeitem(self, *k):
+            """hack to disable the actual behaviour"""
+            self.called = True
+
+    collector = PyCollectorMock()
+    with pytest.warns(RemovedInPytest4Warning):
+        collector.makeitem("foo", "bar")
+    assert collector.called

testing/python/collect.py

@@ -8,10 +8,6 @@ import pytest
 from _pytest.main import EXIT_NOTESTSCOLLECTED
 from _pytest.nodes import Collector
 
-ignore_parametrized_marks = pytest.mark.filterwarnings(
-    "ignore:Applying marks directly to parameters"
-)
-
 
 class TestModule(object):
     def test_failing_import(self, testdir):
@@ -456,12 +452,20 @@ class TestGenerator(object):
 
 
 class TestFunction(object):
+    @pytest.fixture
+    def ignore_parametrized_marks_args(self):
+        """Provides arguments to pytester.runpytest() to ignore the warning about marks being applied directly
+        to parameters.
+        """
+        return ("-W", "ignore:Applying marks directly to parameters")
+
     def test_getmodulecollector(self, testdir):
         item = testdir.getitem("def test_func(): pass")
         modcol = item.getparent(pytest.Module)
         assert isinstance(modcol, pytest.Module)
         assert hasattr(modcol.obj, "test_func")
 
+    @pytest.mark.filterwarnings("default")
     def test_function_as_object_instance_ignored(self, testdir):
         testdir.makepyfile(
             """
@@ -472,8 +476,14 @@ class TestFunction(object):
             test_a = A()
         """
         )
-        reprec = testdir.inline_run()
-        reprec.assertoutcome()
+        result = testdir.runpytest()
+        result.stdout.fnmatch_lines(
+            [
+                "collected 0 items",
+                "*test_function_as_object_instance_ignored.py:2: "
+                "*cannot collect 'test_a' because it is not a function.",
+            ]
+        )
 
     def test_function_equality(self, testdir, tmpdir):
         from _pytest.fixtures import FixtureManager
@@ -662,7 +672,7 @@ class TestFunction(object):
         rec = testdir.inline_run()
         rec.assertoutcome(passed=1)
 
-    @ignore_parametrized_marks
+    @pytest.mark.filterwarnings("ignore:Applying marks directly to parameters")
     def test_parametrize_with_mark(self, testdir):
         items = testdir.getitems(
             """
@@ -748,8 +758,7 @@ class TestFunction(object):
         assert colitems[2].name == "test2[a-c]"
         assert colitems[3].name == "test2[b-c]"
 
-    @ignore_parametrized_marks
-    def test_parametrize_skipif(self, testdir):
+    def test_parametrize_skipif(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -761,11 +770,10 @@ class TestFunction(object):
                 assert x < 2
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 2 passed, 1 skipped in *")
 
-    @ignore_parametrized_marks
-    def test_parametrize_skip(self, testdir):
+    def test_parametrize_skip(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -777,11 +785,10 @@ class TestFunction(object):
                 assert x < 2
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 2 passed, 1 skipped in *")
 
-    @ignore_parametrized_marks
-    def test_parametrize_skipif_no_skip(self, testdir):
+    def test_parametrize_skipif_no_skip(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -793,11 +800,10 @@ class TestFunction(object):
                 assert x < 2
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 1 failed, 2 passed in *")
 
-    @ignore_parametrized_marks
-    def test_parametrize_xfail(self, testdir):
+    def test_parametrize_xfail(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -809,11 +815,10 @@ class TestFunction(object):
                 assert x < 2
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 2 passed, 1 xfailed in *")
 
-    @ignore_parametrized_marks
-    def test_parametrize_passed(self, testdir):
+    def test_parametrize_passed(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -825,11 +830,10 @@ class TestFunction(object):
                 pass
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 2 passed, 1 xpassed in *")
 
-    @ignore_parametrized_marks
-    def test_parametrize_xfail_passed(self, testdir):
+    def test_parametrize_xfail_passed(self, testdir, ignore_parametrized_marks_args):
         testdir.makepyfile(
             """
             import pytest
@@ -841,7 +845,7 @@ class TestFunction(object):
                 pass
         """
         )
-        result = testdir.runpytest()
+        result = testdir.runpytest(*ignore_parametrized_marks_args)
         result.stdout.fnmatch_lines("* 3 passed in *")
 
     def test_function_original_name(self, testdir):
@@ -1468,6 +1472,7 @@ def test_collect_functools_partial(testdir):
     result.assertoutcome(passed=6, failed=2)
 
 
+@pytest.mark.filterwarnings("default")
 def test_dont_collect_non_function_callable(testdir):
     """Test for issue https://github.com/pytest-dev/pytest/issues/331
 
@@ -1490,7 +1495,7 @@ def test_dont_collect_non_function_callable(testdir):
     result.stdout.fnmatch_lines(
         [
             "*collected 1 item*",
-            "*cannot collect 'test_a' because it is not a function*",
+            "*test_dont_collect_non_function_callable.py:2: *cannot collect 'test_a' because it is not a function*",
             "*1 passed, 1 warnings in *",
         ]
     )

testing/python/metafunc.py

@@ -217,7 +217,7 @@ class TestMetafunc(object):
     def test_idval_hypothesis(self, value):
         from _pytest.python import _idval
 
-        escaped = _idval(value, "a", 6, None)
+        escaped = _idval(value, "a", 6, None, item=None, config=None)
         assert isinstance(escaped, str)
         if PY3:
             escaped.encode("ascii")
@@ -244,7 +244,7 @@ class TestMetafunc(object):
             ),
         ]
         for val, expected in values:
-            assert _idval(val, "a", 6, None) == expected
+            assert _idval(val, "a", 6, None, item=None, config=None) == expected
 
     def test_bytes_idval(self):
         """unittest for the expected behavior to obtain ids for parametrized
@@ -262,7 +262,7 @@ class TestMetafunc(object):
             (u"αρά".encode("utf-8"), "\\xce\\xb1\\xcf\\x81\\xce\\xac"),
         ]
         for val, expected in values:
-            assert _idval(val, "a", 6, None) == expected
+            assert _idval(val, "a", 6, idfn=None, item=None, config=None) == expected
 
     def test_class_or_function_idval(self):
         """unittest for the expected behavior to obtain ids for parametrized
@@ -278,7 +278,7 @@ class TestMetafunc(object):
 
         values = [(TestClass, "TestClass"), (test_function, "test_function")]
         for val, expected in values:
-            assert _idval(val, "a", 6, None) == expected
+            assert _idval(val, "a", 6, None, item=None, config=None) == expected
 
     @pytest.mark.issue250
     def test_idmaker_autoname(self):
@@ -383,44 +383,7 @@ class TestMetafunc(object):
         )
         assert result == ["a-a0", "a-a1", "a-a2"]
 
-    @pytest.mark.issue351
-    def test_idmaker_idfn_exception(self):
-        from _pytest.python import idmaker
-        from _pytest.recwarn import WarningsRecorder
-
-        class BadIdsException(Exception):
-            pass
-
-        def ids(val):
-            raise BadIdsException("ids raised")
-
-        rec = WarningsRecorder()
-        with rec:
-            idmaker(
-                ("a", "b"),
-                [
-                    pytest.param(10.0, IndexError()),
-                    pytest.param(20, KeyError()),
-                    pytest.param("three", [1, 2, 3]),
-                ],
-                idfn=ids,
-            )
-
-        assert [str(i.message) for i in rec.list] == [
-            "Raised while trying to determine id of parameter a at position 0."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-            "Raised while trying to determine id of parameter b at position 0."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-            "Raised while trying to determine id of parameter a at position 1."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-            "Raised while trying to determine id of parameter b at position 1."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-            "Raised while trying to determine id of parameter a at position 2."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-            "Raised while trying to determine id of parameter b at position 2."
-            "\nUpdate your code as this will raise an error in pytest-4.0.",
-        ]
-
+    @pytest.mark.filterwarnings("default")
     def test_parametrize_ids_exception(self, testdir):
         """
         :param testdir: the instance of Testdir class, a temporary
@@ -438,13 +401,14 @@ class TestMetafunc(object):
                     pass
             """
         )
-        with pytest.warns(DeprecationWarning):
-            result = testdir.runpytest("--collect-only")
+        result = testdir.runpytest("--collect-only")
         result.stdout.fnmatch_lines(
             [
                 "<Module 'test_parametrize_ids_exception.py'>",
                 "  <Function 'test_foo[a]'>",
                 "  <Function 'test_foo[b]'>",
+                "*test_parametrize_ids_exception.py:6: *parameter arg at position 0*",
+                "*test_parametrize_ids_exception.py:6: *parameter arg at position 1*",
             ]
         )
 

testing/python/test_deprecations.py

@@ -1,22 +0,0 @@
-import pytest
-
-from _pytest.python import PyCollector
-
-
-class PyCollectorMock(PyCollector):
-    """evil hack"""
-
-    def __init__(self):
-        self.called = False
-
-    def _makeitem(self, *k):
-        """hack to disable the actual behaviour"""
-        self.called = True
-
-
-def test_pycollector_makeitem_is_deprecated():
-
-    collector = PyCollectorMock()
-    with pytest.deprecated_call():
-        collector.makeitem("foo", "bar")
-    assert collector.called

testing/test_assertion.py

@@ -1075,17 +1075,27 @@ def test_diff_newline_at_end(monkeypatch, testdir):
     )
 
 
+@pytest.mark.filterwarnings("default")
 def test_assert_tuple_warning(testdir):
+    msg = "assertion is always true"
     testdir.makepyfile(
         """
         def test_tuple():
             assert(False, 'you shall not pass')
     """
     )
-    result = testdir.runpytest("-rw")
-    result.stdout.fnmatch_lines(
-        ["*test_assert_tuple_warning.py:2", "*assertion is always true*"]
+    result = testdir.runpytest()
+    result.stdout.fnmatch_lines(["*test_assert_tuple_warning.py:2:*{}*".format(msg)])
+
+    # tuples with size != 2 should not trigger the warning
+    testdir.makepyfile(
+        """
+        def test_tuple():
+            assert ()
+    """
     )
+    result = testdir.runpytest()
+    assert msg not in result.stdout.str()
 
 
 def test_assert_indirect_tuple_no_warning(testdir):

testing/test_assertrewrite.py

@@ -759,16 +759,16 @@ def test_rewritten():
         testdir.makepyfile("import a_package_without_init_py.module")
         assert testdir.runpytest().ret == EXIT_NOTESTSCOLLECTED
 
-    def test_rewrite_warning(self, pytestconfig, monkeypatch):
-        hook = AssertionRewritingHook(pytestconfig)
-        warnings = []
-
-        def mywarn(code, msg):
-            warnings.append((code, msg))
-
-        monkeypatch.setattr(hook.config, "warn", mywarn)
-        hook.mark_rewrite("_pytest")
-        assert "_pytest" in warnings[0][1]
+    def test_rewrite_warning(self, testdir):
+        testdir.makeconftest(
+            """
+            import pytest
+            pytest.register_assert_rewrite("_pytest")
+        """
+        )
+        # needs to be a subprocess because pytester explicitly disables this warning
+        result = testdir.runpytest_subprocess()
+        result.stdout.fnmatch_lines("*Module already imported*: _pytest")
 
     def test_rewrite_module_imported_from_conftest(self, testdir):
         testdir.makeconftest(

testing/test_cacheprovider.py

@@ -31,6 +31,7 @@ class TestNewAPI(object):
         val = config.cache.get("key/name", -2)
         assert val == -2
 
+    @pytest.mark.filterwarnings("default")
     def test_cache_writefail_cachfile_silent(self, testdir):
         testdir.makeini("[pytest]")
         testdir.tmpdir.join(".pytest_cache").write("gone wrong")
@@ -39,6 +40,9 @@ class TestNewAPI(object):
         cache.set("test/broken", [])
 
     @pytest.mark.skipif(sys.platform.startswith("win"), reason="no chmod on windows")
+    @pytest.mark.filterwarnings(
+        "ignore:could not create cache path:pytest.PytestWarning"
+    )
     def test_cache_writefail_permissions(self, testdir):
         testdir.makeini("[pytest]")
         testdir.tmpdir.ensure_dir(".pytest_cache").chmod(0)
@@ -47,6 +51,7 @@ class TestNewAPI(object):
         cache.set("test/broken", [])
 
     @pytest.mark.skipif(sys.platform.startswith("win"), reason="no chmod on windows")
+    @pytest.mark.filterwarnings("default")
     def test_cache_failure_warns(self, testdir):
         testdir.tmpdir.ensure_dir(".pytest_cache").chmod(0)
         testdir.makepyfile(
@@ -414,13 +419,7 @@ class TestLastFailed(object):
         )
 
         result = testdir.runpytest(test_a, "--lf")
-        result.stdout.fnmatch_lines(
-            [
-                "collected 2 items",
-                "run-last-failure: run all (no recorded failures)",
-                "*2 passed in*",
-            ]
-        )
+        result.stdout.fnmatch_lines(["collected 2 items", "*2 passed in*"])
 
         result = testdir.runpytest(test_b, "--lf")
         result.stdout.fnmatch_lines(
@@ -559,19 +558,6 @@ class TestLastFailed(object):
         testdir.runpytest("-q", "--lf")
         assert os.path.exists(".pytest_cache/v/cache/lastfailed")
 
-    @pytest.mark.parametrize("quiet", [True, False])
-    @pytest.mark.parametrize("opt", ["--ff", "--lf"])
-    def test_lf_and_ff_obey_verbosity(self, quiet, opt, testdir):
-        testdir.makepyfile("def test(): pass")
-        args = [opt]
-        if quiet:
-            args.append("-q")
-        result = testdir.runpytest(*args)
-        if quiet:
-            assert "run all" not in result.stdout.str()
-        else:
-            assert "run all" in result.stdout.str()
-
     def test_xfail_not_considered_failure(self, testdir):
         testdir.makepyfile(
             """
@@ -630,6 +616,23 @@ class TestLastFailed(object):
         assert self.get_cached_last_failed(testdir) == []
         assert result.ret == 0
 
+    @pytest.mark.parametrize("quiet", [True, False])
+    @pytest.mark.parametrize("opt", ["--ff", "--lf"])
+    def test_lf_and_ff_prints_no_needless_message(self, quiet, opt, testdir):
+        # Issue 3853
+        testdir.makepyfile("def test(): assert 0")
+        args = [opt]
+        if quiet:
+            args.append("-q")
+        result = testdir.runpytest(*args)
+        assert "run all" not in result.stdout.str()
+
+        result = testdir.runpytest(*args)
+        if quiet:
+            assert "run all" not in result.stdout.str()
+        else:
+            assert "rerun previous" in result.stdout.str()
+
     def get_cached_last_failed(self, testdir):
         config = testdir.parseconfigure()
         return sorted(config.cache.get("cache/lastfailed", {}))

testing/test_capture.py

@@ -18,7 +18,9 @@ from _pytest.capture import CaptureManager
 from _pytest.main import EXIT_NOTESTSCOLLECTED
 
 
-needsosdup = pytest.mark.xfail("not hasattr(os, 'dup')")
+needsosdup = pytest.mark.skipif(
+    not hasattr(os, "dup"), reason="test needs os.dup, not available on this platform"
+)
 
 
 def tobytes(obj):
@@ -61,9 +63,8 @@ class TestCaptureManager(object):
         pytest_addoption(parser)
         assert parser._groups[0].options[0].default == "sys"
 
-    @needsosdup
     @pytest.mark.parametrize(
-        "method", ["no", "sys", pytest.mark.skipif('not hasattr(os, "dup")', "fd")]
+        "method", ["no", "sys", pytest.param("fd", marks=needsosdup)]
     )
     def test_capturing_basic_api(self, method):
         capouter = StdCaptureFD()

testing/test_config.py

@@ -135,13 +135,13 @@ class TestConfigCmdlineParsing(object):
         """
         )
         testdir.makefile(
-            ".cfg",
+            ".ini",
             custom="""
             [pytest]
             custom = 1
         """,
         )
-        config = testdir.parseconfig("-c", "custom.cfg")
+        config = testdir.parseconfig("-c", "custom.ini")
         assert config.getini("custom") == "1"
 
         testdir.makefile(
@@ -155,8 +155,8 @@ class TestConfigCmdlineParsing(object):
         assert config.getini("custom") == "1"
 
     def test_absolute_win32_path(self, testdir):
-        temp_cfg_file = testdir.makefile(
-            ".cfg",
+        temp_ini_file = testdir.makefile(
+            ".ini",
             custom="""
             [pytest]
             addopts = --version
@@ -164,8 +164,8 @@ class TestConfigCmdlineParsing(object):
         )
         from os.path import normpath
 
-        temp_cfg_file = normpath(str(temp_cfg_file))
-        ret = pytest.main("-c " + temp_cfg_file)
+        temp_ini_file = normpath(str(temp_ini_file))
+        ret = pytest.main(["-c", temp_ini_file])
         assert ret == _pytest.main.EXIT_OK
 
 
@@ -605,6 +605,26 @@ def test_plugin_preparse_prevents_setuptools_loading(testdir, monkeypatch, block
         )
 
 
+@pytest.mark.parametrize(
+    "parse_args,should_load", [(("-p", "mytestplugin"), True), ((), False)]
+)
+def test_disable_plugin_autoload(testdir, monkeypatch, parse_args, should_load):
+    pkg_resources = pytest.importorskip("pkg_resources")
+
+    def my_iter(name):
+        raise AssertionError("Should not be called")
+
+    class PseudoPlugin(object):
+        x = 42
+
+    monkeypatch.setenv("PYTEST_DISABLE_PLUGIN_AUTOLOAD", "1")
+    monkeypatch.setattr(pkg_resources, "iter_entry_points", my_iter)
+    monkeypatch.setitem(sys.modules, "mytestplugin", PseudoPlugin())
+    config = testdir.parseconfig(*parse_args)
+    has_loaded = config.pluginmanager.get_plugin("mytestplugin") is not None
+    assert has_loaded == should_load
+
+
 def test_cmdline_processargs_simple(testdir):
     testdir.makeconftest(
         """
@@ -763,13 +783,14 @@ def test_collect_pytest_prefix_bug(pytestconfig):
     assert pm.parse_hookimpl_opts(Dummy(), "pytest_something") is None
 
 
-class TestWarning(object):
+class TestLegacyWarning(object):
+    @pytest.mark.filterwarnings("default")
     def test_warn_config(self, testdir):
         testdir.makeconftest(
             """
             values = []
-            def pytest_configure(config):
-                config.warn("C1", "hello")
+            def pytest_runtest_setup(item):
+                item.config.warn("C1", "hello")
             def pytest_logwarning(code, message):
                 if message == "hello" and code == "C1":
                     values.append(1)
@@ -782,24 +803,31 @@ class TestWarning(object):
                 assert conftest.values == [1]
         """
         )
-        reprec = testdir.inline_run()
-        reprec.assertoutcome(passed=1)
+        result = testdir.runpytest()
+        result.stdout.fnmatch_lines(
+            ["*hello", "*config.warn has been deprecated*", "*1 passed*"]
+        )
 
-    def test_warn_on_test_item_from_request(self, testdir, request):
+    @pytest.mark.filterwarnings("default")
+    @pytest.mark.parametrize("use_kw", [True, False])
+    def test_warn_on_test_item_from_request(self, testdir, use_kw):
+        code_kw = "code=" if use_kw else ""
+        message_kw = "message=" if use_kw else ""
         testdir.makepyfile(
             """
             import pytest
 
             @pytest.fixture
             def fix(request):
-                request.node.warn("T1", "hello")
+                request.node.warn({code_kw}"T1", {message_kw}"hello")
 
             def test_hello(fix):
                 pass
-        """
+        """.format(
+                code_kw=code_kw, message_kw=message_kw
+            )
         )
         result = testdir.runpytest("--disable-pytest-warnings")
-        assert result.parseoutcomes()["warnings"] > 0
         assert "hello" not in result.stdout.str()
 
         result = testdir.runpytest()
@@ -808,6 +836,7 @@ class TestWarning(object):
             ===*warnings summary*===
             *test_warn_on_test_item_from_request.py::test_hello*
             *hello*
+            *test_warn_on_test_item_from_request.py:7:*Node.warn(code, message) form has been deprecated*
         """
         )
 
@@ -827,7 +856,7 @@ class TestRootdir(object):
     @pytest.mark.parametrize("name", "setup.cfg tox.ini pytest.ini".split())
     def test_with_ini(self, tmpdir, name):
         inifile = tmpdir.join(name)
-        inifile.write("[pytest]\n")
+        inifile.write("[pytest]\n" if name != "setup.cfg" else "[tool:pytest]\n")
 
         a = tmpdir.mkdir("a")
         b = a.mkdir("b")
@@ -873,11 +902,14 @@ class TestRootdir(object):
 class TestOverrideIniArgs(object):
     @pytest.mark.parametrize("name", "setup.cfg tox.ini pytest.ini".split())
     def test_override_ini_names(self, testdir, name):
+        section = "[pytest]" if name != "setup.cfg" else "[tool:pytest]"
         testdir.tmpdir.join(name).write(
             textwrap.dedent(
                 """
-            [pytest]
-            custom = 1.0"""
+            {section}
+            custom = 1.0""".format(
+                    section=section
+                )
             )
         )
         testdir.makeconftest(

testing/test_junitxml.py

@@ -1005,6 +1005,7 @@ def test_record_property_same_name(testdir):
     pnodes[1].assert_attr(name="foo", value="baz")
 
 
+@pytest.mark.filterwarnings("default")
 def test_record_attribute(testdir):
     testdir.makepyfile(
         """
@@ -1023,7 +1024,7 @@ def test_record_attribute(testdir):
     tnode.assert_attr(bar="1")
     tnode.assert_attr(foo="<1")
     result.stdout.fnmatch_lines(
-        ["test_record_attribute.py::test_record", "*record_xml_attribute*experimental*"]
+        ["*test_record_attribute.py:6:*record_xml_attribute is an experimental feature"]
     )
 
 

testing/test_mark.py

@@ -16,7 +16,7 @@ from _pytest.mark import (
 from _pytest.nodes import Node
 
 ignore_markinfo = pytest.mark.filterwarnings(
-    "ignore:MarkInfo objects:_pytest.deprecated.RemovedInPytest4Warning"
+    "ignore:MarkInfo objects:pytest.RemovedInPytest4Warning"
 )
 
 
@@ -1039,10 +1039,19 @@ class TestKeywordSelection(object):
         ),
     ],
 )
-@pytest.mark.filterwarnings("ignore")
+@pytest.mark.filterwarnings("default")
 def test_parameterset_extractfrom(argval, expected):
-    extracted = ParameterSet.extract_from(argval)
+    from _pytest.deprecated import MARK_PARAMETERSET_UNPACKING
+
+    warn_called = []
+
+    class DummyItem:
+        def warn(self, warning):
+            warn_called.append(warning)
+
+    extracted = ParameterSet.extract_from(argval, belonging_definition=DummyItem())
     assert extracted == expected
+    assert warn_called == [MARK_PARAMETERSET_UNPACKING]
 
 
 def test_legacy_transfer():

testing/test_nodes.py

@@ -19,3 +19,14 @@ from _pytest import nodes
 def test_ischildnode(baseid, nodeid, expected):
     result = nodes.ischildnode(baseid, nodeid)
     assert result is expected
+
+
+def test_std_warn_not_pytestwarning(testdir):
+    items = testdir.getitems(
+        """
+        def test():
+            pass
+    """
+    )
+    with pytest.raises(ValueError, match=".*instance of PytestWarning.*"):
+        items[0].warn(UserWarning("some warning"))

testing/test_pytester.py

@@ -83,6 +83,57 @@ def test_testdir_runs_with_plugin(testdir):
     result.assert_outcomes(passed=1)
 
 
+def test_runresult_assertion_on_xfail(testdir):
+    testdir.makepyfile(
+        """
+        import pytest
+
+        pytest_plugins = "pytester"
+
+        @pytest.mark.xfail
+        def test_potato():
+            assert False
+    """
+    )
+    result = testdir.runpytest()
+    result.assert_outcomes(xfailed=1)
+    assert result.ret == 0
+
+
+def test_runresult_assertion_on_xpassed(testdir):
+    testdir.makepyfile(
+        """
+        import pytest
+
+        pytest_plugins = "pytester"
+
+        @pytest.mark.xfail
+        def test_potato():
+            assert True
+    """
+    )
+    result = testdir.runpytest()
+    result.assert_outcomes(xpassed=1)
+    assert result.ret == 0
+
+
+def test_xpassed_with_strict_is_considered_a_failure(testdir):
+    testdir.makepyfile(
+        """
+        import pytest
+
+        pytest_plugins = "pytester"
+
+        @pytest.mark.xfail(strict=True)
+        def test_potato():
+            assert True
+    """
+    )
+    result = testdir.runpytest()
+    result.assert_outcomes(failed=1)
+    assert result.ret != 0
+
+
 def make_holder():
     class apiclass(object):
         def pytest_xyz(self, arg):

testing/test_recwarn.py

@@ -7,7 +7,7 @@ from _pytest.recwarn import WarningsRecorder
 
 
 def test_recwarn_functional(testdir):
-    reprec = testdir.inline_runsource(
+    testdir.makepyfile(
         """
         import warnings
         def test_method(recwarn):
@@ -16,8 +16,8 @@ def test_recwarn_functional(testdir):
             assert isinstance(warn.message, UserWarning)
     """
     )
-    res = reprec.countoutcomes()
-    assert tuple(res) == (1, 0, 0), res
+    reprec = testdir.inline_run()
+    reprec.assertoutcome(passed=1)
 
 
 class TestWarningsRecorderChecker(object):

testing/test_resultlog.py

@@ -13,6 +13,9 @@ from _pytest.resultlog import (
 )
 
 
+pytestmark = pytest.mark.filterwarnings("ignore:--result-log is deprecated")
+
+
 def test_generic_path(testdir):
     from _pytest.main import Session
 

testing/test_terminal.py

@@ -1047,20 +1047,21 @@ def test_terminal_summary(testdir):
     )
 
 
+@pytest.mark.filterwarnings("default")
 def test_terminal_summary_warnings_are_displayed(testdir):
     """Test that warnings emitted during pytest_terminal_summary are displayed.
     (#1305).
     """
     testdir.makeconftest(
         """
+        import warnings
         def pytest_terminal_summary(terminalreporter):
-            config = terminalreporter.config
-            config.warn('C1', 'internal warning')
+            warnings.warn(UserWarning('internal warning'))
     """
     )
-    result = testdir.runpytest("-rw")
+    result = testdir.runpytest()
     result.stdout.fnmatch_lines(
-        ["<undetermined location>", "*internal warning", "*== 1 warnings in *"]
+        ["*conftest.py:3:*internal warning", "*== 1 warnings in *"]
     )
     assert "None" not in result.stdout.str()
 
@@ -1230,6 +1231,22 @@ class TestProgressOutputStyle(object):
             ]
         )
 
+    def test_count(self, many_tests_files, testdir):
+        testdir.makeini(
+            """
+            [pytest]
+            console_output_style = count
+        """
+        )
+        output = testdir.runpytest()
+        output.stdout.re_match_lines(
+            [
+                r"test_bar.py \.{10} \s+ \[10/20\]",
+                r"test_foo.py \.{5} \s+ \[15/20\]",
+                r"test_foobar.py \.{5} \s+ \[20/20\]",
+            ]
+        )
+
     def test_verbose(self, many_tests_files, testdir):
         output = testdir.runpytest("-v")
         output.stdout.re_match_lines(
@@ -1240,11 +1257,38 @@ class TestProgressOutputStyle(object):
             ]
         )
 
+    def test_verbose_count(self, many_tests_files, testdir):
+        testdir.makeini(
+            """
+            [pytest]
+            console_output_style = count
+        """
+        )
+        output = testdir.runpytest("-v")
+        output.stdout.re_match_lines(
+            [
+                r"test_bar.py::test_bar\[0\] PASSED \s+ \[ 1/20\]",
+                r"test_foo.py::test_foo\[4\] PASSED \s+ \[15/20\]",
+                r"test_foobar.py::test_foobar\[4\] PASSED \s+ \[20/20\]",
+            ]
+        )
+
     def test_xdist_normal(self, many_tests_files, testdir):
         pytest.importorskip("xdist")
         output = testdir.runpytest("-n2")
         output.stdout.re_match_lines([r"\.{20} \s+ \[100%\]"])
 
+    def test_xdist_normal_count(self, many_tests_files, testdir):
+        pytest.importorskip("xdist")
+        testdir.makeini(
+            """
+            [pytest]
+            console_output_style = count
+        """
+        )
+        output = testdir.runpytest("-n2")
+        output.stdout.re_match_lines([r"\.{20} \s+ \[20/20\]"])
+
     def test_xdist_verbose(self, many_tests_files, testdir):
         pytest.importorskip("xdist")
         output = testdir.runpytest("-n2", "-v")

testing/test_warnings.py

@@ -37,17 +37,15 @@ def pyfile_with_warnings(testdir, request):
     )
 
 
-@pytest.mark.filterwarnings("always")
+@pytest.mark.filterwarnings("default")
 def test_normal_flow(testdir, pyfile_with_warnings):
     """
-    Check that the warnings section is displayed, containing test node ids followed by
-    all warnings generated by that test node.
+    Check that the warnings section is displayed.
     """
     result = testdir.runpytest()
     result.stdout.fnmatch_lines(
         [
             "*== %s ==*" % WARNINGS_SUMMARY_HEADER,
-            "*test_normal_flow.py::test_func",
             "*normal_flow_module.py:3: UserWarning: user warning",
             '*  warnings.warn(UserWarning("user warning"))',
             "*normal_flow_module.py:4: RuntimeWarning: runtime warning",
@@ -55,7 +53,6 @@ def test_normal_flow(testdir, pyfile_with_warnings):
             "* 1 passed, 2 warnings*",
         ]
     )
-    assert result.stdout.str().count("test_normal_flow.py::test_func") == 1
 
 
 @pytest.mark.filterwarnings("always")
@@ -302,3 +299,204 @@ def test_filterwarnings_mark_registration(testdir):
     )
     result = testdir.runpytest("--strict")
     assert result.ret == 0
+
+
+@pytest.mark.filterwarnings("always")
+def test_warning_captured_hook(testdir):
+    testdir.makeconftest(
+        """
+        from _pytest.warnings import _issue_config_warning
+        def pytest_configure(config):
+            _issue_config_warning(UserWarning("config warning"), config)
+    """
+    )
+    testdir.makepyfile(
+        """
+        import pytest, warnings
+
+        warnings.warn(UserWarning("collect warning"))
+
+        @pytest.fixture
+        def fix():
+            warnings.warn(UserWarning("setup warning"))
+            yield 1
+            warnings.warn(UserWarning("teardown warning"))
+
+        def test_func(fix):
+            warnings.warn(UserWarning("call warning"))
+            assert fix == 1
+        """
+    )
+
+    collected = []
+
+    class WarningCollector:
+        def pytest_warning_captured(self, warning_message, when, item):
+            imge_name = item.name if item is not None else ""
+            collected.append((str(warning_message.message), when, imge_name))
+
+    result = testdir.runpytest(plugins=[WarningCollector()])
+    result.stdout.fnmatch_lines(["*1 passed*"])
+
+    expected = [
+        ("config warning", "config", ""),
+        ("collect warning", "collect", ""),
+        ("setup warning", "runtest", "test_func"),
+        ("call warning", "runtest", "test_func"),
+        ("teardown warning", "runtest", "test_func"),
+    ]
+    assert collected == expected
+
+
+@pytest.mark.filterwarnings("always")
+def test_collection_warnings(testdir):
+    """
+    Check that we also capture warnings issued during test collection (#3251).
+    """
+    testdir.makepyfile(
+        """
+        import warnings
+
+        warnings.warn(UserWarning("collection warning"))
+
+        def test_foo():
+            pass
+    """
+    )
+    result = testdir.runpytest()
+    result.stdout.fnmatch_lines(
+        [
+            "*== %s ==*" % WARNINGS_SUMMARY_HEADER,
+            "*collection_warnings.py:3: UserWarning: collection warning",
+            '  warnings.warn(UserWarning("collection warning"))',
+            "* 1 passed, 1 warnings*",
+        ]
+    )
+
+
+@pytest.mark.filterwarnings("always")
+def test_mark_regex_escape(testdir):
+    """@pytest.mark.filterwarnings should not try to escape regex characters (#3936)"""
+    testdir.makepyfile(
+        r"""
+        import pytest, warnings
+
+        @pytest.mark.filterwarnings(r"ignore:some \(warning\)")
+        def test_foo():
+            warnings.warn(UserWarning("some (warning)"))
+    """
+    )
+    result = testdir.runpytest()
+    assert WARNINGS_SUMMARY_HEADER not in result.stdout.str()
+
+
+@pytest.mark.filterwarnings("default")
+@pytest.mark.parametrize("ignore_pytest_warnings", ["no", "ini", "cmdline"])
+def test_hide_pytest_internal_warnings(testdir, ignore_pytest_warnings):
+    """Make sure we can ignore internal pytest warnings using a warnings filter."""
+    testdir.makepyfile(
+        """
+        import pytest
+        import warnings
+
+        warnings.warn(pytest.PytestWarning("some internal warning"))
+
+        def test_bar():
+            pass
+    """
+    )
+    if ignore_pytest_warnings == "ini":
+        testdir.makeini(
+            """
+            [pytest]
+            filterwarnings = ignore::pytest.PytestWarning
+        """
+        )
+    args = (
+        ["-W", "ignore::pytest.PytestWarning"]
+        if ignore_pytest_warnings == "cmdline"
+        else []
+    )
+    result = testdir.runpytest(*args)
+    if ignore_pytest_warnings != "no":
+        assert WARNINGS_SUMMARY_HEADER not in result.stdout.str()
+    else:
+        result.stdout.fnmatch_lines(
+            [
+                "*== %s ==*" % WARNINGS_SUMMARY_HEADER,
+                "*test_hide_pytest_internal_warnings.py:4: PytestWarning: some internal warning",
+                "* 1 passed, 1 warnings *",
+            ]
+        )
+
+
+class TestDeprecationWarningsByDefault:
+    """
+    Note: all pytest runs are executed in a subprocess so we don't inherit warning filters
+    from pytest's own test suite
+    """
+
+    def create_file(self, testdir, mark=""):
+        testdir.makepyfile(
+            """
+            import pytest, warnings
+
+            warnings.warn(DeprecationWarning("collection"))
+
+            {mark}
+            def test_foo():
+                warnings.warn(PendingDeprecationWarning("test run"))
+        """.format(
+                mark=mark
+            )
+        )
+
+    def test_shown_by_default(self, testdir):
+        self.create_file(testdir)
+        result = testdir.runpytest_subprocess()
+        result.stdout.fnmatch_lines(
+            [
+                "*== %s ==*" % WARNINGS_SUMMARY_HEADER,
+                "*test_shown_by_default.py:3: DeprecationWarning: collection",
+                "*test_shown_by_default.py:7: PendingDeprecationWarning: test run",
+                "* 1 passed, 2 warnings*",
+            ]
+        )
+
+    def test_hidden_by_ini(self, testdir):
+        self.create_file(testdir)
+        testdir.makeini(
+            """
+            [pytest]
+            filterwarnings = once::UserWarning
+        """
+        )
+        result = testdir.runpytest_subprocess()
+        assert WARNINGS_SUMMARY_HEADER not in result.stdout.str()
+
+    def test_hidden_by_mark(self, testdir):
+        """Should hide the deprecation warning from the function, but the warning during collection should
+        be displayed normally.
+        """
+        self.create_file(
+            testdir, mark='@pytest.mark.filterwarnings("once::UserWarning")'
+        )
+        result = testdir.runpytest_subprocess()
+        result.stdout.fnmatch_lines(
+            [
+                "*== %s ==*" % WARNINGS_SUMMARY_HEADER,
+                "*test_hidden_by_mark.py:3: DeprecationWarning: collection",
+                "* 1 passed, 1 warnings*",
+            ]
+        )
+
+    def test_hidden_by_cmdline(self, testdir):
+        self.create_file(testdir)
+        result = testdir.runpytest_subprocess("-W", "once::UserWarning")
+        assert WARNINGS_SUMMARY_HEADER not in result.stdout.str()
+
+    def test_hidden_by_system(self, testdir, monkeypatch):
+        self.create_file(testdir)
+        monkeypatch.setenv(str("PYTHONWARNINGS"), str("once::UserWarning"))
+        result = testdir.runpytest_subprocess()
+        assert WARNINGS_SUMMARY_HEADER not in result.stdout.str()

tox.ini

@@ -209,6 +209,9 @@ norecursedirs = .tox ja .hg cx_freeze_source testing/example_scripts
 xfail_strict=true
 filterwarnings =
     error
+    ignore:yield tests are deprecated, and scheduled to be removed in pytest 4.0:pytest.RemovedInPytest4Warning
+    ignore:Metafunc.addcall is deprecated and scheduled to be removed in pytest 4.0:pytest.RemovedInPytest4Warning
+    ignore:Module already imported so cannot be rewritten:pytest.PytestWarning
     # produced by path.local
     ignore:bad escape.*:DeprecationWarning:re
     # produced by path.readlines
@@ -217,8 +220,8 @@ filterwarnings =
     ignore:.*type argument to addoption.*:DeprecationWarning
     # produced by python >=3.5 on execnet (pytest-xdist)
     ignore:.*inspect.getargspec.*deprecated, use inspect.signature.*:DeprecationWarning
-    #pytests own futurewarnings
-    ignore::_pytest.experiments.PytestExerimentalApiWarning
+    # pytest's own futurewarnings
+    ignore::pytest.PytestExperimentalApiWarning
 pytester_example_dir = testing/example_scripts
 [flake8]
 max-line-length = 120