Previously, the test failed when the directory was not present, which could have been caused for example by invoking the tests with PYTHONDONTWRITEBYTECODE=1. Fixes https://github.com/pytest-dev/pytest/issues/5664