20250227

test coverage threshold with pytest-cov

We have a GitHub Actions workflow that runs tests and checks test coverage. I noticed that a run didn’t fail even though it seemed like it should have at first glance.

The confusing part is that, it said “FAIL” in the result messages by pytest-cov, but did not exit with code 1 (Instead, we got exit code 0), which

When the task fails

Run poetry run pytest --cov=. --cov-fail-under=72
FAIL Required test coverage of 72% not reached. Total coverage: 70.60%

Error: Process completed with exit code 1.

When the task succeeds even though the test coverage is under the given value

Run poetry run pytest --cov=. --cov-fail-under=71
FAIL Required test coverage of 71% not reached. Total coverage: 70.60%

# exit with code 0. thus, the GitHub Actions workflow does not fail.

I didn’t have much time today to dive deep into the code base (sorry!), but it looks like the comparition is being made using the rounded value with the specified precision.

pytest-cov/src/pytest_cov/plugin.py

class CovPlugin:
    def pytest_terminal_summary(self, terminalreporter):
        if self.options.cov_fail_under is not None and self.options.cov_fail_under > 0:
            failed = self.cov_total < self.options.cov_fail_under
            message = '{fail}Required test coverage of {required}% {reached}. ' 'Total coverage: {actual:.2f}%\n'.format(
                required=self.options.cov_fail_under,
                actual=self.cov_total,
                fail='FAIL ' if failed else '',
                reached='not reached' if failed else 'reached',
            )

This condition in pytest-cov reports FAIL by comparing (unrounded?) coverage total and the given value for --cov-fail-under. However,

coveragepy/coverage/results.py

def should_fail_under(total: float, fail_under: float, precision: int) -> bool:
    """Determine if a total should fail due to fail-under.

    `total` is a float, the coverage measurement total. `fail_under` is the
    fail_under setting to compare with. `precision` is the number of digits
    to consider after the decimal point.

    Returns True if the total should fail.

    """
    # We can never achieve higher than 100% coverage, or less than zero.
    if not (0 <= fail_under <= 100.0):
        msg = f"fail_under={fail_under} is invalid. Must be between 0 and 100."
        raise ConfigError(msg)

    # Special case for fail_under=100, it must really be 100.
    if fail_under == 100.0 and total != 100.0:
        return True

    return round(total, precision) < fail_under

The condition rounds the total test coverage, leading to the mismatch.

This might be related to this issue #638

Anyway, I’m glad I could find out what happened.

TODO:

index 20250226 20250228