During the upgrade to SilverStripe 4 we adopted codecov.io across the board to measure code coverage in our modules and provide CI status checks in GitHub letting us know when they change.
The default criteria for codecov.io is (from what I can tell) an algorithmic based policy where:
- overall project coverage shouldn’t decrease at all
- the coverage for the patch you contribute should have at least the overall project coverage covered in it
If either of these conditions fail, the CI checks fail.
Generally speaking we always ignore these checks and merge anyway. This means that the overall “status checks” have often failed for PRs and commits in GitHub, even if Travis (which is what we really care about) is passing.
I see two solutions to this:
- We can adjust the default configuration to be more lenient on coverage changes: Codecov. This can be done globally for
silverstriperepositories by using “organisation configuration” in the codecov.io web UI.
- Remove the codecov.io status checks altogether.
For option 1, we would need to decide what is the absolute baseline we’d expect before we actively say “no, this pull request doesn’t have enough test coverage for us to merge it yet” (in nicer words!). Perhaps an overall reduction of 10% coverage for the project would be OK, while allowing 0% coverage for the patch? I suggest this because sometimes private static changes will show up as 0% coverage, or fixing a typo somewhere.
Benefits for option 1: still surface code coverage metrics on every PR, configuration can be changed once for everything.
For option 2, we could still run coverage in a way like we do with framework now where it’s on a cron schedule only, but doesn’t run on every PR.
Benefits for option 2: Travis builds generally will be faster (since we’re not running coverage every time). Drawback: we have to change
.travis.yml files in every repo, over time.
It’d be cool to get some thoughts on either option here!