Postmortem: Fetch Test April 2020

A Fetch Test was ran in April 2020. The process to run took a little over 24 hours to complete. In result of time constraints, only entirely disabled rulesets were the focus.

Results

Full scan and amendments made in #19339, resulted in 1793 impacted rulesets, of which the majority were disabled.

After Test

After review of false positives, giving recently expired certificates a grace period for COVID-19, and re-adding onion rulesets that were turned off; it became apparent that the check_rules.py script should be amended to mitigate these false positives

Process Notes

Doing a full scan resulted in large set to review. However with patches created from the Fetch test, the review for errors was able to be sampled by maintainers to be able to catch the false positives and issues listed. Resulting in being able to clean rulesets overall with the guidance from the sample. In the future, fetch test will be ran in smaller batches that are more reviewable.

New Fetch Test Rubric

https://github.com/EFForg/https-everywhere/wiki/Fetch-Test-Rubric

Issues to amend before next Fetch Test

• Amend Check Rules Script for Future Fetch Tests

• Travis fetch test should not pass for revoked certificates