|
pulsatrix 1.1.0
|
How CompareBenchmarks decides that a candidate got worse. More...
#include <benchmark.hpp>
Public Attributes | |
| double | time_tolerance = 0.10 |
| A time regresses when the candidate's median exceeds the baseline's by more than this fraction. | |
| double | metric_tolerance = 1e-5 |
| A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics such as conservation error are near zero, so a fraction of the baseline would be meaningless). | |
| double | significance = 0.05 |
| With at least three rounds on each side, a time also needs the candidate's rounds to be slower than the baseline's with this one-sided Mann-Whitney p-value. Separate processes of the same build can differ by 20% on sub-millisecond work, so medians alone raise false alarms; a real slowdown makes every round slower. | |
How CompareBenchmarks decides that a candidate got worse.
| double pulsatrix::CompareOptions::metric_tolerance = 1e-5 |
A metric regresses when the candidate exceeds the baseline by more than this absolute amount (metrics such as conservation error are near zero, so a fraction of the baseline would be meaningless).
| double pulsatrix::CompareOptions::significance = 0.05 |
With at least three rounds on each side, a time also needs the candidate's rounds to be slower than the baseline's with this one-sided Mann-Whitney p-value. Separate processes of the same build can differ by 20% on sub-millisecond work, so medians alone raise false alarms; a real slowdown makes every round slower.
| double pulsatrix::CompareOptions::time_tolerance = 0.10 |
A time regresses when the candidate's median exceeds the baseline's by more than this fraction.