The flake budget

A check has to fail twice in a row before this desk calls it a finding.

Why

A crawl of 74 hosts runs about 800 checks. If one check in five hundred fails for a reason that has nothing to do with the host, a clean network still produces a finding or two every single run.

What happens next is predictable. The first few get investigated. Then someone notices they are usually nothing. Then the whole report gets skimmed. A report that is usually wrong in a small way is worse than no report, because it teaches people that red is normal.

The rule

Each run compares its failures against the previous run. A check failing for the first time is recorded, but held back. A check failing twice in a row is promoted to a finding. A check that passes resets its streak to zero.

The near-misses are kept in the report file rather than discarded, which is what lets the streak survive into the next run.

The number I publish

Every report carries this, beside the totals:

"flakeBudget": {
  "requiredConsecutiveFailures": 2,
  "suppressed": 3
}

suppressed is how many failures the rule hid from you in that run.

That number is the point. A flake budget you cannot see is not a budget, it is a filter, and a filter you cannot see is just a lie with good intentions. If suppressed starts climbing, the rule is covering for something and I want that visible in the same file as the findings it is covering.

What it costs

A real outage that starts and ends between two runs never becomes a finding. That is a genuine miss, and it is the price. Sweep is not a monitor and does not page anyone; it is a scheduled audit whose output is a file in git. For that job, missing a blip is cheaper than crying wolf.

If this network ever needs something that catches a blip, that is a different tool with a different budget, and it should say so on its own page.

Verifying the rule rather than trusting it

The flake budget is logic, so it is tested like logic. In the sweep repo:

npm test

The tests assert that a first failure is held back, that a repeat is promoted, that a pass resets the streak, and that the run a finding was first seen in survives the promotion. If someone loosens the rule, those fail.