Exposure Validation Finder

Lesson 7 of 8

Metrics that hold up

A good validation metric changes when the defences change and stays still when they do not. Many popular numbers fail that test.

The test for a metric

Ask two questions of any number you plan to report. If the defences improved, would this number improve? If nothing changed in the defences but the test set changed, would this number move anyway? A metric that passes the first and fails the second is worth reporting. One that moves with the test set is measuring the tool, not you.

Control metrics

Prevention and detection rates per control, on a fixed test set. Choose a set of techniques that matter to you, for example the thirty most used against your sector, and run the same set every cycle. Report the share prevented, the share detected but not prevented, and the share missed, for each control. Keep the set fixed for at least a year; add new techniques as a separate set so the trend stays comparable.

Time to detect. Validation tools know exactly when each technique ran, so they can measure the gap to the first alert. Report the median and the slowest cases, not only the mean. Agree what counts as detection: an event logged, an alert raised, or an analyst acting on it. SCYTHE, for example, describes tracking pass, degraded and missed results and mean time to detect across scheduled tests.

Drift events. The number of controls that passed last cycle and failed this one. This is one of the most useful numbers validation produces, because nothing else measures it. Each drift event should have a cause written next to it.

Path metrics

Targets reachable from each starting point. For each named crown-jewel asset, record whether a test from each starting point reached it. A yes-or-no grid, repeated each cycle, is easy to read and hard to argue with.

Steps to target. When a target is reachable, how many steps did it take? Longer paths give defenders more chances to detect the attacker. A path that grew from three steps to seven after a fix shows progress even before it is closed.

Choke points closed. The number of high-traffic steps removed this cycle, with the number of paths each one broke.

Remediation metrics

Retest pass rate. Of the fixes marked done, the share that passed the same test afterwards. If this number is below about four in five, the fix process has a problem worth looking at before anything else.

Time from finding to verified fix. Measured to the passing retest, not the closed ticket. Report it per owner team so the slow queues are visible.

Open exceptions with expiry dates. How many accepted risks exist, and how many have passed their expiry without review.

Numbers to treat with care

ATT&CK coverage percentages. "We cover 85% of ATT&CK" usually means the tool has at least one test for 85% of techniques. It says nothing about how many procedures per technique, or whether your controls stop them. MITRE also changes the framework twice a year. Version 19, released 28 April 2026, split the Defense Evasion tactic in two, so a coverage figure from before that date is not comparable with one after.

Vendor posture scores. Many platforms compute a single score from test results. Each vendor defines its own formula, so the score cannot be compared across products and can change when the vendor changes the formula. Use it inside one tool as a trend if you like; never compare it with another tool's score.

Total findings. The count of findings mostly measures how much you tested. More tests produce more findings. Report findings fixed and verified instead.

Library size. Vendors publish attack library sizes, and they are hard to compare. Cymulate's AWS listing claims more than 100,000 attack scenarios; other vendors count differently or not at all. A larger library is not a better test of your defences unless the tests you run are the ones that matter to you.

How to report upward

Executives need three things from validation: whether the most important assets are reachable, whether key controls work, and whether fixes are getting done. One slide can carry all three: the target reachability grid, the prevention and detection rates on the fixed set, and the retest pass rate. Add the drift events if any were serious.

Keep the same slide every quarter. The value is in the comparison, and a new format every quarter destroys it.

What to take from this lesson

Pick four or five metrics before the first run and record a baseline. Choose at least one from each group: control, path and remediation. Write down how each is calculated so the number means the same thing next year, when someone else may be reporting it. The final lesson uses these numbers to make the case for a budget.


Previous: Turning findings into fixes. Next: Building the business case.