Evidence

These are replays of real CI history with no peeking: for each target run, history is rebuilt only from runs before it, and import graphs are built at that commit. The repos were picked for having JUnit artifacts and real failures, not for flattering results. The method is in the eval methodology; every number below is in eval results.

Selection replays

The caveats matter as much:

• rdflib: failures cluster (one bad change can fail hundreds of tests), so 140 of 141 in the 18 selective runs is the more honest figure. The miss, test_service_with_bound_solutions, calls a remote SPARQL service and had failed in 2 earlier unrelated runs: it's flaky. Its 10.4% comes from two lint-only PRs.
• substrate: without declared rules, selective runs caught only 26 of 38; the misses were end-to-end suites no import edge reaches. Two [[depends]] rules close the gap at the cost of most of the savings (5.1% skipped vs 19.5%). Only 8 days of history.
• Cirq measures savings, not safety: there are no PR test results. Before a fix, a nearly empty import graph made it claim an unsafe 59.8%.
• kubo used older (week-2) code, and every failing run fell back to the full suite, so selective recall is unmeasured there.

Triage eval

Every run with failures was triaged by the production engine as of just before it, then scored against proxy labels from what happened afterwards.

• Regressions called flaky, the worst error: 0 on rdflib; 5 of 406 labelled regressions on substrate, all on tests with earlier same-commit flakes; 4 are likely label noise.
• new_regression held up: 319 of 319 labelled substrate failures, and 10 of 11 rdflib fix episodes with a known diff.
• pre_existing: 2,092 of 2,092 on substrate, including 759 of 759 called at medium confidence after a single failing main run.
• flaky precision is mostly by construction: 678 of 683 flaky verdicts come from a pass of the same commit, the same fact the flaky label uses.
• Unknown when unsure: 12.6% of substrate's failures and 14% of rdflib's, mostly runs whose diff was never recorded.

Caveats: the labels are proxies that partly share evidence with the rules, and episodes are few (12 on rdflib, 34 on substrate), so treat percentages as counts. Blind spot: on rdflib, where every change links to every test and CI never re-runs a commit, the remote-service tests were called new_regression 12 times out of 13.

Shadow mode

No live shadow-mode recall numbers are published yet; that comes after launch. The mechanism is tested end to end: the project's action-e2e.yml runs all three actions against a real server on every push and PR, including a shadow run with a miss. The screenshot is that comment; the tests are the workflow's fixtures, not a real project's.

Live numbers will be raw counts: failures are rare on healthy repos, and zero misses in n failures only bounds the miss rate below about 3/n at 95%.

Performance

Measured on 2026-10-19 on a Windows laptop, Postgres 16 in Docker; results identical before and after.

/select takes about 0.4 s on rdflib's 18,000 tests (0.3–1.4 s across the measured changes). Not everything is fast: the repo-wide stats rollup takes 119 s on rdflib, and that wasn't changed.