Introducing siftwise · Part 8 of 10
Limits, and what the numbers don't prove
siftwise is alpha: APIs and the schema may change before 1.0. Known limits:
- One API token per server, with read and write access to every repo on it; no per-repo or read-only tokens.
- Merge-queue (
merge_group), scheduled and manual runs always run the full suite: there's no diff base. - Java tests are recorded but always run in full. Jest selection needs file paths in the
JUnit and isn't evaluated on a real repo.
rerun-unknownre-runs pytest only. - The Action is tested on Linux runners only, and the server image is linux/amd64 only.
- Renamed or moved tests start with no history.
And what the evals don't show:
- recall on repos with dynamic dependencies the graphs can't see (plugins loaded by name
from outside the repo, data-driven tests), beyond what
[[depends]]covers; - behaviour on languages other than Python and Go;
- live recall: every number here replays past CI history of four repos;
- that flaky verdicts generalise, since most re-read the label's own evidence.