How it works

flowchart TD
    A["PR or agent branch"] --> B["select: git diff + import graphs"]
    B --> C["run: selected tests, or the full suite"]
    C --> D["record: JUnit per matrix leg"]
    D --> E["rollup: per-test history"]
    E --> F["triage: a verdict per failure"]
    F --> G["shadow compare: caught or missed"]
    G --> H["sticky PR comment"]
    DB[("Postgres
per-test history")]
    B <-.-> DB
    D -.-> DB
    F <-.-> DB

1. Select. The client sends the changed files and what they link to; the server returns a mode (selective or full), each selected test with its reasons, and a ready-to-run command.
2. Run. CI runs that command, or its full suite when the mode is full.
3. Record. One idempotent JUnit upload per matrix leg.
4. Rollup. Per-test history is recomputed over a 90-day window, once per CI run, after every leg.
5. Triage. Every failing test gets a verdict from history before the run.
6. Shadow compare. On a shadow run, each failure is checked against the stored selection: caught, missed (with a cause), or flaky (counted apart).

Selection signals#

• Path mapping: foo.py → test_foo.py, Go package directories, JS/TS test files.
• Import graphs: Go via go list -deps -test; Python via a static graph including parent packages, conftests and module names in strings.
• Co-change from past failing runs, and tests recently failing on main.
• Declared dependencies no import graph can see, as [[depends]] rules in .siftwise.toml, plus always_run globs.

always_run = ["tests/smoke/**", "tests.test_health::*"]  # test file paths or test IDs
known_flaky = ["tests/test_net.py"]       # triage: unlinked failures are flaky

[[depends]]                               # dependencies no import graph can see
tests = "test/cli/**"                     # test file paths, or <Go package dir>/<TestName>
on = ["**/*.go", "!**/*_test.go"]         # ! excludes

A changed or imported test file runs whole, and a changed Go package runs without -run, so tests added by the change run too.

Safe fallbacks#

When unsure, run the full suite: missing a real failure is worse than running extra tests. That covers an unknown or empty diff, no history, a changed root build or CI file, a source file no signal covers, or a selected test with no runner. Docs, lint-only configs and generated files never force it.

Triage verdicts#

Ordered rules decide. A test that passed on the same commit is flaky; one failing on main is pre-existing; one linked to the change, with no flaky evidence, is a new regression; otherwise it's unknown, with low confidence. Triage never guesses and never peeks: only runs before the triaged one count.

new_regression  high   tests.test_cart::test_total
    tests/test_cart.py tests changed src/cart.py
pre_existing    high   tests.test_db::test_query
    failing on main on ubuntu-py3.12 since 4f2a9c1 (3 main runs)
flaky           medium tests.test_net::test_fetch
    flaky 1 time in 90 days (1 unrelated fail-then-pass)
siftwise: 3 failing test(s): 1 new regression, 1 pre existing, 1 flaky, 0 unknown