Severity levels for run comparisons #1

New issue

Closed

opened 2026-08-08 20:17:15 +00:00 by bjoern · 0 comments

bjoern commented

2026-08-08 20:17:15 +00:00

Owner

As an admin running the ActivityPub compatibility tester,
I want each difference between test runs to carry a severity level (info/low/medium/high/critical), computed from configurable rules,
so that I can immediately tell from the terminal, the TUI Compare screen, or a notification whether a detected change is benign noise or a serious regression — without digging into raw diffs.

Acceptance criteria:

Every DiffEntry produced by a comparison carries a severity with a backward-compatible default (info); old result files without the field still load.
With an unconfigured comparison, all diffs fall back to the configured severity.default level.
Severity is computed in two stages from configurable rules:
- Stage A: static regex rules matching category, endpoint, change_type, diff field, and old_value/new_value.
- Stage B: JSON-body evaluation that parses the newer run's stored response_body via a standard-library dotted/bracket path and optionally compares it to a previous value or a threshold.
The first matching rule wins; unmatched diffs fall back to severity.default.
aptester -c config/default.yaml compare <run1.json> <run2.json> and the differel printed after run show a color-coded severity column; the TUI Compare screen shows the severity label.
Notifications (webhook + ntfy) include the severity per diff, and a min_notification_severity threshold filters which changes trigger a notification.
Desired/benign changes can be tagged info/low (e.g. version commit-hash flips, user/status count drift), regressions high/critical (e.g. an endpoint removed, or a status flipping pass → fail/error).
No new runtime dependency — the JSON path extractor is implemented in the standard library.
Documentation (default.sample.yaml, CHANGELOG.md) and tests stay in sync.

**As** an admin running the ActivityPub compatibility tester, **I want** each difference between test runs to carry a severity level (`info/low/medium/high/critical`), computed from configurable rules, **so that** I can immediately tell from the terminal, the TUI Compare screen, or a notification whether a detected change is benign noise or a serious regression — without digging into raw diffs. Acceptance criteria: 1. [x] Every DiffEntry produced by a comparison carries a severity with a backward-compatible default (info); old result files without the field still load. 2. [x] With an unconfigured comparison, all diffs fall back to the configured severity.default level. 3. [x] Severity is computed in two stages from configurable rules: - Stage A: static regex rules matching category, endpoint, change_type, diff field, and old_value/new_value. - Stage B: JSON-body evaluation that parses the newer run's stored response_body via a standard-library dotted/bracket path and optionally compares it to a previous value or a threshold. 4. [x] The first matching rule wins; unmatched diffs fall back to severity.default. 5. [x] aptester -c config/default.yaml compare <run1.json> <run2.json> and the differel printed after run show a color-coded severity column; the TUI Compare screen shows the severity label. 6. [x] Notifications (webhook + ntfy) include the severity per diff, and a min_notification_severity threshold filters which changes trigger a notification. 7. [x] Desired/benign changes can be tagged info/low (e.g. version commit-hash flips, user/status count drift), regressions high/critical (e.g. an endpoint removed, or a status flipping pass → fail/error). 8. [x] No new runtime dependency — the JSON path extractor is implemented in the standard library. 9. [x] Documentation (default.sample.yaml, CHANGELOG.md) and tests stay in sync.

bjoern added the

enhancement

label

2026-08-08 20:17:15 +00:00

bjoern self-assigned this

2026-08-08 20:17:15 +00:00

bjoern added reference feature/0001-severity-levels

2026-08-08 21:14:43 +00:00

bjoern added a new dependency

2026-08-16 16:06:22 +00:00

#2 Display filter for diff severity

bjoern referenced this issue