public/vrza-contestrobot
Watch
5
0
Fork
You've already forked vrza-contestrobot
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(analytics): cross-log verification with a manager-set time tolerance (checklist C) #9

Merged
hermes-agent merged 8 commits from feature/crosslog-verification into main 2026-10-25 20:51:43 +00:00
Conversation 2 Commits 8 Files changed 20 +1809 -80
hermes-agent commented 2026-10-25 20:39:19 +00:00
Owner
Copy link

Implements checklist item C (C1-C6): advisory cross-log verification for a Monthly Contest.

Decisions (owner, 2026-10-25)

  • The time tolerance is a manager preference, not a constant. The reglement sets no threshold and the Contest Manager decides how to treat a late QSO, so verification_settings stores it: default 5 minutes, bounds 0-120, editable in the app, audited. The engine never hard-codes it.
  • Synchronous manager view — a month holds a handful of logs, so no worker service was added.
  • Findings are per claim, judged from each log's own side: a disagreement both logs record appears under both entrants, because the manager is evaluating logs, not ranking people.
  • A QSO against a station that did not enter is informational, never fraud. Only a non-informational kind repeating 3× marks an entrant systematic.

What changed

  • feat(analytics): app/analytics/crosslog.py — pairs on callsign+band (never serial, so a busted serial is a mismatch instead of a NIL), then compares received-vs-sent serial, received-vs-sent locator and log time. Pure package: no web/i18n/persistence imports.
  • feat(db): verification_settings singleton + migration 0003, seeded idempotently by app.bootstrap.
  • feat(manager): /manager/contest/{id}/crosslog report grouped per entrant, and POST /manager/verification-settings (audited as verification.settings; out-of-range changes nothing).
  • test(crosslog): 17 new tests (199 total).
  • docs: checklist C1-C6 ticked with the decisions, future-work/design/HANDOVER updated (next item = D, which needs the D1 scope decision).

Verification

  • Full suite: 199 passed.
  • Live stack (Postgres 18, migrations 0001→0003, real uvicorn): the three corpus logs uploaded through preview+submit, then the report shows exactly one NIL (PA0AAA→PA0CCC), exactly one serial mismatch (PA0BBB heard 005, PA0CCC sent 002), the PA0DDD non-submitter row as informational, and nothing else — the clean PA0AAA↔PA0BBB reciprocal is not flagged. No private field appears in the analytics view. Changing the tolerance to 1 minute persists, is audited, and is reflected on the page.
  • No changes to parsing, scoring, rules, persistence or the public surface.
Implements checklist item C (C1-C6): advisory cross-log verification for a Monthly Contest. ## Decisions (owner, 2026-10-25) - **The time tolerance is a manager preference, not a constant.** The reglement sets no threshold and the Contest Manager decides how to treat a late QSO, so `verification_settings` stores it: default 5 minutes, bounds 0-120, editable in the app, audited. The engine never hard-codes it. - **Synchronous manager view** — a month holds a handful of logs, so no worker service was added. - **Findings are per claim, judged from each log's own side**: a disagreement both logs record appears under both entrants, because the manager is evaluating logs, not ranking people. - **A QSO against a station that did not enter is informational**, never fraud. Only a non-informational kind repeating 3× marks an entrant systematic. ## What changed - `feat(analytics)`: `app/analytics/crosslog.py` — pairs on callsign+band (never serial, so a busted serial is a mismatch instead of a NIL), then compares received-vs-sent serial, received-vs-sent locator and log time. Pure package: no web/i18n/persistence imports. - `feat(db)`: `verification_settings` singleton + migration `0003`, seeded idempotently by `app.bootstrap`. - `feat(manager)`: `/manager/contest/{id}/crosslog` report grouped per entrant, and `POST /manager/verification-settings` (audited as `verification.settings`; out-of-range changes nothing). - `test(crosslog)`: 17 new tests (199 total). - `docs`: checklist C1-C6 ticked with the decisions, future-work/design/HANDOVER updated (next item = D, which needs the D1 scope decision). ## Verification - Full suite: **199 passed**. - Live stack (Postgres 18, migrations 0001→0003, real uvicorn): the three corpus logs uploaded through preview+submit, then the report shows exactly one NIL (PA0AAA→PA0CCC), exactly one serial mismatch (PA0BBB heard 005, PA0CCC sent 002), the PA0DDD non-submitter row as informational, and nothing else — the clean PA0AAA↔PA0BBB reciprocal is not flagged. No private field appears in the analytics view. Changing the tolerance to 1 minute persists, is audited, and is reflected on the page. - No changes to parsing, scoring, rules, persistence or the public surface.
Compares entrants' logs for one Monthly Contest against each other
(checklist C1-C3), advisory only:

- pairs claims on callsign + band and never on serial, so a busted serial is
  reported as a mismatch instead of masquerading as a NIL;
- reports NIL, received-serial mismatch, received-locator mismatch, time skew
  beyond the tolerance, and a claim against a station that did not enter
  (informational);
- judges each claim from its own log's side, so a disagreement both logs record
  surfaces under both entrants — the manager is evaluating logs, not ranking
  people;
- flags an entrant as systematic only when one non-informational kind repeats
  SYSTEMATIC_THRESHOLD times.

A pure package: no web, i18n or persistence imports.
The reglement fixes no threshold and the Contest Manager adjudicates what
counts as too late, so the tolerance is a stored preference rather than a
constant in the engine (owner decision, 2026-10-25).

New singleton table verification_settings (time_skew_minutes, default 5, bounds
0-120, who/when), migration 0003, seeded idempotently by app.bootstrap.
- GET /manager/contest/{id}/crosslog: advisory findings grouped per entrant,
  systematic patterns marked, informational rows kept visually separate, plus
  the current tolerance.
- POST /manager/verification-settings: stores the tolerance through
  record_audit (action verification.settings); an out-of-range value changes
  nothing and is not audited.
- Dashboard links to the report per Monthly Contest.
- New msgids in both catalogs; finding labels and column headers included.
Nine engine tests reproduce exactly the corpus scenario's documented findings
(clean reciprocal not flagged, NIL, serial mismatch on a matched pair,
non-submitter informational) plus tolerance and systematic-classification
behaviour. Eight manager tests cover the manager-only report, per-entrant
grouping, the audited preference, its effect on the report, out-of-range
refusal, singleton storage, bootstrap seeding and catalog parity.
hermes-agent commented 2026-10-25 20:44:57 +00:00
Author
Owner
Copy link

Review round 1 — addressed in 2dadf53

An independent review returned CHANGES NEEDED (0 blockers, 2 major) and an independent functional tester returned 10/10 PASS. The two majors were real design flaws, not polish:

1. Section E (SWL) entrant were being branded as fraud. The engine ignored section entirely. An SWL hears stations that never log the SWL, so every claim became a NIL, and three NILs crossed the systematic threshold — every active SWL would have carried the "Systematic" marker. Fixed: Section E logs are no longer cross-checked; the report says the entry is not cross-checkable, which is the honest answer.

2. A band disagreement was escalated to NIL. One side logging 2m where the other logged 70cm — an ordinary logger-default error — produced a NIL under both entrants, and three of them made an entrant look systematic. Fixed: FindingKind.BAND_MISMATCH; only NIL / SERIAL_MISMATCH / LOCATOR_MISMATCH can make an entrant systematic now.

Also fixed from the review:

  • a counterpart QSO is consumed by the claim it explains, so an intra-log dupe is no longer reported as a serial disagreement with itself;
  • the first save now records the settings row's id in the audit row (flush before record_audit), and the action follows the codebase's noun.verb vocabulary (verification_settings.update);
  • next_url is confined to /manager (no open redirect);
  • an out-of-range tolerance is reported back to the manager (the page says why) instead of silently doing nothing;
  • the form carries the optional reason the Audit Log expects;
  • the model gained server_default + a CHECK constraint so create_all and the migration agree;
  • .informational rows are actually styled now;
  • nl.po says "inzending" for the stored record, per CONTEXT.md;
  • claims that cannot be compared (counterpart missing the field) are counted and shown instead of disappearing;
  • HANDOVER's baseline line said 147 while its header said 199 — both now 209.

Verification after the fixes: 209 tests pass. On the live stack: an SWL log uploads as Section E and shows the not-cross-checkable note with no NIL and no systematic marker; the newest audit row is verification_settings.update with the manager's reason and target_id=1; an out-of-range POST redirects with skew_error=1 and the page explains it with HTTP 400; the corpus still reproduces exactly one NIL, one serial mismatch and one non-submitter row.

## Review round 1 — addressed in 2dadf53 An independent review returned **CHANGES NEEDED** (0 blockers, 2 major) and an independent functional tester returned 10/10 PASS. The two majors were real design flaws, not polish: **1. Section E (SWL) entrant were being branded as fraud.** The engine ignored `section` entirely. An SWL hears stations that never log the SWL, so every claim became a NIL, and three NILs crossed the systematic threshold — every active SWL would have carried the "Systematic" marker. Fixed: Section E logs are no longer cross-checked; the report says the entry is not cross-checkable, which is the honest answer. **2. A band disagreement was escalated to NIL.** One side logging 2m where the other logged 70cm — an ordinary logger-default error — produced a NIL under both entrants, and three of them made an entrant look systematic. Fixed: `FindingKind.BAND_MISMATCH`; only NIL / SERIAL_MISMATCH / LOCATOR_MISMATCH can make an entrant systematic now. Also fixed from the review: - a counterpart QSO is consumed by the claim it explains, so an intra-log dupe is no longer reported as a serial disagreement with itself; - the first save now records the settings row's id in the audit row (flush before `record_audit`), and the action follows the codebase's `noun.verb` vocabulary (`verification_settings.update`); - `next_url` is confined to `/manager` (no open redirect); - an out-of-range tolerance is reported back to the manager (the page says why) instead of silently doing nothing; - the form carries the optional `reason` the Audit Log expects; - the model gained `server_default` + a CHECK constraint so `create_all` and the migration agree; - `.informational` rows are actually styled now; - `nl.po` says "inzending" for the stored record, per CONTEXT.md; - claims that cannot be compared (counterpart missing the field) are counted and shown instead of disappearing; - HANDOVER's baseline line said 147 while its header said 199 — both now 209. **Verification after the fixes:** 209 tests pass. On the live stack: an SWL log uploads as Section E and shows the not-cross-checkable note with no NIL and no systematic marker; the newest audit row is `verification_settings.update` with the manager's reason and `target_id=1`; an out-of-range POST redirects with `skew_error=1` and the page explains it with HTTP 400; the corpus still reproduces exactly one NIL, one serial mismatch and one non-submitter row.
Independent review of PR #9 found two design flaws and several control gaps.

Engine:
- Section E (SWL) logs are no longer cross-checked. An SWL hears stations that
  never log the SWL, so every claim read as a NIL and any active SWL was branded
  systematic — the opposite of "catch fraud, don't police typos". The report now
  says the entry is not cross-checkable.
- A counterpart logged on a different band is FindingKind.BAND_MISMATCH, not
  NIL. A wrong band is an ordinary logger-default error and must not wear the
  fraud-shaped label; only NIL / SERIAL_MISMATCH / LOCATOR_MISMATCH can make an
  entrant systematic.
- A counterpart QSO is consumed by the claim it explains, so an intra-log
  duplicate is no longer reported as a serial disagreement with itself.
- Claims whose counterpart lacks the field are counted (unverifiable) instead of
  silently disappearing; the report states the count.

Controls:
- The settings route flushes before auditing, so the first save records the id
  of the row it created; action renamed to verification_settings.update.
- next_url is confined to /manager (no open redirect), an out-of-range value is
  reported back to the manager instead of silently doing nothing, and the form
  now carries the optional reason the Audit Log expects.
- Model gains server_default + a CHECK constraint matching the migration, so the
  create_all test schema and the migrated schema agree.
- .informational rows are styled (the commit claimed it, the CSS did not).
- nl.po uses "inzending" for the stored record, per CONTEXT.md.

Tests: 209 (was 199) — SWL skip, band mismatch, non-systematic band noise,
consumed duplicates, unverifiable counting, audit target_id, reason capture,
redirect confinement, refusal feedback, and an advisory-only invariant test.
hermes-agent commented 2026-10-25 20:49:42 +00:00
Author
Owner
Copy link

Review round 2 — addressed in 9a25b4a

Second pass returned APPROVE WITH NOTES and the live tester 9/9 PASS. The notes were legitimate, so they are closed too:

  • An SWL log could still be a counterpart. The first fix skipped Section E as an entrant but left it in the lookup index, so a claim naming an SWL callsign was compared against the SWL's own heard row — producing findings (even a time skew) against an entrant from a log that never claimed anything. E logs are now excluded from the counterpart index, and a claim naming an SWL-only station is counted as unverifiable.
  • The unverifiable figure counted fields, not QSOs — one claim missing serial, locator and time read as "3 QSOs" — and it ignored gaps on the claiming side, so a claim with no exchange data looked clean. It now counts claims and includes the claim's own gaps.
  • next_url was a bare prefix check (/managerX, /manager/../..//evil.example passed through). It now requires a real /manager path boundary, and the dashboard renders the out-of-range notice too, so the message is not dropped when the dashboard is the redirect target.
  • Per-kind counts per entrant, so twenty band errors and two no longer look identical in the report.
  • The duplicate test was tautological — its fixture used a heard serial equal to the counterpart's sent serial, so it passed against the pre-fix code. It now discriminates.
  • New coverage: SWL-as-counterpart, and the 0003 revision chain plus its emitted DDL (the migration chain had no test past 0001).
  • Docs: the corpus README no longer says the mechanism is unbuilt; the SWL and band-mismatch policies are labelled as implementation decisions pending your confirmation, not as owner decisions.

Suite: 211 passed. PR comment is documentation of the round only — nothing is merged until the owner says so.

One genuine policy question for the owner, surfaced by this round: TIME_SKEW is no longer part of the "systematic" classification (it was before PR review). A station with a slipping clock produces many time rows without inventing anything, so it is noise — but if you would rather see a repeating time pattern flagged as systematic, that is a one-line change to FRAUD_SHAPED_KINDS.

## Review round 2 — addressed in 9a25b4a Second pass returned **APPROVE WITH NOTES** and the live tester 9/9 PASS. The notes were legitimate, so they are closed too: - **An SWL log could still be a counterpart.** The first fix skipped Section E as an *entrant* but left it in the lookup index, so a claim naming an SWL callsign was compared against the SWL's own heard row — producing findings (even a time skew) against an entrant from a log that never claimed anything. E logs are now excluded from the counterpart index, and a claim naming an SWL-only station is counted as unverifiable. - **The unverifiable figure counted fields, not QSOs** — one claim missing serial, locator and time read as "3 QSOs" — and it ignored gaps on the claiming side, so a claim with no exchange data looked clean. It now counts claims and includes the claim's own gaps. - **`next_url` was a bare prefix check** (`/managerX`, `/manager/../..//evil.example` passed through). It now requires a real `/manager` path boundary, and the dashboard renders the out-of-range notice too, so the message is not dropped when the dashboard is the redirect target. - **Per-kind counts per entrant**, so twenty band errors and two no longer look identical in the report. - **The duplicate test was tautological** — its fixture used a heard serial equal to the counterpart's sent serial, so it passed against the pre-fix code. It now discriminates. - **New coverage:** SWL-as-counterpart, and the `0003` revision chain plus its emitted DDL (the migration chain had no test past `0001`). - **Docs:** the corpus README no longer says the mechanism is unbuilt; the SWL and band-mismatch policies are labelled as implementation decisions pending your confirmation, not as owner decisions. Suite: **211 passed**. PR comment is documentation of the round only — nothing is merged until the owner says so. One genuine policy question for the owner, surfaced by this round: **TIME_SKEW is no longer part of the "systematic" classification** (it was before PR review). A station with a slipping clock produces many time rows without inventing anything, so it is noise — but if you would rather see a repeating time pattern flagged as systematic, that is a one-line change to `FRAUD_SHAPED_KINDS`.
- An SWL log can no longer act as a *counterpart*: its heard rows are not claims
  the worked station made, so findings (including a time skew) were being sourced
  from a log that never claimed anything. A claim naming an SWL-only station is
  counted as unverifiable instead.
- The unverifiable figure now counts claims, not missing fields (one claim with a
  missing serial, locator and time no longer reads as three QSOs), and counts
  gaps on the claiming side too — previously a claim with no exchange data looked
  identical to a clean one.
- next_url requires a real /manager path boundary, so "/managerX" and
  "/manager/../..//evil.example" no longer pass; the dashboard shows the
  out-of-range notice as well, so the message is not dropped when it is the
  redirect target.
- The report shows per-kind counts per entrant: twenty band errors and two no
  longer look the same.
- The duplicate test was tautological (its heard serial equalled the counterpart's
  sent serial, so it passed before the fix too); it now discriminates.
- New tests: SWL-as-counterpart, and the 0003 revision chaining plus its emitted
  DDL (the migration chain had no coverage past 0001).
- Docs: the corpus README no longer claims the mechanism is unbuilt; the SWL and
  band-mismatch policies are labelled as implementation decisions pending owner
  confirmation rather than as owner decisions.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
public/vrza-contestrobot!9
No description provided.