Skip to content
Testament
How it’s tested

This dashboard is tested the same way as the projects it reports on.

A test dashboard that isn’t tested is just a claim. Testament is built test-first. Its own CI runs every suite on every pull request and reports the results here, alongside everything else.

Principles

Five rules from the specification. Each one exists because breaking it has cost real time on a real project.

  1. Test-first

    Every phase starts with failing tests written from its acceptance criteria. A phase is done when each criterion has a passing automated test or a recorded manual check.

  2. Real fixtures

    Parsers are tested against actual result files captured from Ostomate2, RouteServe and Testament itself: Gradle JUnit XML, Jest JSON, Vitest and Playwright output, JaCoCo and istanbul coverage.

  3. Normalize on the server

    Projects send the files their tools already produce. Testament does the parsing, so a project’s CI step stays a few lines long.

  4. Never fail the caller’s build

    If Testament is down, a project’s CI stays green. The reporting step always exits 0 and logs a warning.

  5. No swallowed failures

    In Testament’s own pipeline, nothing is allowed to fail quietly. No || true, no continue-on-error on test steps.

What runs, and what it covers

Parsers are tested against result files captured from Ostomate2 and RouteServe, not hand-written samples. Integration tests run against a local Supabase, including the row-level security that hides private projects from the public.

LayerToolScope
UnitVitestParsers against real fixtures, layer resolution, stat calculations, API key hashing. 90% line floor.The alert detail schemas; the daily job’s step order and failure handling; the cron route’s secret check; bot and link-preview classification of user agents; tracked-link token generation and lookup; admin sign-in: the sign-in action’s one neutral answer, the sign-in link’s confirm route, the owner allowlist check, and the guard every admin page and action calls first.
ContractVitestThe shared GitHub Action every project reports through, run against a local server. Runs in the unit job; counted as Unit.
ComponentVitestTesting LibraryEach component built from the design, rendered in jsdom: its words, states and links, and its stylesheet against the design’s values.
IntegrationVitestlocal SupabaseThe ingestion endpoint end to end against local Supabase: idempotent retries, transactions and rollups, and row-level security checked with the anonymous client, so private projects stay hidden.The alert rules (stale, count drop, empty report, below floor), which are SQL, against local Supabase with captured fixtures. The daily job against local Supabase: heartbeat, opening stale alerts, and pruning results at 180 days and visits and tracked sessions at 365. Visit recording, with bots and previews kept out of every count and no raw IP address stored in visits or tracked sessions. Admin sign-in and its actions (create, acknowledge, rotate, delete) against local Supabase Auth with a real session, each refused without one.
E2EPlaywrightA seeded database behind the landing, project, run and test-history pages; visibility rules in the rendered UI; every public page again with JavaScript turned off.The tracked-link flow: open a link, check its cookie and redirect, see the lead project first, and find the visits stored against the link; nothing the visitor sees names the company, role, contact or token. Admin sign-in with the email opened from a local mail catcher, a spent or made-up link and another address. Admin’s sections with seeded links and visits, each action, and every action refused once the session has gone. The run page’s pruned-results notice, after a real prune.
AccessibilityaxePlaywrightcontrast scriptaxe on every public page, counted with E2E. A script in the unit job checks the contrast of the design’s token pairs in both themes.
VisualPlaywrightSnapshots of each page at desktop and phone widths, light and dark, compared inside a pinned container image so rendering can’t drift. Counted with E2E.
Leak sweepPlaywrightWalks every page that can show a private project as a visitor and fails if any private project’s failure text, stack trace or source link appears anywhere. Counted with E2E.Also checks that admin without a session shows only the sign-in form and nothing hidden.
Coverage floor
90% lines
Enforced by Vitest thresholds in the unit job. Below it, CI fails.
On every pull request
Lint, typecheck, unit, integration, E2E
Failure masking
None
No || true, no continue-on-error on test steps.

Why it watches for silence

Two incidents from Ostomate2’s post-mortems shaped how results are shown. A green dashboard means nothing if the pipeline behind it has stopped running, or if a suite passes without executing anything.

Incident

CI was silently dead for 11 days

Now: each project has an expected cadence. When a project goes quiet past it, its card and project page say so, for example “No report in 12 days”. A daily check flags it on the admin page.
Incident

E2E reported green without ever passing

Now: a run with zero tests executed is marked Empty and shown as a problem, never as a pass. A sudden drop in a report’s test count raises an alert too.

Testament’s own results

Since , Testament’s CI posts its results to this site as the project testpulse, through the same shared GitHub Action Ostomate2 and RouteServe use.

Passed·main·051d1b7
3,615tests ·0 failed·0 skipped ·18m 32s
  • E2E
    2818%
  • Integration
    3058%
  • Component
    74921%
  • Unit
    2,28063%

Only spec §8 layers are counted. The axe, visual and leak-sweep checks run inside the Playwright suite and count as E2E; the Action’s contract tests run in the unit job and count as Unit. Each test counts once, in the layer Testament’s layer rules give it, whichever report it arrived in.

Full project page →

Coverage against the floor

Lines, latest default-branch run
unit98.9%floor 90%

unit: 4,437 of 4,484 lines covered.

Build progress

As of
  1. Phase 0FoundationsDone
  2. Phase 1Schema, parsers, ingestionDone
  3. Phase 2Ostomate2 reporting liveDone
  4. Phase 3RouteServe reporting liveDone
  5. Phase 4BackfillDone
  6. Phase 5Public siteDone
  7. Phase 6Alerts, tracked links, adminDone
  8. Phase 7LaunchIn progress