Baseline checks, calibrated mutations, differential oracles, false-block reporting, and harness-failure categories keep green verdicts from becoming self-certification.

A verification system cannot use its own green verdict as the sole evidence that it works.

If a check is wired incorrectly, silently skipped, or incapable of detecting the failure it claims to cover, every run can pass.

The harness needs tests of its own sensitivity.

Establish a green baseline first

A mutation campaign should run the unmodified baseline under the same command before measuring mutants.

If the baseline is red, a red mutant does not prove the mutation was caught. The environment or existing repository state may be responsible.

Baseline failure is a campaign error, not a killed mutant.

Mutations should target named invariants

Whole-repository random mutation can be expensive and hard to interpret. A bounded campaign can begin with explicit replacements tied to deterministic policy boundaries:

required-value handling;
unknown flags;
numeric boundaries;
boolean policy;
empty profiles;
wiring that ensures important test lanes actually run.

Each mutant should include the exact source change and the rationale for which test ought to kill it.

Survival, timeout, or measurement failure keeps the campaign red.

Differential oracles test emulation

A lightweight module loader may emulate selected bundler behavior for fast candidate checks. Its ability to load a file is not enough.

The same fixture can run through the real candidate bundler and the emulation. The oracle compares load-versus-fail and the resulting value class.

If one returns a string and the other a function, the emulation is wrong even though neither threw.

Known gaps remain catalogued and produce yellow coverage results. Unknown asset queries or extensions generate draft telemetry for a deliberate future fixture; the runtime does not silently teach itself a passing expectation.

Independent graders need exact provenance

Longitudinal evaluation should preserve the model, prompt, reasoning mode, harness version, repository state, environment, budget, raw events, artifacts, and grader configuration.

Without those inputs, a score cannot be reproduced or compared fairly across campaigns.

The evaluator should also distinguish:

true catches;
false blocks;
missed regressions;
workflow failures;
environment failures;
harness failures;
transfer-policy violations;
false confidence.

Only reporting successful catches creates the same selective evidence problem the product is meant to prevent.

Coverage lanes must be executable

A test file listed in documentation but not invoked by any script is imaginary coverage.

The harness can mechanically assert that memory, performance, mutation, browser, and heavy fixture tests are registered in their owning lanes. Changes to lane lists then fail CI when they strand a test.

The uncomfortable result is useful

A false block means the harness stopped a safe change. That costs engineering time and should be measured. A missed mutant means an advertised boundary lacks sensitivity. A flaky environment means the run cannot support a candidate verdict.

These are product findings, not embarrassing noise to remove from the report.

KodeProof’s Evidence Lab is built around this principle: a verifier earns trust by demonstrating where it catches faults, where it overreacts, where it cannot observe, and whether its own machinery behaved correctly. A system that never reports a harness problem has probably not learned to look for one.

A Verification Harness Must Test Its Own Ability to Fail