Test Harness Deep Dive

The Specimen and the Run

Everything on this page runs against one small service, fwvault: a firmware artifact intake. You POST a blob, it parses the header, asks a signing oracle about the digest, applies policy, and stores what survives. It exists to have seams.

   the specimen, and which section tests which part

   POST /artifact
        │
        ▼
   parse.py    UF2 / ELF header walkers      ── §10 parser  §11 hostile
        │
        ▼
   signing.py  ask the oracle about a digest  ── §08 transports
        │              └─ transport is a PARAMETER
        ▼
   policy.py   allow / refuse, with a code    ── §12 guardrails
        │
        ▼
   store.py    content-addressed, atomic      ── tmp_path
        │
        ▼
   app.py      raw ASGI, no framework         ── §09 asgi
   cli.py      argv in, exit code out         ── §14 cli
   testing.py  specimen builders + doubles, SHIPPED IN THE PACKAGE

What a pytest run actually does

   one run, five phases. Most confusion is about which phase you are in.

   1 · CONFIG     find rootdir, read pyproject/pytest.ini, load plugins,
                  register markers, apply addopts
        ▼
   2 · COLLECT    walk testpaths, import every test module, discover
                  conftest.py per directory, build the item tree
                  ↳ an ImportError here kills the FILE, not one test
        ▼
   3 · MODIFY     pytest_collection_modifyitems -- deselect, reorder, mark
        ▼
   4 · RUN        per item:  setup fixtures  →  call the test  →  teardown
                             (outermost first)      (reverse order)
        ▼
   5 · REPORT     pass / fail / skip / xfail, plus -rs reasons

Terms

Every word this page uses, with a one-line meaning and the section that explains it. Generated from terms.py, which is also where GLOSSARY.md and the glossary side of the printed card come from. They held separate copies once and drifted.

the basics

fixtures

running and selecting

test doubles

kinds of test

test quality

an AI author

Collection & Config

Before a single assertion runs, pytest has decided where the project root is, which files are tests, which fixtures are visible where, and what flags are in force. Most "why is this test not running" questions are answered here.

   what counts as a test, by default

   files       test_*.py  or  *_test.py       ← testpaths, then rootdir
   classes     Test*      (and no __init__ method)
   functions   test*      ← note: test* not test_*

   that last one bites. A helper named `tests_in()` is COLLECTED as a
   test. If it is a generator, pytest refuses the whole file:

     'yield' keyword is allowed in fixtures, but not in tests

   and a collection error takes down every other test in that file.
   This page's own hygiene file hit it. Name helpers `find_tests`.

conftest.py is a plugin, not a junk drawer

   fixture visibility follows the DIRECTORY tree, not imports

   repo/
     conftest.py            ← visible to everything below
     tests/
       conftest.py          ← visible to tests/ and below; shadows the above
       test_parse.py            sees both, nearest wins
       integration/
         conftest.py        ← only here
         test_slow.py

   A fixture defined in a conftest needs NO import in a test file.
   That is the feature and the trap: a name with no visible origin.

tests/conftest.py: the three jobs# 1. extend pytest itself
def pytest_addoption(parser):
    parser.addoption("--runnet", action="store_true", default=False,
                     help="run tests marked `net`, which make real connections")

def pytest_collection_modifyitems(config, items):
    # runs after collection, before the first test.
    # this is where a marker becomes behaviour.
    if config.getoption("--runnet"): return
    skip = pytest.mark.skip(reason="needs a real network; pass --runnet")
    for item in items:
        if "net" in item.keywords:
            item.add_marker(skip)

# 2. isolate the process, once, for everything
@pytest.fixture(autouse=True)
def isolate_environment(monkeypatch, tmp_path_factory):
    home = tmp_path_factory.mktemp("fwvault_home")
    monkeypatch.setenv("HOME", str(home))
    monkeypatch.setenv("USERPROFILE", str(home))   # expanduser reads this on Windows
    monkeypatch.setenv("FWVAULT_WALKER_RAISE", "1")

# 3. hand out specimens and collaborators -- the fixtures themselves

pyproject.toml: the flags that are always on

[tool.pytest.ini_options]testpaths = ["tests"]
pythonpath = ["src"]              # src-layout, no install needed to run

# -rs so a skip explains itself. A run that says "12 skipped" and
# nothing else is a run whose coverage you cannot describe.
addopts = "-rs --strict-markers"

markers = [
    "slow: takes real wall-clock time",
    "net: would touch a real network (never enabled in CI)",
]

filterwarnings = [
    # our own deprecations are errors: a warning nobody reads is a
    # rename nobody did. Scoped, so a dependency's schedule is not ours.
    "error::DeprecationWarning:fwvault.*",
    "ignore::DeprecationWarning:starlette.*",
]

Fixtures: Scope, Teardown, Factories

A fixture is a named setup whose result pytest caches per scope and tears down in reverse order of setup. Everything else about fixtures is a consequence of that sentence.

   build order is DEPENDENCY order; teardown is its exact reverse

   @fixture outer                     @fixture inner(outer)
       setup                              setup
       yield       ──────────────►      yield      ──────►   test body
       teardown    ◄─────────────       teardown   ◄──────

   outer:setup → inner:setup → TEST → inner:teardown → outer:teardown

   Code after `yield` runs even if the test FAILS.
   It does NOT run if setup before the yield raised.
   → put anything that can fail AFTER everything that must be cleaned up.

Scope: pick the narrowest that is not painfully slow

The five shapes worth knowing

tests/conftest.py# 1. plain value
@pytest.fixture
def uf2():
    # the smallest thing that is genuinely valid. Two blocks, not two hundred.
    return build_uf2(blocks=2)

# 2. FACTORY -- the test gets the function, not a value
@pytest.fixture
def make_uf2():
    return build_uf2          # now a test can build three different ones

# 3. yield, for teardown
@pytest.fixture
def vault(tmp_path):
    return Store(tmp_path / "vault")   # tmp_path cleans itself up

# 4. PARAMETRIZED -- every test that requests it runs once per param
@pytest.fixture(params=["uf2", "elf"], ids=["uf2", "elf"])
def any_artifact(request):
    return build_uf2(blocks=2) if request.param == "uf2" else build_elf()

# 5. COMPOSED -- the whole service, wired to fakes, in one line,
#    because every collaborator in app.py is a parameter
@pytest.fixture
def app(vault, client, policy):
    return create_app(store=vault, client=client, policy=policy)

factory: rightdef test_sizes(make_uf2):
    small = make_uf2(blocks=1)
    large = make_uf2(blocks=9)
    assert len(large) == 9 * len(small)

# the subject is visible in the test

fixture-per-variant: wrongdef test_sizes(one_block, nine_blocks):
    assert len(nine_blocks) == 9 * len(one_block)

# what IS nine_blocks? open conftest.
# now do that forty more times.

autouse, and the two ways to misuse it

An autouse fixture applies to every test in its directory whether the test asks or not. Correct for process-wide state any test could touch by accident and no test should have to remember: the environment, a fake home, a global registry. Wrong for anything a test could reasonably want to opt out of -- that gives you a suite where nobody can explain what a test is actually running against.

the anti-pattern, named@pytest.fixture
def over_helpful_fixture(vault, uf2):
    assert len(vault) == 0       # ← WRONG PLACE
    return vault, uf2

# when that fails, EVERY test using this fixture reports an ERROR,
# with a traceback pointing at conftest.py instead of at the
# behaviour that broke.
#
#   assertions belong in tests. fixtures build things.

Overriding: nearest conftest wins, silently

A fixture defined in a test module shadows one of the same name from conftest.py, for that module only. Nothing anywhere announces it.

   resolution order for the name `policy`

   tests/test_fixtures.py    def policy()  ← wins, for this file
   tests/conftest.py             def policy()
   conftest.py                       def policy()

   nothing says a name has been shadowed. a test three hundred lines
   below is now running against a different object than its neighbour
   in the next file, and both are called `policy`.

   an override can also EXTEND what it shadows, by requesting the same
   name -- the lookup for the PARAMETER starts one level up:

     @pytest.fixture
     def strict_policy(policy):        ← this module's override
         return replace(policy, require_signature=True)

Use it when a whole file genuinely needs a different baseline. Never as a quick fix for one test: that test should ask for what it needs by name.

Two smaller tools

fixture(name=) and addfinalizer# the function is _make_digest; the fixture is `digest`.
# frees the good name for a normal import in the same module, and
# stops a linter flagging the fixture function as unused.
@pytest.fixture(name="digest")
def _make_digest(uf2):
    return digest_of(uf2)


# addfinalizer is the older API. `yield` covers almost every case and
# reads better. addfinalizer still wins when the cleanup should be
# registered CONDITIONALLY -- only once the thing exists:
@pytest.fixture
def conditional_cleanup(request, tmp_path):
    target = tmp_path / "artifact.bin"
    target.write_bytes(bytes(16))
    request.addfinalizer(target.unlink)   # registered only now that it exists
    return target

# with `yield`, the teardown half runs even when the setup half failed,
# and has to defend itself against a thing that was never created.

Parametrize & IDs

One function, many cases, each reported as its own test. The mechanics are easy; the value is entirely in the IDs.

   stacked decorators are a CARTESIAN PRODUCT

   @parametrize("bit64", [True, False], ids=["elf64", "elf32"])
   @parametrize("little", [True, False], ids=["le", "be"])
   def test_elf_entry(bit64, little): ...

                    ┌──────────────┬──────────────┐
                    │   elf64      │   elf32      │
        ┌───────────┼──────────────┼──────────────┤
        │    le     │ [le-elf64]   │ [le-elf32]   │
        ├───────────┼──────────────┼──────────────┤
        │    be     │ [be-elf64]   │ [be-elf32]   │
        └───────────┴──────────────┴──────────────┘

   4 tests from 2 lines. All four are real files somebody has --
   entry point width and byte order are independent header fields.

ids: three ways, all better than the default# 1. an explicit list -- best when the cases have names
@pytest.mark.parametrize(
    "family,expected",
    [(RP2040, "RP2040"), (NRF52840, "NRF52840"), (UNKNOWN_FAMILY, None)],
    ids=["rp2040", "nrf52840", "unknown"],
)

# 2. a callable -- best for computed cases
@pytest.mark.parametrize("bad_block", [0, 1, 2], ids=lambda n: "block%d" % n)

# 3. repr -- best for byte specimens
@pytest.mark.parametrize("blob", [b"", b"\x00", b"\x7fEL"], ids=repr)
#   → test_short_buffers[b'\x7fEL']

When parametrize is the wrong tool

One more habit worth building: when a sweep is shared with a test that checks the sweep, make it one constant. This page's own hostile-input file had a truncation sweep stepping by 97 -- which never lands on a multiple of 512, so every one of its twenty cases hit the same early branch and none ever reached the walker. The meta-test caught it. Two copies of the list would have drifted silently.

tests/test_hostile.py# One list, used by the sweep AND by the meta-test that checks the sweep.
SWEEP = sorted(set(range(0, 1537, 64)) | {1, 7, 511, 513, 1535})

@pytest.mark.parametrize("length", SWEEP)
def test_every_truncation_is_survivable(length):
    _parses_or_raises_parse_error(VALID[:length])

def test_the_sweep_actually_covers_both_outcomes():
    outcomes = {_parses_or_raises_parse_error(VALID[:n]) is not None for n in SWEEP}
    assert outcomes == {True, False}, (
        "every case in SWEEP had the same outcome, so the sweep is "
        "testing one branch twenty-five times")

Indirect: send the parameter to a fixture

Plain parametrize hands the value to the test. indirect hands it to a fixture first, so each case gets real setup and real teardown.

tests/test_parametrize.py@pytest.fixture
def vault_with(request, tmp_path):
    store = Store(tmp_path / "v")
    for n in range(1, request.param + 1):
        store.put(...)
    yield store
    # teardown a plain parametrize cannot express: where a socket, a
    # process or a database transaction would close.

@pytest.mark.parametrize("vault_with", [0, 1, 5], indirect=True,
                         ids=lambda n: "%d-stored" % n)
def test_vault_size(vault_with):
    assert len(vault_with) == vault_with.expected

# indirect=["artifact"] names WHICH params get routed; the others
# arrive normally. Worth it when each case needs setup or teardown.
# NOT worth it when the fixture is just a function call -- that is
# what a factory fixture is for.

pytest_generate_tests: build the list at collection

the hook behind parametrizedef pytest_generate_tests(metafunc):
    # runs once per test function, at collection time.
    if "known_family" in metafunc.fixturenames:
        families = importlib.import_module("fwvault.parse").FAMILIES
        metafunc.parametrize(
            "known_family,expected_name",
            sorted(families.items()),
            ids=[name.lower() for _fid, name in sorted(families.items())])

# the cases come from the SOURCE OF TRUTH rather than a copy of it.
# add a supported chip and a test appears; nobody has to remember.
#
# the cost: a reader of the file cannot see the cases without running
# `pytest --collect-only`. pay it only when keeping a literal list in
# sync is the bigger risk.

Assertions & Failure

pytest rewrites the bytecode of your test modules so a bare assert reports the values of its subexpressions. That is why you never need assertEqual. It also means the quality of a failure message is a property of how you wrote the assertion.

tells you nothingassert result

# E  assert None


assert len(hits) > 0

# E  assert 0 > 0

tells you what brokeassert codes(image, SIGNED, policy) == ["OVERSIZE"]

# E  assert ['UNSIGNED'] == ['OVERSIZE']
# E    At index 0 diff: 'UNSIGNED' != 'OVERSIZE'

assert not offenders, "undocumented flags: " + ", ".join(offenders)

# E  AssertionError: undocumented flags: ingest vault

Exceptions: assert the message and the object, not just the class

pytest.raises# the type alone. Passes when the message says "None".
with pytest.raises(ParseError):
    sniff(blob)

# + the message. `match` is re.search, so it is a substring by default.
with pytest.raises(ParseError, match=r"unrecognised magic deadbeef"):
    sniff(b"\xde\xad\xbe\xef" + b"\x00" * 8)

# + the OBJECT. TruncatedError carries offset/needed/available
# precisely so a test can say WHERE.
with pytest.raises(TruncatedError) as excinfo:
    parse(build_elf(truncate_to=26))
assert excinfo.value.needed == 32
assert excinfo.value.available == 26

The assertion vocabulary

One behaviour per test

A test with four unrelated asserts reports the first failure and hides the other three. Two asserts about the same decision seen from two sides are fine and often better together:

two views of one decisiondef test_oversized_payload_size_is_clamped_and_warned():
    image = parse(build_uf2(blocks=2, oversized_payload=9999))
    assert image.payload_bytes == UF2_PAYLOAD_MAX + 256   # the clamp
    assert any("exceeds 476" in w for w in image.warnings)  # and it SAID so

# a version that clamps silently is the bug this test exists to catch,
# and it needs both halves to catch it.

caplog: two things people miss

tests/test_transport.pydef test_a_retry_is_logged(caplog, clock):
    # 1. SET THE LEVEL. pytest captures at WARNING by default, so an
    #    log.info() you are asserting on never arrives, and the test
    #    fails looking like the code is wrong.
    caplog.set_level(logging.WARNING, logger="fwvault.signing")

    script = ScriptedTransport(flaky(2, Response(200, signed_body())))
    SigningClient("https://x", transport=script, clock=clock, retries=3).verify(d)

    # 2. ASSERT ON RECORDS, NOT ON TEXT. a record has .levelname, .name
    #    and .getMessage(). caplog.text is whatever the formatter felt
    #    like, and asserting on it couples the test to a format string
    #    nobody thinks of as an interface.
    retries = [r for r in caplog.records if "attempt" in r.getMessage()]
    assert len(retries) == 2
    assert all(r.levelname == "WARNING" for r in retries)
    assert all(r.name == "fwvault.signing" for r in retries)

Markers, xfail & Flakes

Four ways a test can not-run, and they mean completely different things. The one people get wrong is xfail, and it fails quietly.

   skip          this does not apply here        platform, missing dep
   skipif(cond)  the same, decided at collection
   xfail         this is BROKEN and we know     a bug with a ticket
   xfail(strict) ...and tell me the MOMENT it stops being broken
   deselected    a marker the run excluded      -m 'not net'

   ─────────────────────────────────────────────────────────────

   a plain xfail that starts passing reports XPASS and stays GREEN.

   so: the bug gets fixed, nobody notices, the marker sits there for
   two years, and the test has been asserting nothing that whole time.

   strict=True turns that XPASS into a FAILURE. it is the only
   setting that makes xfail a temporary state rather than a permanent
   one. set xfail_strict = true in pyproject and get it by default.

silently green forever@pytest.mark.xfail(reason="bug FW-118")
def test_the_bug_is_fixed_now():
    assert True

# 1 xpassed. exit code 0.

tells you to delete the marker@pytest.mark.xfail(strict=True, reason="bug FW-118")
def test_the_bug_is_fixed_now():
    assert True

# 1 failed. exit code 1.

pytest.param: marking one case

A bare tuple in a parametrize list cannot carry an id or a marker. pytest.param wraps one case so it can carry both.

tests/test_markers.py@pytest.mark.parametrize(
    "family,expected",
    [
        pytest.param(0xE48BFF56, "RP2040", id="rp2040"),
        pytest.param(0xADA52840, "NRF52840", id="nrf52840"),
        pytest.param(
            UNKNOWN_FAMILY, "SOME-FUTURE-CHIP",
            id="unknown",
            marks=pytest.mark.xfail(
                strict=True,
                reason="we do not name unknown families; parse returns None"),
        ),
    ],
)
def test_family_naming_with_one_expected_failure(family, expected):
    assert parse(build_uf2(blocks=1, family=family)).family == expected

# the third case is a real, documented gap, stated IN THE SUITE rather
# than in a comment. The day someone adds that family to the table,
# this fails and tells them to delete the marker.

selecting one case out@pytest.mark.parametrize(
    "blocks", [1, 2, pytest.param(600, marks=pytest.mark.slow, id="600-blocks")])
def test_marks_can_select_out_one_case(blocks): ...

# `pytest -m 'not slow'` drops ONLY the 600-block case. Without
# pytest.param the marker goes on the whole function and takes the
# two cheap cases with it.

usefixtures, for a fixture with nothing to return

side-effect-only fixtures@pytest.mark.usefixtures("strict_umask")
def test_something():
    ...

# the dependency is still declared and still visible; it just is not
# pretending to be a value your linter has to ignore.
#
# do NOT use it to hide a fixture whose value you actually want. the
# signature is where a reader looks for a test's inputs.

The flaky test

A flaky test is a bug report, against the test or against the code. It is never noise: something is genuinely non-deterministic and you have found it.

   the causes, in the order they actually occur

   unseeded randomness          → fix the seed        (§11)
   a real clock or a real sleep → inject a clock      (§08)
   shared state between tests   → narrow the scope    (§15)
   a fixed path two workers race→ tmp_path           (§15)
   a real network call          → a marker + a fake  (§08)
   dict / set ordering assumed  → sort, or compare sets

   ─────────────────────────────────────────────────────────────

   QUARANTINE, do not rerun.

   move it to a marker that CI deselects, with a ticket. it stops
   blocking and it stops lying. `pytest-rerunfailures` is a last
   resort for a genuinely external dependency you do not control;
   `--reruns 3` on your own code converts a real bug into a slower
   green run, permanently.

   the tell that you have this wrong: anyone on the team has ever
   said "just re-run it".

The Five Test Doubles

The words are Gerard Meszaros's. They are worth using precisely because "mock" has drifted into meaning "any object I made up". The distinction is not pedantry: it predicts how your test fails.

   the ladder, in order of how much they know about themselves

   DUMMY    passed to satisfy a signature, never used
              └─ `evaluate(image, None, unchecked_policy)`

   STUB     returns canned answers. no state worth asking about
              └─ 4 lines. cannot tell you the URL was right.

   FAKE     a real, working, SIMPLIFIED implementation
              └─ the one to reach for. It can be WRONG in a way a test notices.

   SPY      a stub that records how it was called
              └─ for asserting on calls that DID NOT happen

   MOCK     a spy with expectations built in. it asserts on ITSELF
              └─ fails inside the double, so the message describes a call
                 that did not happen rather than a behaviour that is wrong

stub: cannot model a 404class StubTransport:
    def __init__(self, response):
        self.response = response

    def request(self, method, url, body=None, headers=None):
        return self.response

# answers everything identically, so it passes
# against a client that builds a nonsense URL.

fake: models both outcomesclass FakeOracle:
    def __init__(self, known=None):
        self.known = dict(known or {})

    def request(self, method, url, body=None, headers=None):
        digest = url.rsplit("/", 1)[-1]
        if digest not in self.known:
            return Response(404, b'{"error":"unknown"}')
        return Response(200, signed_body(self.known[digest]))

Only the fake can express the distinction the whole service is built around: a 404 is a definite no, not an outage. A stub returning 200 forever cannot fail to model a 404, because it was never modelling anything -- so a suite built on stubs never checks it.

The mock failure mode, twice

unittest.mock, as people actually use itfrom unittest.mock import Mock

transport = Mock(spec=["request"])            # spec= is NOT optional
transport.request.return_value = Response(200, signed_body())

client = SigningClient("https://x", transport=transport, clock=clock)
client.verify("c" * 64)

transport.request.assert_called_once_with(
    "GET", "https://x/keys/" + "c" * 64, headers={"accept": "application/json"})

# FAILURE 1: the message.
#   "Expected call: request('GET', ...)" -- nothing about which
#   BEHAVIOUR is wrong. Compare a spy: you get a URL diff.

# FAILURE 2: a Mock without spec answers ANY attribute with another Mock.
loose = Mock()
loose.this_method_does_not_exist().nor_does_this_one()   # passes. silently.
transport.reqeust                                        # spec= makes this raise

Ship your doubles

A library whose only seam is a transport protocol owes its users a working fake for that protocol -- the way httpx ships MockTransport and Django ships a test client. If the fake lives in tests/, every downstream user writes their own and every one of them gets a detail wrong. fwvault.testing is a shipped module, and §17 asserts its signatures still match the real transport.

fwvault/testing.py: a scripted double for sequencesclass ScriptedTransport:
    """Answers a fixed sequence, one per call. For testing retry loops,
    where WHEN a response arrives is the behaviour under test."""

    def request(self, method, url, body=None, headers=None):
        self.calls.append((method, url, body, headers or {}))
        if not self.script:
            # Running off the end is a HARD FAILURE, not a repeat of the
            # last item. A double that quietly keeps answering turns
            # "retried 3 times" and "retried 300 times" into the same
            # passing test.
            raise AssertionError(
                "ScriptedTransport exhausted: call {} was not scripted"
                .format(len(self.calls)))
        item = self.script.pop(0)
        if isinstance(item, Exception):
            raise item
        return item

Monkeypatching: Every Method, and When Not To

monkeypatch is a function-scoped fixture that records every change it makes and undoes all of them at teardown, in reverse order, whether the test passed, failed, or exploded. That automatic undo is the entire reason to use it over os.environ[...] = x or a hand-rolled try/finally.

Patch where the name is LOOKED UP

   this is the single most common patching mistake, and it fails
   by the test PASSING against unpatched code.

   works                             does nothing

   parse.py                          parse.py
     from . import families           from .families import FAMILIES
     ...                                 ...
     families.FAMILIES[fam]              FAMILIES[fam]
            ▲                                ▲
            │ read at CALL time               │ bound at IMPORT time
            │                                │ into a SECOND namespace
   patch families.FAMILIES            patch families.FAMILIES
   → the parser sees it              → the parser never looks there again

   the from-import already copied the reference.

Four traps, all real, all found writing this repo

1: raising=False disables the only signal you have# patching a name that does not exist is almost always a typo or a
# rename you have not noticed, so monkeypatch raises.
monkeypatch.setattr(parse_module, "FAMILEIS", {})                 # AttributeError
monkeypatch.setattr(parse_module, "FAMILEIS", {}, raising=False)  # created,
                                                                # and aimed at nothing

2: patching a constant does not move an already-bound default@dataclass(frozen=True)
class Manifest:
    schema_version: int = SCHEMA_VERSION     # ← evaluated ONCE, at import

monkeypatch.setattr("fwvault.store.SCHEMA_VERSION", 99)
Manifest(...).schema_version                 # still 3.

# same shape as `def f(x=CONST)`, same shape as `from x import CONST`.
# if a value must be patchable, it has to be READ at call time.

3: the string target form walks getattr, and can be ambushed# fwvault/__init__.py does `from .parse import parse`, which rebinds
# the attribute `fwvault.parse` from the MODULE to the FUNCTION.

monkeypatch.setattr("fwvault.parse.UF2_PAYLOAD_MAX", 8)
# AttributeError: 'function' object at fwvault.parse has no attribute ...

import fwvault.parse as m       # does NOT save you -- `as` getattrs too
m = importlib.import_module("fwvault.parse")   # reads sys.modules. works.
from fwvault.parse import sniff              # also works. prefer this.

4: sys.modules[name] = None makes `import name` raise# the import-isolation trick. Test the dependency-is-absent path on a
# machine where it is very much present.

monkeypatch.delitem(sys.modules, "httpx", raising=False)  # ← NOT optional
monkeypatch.setitem(sys.modules, "httpx", None)
with pytest.raises(ImportError):
    import httpx

# the delitem first matters: if the module is already imported, the
# None assignment is what takes effect, and forgetting to clear an
# existing entry gives a test that passes for the wrong reason.

The rule

   PATCH what you do NOT own.      urllib, datetime, os.replace,
                                    a third-party client's internals

   INJECT what you DO own.          your transport, your clock,
                                    your store, your policy

   why it matters, concretely:

   A. monkeypatch UrllibTransport.request    passes even if the client
                                             stops using its transport
                                             entirely

   B. pass transport=<a fake>                fails immediately -- the fake
                                             stops being asked anything

monkeypatch.context(): undo early

The fixture undoes everything at teardown, which is too late when one test needs the patched behaviour and then the real behaviour.

patch, assert, unpatch, assert againdef test_monkeypatch_context_undoes_early(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(parse_module, "FAMILIES", {})
        assert parse(build_uf2(blocks=1)).family is None

    assert parse(build_uf2(blocks=1)).family == "RP2040"   # already restored

Mock Transports

The thing that talks to the network is a parameter, not an import. SigningClient never mentions urllib, sockets or hosts -- it holds a transport with a single request method, and the real one is just the default argument.

   the seam. Nothing is patched; the test passes a different object.

   SigningClient
        │  .verify(digest)
        ▼
   transport.request(method, url, body, headers) → Response
        │
        ├── UrllibTransport      real sockets. stdlib. §net marker only
        ├── RecordingTransport   routes + a call log      (spy)
        ├── ScriptedTransport    one answer per call      (sequence)
        └── FakeOracle           a working oracle in RAM  (fake)

   The narrower the seam, the cheaper the fake that fills it.
   `Response` is a 3-field dataclass, deliberately NOT an httpx.Response.

What is actually worth asserting

Not "it returned a Verdict". The interesting assertions are about when, how often, and what it did not do -- and none of those are visible in a return value.

the retry schedule, in zero wall-clock timedef test_backoff_is_exponential_and_instant(clock):
    script = ScriptedTransport(flaky(3, Response(200, signed_body())))
    client = SigningClient("https://x", transport=script, clock=clock, retries=4)
    client.verify("a" * 64)

    assert clock.slept == [0.5, 1.0, 2.0]

# the schedule IS the behaviour. A real sleep here would put 3.5
# seconds into every CI run to test arithmetic.

def test_no_sleep_on_the_first_attempt(clock):
    # an off-by-one in a backoff loop costs every caller half a second
    # on the HAPPY path, and no return value shows it.
    assert clock.slept == []

The status taxonomy this is all for

   what the oracle said              what it MEANS

   200 {"signed": true}    ──►   Verdict(signed=True)      an answer
   404                     ──►   Verdict(signed=False)     an answer. NOT retried.
   401 / 403 / 400         ──►   VaultUnavailable          OUR credentials are wrong
   500 / 502 / 503         ──►   retry, then VaultUnavailable
   connection reset        ──►   retry, then VaultUnavailable

   VaultUnavailable is NEVER Verdict(signed=False).

   An outage rendered as "unsigned" turns into a wall of confident
   refusals indistinguishable from real ones, and nobody finds it
   until someone asks why every build failed overnight.

httpx: two transports, opposite ends of the wire

the confusion worth clearing up once# ASGITransport substitutes the SERVER -- the thing that answers us.
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://vault") as c:
    r = await c.post("/artifact", content=uf2)

# MockTransport substitutes the THING WE CALL -- the far side.
def handler(request):
    return httpx.Response(200, json={"signed": True, "signer": "ci-builder"})

httpx_client = httpx.Client(transport=httpx.MockTransport(handler))

# same library, opposite directions, and they get confused constantly.

The gap you cannot close, named rather than hidden

tests/test_transport.py@pytest.mark.net
def test_urllib_transport_against_a_real_server():
    """The one thing no fake can cover: that UrllibTransport speaks HTTP.

    Deselected by default; `pytest --runnet` opts in. The skip reason prints
    on every run because pyproject sets -rs, so the gap stays VISIBLE instead
    of being quietly covered by a fake of the very class under test."""
    response = UrllibTransport(timeout=5.0).request("GET", "https://example.com/")
    assert response.status == 200


def test_urllib_transport_converts_socket_errors(monkeypatch):
    # what CAN be tested without a socket: that the error TRANSLATION
    # is right. urlopen is PATCHED here, and that is correct -- urllib
    # is not our seam, we do not own it, there is no argument to pass.
    monkeypatch.setattr(urllib.request, "urlopen", boom)
    with pytest.raises(TransportError, match="unreachable"):
        UrllibTransport().request("GET", "https://x/")

ASGI & Fake Clients

An ASGI app is a coroutine taking three arguments. That is the whole protocol. Every framework you have used is a very good router and middleware stack wrapped around those three names, and every test client you have used is something that builds a scope, feeds a receive, and collects the sends.

   async def app(scope, receive, send)

   scope     a dict describing the connection
             {"type": "http", "method": "POST", "path": "/artifact",
              "headers": [(b"content-type", b"application/octet-stream")], ...}

   receive   an awaitable you CALL to pull events IN
             → {"type": "http.request", "body": b"...", "more_body": True}
             → {"type": "http.request", "body": b"...", "more_body": False}
             → {"type": "http.disconnect"}

   send      a coroutine you CALL to push events OUT
             ← {"type": "http.response.start", "status": 201, "headers": [...]}
             ← {"type": "http.response.body", "body": b'{"digest":...}'}

   plus one more scope type nobody handles until it hangs:
   lifespan  {"type": "lifespan"} → startup / shutdown, each needing a reply

The whole test client, in twenty lines

tests/test_asgi_raw.pydef call(app, method="GET", path="/", body=b"", headers=None, chunks=None):
    """One request, start to finish. Returns (status, headers, parsed body)."""
    incoming = list(chunks) if chunks is not None else [body]
    scope = {
        "type": "http", "asgi": {"version": "3.0"}, "method": method,
        "path": path, "query_string": b"",
        "headers": [(k.lower().encode(), v.encode())
                    for k, v in (headers or {}).items()],
    }

    async def receive():
        if incoming:
            part = incoming.pop(0)
            return {"type": "http.request", "body": part,
                    "more_body": bool(incoming)}
        return {"type": "http.disconnect"}

    sent = []
    async def send(message):
        sent.append(message)

    asyncio.run(app(scope, receive, send))

    start = next(m for m in sent if m["type"] == "http.response.start")
    payload = b"".join(m.get("body", b"") for m in sent
                       if m["type"] == "http.response.body")
    return start["status"], dict(start["headers"]), json.loads(payload)

Three clients, one behaviour

What only the raw harness can do

things no HTTP client will let you send# 1. choose the chunk boundaries. The reassembly loop is exactly
#    where an off-by-one costs you a corrupted upload.
call(app, "POST", "/artifact", chunks=[uf2[:100], uf2[100:700], uf2[700:]])

# 2. prove the size limit fires BEFORE the whole body is buffered.
#    a service that buffers 900 MB and THEN checks has no limit.
app = create_app(store=vault, client=client, policy=policy, max_body=1024)
status, _h, body = call(app, "POST", "/artifact", chunks=[...])
assert (status, body["error"]) == (413, "OVERSIZE")

# 3. hang up mid-body. The app must not respond to a closed
#    connection, and must not raise.
events = [{"type": "http.request", "body": uf2[:100], "more_body": True},
          {"type": "http.disconnect"}]
assert sent == []

# 4. drive lifespan by hand, and assert both events are answered.
assert lifespan(app) == ["lifespan.startup.complete", "lifespan.shutdown.complete"]

Async fixtures need an async plugin

the "coroutine was never awaited" answer@pytest.fixture
def anyio_backend():
    # anyio's plugin parametrizes over backends via this fixture.
    # pinning it stops the run also spawning a trio pass.
    return "asyncio"

@pytest.fixture
async def http(app):
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport,
                                 base_url="http://vault") as client:
        yield client

@pytest.mark.anyio
async def test_ingest_over_httpx(http, uf2):
    response = await http.post("/artifact", content=uf2)
    assert response.status_code == 201

# no plugin → pytest collects the coroutine as if it were a test,
# never awaits it, and reports a pass with a RuntimeWarning.

Parser Tests: Walkers, Builders, Corpora

Two parsers, one shape: a generator that walks structure and yields records, wrapped by a function that folds those records into a result. Keeping the walk separate from the fold is what makes a parser testable at two levels.

   walk / fold, and what each half buys you

   bytes ──► walk_uf2() ──► Block, Block, Block ──► parse_uf2() ──► Image
                │                                       │
                │ a GENERATOR                           │ the FOLD
                │ · 40 MB costs one block of RAM        │ · testable with hand-built
                │ · a caller can stop early             │   records and NO bytes
                │ · assert on block 10 of 900           │ · where warnings accumulate
                │   without building a manifest         │

   raise vs warn is THE line, and getting it wrong either way is a bug:
     raise too eagerly  → half the real world is unparseable
     warn too eagerly   → a corrupt image ships

The UF2 block, which is what all of this parses

   512 bytes, three magic numbers, 476 usable payload bytes

   offset   size   field
   ──────────────────────────────────────────────────────────────────
     0       4     magicStart0   0x0A324655   "UF2\n"
     4       4     magicStart1   0x9E5D5157
     8       4     flags         0x2000 = familyID present
    12       4     targetAddr    where this block is flashed
    16       4     payloadSize   ≤ 476, and a parser must clamp
    20       4     blockNo
    24       4     numBlocks     declared, not verified
    28       4     fileSize / familyID
    32     476     data
   508       4     magicEnd      0x0AB16F30
   ──────────────────────────────────────────────────────────────────

   the third magic at 508 is the POINT of the format:
   it makes a block self-identifying even when found mid-stream.

Build specimens, do not commit them

A builder turns keyword arguments into bytes, and every defect argument names a block index. A test then reads as "block 1's end magic is wrong" rather than as a byte offset the reader has to divide by 512 in their head.

fwvault/testing.pydef build_uf2(blocks=2, family=RP2040, payload_size=256, base_addr=0x10000000,
              flags=None, bad_start_magic=None, bad_end_magic=None,
              wrong_num_blocks=None, oversized_payload=None,
              shuffle_block_no=False, trailing_garbage=0, addr_step=None):
    ...

# reads as a sentence:
build_uf2(blocks=3, bad_end_magic=1)              # block 1's end magic is wrong
build_uf2(blocks=10, shuffle_block_no=True)      # blockNo fields out of order
build_uf2(blocks=2, wrong_num_blocks=0xFFFFFFFF) # claims 4 billion blocks

Test the builder

the one everybody skipsdef test_the_builder_builds_what_it_claims():
    """A corpus generator is code, and untested code in a test corpus
    produces tests that pass against the wrong bytes.

    This asserts the raw layout DIRECTLY, without going through the parser --
    otherwise a matching pair of bugs in builder and parser cancels out and
    the suite stays green."""
    blob = build_uf2(blocks=1, family=RP2040, payload_size=128)
    assert len(blob) == 512
    assert struct.unpack_from("<I", blob, 0)[0] == UF2_MAGIC0
    assert struct.unpack_from("<I", blob, 16)[0] == 128
    assert struct.unpack_from("<I", blob, 508)[0] == UF2_MAGIC_END

Sniff by magic, never by filename

and locate every finding absolutelydef test_block_offsets_are_absolute():
    # so a finding can be handed to `xxd -s`. Relative offsets are the
    # reason people stop trusting parser output.
    blocks = list(walk_uf2(build_uf2(blocks=3)))
    assert [b.offset for b in blocks] == [0, 512, 1024]

@pytest.mark.parametrize("bad_block", [0, 1, 2], ids=lambda n: "block%d" % n)
def test_bad_end_magic_is_located(bad_block):
    # parametrizing over WHICH block is defective catches an off-by-one
    # in the offset arithmetic that a single specimen would not.
    with pytest.raises(ParseError) as excinfo:
        list(walk_uf2(build_uf2(blocks=3, bad_end_magic=bad_block)))
    assert excinfo.value.offset == bad_block * 512 + 508

The negative case people forget

without this, a parser that warns about everything passesdef test_a_clean_image_warns_about_nothing():
    assert parse(build_uf2(blocks=4)).warnings == ()

Why parsers get more testing than other code

Four properties, and no other kind of code has all four.

The techniques, in increasing order of return

   cheapest first. each one finds what the one above it cannot.

   1 corpus + golden files     catches regressions. catches nothing new.
   2 truncation sweeps         an interrupted upload is the commonest
                              corrupt file alive. cheapest real finder.
   3 bit flips, seeded fuzz    field boundaries beat random offsets
                              by orders of magnitude.
   4 metamorphic relations     no oracle needed. parse, reserialize,
                              reparse, and assert a fixed point.
   5 differential              the strongest, when a second
                              implementation exists.
   6 structural generation     generate VALID inputs so the fuzzer
                              spends its budget past the header checks
                              instead of bouncing off them.

   and the sweep itself needs a test. this repo's truncation sweep
   stepped by 97, which never lands on a multiple of 512, so all
   twenty cases took the same early branch and none reached the
   walker. a meta-test caught it. see §13.

Metamorphic relations: assertions with no oracle

Every other technique needs somebody to know the right answer. An example test states it, a differential test borrows it, a property test implies it. A metamorphic relation needs none: it says how two runs must relate, never what either one should produce.

the four shapes worth learning# ROUND TRIP: the strongest, when an inverse exists
assert serialize_uf2(walk_uf2(blob)) == blob
#   byte for byte. anything the walker drops shows up as a diff,
#   including fields nothing else in the suite reads.

# INVARIANCE: changing X must NOT change Y
scrambled = serialize_uf2(replace(b, data=random_bytes()) for b in blocks)
assert parse(scrambled) == parse(blob)
#   payload is opaque to a HEADER parser. catches a parser reading
#   a length or a flag out of the data area.

# EQUIVARIANCE: changing X must change Y by a stated amount
after = parse(serialize_uf2(blocks_of(base) + blocks_of(added)))
assert after.payload_bytes == before.payload_bytes + 200 * extra
#   not "the total is 768". the relation holds whatever the
#   numbers are, so it survives the specimen changing.

# IDEMPOTENCE: twice is once
assert serialize_uf2(walk_uf2(once)) == once

Why this is the technique that survives an AI author

A relation is a claim about the problem, not about the code, so it cannot be transcribed from the implementation's behaviour. Ask a model what parse returns for a given blob and it can read the answer off the source. Ask it whether reordering blocks may change the payload total and it has to reason about UF2. See §19, and test_oracle_independence.py for what happens when the assertions do come from the implementation.

Two ideas worth stealing from other parser traditions

The walk/fold split at the top of this section is the same idea in a byte-parser: walk_uf2 is the sub-parser, parse_uf2 composes it, and both are reachable on their own. serialize_uf2 is what makes the pair invertible, which is what buys the round-trip relation.

Hostile Input, Fuzz & Properties

A parser's contract under bad input is narrow and absolute: for any byte string, parse() either returns an Image or raises ParseError. Nothing else. Not IndexError, not struct.error, not MemoryError, not a hang -- every one of those is the same defect wearing a different exception class, and every one reaches a client as a 500 with a stack trace in it.

   four techniques, and what each one is actually good at

   TRUNCATION SWEEP   every prefix of a valid file
                      cheap · exhaustive · finds more real bugs than the rest combined
                      an interrupted upload is the commonest corrupt file in the world

   BIT FLIPS          one byte changed, walking the header FIELD BOUNDARIES
                      12 named offsets do what a fuzzer needs a million iterations for

   SEEDED FUZZ        random garbage from a FIXED seed
                      reproducible from the test ID alone. no artifact to attach.

   PROPERTY-BASED     the machine finds the specimen, then SHRINKS it
                      1,847 bytes of noise → b'\x00' and the same traceback

Guided beats unguided, by three orders of magnitude

why pure random bytes prove almost nothing here@pytest.mark.parametrize("seed", range(8))
def test_valid_prefix_random_tail(seed):
    """The interesting shape: enough valid structure to get PAST sniff,
    then garbage.

    Pure random bytes almost never reach the walker at all -- they fail the
    magic check in the first four bytes and prove nothing about the code past
    it. This is why unguided fuzzing of a parser with a magic number spends
    99.9% of its budget on the same branch."""
    rng = random.Random(1000 + seed)
    tail = bytes(rng.randrange(256) for _ in range(512))
    _parses_or_raises_parse_error(VALID[:512] + tail)

Properties, in order of strength

tests/test_property.py@COMMON
@given(blocks=st.integers(min_value=1, max_value=12),
       payload=st.integers(min_value=0, max_value=UF2_PAYLOAD_MAX))
def test_uf2_round_trip(blocks, payload):
    image = parse(build_uf2(blocks=blocks, payload_size=payload, family=RP2040))
    assert image.block_count == blocks
    assert image.payload_bytes == blocks * payload
    assert image.size == blocks * UF2_BLOCK_SIZE

@COMMON
@given(entry=st.integers(min_value=0, max_value=2**64 - 1),
       bit64=st.booleans(), little=st.booleans())
def test_elf_entry_round_trips_at_every_width(entry, bit64, little):
    # 32-bit ELF cannot hold a 64-bit entry point, so the precondition
    # is stated with `assume` rather than by narrowing the strategy --
    # hypothesis tracks rejections and complains if the filter is too
    # aggressive, which beats a silently narrower search.
    assume(bit64 or entry < 2**32)
    assert parse(build_elf(entry=entry, bit64=bit64, little=little)).entry == entry

Resource exhaustion, without a timing assertion

the classic parser DoSdef test_a_declared_block_count_does_not_allocate():
    """A length field is trusted and pre-allocated. A two-block file
    claiming four billion blocks must cost two blocks of work.

    Asserted by wall-clock-free means -- drive the walker to completion and
    check the count -- because a timing assertion on a loaded CI runner is a
    flake generator."""
    image = parse(build_uf2(blocks=2, wrong_num_blocks=0xFFFFFFFF))
    assert image.block_count == 2
    assert image.declared_blocks == 0xFFFFFFFF

Guardrails: Proving a Refusal

A policy test has three obligations, and most suites discharge one.

   1. the rule FIRES on a violating artifact        everyone does this
   2. the rule does NOT fire on a compliant one      half of everyone
   3. the rule fires for the RIGHT REASON            almost nobody

   why three rots:

   an oversized UNSIGNED artifact is rejected by TWO rules.
   a test asserting "rejected" passes forever after the size check is
   accidentally deleted, because the signature check is still there.

        ┌─────────────────────────────────────────────┐
        │  evaluate() returns EVERY hit               │  → can state #3
        │  enforce()  raises the FIRST by precedence  │  → what a client sees
        └─────────────────────────────────────────────┘

Precedence is behaviour, not dict ordering

fwvault/policy.pyPRECEDENCE = (
    "MALFORMED",          # parser said no, before policy is consulted
    "OVERSIZE",
    "EMPTY_PAYLOAD",
    "UNKNOWN_FAMILY",
    "DENIED_MACHINE",
    "NOT_MAIN_FLASH",
    "REVOKED_KEY",        # a STRONGER statement than unsigned...
    "UNSIGNED",           # ...and must not be masked by it
    "TOO_MANY_WARNINGS",
)

# codes are a PUBLIC CONTRACT. Clients branch on `code`, never on
# `detail`, so the codes are frozen in test_contract.py while the
# wording of `detail` stays free to improve.

the assertion a first-hit design cannot makedef test_every_applicable_rule_fires_at_once(policy):
    """One artifact, five independent violations, five codes.

    If OVERSIZE stops firing, this list gets shorter and the test fails BY
    NAME -- rather than staying green because UNSIGNED still rejects the
    same file."""
    image = parse(build_uf2(blocks=2, family=UNKNOWN_FAMILY, payload_size=0))
    tight = replace(policy, max_bytes=100)

    assert codes(image, UNSIGNED, tight, flags=UF2_FLAG_NOT_MAIN_FLASH) == [
        "OVERSIZE", "EMPTY_PAYLOAD", "UNKNOWN_FAMILY",
        "NOT_MAIN_FLASH", "UNSIGNED",
    ]

The line the whole service is built around

   three kinds of bad thing, and conflating any two is how a service
   starts lying about its inputs

   ParseError        the blob is malformed        a fact about the ARTIFACT
                                                  → 422 MALFORMED, with an offset

   PolicyRejection   parsed fine, we refuse it    a DECISION
                                                  → 422 <CODE>

   VaultUnavailable  we could not tell            a fact about US
                                                  → 503 + Retry-After. NEVER a rejection.

   VaultUnavailable is not a subclass of PolicyRejection and never
   converts into one. A signing oracle that times out and is reported
   as "unsigned" turns an outage into a stream of confident refusals,
   and the refusals look exactly like the real ones.

tests/test_asgi_raw.py: the headline assertiondef test_an_oracle_outage_is_503_and_never_a_rejection(vault, policy, uf2):
    class DeadClient:
        def verify(self, digest):
            raise VaultUnavailable("oracle down")

    app = create_app(store=vault, client=DeadClient(), policy=policy)
    status, headers, body = call(app, "POST", "/artifact", uf2)

    assert status == 503
    assert body["error"] == "ORACLE_UNAVAILABLE"
    assert headers[b"retry-after"] == b"5"
    assert len(vault) == 0, "nothing is stored when we could not verify it"

The meta-test that keeps the guardrails complete

no code in PRECEDENCE may go untesteddef test_every_rejection_code_is_exercised_somewhere():
    """Implemented by RUNNING the suite's own specimens rather than by
    scraping source text, so it cannot be satisfied by a code appearing in a
    comment.

    When you add a rule, this fails until you add a case -- which is the only
    reliable way a guardrail suite stays complete as the policy grows."""
    seen = set()
    for image, verdict, pol, flags in specimens:
        seen.update(codes(image, verdict, pol, flags))

    untested = set(PRECEDENCE) - seen - {"MALFORMED"}
    assert not untested, (
        "these policy codes are never produced by any specimen in the suite, "
        "so nothing would notice if the rule stopped firing: "
        + ", ".join(sorted(untested)))

Path safety: regenerate, do not sanitise

   the traversal test everyone writes, and why it is not the fix

   "../../etc/passwd"    "..%2f..%2fetc"    ".../....//"    "\\..\\..\\"

   this list can never be complete. There is always another encoding.

   so the design does not rely on it:

     the key is not SANITISED, it is REGENERATED.
     `put()` hashes the bytes itself and ignores whatever the
     client called them. `has()` on a hostile string is just a miss.

   the parametrized sweep still exists -- but it documents the threat,
   it does not implement the defence. And it gets a companion:

     def test_the_traversal_test_is_not_vacuous(vault, uf2):
         # if has() returned False for EVERYTHING, the whole sweep
         # passes against a store that does not work.
         digest, _ = vault.put(uf2, manifest_for(uf2))
         assert vault.has(digest) is True

Differential & Oracle Tests

When a parser gets complicated enough that you cannot write down the expected output by hand, write a second, dumber implementation and compare them. The naive one is obviously correct and much too slow; the real one is fast and subtle.

   the wiring

                    ┌────────────────────┐
      specimen ──┬──►  walk_uf2()          ├──► blocks  ─┐
                 │  │ fast, subtle       │            │
                 │  └────────────────────┘            ├──► == ?
                 │  ┌────────────────────┐            │
                 └──►  _reference_walk()   ├──► blocks  ─┘
                    │ naive, obvious     │
                    └────────────────────┘

   both OUTCOMES are compared, not just both outputs.
   agreeing on the block list is not enough if one raises where the
   other does not.

   the ORACLE variant: same shape, but the reference is a trusted
   external tool -- your parser vs readelf, your decoder vs ffmpeg --
   and the corpus is real files instead of generated ones.

tests/test_differential.py# one test per specimen, NAMED by specimen. Adding a corpus entry adds
# a test with no assertion to write -- the whole economics of this.

CORPUS = {
    "one-block":         build_uf2(blocks=1),
    "many-blocks":       build_uf2(blocks=17),
    "no-family":         build_uf2(blocks=2, family=None, flags=0),
    "zero-payload":      build_uf2(blocks=3, payload_size=0),
    "max-payload":       build_uf2(blocks=3, payload_size=476),
    "oversized-payload": build_uf2(blocks=2, oversized_payload=5000),
    "backwards-addr":    build_uf2(blocks=3, addr_step=-256),
    "bad-end-magic":     build_uf2(blocks=3, bad_end_magic=2),
    "ragged":            build_uf2(blocks=2) + b"\x00" * 3,
    "empty":             b"",
}

@pytest.mark.parametrize("name", sorted(CORPUS), ids=sorted(CORPUS))
def test_the_walker_agrees_with_the_reference(name):
    ours,   our_error   = _run(walk_uf2, CORPUS[name])
    theirs, their_error = _run(_reference_walk, CORPUS[name])

    assert (our_error is None) == (their_error is None), (
        "specimen {!r}: one implementation raised and the other did not "
        "(ours={!r}, reference={!r})".format(name, our_error, their_error))
    if our_error is None:
        assert ours == theirs, "specimen {!r}: block streams differ".format(name)

Differential also works at a higher level than the walk. The reference computes the payload total the obvious way; parse accumulates it during the walk with a clamp. Two routes to one number, and no expected value written down anywhere:

folds agree toodef test_the_folds_agree_on_payload_total():
    blob = build_uf2(blocks=6, payload_size=300, family=RP2040)
    expected = sum(min(b["payload_size"], 476) for b in _reference_walk(blob))
    assert parse(blob).payload_bytes == expected

CLI & Subprocess

Two ways to test a command line, and you want both. The shape that makes the fast way possible costs nothing and is the highest-leverage decision in a CLI's design.

   def main(argv=None, stdout=None, stderr=None, transport=None) -> int

     · takes argv                    does not read sys.argv
     · returns an int                does not call sys.exit
     · writes to streams it was HANDED

   ────────────────────────────────────────────────────────────────

   IN-PROCESS    main(["inspect", path])          microseconds
                 every branch · real tracebacks · the coverage lives here

   SUBPROCESS    python -m fwvault inspect ...    milliseconds each
                 a handful of cases · "works on my machine" dies here

   in-process CANNOT see:
     · whether the package imports from a CLEAN interpreter
     · whether the console entry point exists
     · what the SHELL sees as the exit code
     · stream buffering, encoding, line endings
     · a stray print() somewhere in the import path corrupting your JSON

Exit codes are a contract

in-process, with streams we owndef run(argv, transport=None):
    # explicit StringIOs rather than capsys: capsys also captures pytest's
    # own output and anything a library logs, which makes an exact-match
    # assertion on stdout fragile.
    out, err = io.StringIO(), io.StringIO()
    code = cli.main(argv, stdout=out, stderr=err, transport=transport)
    return code, out.getvalue(), err.getvalue()


def test_findings_go_to_stderr_and_data_to_stdout(tmp_path):
    # so `fwvault --json inspect x | jq` works while warnings stay visible.
    # mixing them means the pipe eats the warnings or the JSON parse
    # fails -- and which one it is depends on the day.
    _code, out, err = run(["--json", "inspect", str(path)])
    json.loads(out)                     # stdout is pure JSON
    assert "warning:" in err

the real process, four casesdef _module_run(args, **kwargs):
    env = dict(os.environ, PYTHONPATH=os.path.join(REPO, "src"))
    return subprocess.run(
        [sys.executable, "-m", "fwvault"] + args,
        capture_output=True, text=True, env=env, cwd=REPO,
        timeout=30,          # ← a test that CAN hang needs a timeout on the CALL
        **kwargs)

def test_stdout_stays_parseable_through_a_real_pipe(artifact):
    # catches a stray print() in the import path, which corrupts the JSON
    # for every downstream consumer and is completely invisible to capsys.
    result = _module_run(["--json", "inspect", artifact])
    assert json.loads(result.stdout)["kind"] == "uf2"

The cheapest documentation test there is

walk the parser, assert every flag has helpdef test_every_subcommand_has_help_text():
    parser = cli.build_parser()
    subparsers = [a for a in parser._actions
                  if hasattr(a, "choices") and isinstance(a.choices, dict)]
    assert subparsers, "no subcommands found; did the parser shape change?"

    missing = ["{} {}".format(name, arg.dest)
               for a in subparsers for name, sub in a.choices.items()
               for arg in sub._actions
               if arg.dest != "help" and not arg.help]
    assert not missing, "undocumented flags: " + ", ".join(missing)

# a flag with no help is a flag nobody can use, and it is invisible
# to every behavioural test in the file.

Meta-Tests: Contract, Lean Install, Hygiene

A test suite is code, it rots like code, and it rots in a way ordinary code does not: silently, and in the direction of passing. A test that stops asserting anything still shows up green. A test that skips on every machine still counts toward "1,204 passed".

   the tier above the tests

   CONTRACT       the SHAPE you publish, pinned
                  __all__ · rejection codes · exit codes · schema version
                  · the route table · the response envelope
                  invisible to every behavioural test

   LEAN INSTALL   the dependency boundary, two ways
                  static: AST-walk for module-level optional imports
                  behavioural: sys.modules[name] = None, then run it

   HYGIENE        the suite as DATA
                  every test asserts · every skip explains itself
                  · no fixed paths · no path outside the repo
                  · no silently disabled test · no duplicate names

   and the rule that keeps this tier honest:
   every check must be SHOWN FIRING at least once.
   a filter with a broken pattern passes everything, forever.

Contract: what a behavioural test cannot see

tests/test_contract.pydef test_the_public_api_is_exactly_what_is_documented():
    # both directions matter. A name that disappeared breaks importers;
    # a name that APPEARED is a promise nobody meant to make, and it is
    # much harder to withdraw a year later.
    assert set(fwvault.__all__) == EXPECTED_API

def test_rejection_codes_are_frozen():
    # clients branch on these strings. Renaming one is a breaking change
    # whether or not anything else in the suite notices.
    #
    # `detail` is deliberately NOT pinned -- the wording should be free
    # to improve, and a test that asserts on prose fails every time
    # someone fixes a typo.
    assert PRECEDENCE == ("MALFORMED", "OVERSIZE", ...)

def test_the_shipped_doubles_match_the_transport_protocol():
    # a fake whose signature drifts from the real transport passes every
    # test in this repo and breaks in every consumer.
    real = inspect.signature(UrllibTransport.request)
    for double in (RecordingTransport, ScriptedTransport):
        assert inspect.signature(double.request) == real, double.__name__

Lean install: the boundary, asserted twice

static: reads source, so it reports EVERY offender in one rundef _module_level_imports(path):
    tree = ast.parse(open(path, encoding="utf-8").read())
    names = set()
    for node in tree.body:          # TOP LEVEL only -- lazy imports are the contract
        if isinstance(node, ast.Import):
            names.update(a.name.split(".")[0] for a in node.names)
        elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
            names.add(node.module.split(".")[0])
    return names

# a test that fails with "fix this, then run me again to find the next
# one" wastes an afternoon.

behavioural: one test per package, so a failure NAMES which one@pytest.mark.parametrize("absent", sorted(OPTIONAL), ids=sorted(OPTIONAL))
def test_the_core_imports_with_every_optional_package_missing(absent, monkeypatch):
    monkeypatch.delitem(sys.modules, absent, raising=False)
    monkeypatch.setitem(sys.modules, absent, None)
    for name in ("fwvault", "fwvault.parse", ...):
        monkeypatch.delitem(sys.modules, name, raising=False)

    assert importlib.import_module("fwvault").parse is not None

# the alternative is a second CI job with a different lockfile, which
# is real work and gets disabled the first time it goes red on a Friday.

Hygiene: treat the test tree as data

tests/test_suite_hygiene.pydef test_every_test_asserts_something():
    for path in TEST_FILES:
        for name, node in find_tests(path):
            has_assert  = any(isinstance(n, ast.Assert) for n in ast.walk(node))
            has_raises  = # .raises / .warns / .approx
            calls_check = # a helper named assert_*
            if not (has_assert or has_raises or calls_check):
                offenders.append("{}:{} {}".format(path.name, node.lineno, name))

    assert not offenders, (
        "these tests assert nothing, so they pass whatever the code does:\n  "
        + "\n  ".join(offenders))

Scanning source without flagging your own prose

the false-positive fixdef code_lines(path):
    """Every line that is CODE: no comments, no docstrings.

    String LITERALS stay in, and that distinction is the whole point. The
    banned things these checks look for -- "/tmp/", "expanduser" -- appear in
    real code AS string literals, so a scanner that strips every string can
    never see the defect. It also has to strip docstrings, or it flags the
    paragraph explaining the rule."""
    skip = set()
    for token in tokenize.generate_tokens(io.StringIO(source).readline):
        if token.type == tokenize.COMMENT:
            skip.update(range(token.start[0], token.end[0] + 1))

    for node in ast.walk(ast.parse(source)):
        # a docstring is exactly an Expr whose value is a str Constant
        if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant) \
                and isinstance(node.value.value, str):
            skip.update(range(node.lineno, (node.end_lineno or node.lineno) + 1))

    for lineno, line in enumerate(source.splitlines(), 1):
        if lineno not in skip:
            yield lineno, line

Distinguish a constant from shared state

mutable module-level state makes test ORDER load-bearing# flag only names something actually MUTATES. A frozen set of expected
# API names assigned once is a constant, and flagging it teaches people
# the check is noise.

MUTATORS = {"append", "add", "extend", "update", "insert",
            "pop", "remove", "clear", "setdefault"}

# CONSTANT = {'a', 'b'}   → not flagged
# STATE = []             → flagged, because STATE.append(1) exists

Coverage, Mutation & CI

Coverage answers "did this line run". It cannot answer the question you actually have.

   100% line coverage is compatible with ZERO assertions.

   def test_everything():
       parse(build_uf2(blocks=3))
       evaluate(image, verdict, policy)
       store.put(blob, manifest)
       # no asserts. every line green.

   ─────────────────────────────────────────────────────────────

   line coverage     did this line execute
   branch coverage   did BOTH sides of this `if` execute
                     --cov-branch. line coverage calls a half-tested
                     conditional covered, which is most of them.
   mutation score    if I BREAK this line, does anything go red

   only the third one is a question about your TESTS. the first two
   are questions about your test RUN.

Mutation testing: break it on purpose

   the loop

   for each small edit the tool can make:
        │   (x < y  →  x <= y,  return v  →  return None,  and  →  or)
        ▼
     apply it to the source
        ▼
     run the whole suite
        ▼
     suite goes RED   → mutant KILLED    the behaviour is checked
     suite stays GREEN → mutant SURVIVED  a hole. executed, unchecked.

   the tools are mutmut and cosmic-ray. both rewrite your source and
   run the suite once per mutant, so they take minutes to hours.
   run one occasionally. never in CI.

A mutation score is never 100%, and the reason has a name. Some edits cannot be detected because they do not change behaviour at all:

The cheap version, which does belong in the suite

Hand-written mutants, each paired with the assertion that should catch it. Milliseconds, lives next to the code, and it fails the day someone weakens a check. It cannot find the mutants nobody thought of, so it does not replace the real tool. It pins the checks that matter most.

tests/test_mutation.py# A mutant is KILLED if the suite goes red, and a test that ERRORS is red
# just like a test that fails. So any exception counts.
#
# pytest.fail.Exception has to be listed separately: a failing
# pytest.raises(...) raises Failed, which derives from BaseException
# rather than Exception, so a bare `except Exception` misses it.
CHECK_FAILURES = (Exception, pytest.fail.Exception)


def assert_caught(check, what):
    """Run a real suite assertion against a mutant and require it to go RED."""
    try:
        check()
    except CHECK_FAILURES:
        return
    raise AssertionError(
        "MUTANT SURVIVED: {}. The suite executes this behaviour but does "
        "not check it.".format(what))

the mutant this whole service is shaped arounddef test_an_outage_rendered_as_unsigned_is_caught(clock):
    # a "helpful" refactor catches VaultUnavailable so the pipeline does
    # not blow up. the service keeps running, the rejections look real,
    # and nobody finds out until someone asks why every build failed.
    class ForgivingClient(SigningClient):
        def verify(self, digest):
            try:
                return super().verify(digest)
            except VaultUnavailable:
                return Verdict(signed=False)          # ← the mutation

    dead = ScriptedTransport([TransportError("down")] * 3)
    client = ForgivingClient("https://x", transport=dead, clock=clock, retries=3)

    def the_suite_assertion():
        with pytest.raises(VaultUnavailable):       # from test_doubles.py
            client.verify("0" * 64)

    assert_caught(the_suite_assertion, "an outage rendered as Verdict(signed=False)")

CI: two jobs, and the first one is the point

.github/workflows/tests.ymljobs:
  bare:                          # pytest and NOTHING else
    steps:
      - run: pip install pytest
      - run: python -m pytest -q

  full:
    strategy:
      fail-fast: false            # one red cell must not hide the others
      matrix:
        os: [ubuntu-latest, windows-latest]
        python-version: ["3.11", "3.13"]
    steps:
      - run: pip install -e ".[test]"
      - run: python -m pytest -q

  coverage:
    steps:
      - run: python -m pytest -q --cov=fwvault --cov-branch --cov-report=term-missing

Testing an AI Author

The argument goes: nobody reads their compiler's assembly output, so nobody will read generated code either, and the objection is sentimental. That argument is better than its critics allow and worse than its advocates think. This section takes it seriously, then takes it apart, then says what the repo does about it.

   the position, in two moves

   Su          lights-out codebases: code no human reads or edits.
               volume makes review impossible, and the chess analogy
               says it will look like a category error to have tried.

   Venturini   the better version. you do not read your compiler's
               output, and NOT because compilers are trusted blindly.
               they have shipped catastrophic bugs. it is because an
               apparatus was built that makes reading unnecessary:
               behavioural tests, type systems, reproducible builds,
               fuzzers, sanitizers, formal methods where it matters.

               you trust the process, not the artifact.

   his diagnosis: that apparatus does not exist for agents yet,
   so the fear is rational rather than nostalgic.

Where the analogy breaks

The numbers

Every figure below is from a paper, and each was checked against the abstract rather than taken from a summary. Sources at the end of the section.

The loop that cannot close

   self-verification                      provenance separation

   model ──► code                        model A ──► code
     │       ▲                                     ▲
     └──► tests ──┘                        model B ──► tests ──┘
          │                                          │
          ▼                                          ▼
       GREEN                                    independent oracle
   and it means nothing:                    still not sufficient,
   both halves share the                    but now a wrong belief
   same wrong belief                        has to be held twice

   tests/test_oracle_independence.py runs exactly this. A parser and
   its tests, both built on one plausible wrong number: the pair is
   self-consistent, green, and wrong. Only the spec catches it.

   no model is involved in that file. it is a property of where the
   assertions came from, and it holds for a human who writes tests by
   running the code and pasting the output.

What this repo measures

tests/test_ai_authored.py puts two suites over the same twelve specimens. One is written in the shape a model produces: sweep the inputs, assert only that nothing escaped the declared exception. The other asserts what each specimen should actually produce. Coverage is counted with a twenty-line sys.settrace tracer so the number owes nothing to a plugin.

measured, not asserted from memory                 lines covered   mutants killed
ai-authored           93            0 / 5
asserting             93            5 / 5
                 # identical coverage. a --cov gate cannot
                 # tell these two files apart.

Three of the five mutants return a plausible Image with the wrong contents rather than raising. That is the failure an intake service actually ships: not a crash, an answer that is quietly false. A never-crash suite rewards it, because refusing to raise is exactly what it checks for.

Eight things to do about it

Where it runs out

   a test can check that the code does what the test says.
   nothing can check that the test says what was meant.

   spec ──► tests ──► code
    ▲
    │
   a human wrote this, and a human has to read it

   the mitigations above all live to the RIGHT of the spec. every one
   of them assumes the tests encode the intent. none of them can check
   that they do.

   the only defence is that the spec is small enough to be read.

This is the one place the apparatus genuinely runs out, and it is worth saying plainly rather than ending on the eight mitigations as though they closed the loop. An agent can verify code against tests all day. It cannot verify that the tests capture what anyone actually wanted.

Sources

Running It

the repo# everything a fresh clone can run -- no install, no dependencies
python -m pytest -q                     # 424 passed, 3 skipped

# plus hypothesis, httpx, starlette
pip install -e .[test]
python -m pytest -q                     # 433 passed, 2 skipped

# the tests that make real connections, off by default
python -m pytest --runnet

Flags worth muscle memory

Plugins, and whether you need them

The order this repo would teach it in

   1  make the code have seams          every collaborator a parameter
   2  write specimen BUILDERS           not a committed corpus
   3  example tests, with good IDs      §04, §05
   4  a FAKE for each seam              §07, §09 -- ship it in the package
   5  the negative cases                the clean specimen that must NOT warn
   6  hostile input                     §12 -- truncation first, it is cheapest
   7  contract tests                    §16 -- before your first release
   8  hygiene tests                     §16 -- before the suite hits 100 files

   patching comes LAST, and only for things you do not own.