pytest console

STRUCTURAL   split a function so both halves are reachable
   walk_uf2()   generator, yields Block records
   parse_uf2()  folds that stream into an Image
   assert on block 10 of 900 without building a
   manifest; test the fold with hand-built records
   and no bytes at all.

INVERSE      serialize_uf2() is walk_uf2 backwards
   buys the round-trip relation, which needs no
   oracle at all. see the EVIDENCE tab.

IMPORT       sys.modules[name] = None
   makes `import name` raise. tests the
   dependency-absent path on a machine that has it.

ENVIRONMENT  FWVAULT_WALKER_RAISE
   a walker bug is a warning in production and a
   traceback in the suite. both states tested.

visibility follows the DIRECTORY tree.
nearest wins. no import needed, anywhere.
repo/conftest.py       everything below
  tests/conftest.py    tests/ and below
    tests/test_x.py    a fixture here SHADOWS both

1. isolate the process (autouse)
@pytest.fixture(autouse=True)
def isolate_environment(monkeypatch, tmp_path_factory):
    home = tmp_path_factory.mktemp("fwvault_home")
    monkeypatch.setenv("HOME", str(home))
    monkeypatch.setenv("USERPROFILE", str(home))  # windows
    monkeypatch.setenv("FWVAULT_WALKER_RAISE", "1")

2. supply specimens and collaborators
3. extend pytest itself
def pytest_addoption(parser):
    parser.addoption("--runnet", action="store_true")

def pytest_collection_modifyitems(config, items):
    # where a marker becomes behaviour

FACTORY: the test gets the FUNCTION
@pytest.fixture
def make_uf2(): return build_uf2
  beats forty fixtures named after their defects

PARAMETRIZED: requester runs once per case
@pytest.fixture(params=["uf2","elf"], ids=[...])
def any_artifact(request): ...

NAMED: function name is not the fixture name
@pytest.fixture(name="digest")
def _make_digest(uf2): ...

OVERRIDE: a module-level fixture shadows conftest
  silently. nothing announces it.

CONDITIONAL CLEANUP: addfinalizer beats yield
    request.addfinalizer(t.unlink)  # only now that
                                    # it exists

@pytest.mark.xfail(reason="bug FW-118")
def test_the_bug():
    assert True        # someone fixed it
# → XPASS. exit 0. GREEN. the marker sits there
# for two years and the test asserts nothing.

@pytest.mark.xfail(strict=True, reason="FW-118")
# → FAILED: "delete this marker."

with pytest.raises(ParseError, match=r"bad magic"):
    parse(blob)
  match is re.SEARCH, not fullmatch.
  ESCAPE YOUR METACHARACTERS. "offset (0x1fc)"
  is a group. a match that matches NOTHING still
  passes if the exception type is right.

with pytest.raises(TruncatedError) as ei: ...
assert ei.value.needed == 32   # the OBJECT, not
                               # just the class

A flaky test is a bug report, against the test or the code. Never noise. Something is genuinely non-deterministic and you have found it.

A library whose only seam is a protocol owes its users a working fake for it, the way httpx ships MockTransport. Otherwise every downstream user writes their own and each gets a detail wrong.

# fwvault/testing.py, part of the package
build_uf2(blocks=3, bad_end_magic=1)   # named defect
build_elf(machine=0x28, bit64=True)
signed_body(signer="ci-builder")

RecordingTransport(routes={...})       # spy
ScriptedTransport([err, err, Response(200, ...)])
flaky(2, Response(200, body))          # the idiom

# running off the end of a script is a HARD FAILURE
AssertionError: ScriptedTransport exhausted:
call 4 was not scripted (GET https://...)
# a double that keeps answering makes "retried 3
# times" and "retried 300 times" the same test.

1 raising=False disables your only signal
  a patch aimed at a typo'd name is a patch
  aimed at nothing, and it passes.

2 patching a constant does not move an
  already-bound default
  @dataclass  x: int = CONST  ← evaluated ONCE at
  class creation. same for def f(x=CONST) and
  `from mod import CONST`.

3 the string form walks getattr
  "pkg.mod.CONST" imports pkg then getattrs along.
  if pkg re-exported a FUNCTION over its own
  submodule name, the walk dies there.
  `import pkg.mod as m` does NOT save you.
  importlib.import_module() reads sys.modules.

4 sys.modules[n] = None makes `import n` raise
  delitem FIRST, or the None assignment is what
  takes effect and you pass for the wrong reason.

Store(root).put(blob, manifest) → (digest, created)
content-addressed. atomic. two-level fan-out.

• Atomic: temp file in the SAME directory plus os.replace. A temp in the system temp dir and a cross-device move is a copy, and a copy is the torn write you were avoiding.
• Idempotent: the same bytes twice is a no-op, not a rewrite.
• Path safety by regeneration: the key is not sanitised, it is hashed from the bytes. A traversal string is just a miss.

# the traversal test everyone writes:
"../../etc/passwd"   "..%2f..%2fetc"   "....//"
this list can never be complete.
# so the design does not rely on it. the sweep
# documents the threat; regeneration IS the defence.

def main(argv=None, stdout=None, stderr=None) -> int
  takes argv. returns an int. writes to streams it
  was HANDED. the highest-leverage decision in a
  CLI's design, and it costs nothing.

• importable from a clean interpreter
• the exit code as the shell sees it
• buffering, encoding, line endings
• a stray print() in the import path corrupting your JSON

what is worth asserting is not "it returned a
Verdict". it is WHEN, HOW OFTEN, and WHAT IT DID
NOT DO. none of those show in a return value.

assert clock.slept == [0.5, 1.0, 2.0]   schedule
assert len(transport.calls) == 1        no retry
assert verdict.stale is True            honesty

fwvault has no container in the loop, so nothing here is demonstrated by a running test. The pattern is the one this repo already uses for the single real HTTP call, and it generalises unchanged.

# the shape: a marker, deselected by default
@pytest.mark.net          # or .docker, .integration
def test_against_the_real_thing():
    ...

# conftest turns the marker into behaviour
def pytest_collection_modifyitems(config, items):
    if config.getoption("--runnet"): return
    skip = pytest.mark.skip(reason="needs a real network;
                                    pass --runnet")
    for item in items:
        if "net" in item.keywords:
            item.add_marker(skip)

A surface most projects never test. These read the test tree as data.

• every test asserts something
• every skip explains itself
• no fixed paths, no path outside the repo
• nothing silently disabled: a bare return at the top leaves no skip line at all
• no duplicate test names across files
• no mutated module-level state
• the published shape is pinned: names, codes, routes, schema version
• the dependency boundary, statically and behaviourally

ROUND TRIP: the strongest, when an inverse exists
assert serialize_uf2(walk_uf2(blob)) == blob

INVARIANCE: changing X must NOT change Y
scrambled = every payload byte randomised
assert parse(scrambled) == parse(blob)

EQUIVARIANCE: change X, predict the change in Y
assert after.payload_bytes ==
       before.payload_bytes + 200 * extra

IDEMPOTENCE: twice is once
assert serialize_uf2(walk_uf2(once)) == once

specimen ─┬─► walk_uf2()          ─┐
          │   fast, subtle        ├─► == ?
          └─► _reference_walk()   ─┘
              naive, obvious, INDEPENDENT

• The reference must be independent. Shared helpers mean a shared bug cancels out and both agree. Asserted structurally here: the reference borrows no constant, import or type from the implementation.
• Compare outcomes, not just outputs. Agreeing on the result is not enough if one raises and the other does not.
• The corpus needs a guard. If every specimen raises, both agree everything is broken and the test is vacuous.
• One test per specimen, named by specimen. Adding a corpus entry adds a test with no assertion to write. That is the economics.

A mutation score is never 100%, and the reason has a name: some edits cannot be detected because they change no behaviour.

# the first attempt at "PRECEDENCE gets alphabetised"
# SURVIVED, and it was not a hole:
real:         MALFORMED, OVERSIZE, EMPTY_PAYLOAD, ...
alphabetical: DENIED_MACHINE, EMPTY_PAYLOAD, ... OVERSIZE

# for the specimen chosen, OVERSIZE came first in
# BOTH orders. the mutation changed nothing
# observable.
OVERSIZE + EMPTY_PAYLOAD is the pair that
actually distinguishes them.

# one surface at a time
pytest tests/test_parse.py tests/test_hostile.py \
       tests/test_metamorphic.py tests/test_differential.py

# by name expression, across files
pytest -k "parse or walk or serialize"
pytest -k "policy or reject"
pytest -k "asgi or transport"

# the meta tier only
pytest tests/test_suite_hygiene.py tests/test_contract.py \
       tests/test_lean_install.py tests/test_cards.py

# everything EXCEPT the slow generated ones
pytest -k "not property and not metamorphic"

# what changed, if you use git
pytest $(git diff --name-only HEAD~1 | grep ^tests/)

# coverage as a MAP, not a score
pytest --cov=fwvault --cov-branch --cov-report=term-missing

# the branch list only, which is what you act on
pytest --cov=fwvault --cov-branch --cov-report=term-missing \
  | grep -E "^src.*[0-9]+%" | sort -t% -k1

# does the suite notice a break? (slow, occasional)
pip install mutmut && mutmut run --paths-to-mutate src/fwvault

# the cheap in-repo version, milliseconds
pytest tests/test_mutation.py tests/test_ai_authored.py -v

# order dependence, the cheapest detector you own
pytest -p xdist -n 4
pytest -p no:randomly            # baseline to compare

# slowest ten. run it monthly.
pytest --durations=10

# after editing terms.py
python gen_glossary.py     # → GLOSSARY.md, cards, frag

# after editing card_content*.py
python build_card.py       # the 3-sheet reference
python build_basics.py     # the 1-sheet beginner card
python build_console.py    # this page

# layout without writing anything
python build_card.py --check
python build_console.py --check

# the deep dive: edit frag.html, NEVER the built page
build.py frag.html testharness_deep-dive.html "<title>"

# the check that catches a stale artifact
pytest tests/test_cards.py

jobs:
  bare: pip install pytest   # and NOTHING else
        # the only environment that can DISPROVE a
        # no-dependencies claim.

  full: matrix [ubuntu, windows] x [3.11, 3.13]
        fail-fast: false   # one red cell must not
                            # hide the others

  cov:  --cov=fwvault --cov-branch --cov-report=term-missing
        # no --cov-fail-under. a threshold is a number
        # people game, and the cheapest way to game it
        # is a test with no assertions.

a test can check the code does what the test says.
nothing can check the test says what was meant.

spec ──► tests ──► code
 ▲
 └─ a human wrote this, and a human has to read it

every mitigation lives to the RIGHT of the spec.
all of them assume the tests encode the intent.
none can check that they do.

the only defence is that the spec is small enough
to be read.