testing-python-libraries

name testing-python-libraries description Designs and implements pytest test suites for Python libraries with fixtures, parametrization, mocking, Hypothesis property-based testing, and CI configuration. Use when creating tests, improving coverage, setting up testing infrastructure, or implementing property-based testing. Python Library Testing Quick Start uv run pytest # Run tests uv run pytest --cov=my_library # With coverage uv run pytest -x # Stop on first failure uv run pytest -k "test_encode" # Run matching tests Pytest Configuration # pyproject.toml [tool.pytest.ini_options] testpaths = [ "tests" ] addopts = "-ra -q --cov=my_library --cov-fail-under=85" [tool.coverage.run] branch = true source = [ "src/my_library" ] Test Structure tests/ ├── conftest.py # Shared fixtures ├── test_encoding.py └── test_decoding.py Essential Patterns Basic test: def test_encode_valid_input (): result = encode( 37.7749 , - 122.4194 ) assert isinstance (result, str ) assert len (result) == 12 Parametrization: @pytest.mark.parametrize( "lat,lon,expected" , [ ( 37.7749 , - 122.4194 , "9q8yy" ), ( 40.7128 , - 74.0060 , "dr5ru" ), ] ) def test_known_values ( lat, lon, expected ): assert encode(lat, lon, precision= 5 ) == expected Fixtures: @pytest.fixture def sample_data (): return [( 37.7749 , - 122.4194 ), ( 40.7128 , - 74.0060 )] def test_batch ( sample_data ): results = batch_encode(sample_data) assert len (results) == 2 Mocking: def test_api_call ( mocker ): mocker.patch( "my_lib.client.fetch" , return_value={ "data" : []}) result = my_lib.get_data() assert result == [] Exception testing: def test_invalid_raises (): with pytest.raises(ValueError, match = "latitude" ): encode( 91.0 , 0.0 ) For detailed patterns, see: FIXTURES.md - Advanced fixture patterns HYPOTHESIS.md - Property-based testing ../project-setup/CI.md - CI/CD test configuration Test Principles Principle Meaning Independent No shared state between tests Deterministic Same result every run Fast Unit tests < 100ms each Focused Test behavior, not implementation Tests That Lie: Avoiding False-Green A passing suite is worthless if it can't fail when the code is wrong. The most expensive bugs ship under green CI. Audit for these anti-patterns — each is a way "all tests pass" can mask broken behavior. Conditional assertions that vacuously pass. If the setup silently fails, a guarded assertion never runs and the test still passes. # BAD — if the Sphinx build fails, no index.html, so nothing is asserted. def test_build_includes_css (): if index_html.exists(): # build broke? test passes anyway. assert "theme.css" in index_html.read_text() # GOOD — assert the precondition, then the behavior. def test_build_includes_css (): assert index_html.exists(), "build produced no index.html" assert "theme.css" in index_html.read_text() Over-permissive assertions. An or that can't fail, or a substring match so loose it accepts wrong output. assert result.returncode == 0 or "html_static_path" in result.stderr # masks real failures assert "ui" in todo.tags # also matches "build" Presence assertions are blind to duplicates. assert X in output proves that at least one X exists. It cannot tell you that there are two, or which one a consumer will honor. That is the exact shape of bugs in generated output — HTML tags, config keys, emitted headers — where a framework's base template and your override each emit one and they disagree. Extract every occurrence and compare the whole list. # BAD — passes with one correct tag, and equally with two conflicting ones. assert '<link rel="canonical"' in html # GOOD — pins both the count and the values. canonicals = re.findall( r'<link rel="canonical" href="([^"]*)"' , html) assert canonicals == [ "https://example.com/guide/" ] When two emitters produce the "same" tag they often don't format it identically (one self-closing, one not), so anchor the pattern on the attribute you care about and stop before the tag close — otherwise the duplicate you're hunting slips past your regex and the test looks clean. The same rule applies to anything countable: assert len(rows) == 2 and the row values, not assert rows . Mocking the thing under test. If you patch _run_command and only assert the argv tokens, the test locks in a command that may not exist — it stays green even after the subcommands or flags it builds are renamed or removed upstream. Mock at the boundary (the subprocess/HTTP call), then assert on the parsed result , not on the arguments you passed in. Fakes that ignore the parameters being tested. A fake client whose list_items returns all canned rows in one call — ignoring after and pagination — cannot exercise the pagination or incremental-sync logic those parameters drive, so it stays unverified. Make fakes honor the parameters whose handling is the point of the test. Smoke tests that import the wrong thing. python -c "import server" can print success by resolving an empty server/ package that shadows the real server.py — a broken wheel that still "imports." Assert a real symbol is reachable ( from server import main; main ), not merely that an import name resolves. Forgotten mock → silent real network calls. A test missing its httpx_mock fixture hits the live API: slow, flaky, rate-limited, and silently exercising nothing deterministic. Add --disable-socket (pytest-socket) so any unmocked network call fails loudly instead of "passing." No-op CI gates. Confirm the gate actually runs the tests: go test ./... / pytest with zero test files is a green no-op. Files excluded via --ignore or pytest.mark.skip "because flaky" often fail deterministically — exclusion hides real breakage, not flakiness. Marker filters ( -m "not integration" ) can deselect the only meaningful tests. Reproduce CI's exact marker expression locally before trusting green. Empty evaluator sets must not mean "all clear." Auditors, policy engines, and validation pipelines often compute a score from the enabled rules. A category filter can accidentally disable every rule; if the scoring code maps results == [] to score = 100 , an unsupported request becomes a silent perfect pass. Treat an empty post-filter rule set as a configuration/error state, or keep an aggregate rule enabled when it is the evaluator for every category. enabled = [rule for rule in rules if rule.supports(requested_categories)] if not enabled: raise NoApplicableRulesError(requested_categories) results = evaluate(enabled, subject) Regression tests must assert more than the final score: request each supported category (and representative combinations), assert that at least one rule ran, and use a known failing subject so a no-op path cannot look clean. result = audit(known_noncompliant_subject, categories=[ "bias" ]) assert result.rules_evaluated > 0 assert result.issues # proves filtering did not bypass evaluation assert result.overall_score < 100 Tests written around a bug. Wrapping a call in try/except RuinError to make it pass documents the bug as acceptable. Assert the correct behavior and let it fail until the bug is fixed (use xfail(strict=True) to track it without red CI). Prove the Test Can Fail: Mutate the Fix Every regression test makes an implicit claim — this would have caught the bug . The only way to check the claim is to put the bug back and watch the test go red. Do it once, while the fix is still fresh in your head; it takes a minute and it is the difference between a regression test and a decoration. # 1. revert the fix (git stash, or hand-edit the guard back to its broken form) # 2. run ONLY the new test — it must FAIL, and for the right reason uv run pytest tests/test_paths.py::test_symlink_escape_rejected -q # 3. restore the fix — it must pass Mutate each half of a compound guard separately. A fix that validates an input and re-checks the resolved result looks redundant until you revert each half on its own. def _resolve_baseline ( name: str ) -> Path: if ".." in name or not NAME_RE.fullmatch(name): # mutation A raise ValueError(name) path = (BASE_DIR / f" {name} .json" ).resolve() if not path.is_relative_to(BASE_DIR.resolve()): # mutation B raise ValueError(name) return path Reverting A is caught by a ../../etc/passwd test. Reverting B is caught only by a symlink placed inside the approved directory that points outside it — a test most suites don't have. If reverting one half leaves the suite green, you have a test gap, not a redundant check: write the test that pins it. Know what "red" looks like for the mutation you chose. A reintroduced bug does not always surface as a clean assertion failure. Remove a retry/iteration cap and the suite hangs instead of failing. Run the mutated suite under an external timeout — timeout 60 uv run pytest -q with no output is the reproduction. Making the cap merely unreachable ( > 10**9 ) is the honest mutation; the off-by-one ( > → >= ) fails loudly and is the cheaper one to re-run day to day. Drop an await or a try in async code and the runner may die with an unhandled rejection and a worker crash rather than a named test failure. Still red — read the output before concluding your test didn't fire. If the mutated run errors during collection , no test ran at all and you have learned nothing about the test. A hand-written fixture is itself a mutation — of reality. When the same person authors both a parser and every fixture it is tested against, both encode the same assumption, and the suite is green against a guard that cannot fire in production. A regex requiring single spaces between tokens matches hand-typed single-space fixtures forever, while the live document renders those tokens across indented lines. Capture at least one fixture from the real source, commit it, and point the guard's test at that. Mutate in Both Directions: A Guard Can Also Fire Too Often A fix that adds a conditional — a warning on incomplete data, a validation check, a flag that suppresses a claim — has two plausible wrong implementations, not one: it never fires (the original bug), or it always fires (over-broad). The regression test you naturally write covers only the first. Mutate both ways. # The fix: warn when a sub-fetch failed, so a partial total isn't read as complete. if failed_sources: log.warning( "counts UNDERCOUNT: no data from %s" , ", " .join(failed_sources)) # Mutation 1 — the original bug. Delete the tracking so the list is always empty. # Mutation 2 — over-broad. Drop the condition, or write `if not sources:`. The control test is the one that catches mutation 2, and it looks vacuous. A test asserting that a fully successful run emits no warning passes on the pre-fix code by construction, so it reads like it proves nothing and gets cut in review. It is the only test that fails when the warning becomes unconditional. Keep it, and name the pairing explicitly in the PR body so a reviewer doesn't delete half the coverage: mutation test that goes red tracking removed (never warns) test_failed_source_warns condition removed (always warns) test_healthy_run_emits_no_warning Truthiness is how guards become over-broad. The over-firing mutation is rarely a deliberate edit — it is if not x: where the intent was "this value is absent". None means unmeasurable / the fetch failed ; [] , 0 , 0.0 and "" are legitimate healthy results that a truthiness check silently folds in with it. # BAD — fires on a resource that genuinely has no referrers, or a real count of 0. if not referrers: warn_incomplete(name) # GOOD — only the sentinel means "we don't know". if referrers is None : warn_incomplete(name) A useful tell that you got this wrong: existing fixtures that pass empty collections start emitting the new warning. If adding a guard turns unrelated tests red, read those failures as the over-broad mutation reporting itself rather than as fixtures needing an update. Attribute each mutation to exactly one test. When a fix has several parts, revert them one at a time and record which test fails for each. "Deleting only the bool short-circuit fails exactly one test" is a checked statement; "the suite covers this" is not. A part whose removal leaves the suite green is a gap. Pick Fixture Values That Separate the Bug From the Fix A test survives mutation most often because its data cannot tell the two implementations apart. Before writing the assertion, work out what the buggy version would produce from your fixture. If it produces the same number, the fixture is the problem — change it, not the assertion. Order-sensitive readings need a fixture whose ends differ. A summary that prints the latest value from an already-sorted frame reads .iloc[0] when the sort is descending and .iloc[-1] when it is ascending; both are one character apart and only one is right. A single-row or flat fixture makes them agree. # BAD — every row carries the same total, so first and last agree and the # assertion holds against either reading. rows = [{ "day" : "2024-01-01" , "cumulative" : 5 }] # GOOD — strictly increasing across three days: 2, then 5, then 9. Only the # correct reading yields 9. rows = [ { "day" : "2024-01-01" , "cumulative" : 2 }, { "day" : "2024-01-02" , "cumulative" : 5 }, { "day" : "2024-01-03" , "cumulative" : 9 }, ] assert "Final cumulative stars: 9" in captured.messages For per-group derived columns, the two readings must disagree on every row. A gap/delta/rank column computed per group and the same column computed over the globally sorted table often coincide for most rows; interleaving two groups is not enough on its own — the values have to come out different. Two items per group, timed so the within-group spacing and the global spacing diverge, is the smallest fixture that pins it: group dates per-group gaps global-sort gaps A 2024-01-01, 2024-01-11 10.0 5.0, 5.0, 10.0 B 2024-01-06, 2024-01-21 15.0 (one blank) Per group there are two blanks and gaps of 10.0 and 15.0; globally there is one blank and no value matches. Assert the exact numbers and the blank count. A fixture that reproduces the bug's magic constant proves nothing. When the defect is a redundant fixed wait of 60 seconds and the legitimate code path also falls back to 60 seconds when a header is missing, a fixture that omits the header passes on both. Supply a value that makes the legitimate result distinct — a reset header five seconds out — so the extra wait is visible. Record the sequence, not the fact. A no-op stub answers "did it sleep?" and nothing else. Collect the arguments instead and assert the whole call sequence; that is what catches one extra call in the middle. # BAD — passes whether the code sleeps once or twice. monkeypatch. setattr ( "mypkg.client.time.sleep" , lambda _s: None ) # GOOD — the sequence is the assertion. sleeps: list [ float ] = [] monkeypatch. setattr ( "mypkg.client.time.sleep" , sleeps.append) ... assert sleeps == [reset_wait, 0.1 ] # reset wait, then the post-success pause Changing Behavior an Existing Test Already Asserts Before you change any output, read what the suite already claims about it. Three situations look identical from a green run: Shape assertions only — isinstance(x, float) , >= 0 , "minutes" in result . The behavior is unspecified : the test passed before your fix and will pass after it, and would have passed on almost any output. Treat this as no coverage, not as coverage you must preserve. The correct value is pinned. Your "fix" is the regression. Stop. The buggy value is pinned. A confident exact-match assertion on wrong output makes the defect read as tested-and-intended.