Verifying External Behavior

Verifying External Behavior

Most integration bugs are not coding errors. They are a belief about someone else's system — a status code, a parameter name, a response shape, a build backend's scoping rule — that was never checked and turned out to be wrong. The code is written correctly against a contract that does not exist.

The fix is not more care. It is a probe: a throwaway command that asks the real system the exact question, before the code is written. Probes cost seconds. The bugs they prevent are silent, ship green, and are found months later.

Probe the exact call you are about to write

Not a similar call, not the documented example — the same endpoint form, the same library version, the same argument spelling.

# A library's real defaults / return shapes, with no venv to create or clean up
uv run --no-project --with somelib python -c "import somelib; print(somelib.Thing())"

# The exact URL, including the collection-vs-item distinction
curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/v1/things
curl -s -o /dev/null -w '%{http_code}\n' https://api.example.com/v1/things/1

# What a build backend actually put in the artifact
uv build && unzip -l dist/*.whl

# A real service instead of a fake, for one minute
docker run --rm -p 6390:6379 redis:7-alpine

Two rules make probes worth the minute they cost:

• Probe before you design around the answer. A probe that confirms a parameter tweak "should" fix a discrepancy is worth more than the tweak.
• Paste the probe command and its output into the PR, comment, or test. A finding with no reproduction decays into folklore, and the next person re-derives it — or, worse, trusts it after it has gone stale.

Permissive clients don't reject wrong arguments — they ignore them

This is the highest-severity class, because the failure returns plausible data. Many HTTP client wrappers forward keyword arguments verbatim into the query string without validating them against their own documented parameter list. The remote service then drops the unknown parameter and applies its default — often "the authenticated user". A misspelled selector (owner= for owner_screen_name=, id_= for id=) does not raise; it silently returns someone else's records.

Never infer "the library would have rejected that" from the library's declared parameter list. Read the bytes that leave the process:

# Probe: intercept the transport and print the outbound request, then stop.
import requests

sent = {}
def spy(self, method, url, **kw):
    sent["url"], sent["params"] = url, kw.get("params")
    raise RuntimeError("probe: request intercepted")

requests.Session.request = spy
try:
    client.favorites(id_=12345)          # the call you were about to ship
except RuntimeError:
    pass
print(sent)      # {'url': '.../favorites/list.json', 'params': {'id_': 12345}}

If the parameter you passed is not in params under the name the service documents, the call is wrong no matter how healthy the response looks.

Two corollaries:

• Pass every selector by keyword. Clients that bind positional arguments in a per-endpoint order will happily accept get_thing(owner, slug) and send slug as owner_id, while a neighbouring method with the same-looking signature is correct by luck.
• Check that the parameter exists at all. Some endpoints have no equivalent of the selector you want. "Forward it under the right name" is not a fix when the right name does not exist — the feature has to be built differently.

Per-endpoint docs do not generalize across sibling endpoints

A status code documented for the single-item form frequently does not apply to the collection form of the same resource. Verified live against a large public REST API: GET /repos/{repo}/issues on a repository with issues disabled returns 200 with an empty array, while GET /repos/{repo}/issues/1 returns 410 Gone. Code written to "tolerate 410 when issues are disabled" therefore has a branch that never fires, and the real path — an empty 200 — falls through to whatever the generic handler does.

Probe the exact URL before writing a tolerated-status branch. If you keep a defensive branch for a status you could not reproduce, label it as defensive and name the path you did observe, so the next reader does not mistake it for verified behaviour.

# Observed: issues-disabled repos return 200 with []. The 410 branch is
# defensive — the item endpoint documents it, the list endpoint never sent it.
if resp.status_code == 410:
    return []

The response shape is part of the contract

List endpoints commonly embed a summary object — a handful of identity fields — rather than the full entity. A "we already have the data" fast path written against the full entity is then always false:

# This check is intended to skip a per-item fetch. Against summary objects that
# carry only {login, id, avatar_url}, it is False for every item — so the
# "fallback" enrichment fetch is the common path, and the loop is N+1.
if all(k in user for k in ("name", "company", "location", "followers")):
    return user
return fetch_user(user["login"])       # runs every time

Before optimizing around a response, print one real element and compare its keys to what your code reads. The same probe settles range and boundary assumptions that otherwise get "handled" defensively forever: if a per-year query provably returns exactly that year's days, the dedup pass guarding against adjacent-year leakage is dead code, and saying so in the PR is more valuable than the code.