Podium Webhook Reliability

Podium Webhook Reliability

Overview

Receive Podium webhooks in production without forged events, double-charged AI side-effects, lost notifications, or out-of-order conversation events. This is not an introductory webhook walkthrough — it is the receiver code your integration runs when Podium retries a 5xx response six times over 24 hours, when a leaked secret lets an attacker POST forged events, when a batch delivery arrives with conversation.deleted ahead of conversation.created, and when on-call needs to drain and replay 800 failed events without re-firing the ones that already succeeded.

The six production failures this skill prevents:

1. Missing signature verification — a webhook endpoint that accepts any POST will accept forged events. An attacker who learns the URL can create phantom contacts, fire phantom review requests, or impersonate a real customer in a webchat. HMAC-SHA256 over the raw request body is non-optional and must run before any handler logic.
2. Replay attacks against a stateless handler — a valid signed event POSTed twice (or 1000 times) re-runs every side effect each time. Signature validity alone is not enough — the receiver must reject events whose timestamp falls outside a 5-minute window AND whose nonce has already been seen.
3. Duplicate event processing from Podium retries — Podium retries webhook delivery on 5xx for up to 24 hours. Without an idempotency cache, every retry re-runs the handler (writes the contact again, fires the review request again, double-charges an AI call). SET NX EX 86400 on the event_id is the cheapest fix that exists.
4. Lost events without a dead-letter queue — if a handler raises and Podium retries six times and gives up, the event is gone. On-call has nothing to replay. Every handler exception must persist the raw signed payload to a DLQ before the response returns 5xx, so the event is recoverable independent of Podium's retry clock.
5. Batch event reordering — Podium can deliver multiple events in one POST and ordering across deliveries is not guaranteed. A naive handler processes conversation.deleted before conversation.created and the system observes a delete on a contact that does not exist. Within a batch, sort by occurred_at before dispatch; across batches, gate causally-dependent handlers on the precondition existing.
6. Timing-attack vulnerability on signature compare — received_sig == computed_sig with == short-circuits on the first byte mismatch. An attacker measures response latency to recover the signature byte-by-byte over a few thousand probes. Always use hmac.compare_digest, which is constant-time over the longer of the two inputs.

Prerequisites

• Python 3.10+ with fastapi, uvicorn, httpx, and redis (in-memory fallback for dev is provided)
• Podium account with an OAuth app authorized for webhook delivery: Settings → Developer → Apps → Webhooks
• The webhook signing secret from the app's Webhooks tab (saved to a secret store — never committed)
• A receiver URL reachable from Podium (publicly resolvable HTTPS endpoint with valid cert)
• Redis 6+ for production dedup + DLQ; an in-memory dict + SQLite file fallback exists for dev
• A podium-auth instance if your handler needs to call back into the Podium API after processing

Instructions

Build in this order. Each section neutralizes one production failure mode.

1. HMAC-SHA256 signature verification on the raw body (neutralizes forgery)

Verify the signature against the raw, unparsed request body. Any framework middleware that JSON-decodes-and-re-encodes before signature check will fail because whitespace and key ordering change. Read the body once, verify, then parse:

import hmac, hashlib
from fastapi import FastAPI, Request, HTTPException, Header

app = FastAPI()
SIGNING_SECRET = os.environ["PODIUM_WEBHOOK_SECRET"].encode("utf-8")

@app.post("/webhooks/podium")
async def receive(request: Request, x_podium_signature: str = Header(None)):
    raw = await request.body()              # bytes — DO NOT decode/re-encode
    if not x_podium_signature:
        raise HTTPException(401, "missing X-Podium-Signature")
    if not verify_signature(raw, x_podium_signature):
        raise HTTPException(401, "signature mismatch")
    # ... continue with replay/dedup/dispatch

def verify_signature(body: bytes, header_value: str) -> bool:
    # Podium signature header format: "t=<unix_ts>,v1=<hex_hmac>"
    # Adapt to current spec — verify against the Podium developer docs at integration time.
    parts = dict(p.split("=", 1) for p in header_value.split(",") if "=" in p)
    ts, sig = parts.get("t"), parts.get("v1")
    if not ts or not sig:
        return False
    signed_payload = f"{ts}.".encode("utf-8") + body
    expected = hmac.new(SIGNING_SECRET, signed_payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)    # constant-time, byte-by-byte safe

The t= timestamp is what makes the next mitigation possible. A signature alone with no timestamp is replayable forever.

2. Replay-attack window (neutralizes timestamp replay)

Reject any event whose signed timestamp is more than 5 minutes from now (in either direction — clock skew goes both ways). This bounds the replay window an attacker has even if they capture a valid signed event off the wire:

import time

REPLAY_WINDOW_SECONDS = 300        # 5 minutes; tune to your clock-skew tolerance

def within_replay_window(ts_str: str) -> bool:
    try:
        ts = int(ts_str)
    except (TypeError, ValueError):
        return False
    return abs(time.time() - ts) <= REPLAY_WINDOW_SECONDS

Wire within_replay_window(parts["t"]) immediately after signature verification. A failed window check is a 401 — do not return 200, do not enqueue, do not log the body (the attacker is probing).

3. Idempotent dedup with SET NX EX 86400 (neutralizes duplicate processing)

Every Podium webhook carries an event_id (or equivalent unique identifier — verify against the current schema). Reject any event whose event_id is already in the dedup cache. Use Redis SET key value NX EX 86400 so the check and the claim are atomic; 86400 seconds matches Podium's 24-hour retry ceiling:

import redis.asyncio as redis

REDIS = redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379/0"))

async def claim_event(event_id: str) -> bool:
    # Returns True if this process is the first to see this event_id.
    # Returns False if the event_id is already in the cache (duplicate).
    return await REDIS.set(f"podium:evt:{event_id}", "1", nx=True, ex=86400)

In the handler:

event = json.loads(raw)
event_id = event["id"]
if not await claim_event(event_id):
    return {"status": "duplicate", "event_id": event_id}     # 200 — Podium stops retrying

Returning 200 on duplicate is correct — Podium has correctly delivered, the receiver has correctly identified it as already processed. The handler is idempotent by construction.

For dev / smoke environments without Redis, fall back to an in-memory set() with a periodic eviction loop. Documented in references/implementation.md.

4. Dead-letter queue before responding 5xx (neutralizes silent event loss)

Wrap every handler invocation in a try/except. On any exception, persist the raw signed payload plus the timestamp plus the signature to the DLQ before letting the exception bubble. The DLQ entry is the recovery anchor — dlq_replay.py can re-POST it to the handler later:

async def safe_dispatch(event: dict, raw: bytes, sig_header: str):
    try:
        await dispatch(event)
    except Exception as e:
        await dlq_persist({
            "event_id": event.get("id"),
            "event_type": event.get("type"),
            "raw_body": raw.decode("utf-8", errors="replace"),
            "signature_header": sig_header,
            "occurred_at": event.get("occurred_at"),
            "received_at": time.time(),
            "exception": f"{type(e).__name__}: {e}",
        })
        raise          # let FastAPI return 5xx; Podium will retry

DLQ backend options (in priority order):

The DLQ is durable independent of the Redis dedup cache. If Redis dies, dedup is degraded but events are still recoverable.