Dongle Crash Analysis

Dongle crash-report analysis

The dongle firmware (phoneblock-dongle/firmware/, an ESP32-PICO-D4 / IDF v5.3 app) writes an ESP core dump to its coredump flash partition on a panic, and on the next boot crashreport_upload_async() (main/crashreport.c) POSTs the raw image to ${BASE}/api/dongle/coredump with the firmware version as a query param. The server stores them per-version, e.g. phoneblock/tmp/crash-reports/<version>/<device-uuid>-<YYYYMMDD>-<HHMMSS>.coredump.

This skill turns one of those files into a symbolized backtrace and a root-cause read. Do the whole flow — a decode against the wrong ELF is worse than useless (see step 2).

0. Find the files

find / -type d -name '<version>' -path '*crash-reports*' 2>/dev/null   # e.g. .../crash-reports/1.5.0
ls -la <dir>

Those directories are sometimes owned by root with mode 0750. If you get "Permission denied" and sudo needs a password, ask the user to run sudo chown -R $USER:$USER <dir> (or chmod -R a+rX) themselves — don't burn turns retrying.

1. Strip the 24-byte flash header

The .coredump is not a bare ELF. It carries a 24-byte ESP flash core-dump header (data_len u32 = full file size, version u32 ≈ 0x0102, …). The real ELF magic \x7fELF begins at offset 24. Confirm, then strip:

f=<file>.coredump
grep -aboF $'\x7fELF' "$f" | head -1          # must print "24:..."
tail -c +25 "$f" > "$SCRATCH/core.elf"        # SCRATCH = your scratchpad dir
xxd -l 8 "$SCRATCH/core.elf"                   # sanity: starts with 7f45 4c46

2. Get the EXACT-version unstripped ELF from the CDN

Critical. Symbolization is only trustworthy against the same build that produced the dump. A local checkout that is even a few commits past the dongle-vX.Y.Z tag mis-resolves every address — IDF-internal frames come out as nonsense (esp_startup_start_app_other_cores, wifi_transmit, …) that look plausible and will send you down a rabbit hole.

release.sh uploads the unstripped ELF to the CDN specifically for this (look for the phoneblock_dongle.elf copy + the espcoredump.py comment in firmware/scripts/release.sh). Pull it by version:

curl -fsSL -o "$SCRATCH/dongle-<ver>.elf" \
  https://cdn.phoneblock.net/dongle/firmware/<ver>/phoneblock_dongle.elf
file "$SCRATCH/dongle-<ver>.elf"                       # "with debug_info, not stripped"
strings "$SCRATCH/dongle-<ver>.elf" | grep -Eo '<ver>' | head   # confirm version baked in

The CDN layout (from release.sh): base https://cdn.phoneblock.net/dongle/firmware/, then <version>/ holds phoneblock_dongle.{bin,elf}, bootloader.bin, partition-table.bin, ota_data_initial.bin, manifest.json; the stable/ and beta/ dirs hold the channel manifest.json that flips per release.

3. Decode with esp-coredump + the xtensa GDB

esp-coredump needs the Xtensa GDB, not system gdb. --chip is a global flag, before the subcommand.

source ~/.espressif/python_env/idf5.3_py3.10_env/bin/activate
GDB=~/.espressif/tools/xtensa-esp-elf-gdb/*/xtensa-esp-elf-gdb/bin/xtensa-esp32-elf-gdb
esp-coredump --chip esp32 info_corefile --gdb $GDB \
    --core "$SCRATCH/core.elf" --core-format elf \
    "$SCRATCH/dongle-<ver>.elf" > "$SCRATCH/decoded-<ver>.txt" 2>/dev/null

Match check: in a correct decode the IDF frames resolve to the release build host path /home/bhu/... and the app frames to /home/bhu/git/phoneblock/.... If instead you see local /home/haui/... paths and absurd IDF frames, the ELF is the wrong build — go back to step 2. dbg_corefile (instead of info_corefile) drops you into an interactive gdb on the dump for deeper poking.

4. Read the dump

Key fields at the top:

• Crashed task name + whether it was in interrupt context.
• exccause: 0x1c LoadProhibitedCause / 0x1d StoreProhibited = deref of an invalid pointer; 0x02 InstrFetchProhibited = jumped through a bad function pointer; 0x09 LoadStoreAlignment; etc.
• excvaddr: the bad address that was accessed. A value outside DRAM (0x3ffb0000–0x40000000) or IRAM is a wild pointer.
• a0–a15: check for ASCII text in supposedly-pointer registers (bytes in the 0x20–0x7e range) — a fingerprint of string/HTTP data written over a struct or heap metadata.
• Stack usage table: USED/FREE per task. A task at/near its limit ⇒ stack overflow (a different failure mode than heap corruption). Healthy free stack everywhere rules that out.