GreptimeDB Fuzz CI Failure Investigation

GreptimeDB Fuzz CI Failure Investigation

Investigate failed fuzz-test jobs for GreptimeTeam/greptimedb from a local checkout. The purpose is to download the failed job's CI output and fuzz artifacts, then explain the likely cause by correlating the evidence with GreptimeDB source.

Always pass --repo GreptimeTeam/greptimedb to gh; local remotes may point to forks. Keep the workflow read-only: do not rerun jobs, cancel workflows, push, comment on PRs, or delete artifacts unless the user explicitly asks.

Scope

Use this skill for fuzz CI only. In .github/workflows/integration.yml, fuzz failures are the CI jobs that use .github/actions/fuzz-test:

• fuzztest — standalone fuzz targets.
• unstable-fuzztest — unstable standalone fuzz target.
• distributed-fuzztest — distributed cluster fuzz targets.
• distributed-fuzztest-with-chaos — distributed fuzz targets with Chaos Mesh.

Each matrix job runs a semantic group of targets on one prepared environment. The group defaults to fail-fast: after the first target failure it captures that target's diagnostics and marks the remaining targets as skipped. The workflow strategy still uses fail-fast: false, so failures do not cancel other groups.

The reusable action .github/actions/fuzz-test/action.yaml delegates each target to .github/scripts/run-fuzz-targets.sh. CI passes FUZZ_BIN_DIR, so the script runs the prebuilt target executable:

<prebuilt-fuzz-binary> -max_total_time=<seconds> \
  -artifact_prefix=<target-dir>/libfuzzer/

Without FUZZ_BIN_DIR, such as during local reproduction, the script falls back to cargo fuzz run <target> --fuzz-dir tests-fuzz -D -s none with the same libFuzzer arguments.

Failed jobs upload one group artifact. Its stable name identifies the job kind, mode, and group, for example:

fuzz-distributed-remote-wal-database-and-regular-table

The artifact contains manifest.json, summary.md, and one directory per target under targets/<target>/. A failed distributed target can include fuzz.log, libFuzzer reproducers, CSV/SQL traces, Kind logs, monitor dumps, and Kubernetes state. Setup failures use targets/setup/.

Inputs

The normal user input is a CI target link: a GitHub Actions job URL for one failed matrix target. Treat that URL as the primary source of truth. Example:

https://github.com/GreptimeTeam/greptimedb/actions/runs/29000666156/job/86062894741

Also accept these fallback inputs:

• A workflow run URL, e.g. https://github.com/GreptimeTeam/greptimedb/actions/runs/29000666156.
• A run id plus a fuzz target, mode name, artifact name, PR, branch, or commit SHA.

Parse ids from URLs:

REPO=GreptimeTeam/greptimedb
RUN_ID=29000666156
JOB_ID=86062894741

When the input is a job URL, do not ask for the target name first. Use the job id to fetch metadata, then derive the group and mode from the job name. A distributed fuzz job name usually contains the CI group tuple:

Fuzz Test (Distributed, <mode>, <group>)
Fuzz Test with Chaos (Distributed, <mode>, <group>)

If only a run URL is provided, list failed jobs and choose the fuzz job matching the group/mode. Ask only when multiple fuzz jobs are plausible and there is no group, mode, artifact, or job-url clue.

Prerequisites

• gh must be installed and authenticated (gh auth status). Actions logs and artifacts commonly require authentication even for a public repo.
• jq, unzip, and standard shell tools are useful for reducing logs.
• Use the local GreptimeDB checkout for source analysis. If the run's head SHA is not the current checkout, inspect the code at the run's SHA before making claims.
• Use a scratch directory outside the source tree, such as /tmp/greptimedb-fuzz-ci.

1. Capture run and job metadata

Create a scratch directory:

mkdir -p /tmp/greptimedb-fuzz-ci/$RUN_ID
cd /tmp/greptimedb-fuzz-ci/$RUN_ID

Fetch run metadata:

gh run view "$RUN_ID" --repo "$REPO" \
  --json databaseId,name,displayTitle,event,headBranch,headSha,status,conclusion,createdAt,updatedAt,url,workflowName \
  > run.json

Fetch all jobs through the REST API:

gh api "repos/$REPO/actions/runs/$RUN_ID/jobs?per_page=100" \
  --paginate --jq '.jobs[]' \
  | jq -s '{jobs: .}' > jobs.json

List failed fuzz-like jobs:

jq -r '
  .jobs[]
  | select(.conclusion != null and .conclusion != "success")
  | select(.name | test("Fuzz Test|fuzz"; "i"))
  | [.id, .name, .status, .conclusion, .html_url] | @tsv
' jobs.json

For a known JOB_ID, capture job name and step outcomes:

jq -r --arg id "$JOB_ID" '
  .jobs[] | select((.id | tostring) == $id) |
  "job=\(.name) conclusion=\(.conclusion) url=\(.html_url)",
  (.steps[]? | [.number, .name, .status, .conclusion, .started_at, .completed_at] | @tsv)
' jobs.json > job-summary.txt

Record the fuzz job kind, matrix mode, group, failed step, event type, branch, and head SHA. Do not assume main for PR or merge-queue runs. The failed target is authoritative in the downloaded artifact's manifest.json; do not guess it from the group name.

2. Download CI output for the failed job

Download the selected job's log; this is the authoritative target CI output for the cargo fuzz run command:

gh run view "$RUN_ID" --repo "$REPO" --job "$JOB_ID" --log > job-$JOB_ID.log

Download the full workflow log only when setup/build context matters:

gh run view "$RUN_ID" --repo "$REPO" --log > run-$RUN_ID.log

Fallback if gh run view --log is incomplete. The job logs endpoint returns a plain text stream, while the full run logs endpoint returns a zip archive:

gh api "repos/$REPO/actions/jobs/$JOB_ID/logs" > job-$JOB_ID.log
gh api "repos/$REPO/actions/runs/$RUN_ID/logs" > run-$RUN_ID-logs.zip
unzip -oq run-$RUN_ID-logs.zip -d run-logs

In the job log, inspect the Run Fuzz Test step first. Capture:

• exact target name;
• max_total_time;
• panic/assertion/backtrace or libFuzzer crash output;
• generated crash/reproducer path if printed;
• timeout or resource symptoms.

3. Find and download fuzz artifacts

List artifacts:

gh api "repos/$REPO/actions/runs/$RUN_ID/artifacts?per_page=100" \
  --paginate --jq '.artifacts[]' \
  | jq -s '{artifacts: .}' > artifacts.json
jq -r '.artifacts[] | [.id, .name, .expired, .size_in_bytes] | @tsv' artifacts.json

Match the one group artifact by job kind, mode slug, and group slug. If the display name does not make the slug obvious, list the run artifacts and choose the unique fuzz-* artifact whose suffix matches the group. Do not search for separate Kind, monitor, or CSV artifacts; these are target subdirectories in the group artifact.

Download exact artifacts when possible:

ARTIFACT='fuzz-distributed-remote-wal-database-and-regular-table'
gh run download "$RUN_ID" --repo "$REPO" --name "$ARTIFACT" --dir artifacts/"$ARTIFACT"

Fallback by artifact id:

ARTIFACT_ID=$(jq -r --arg name "$ARTIFACT" '.artifacts[] | select(.name == $name and (.expired | not)) | .id' artifacts.json | head -n 1)
if [ -z "$ARTIFACT_ID" ]; then
  echo "Artifact not found or expired: $ARTIFACT"
  exit 1
fi
gh api "repos/$REPO/actions/artifacts/$ARTIFACT_ID/zip" > artifact-$ARTIFACT_ID.zip
mkdir -p artifacts/"$ARTIFACT"
unzip -oq artifact-$ARTIFACT_ID.zip -d artifacts/"$ARTIFACT"

After download, read the manifest before the bulk logs:

jq -r '.[] | [.target, .status, .exit_code, .after_prior_failure, .artifact_collection] | @tsv' \
  artifacts/"$ARTIFACT"/manifest.json

Select the first failure entry as the primary target. Entries with skipped_after_failure did not run; entries with after_prior_failure=true ran against a potentially contaminated shared environment. Note expired or missing artifacts explicitly; retention is currently short (retention-days: 3).

4. Understand artifact contents

Evidence for target <target> is rooted at artifacts/$ARTIFACT/targets/<target>/. result.json records timestamps, status, exit code, fuzz budget, prior-failure provenance, and artifact collection status. fuzz.log is the complete libFuzzer output from the prebuilt target executable in CI; local fallback runs it through cargo fuzz run.

Kind logs under targets/<target>/kind/ come from an immediate failure snapshot and usually include Kubernetes state plus container logs. Prioritize:

• GreptimeDB component logs: frontend, datanode, metasrv, flownode, monitor.
• Kubernetes pod descriptions/events: restarts, OOMKilled, scheduling, image pull, volume, DNS, and network issues.
• External service logs: etcd, Kafka, Minio, Chaos Mesh, Kafka WAL helper.

Monitor dumps under targets/<target>/monitor/ are collected by .github/scripts/collect-fuzz-monitor-artifacts.sh. They may include:

• state.log, sql.log, copy.log, port-forward.log;
• *.show_create_table.sql for _gt_logs and OpenTelemetry trace tables;
• parquet dumps for _gt_logs, opentelemetry_traces, opentelemetry_traces_operations, and opentelemetry_traces_services.

CSV and SQL traces are under targets/<target>/csv/. LibFuzzer crash or timeout reproducers are under targets/<target>/libfuzzer/. Kubernetes descriptions and events are under targets/<target>/kubernetes/. Standalone service logs are under targets/<target>/service/ when available.

5. Build a layered diagnosis model

Before drilling into a specific error string, build a small macro-level model of the failure. This prevents overfitting on the loudest symptom, such as libFuzzer: deadly signal, Internal error, or a cleanup warning.

5.1 Create a failure timeline

Record the key timestamps in order:

• job start and setup steps;
• external dependencies ready, such as etcd, Kafka, Minio, and Chaos Mesh;
• GreptimeDB cluster ready;
• fuzz binary starts;
• first generated operation or SQL visible in the target log;
• first suspicious component log line;
• user-visible fuzz failure or panic;
• artifact collection and cleanup.

Use the timeline to classify the failure phase: setup, workload execution, storage/WAL/write path, query path, cleanup, or runner infrastructure.