Weather Model Data Fetching

Weather Model Data Fetching

Overview

Fetch numerical weather prediction data without treating a multi-gigabyte GRIB2 file as one indivisible download. Prefer an existing project adapter or Herbie; use direct object-store byte ranges only when the supported path cannot express the request.

This skill covers transport, inventory selection, caching, and verification. It does not interpret the forecast or decide whether a model is meteorologically appropriate.

When to Use This Skill

• A task needs GFS, GEFS, HRRR, RAP, NAM, IFS, or similar model output.
• Data lives in a public AWS S3 bucket, NOMADS, or a public cloud mirror.
• The input is GRIB2 and only selected variables or levels are needed.
• A point, sounding, time series, map, or batch job needs a reliable fetch path.
• A download is missing, partial, unexpectedly large, slow, or hard to resume.

Do not activate this skill for ordinary weather-forecast questions that do not require model files.

Define the Request First

Resolve these values before downloading:

• model and product;
• initialization cycle in UTC;
• forecast hour and therefore valid time (valid = initialization + lead);
• ensemble member when applicable;
• variables, vertical levels, and surface fields;
• point, region, or full-grid output;
• cache location and when the downloaded data may be deleted.

Confirm that the cycle is complete, the forecast hour exists for that cycle, and the requested location is inside the model domain. A recent 404 often means the cycle is not published yet; step back to a completed cycle instead of retrying indefinitely.

Choose the Smallest Retrieval Route

1. Reuse the project's existing fetch/cache abstraction when it already handles the model.
2. Use Herbie for a supported GRIB2 model. It discovers AWS, NOMADS, Google, Azure, and other configured sources and understands their key layouts.
3. Use a provider-native point or Zarr endpoint when the task needs a tiny spatial slice from many times or members.
4. Use direct S3 or HTTPS object access when the key is known and no suitable adapter exists.

Do not recursively list a large public bucket to discover one run. Build the documented prefix for the model, cycle, product, forecast hour, and member, then probe that exact object and its inventory.

Use an explicit provider priority and record the provider that succeeded. A fallback must refer to the same model run, product, member, and forecast hour; never silently substitute a different forecast.

Subset GRIB2 by Inventory

GRIB2 files contain consecutive messages. A companion inventory such as .idx, .grib2.idx, or .grb2.inv records each message's starting byte.

1. Fetch the small inventory first.
2. Inspect its actual rows before writing a regex.
3. Select exact variables, levels, and forecast-step records.
4. Set each selected message's end byte to one less than the next message's start; request the final selected message through EOF when no end is known.
5. Coalesce adjacent selected messages into one range.
6. Issue one Range: bytes=START-END request per range. S3 does not support multiple ranges in one GetObject request.
7. Require 206 Partial Content and a matching Content-Range. If a server answers 200, do not append the whole object as though it were a fragment.
8. Pin the object's length and identity (ETag and/or Last-Modified) while downloading. Discard fragments if the object changes.
9. Assemble into a temporary file, verify it with a GRIB decoder, then rename atomically into the cache.

A GRIB message contains one field over its grid. Message-range subsetting saves variables and levels, not geography. A point request still downloads the full grid for every selected message unless the provider offers a point, regional, Zarr, or other chunked endpoint.

Herbie Example

Use the current search argument; searchString is deprecated. Start from the inventory, fail on an empty match, and keep the download directory explicit.

from pathlib import Path

from herbie import Herbie

PRESSURE_FIELDS = (
    r":(?:HGT|TMP|RH|SPFH|UGRD|VGRD):\d+(?:\.\d+)? mb:"
)


def fetch_hrrr_pressure_run(initialization, forecast_hour, cache_dir):
    cache_dir = Path(cache_dir)
    h = Herbie(
        initialization,
        model="hrrr",
        product="prs",
        fxx=forecast_hour,
        priority=["aws", "nomads", "google", "azure"],
        save_dir=cache_dir,
        verbose=False,
    )

    selected = h.inventory(PRESSURE_FIELDS)
    if selected.empty:
        raise RuntimeError("inventory matched no pressure-level fields")

    downloaded = h.download(PRESSURE_FIELDS, errors="raise")
    path = Path(downloaded) if downloaded is not None else None
    if path is None or not path.is_file() or path.stat().st_size == 0:
        raise RuntimeError("GRIB2 subset was not materialized")

    return path, {
        "model": h.model,
        "product": h.product,
        "initialization": h.date.isoformat(),
        "forecast_hour": h.fxx,
        "valid_time": h.valid_date.isoformat(),
        "provider": h.grib_source,
        "remote_object": str(h.grib),
        "messages": len(selected),
    }

For xarray output, call h.xarray(search, ...) and handle either one xarray.Dataset or a list of incompatible GRIB hypercubes. Merge only groups whose coordinates and dimensions are compatible, and close every dataset when finished.

Public AWS S3 Diagnostics

NOAA Open Data buckets allow unsigned reads. --no-sign-request prevents the AWS CLI from loading credentials; it does not disable TLS verification.

aws s3 ls --no-sign-request s3://noaa-hrrr-bdp-pds/hrrr.YYYYMMDD/conus/

aws s3api get-object --no-sign-request \
  --bucket noaa-hrrr-bdp-pds \
  --key "hrrr.YYYYMMDD/conus/hrrr.tHHz.wrfprsfFF.grib2" \
  --range "bytes=START-END" fragment.grib2

Use these commands to inspect a documented public object or reproduce one known range. For normal multi-message assembly, reuse Herbie or the project's tested downloader instead of scripting binary concatenation in shell.

Complete Sounding Contract

A pressure-level file alone may not contain a usable surface row. Before building a vertical profile, require:

• all published isobaric levels for geopotential height, temperature, a moisture variable (dew point, relative humidity, or specific humidity), and U/V wind;
• surface pressure and terrain or surface height;
• 2 m temperature and moisture;
• 10 m U/V wind.

Some providers split pressure and surface fields into separate products. Fetch and join the companion product from the same run, or reject the request with a list of missing fields. Do not fabricate a ground row or silently reduce the profile to a short mandatory-level list.

After decoding, sort pressure monotonically, remove duplicate levels, normalize units and longitude conventions, and run the consuming project's profile QC.