Skip to content
Mitchel Carson

In progress · Case study

HYDRA

Watershed forecasting research: reforecast generation and deep-learning post-processing for NOAA NextGen streamflow forecasts

In progress

KeywordsPyTorch · LSTM · Transformer · Mamba (state-space) · NextGen · Google Cloud · xarray

Research DetailsGitHub repository

Summary

Problem and impact

Problem

Experiencing Hurricane Helene in Boone is what motivated HYDRA. Short-lead streamflow forecasts from NOAA’s NextGen framework carry errors that a learned post-processor may be able to reduce, and HYDRA asks whether a model trained on reforecasts generated with the same framework can improve forecasts at 1–18 hour lead times without leaking future information.

Impact

The work has two parts: NextGen reforecast generation software with Google Cloud workflows that prepare the data, and a post-processing model evaluated by site and lead time. Results are pending; a results manuscript for Water Resources Research is in preparation, and a software paper for Environmental Modelling & Software is planned.

No performance number is reported on this site until the analysis is complete; results will be published with the manuscripts.

Case study

Approach, architecture, evaluation, delivery

01Technical Approach

  1. Built NextGen reforecast generation software and Google Cloud workflows to acquire, validate, and align weather, streamflow, and forecast data with traceable provenance.
  2. Designing forecast post-processing for 1–18 hour lead times, comparing LSTM, Transformer, and Mamba-style models on identical inputs and splits.
  3. Evaluating with leakage-aware temporal splits and hydrologic metrics (RMSE, NSE, KGE), reporting performance by site and lead time.
  4. Preserving initialization, lead, valid-time, version, and source metadata so every training example traces back to the forecast that produced it.
  5. Preparing a results manuscript for Water Resources Research; a software paper for Environmental Modelling & Software is planned.

02Architecture Decisions

  1. Reforecast generation software drives NOAA’s NextGen framework to produce retrospective forecasts with consistent metadata.
  2. A post-processing model consumes reforecasts and observations to produce improved forecasts at 1–18 hour lead times.
  3. Config-driven training and evaluation with strict train/validation/test time boundaries.

03Reliability and Evaluation

  1. Leakage-aware splitting by time and lead time to match operational inference constraints.
  2. Reproducible runs through fixed seeds, immutable data artifacts, and tracked configuration snapshots.
  3. Performance tracked by site, lead time, and hydrologic metric instead of a single aggregate score.

04Delivery and Operations

  1. Packaged data acquisition and reforecast generation as reusable software for a planned software paper; the data tooling is public.
  2. Produced versioned artifacts, diagnostics, and technical documentation for scientific review.
  3. Maintained a research website while reserving claims for the completed analysis.

05Results

  1. Reforecast generation software and Google Cloud data workflows are in place; the NextGen_Hydra repository is public.
  2. Model comparison and lead-time evaluation are in progress; results will be posted when the analysis is complete.
  3. A results manuscript is in preparation, and a software paper is planned.

06What I Learned

  1. Post-processing depends as much on how reforecasts are generated and aligned as on model architecture.
  2. Reproducible data and configuration lineage make changing scientific results auditable.
  3. Research claims should evolve with the evidence rather than outrun the analysis.
Back to projectsResearch DetailsGitHub repository