Five Notebooks to represent water budgets from compartment models (with EPNs inside)

Since the paper on GEOtop (Rigon et al., 2006), which claimed that a hydrological model should look at the water and energy budgets, rather than at single fluxes, in order to gain real knowledge of the system, I have been struggling to build models that respond to this effort. Estimating a discharge, an evaporation, a snow water equivalent one at a time is not the same thing as closing a budget: it is the budget — the simultaneous accounting of inputs, outputs and storage variations — that constrains the pieces to be mutually coherent, and it is from that coherence that understanding comes.

Detail of a Giacomo Balla artwork

The effort has grown in various directions since then. On the conceptual and strategic side, it evolved into the vision of Digital eARth Twin Hydrology systems, the DARTHs (Rigon et al., 2022), discussed in several posts on this blog (see the posts on DARTHs). On the software side, it required a complete rebuilding of the informatics of these systems, deployed in the GEOframe system: a component-based infrastructure built on the OMS3 framework (David et al., 2013; Formetta et al., 2014), whose NewAGE branch (Bancheri et al., 2020) is the one we routinely use for operational and research modelling (see the GEOframe posts). On the formal side, it produced the Extended Petri Nets (EPN) formalism (Bancheri, Serafin and Rigon, 2019), which gives a graphical and mathematical grammar for writing budget-based models: places for storages, transitions for fluxes, controllers for the variables that regulate rates without exchanging mass, splitters for the partition of fluxes.

In all of this effort, one persistent issue has been how to represent the water budget in a compact and coherent way (on the energy budget we are still working). Compact, because a budget involves many time series at once — inputs, outputs, several storages — and they must be readable on a single temporal abscissa. Coherent, because the representation should not merely juxtapose the series but make their mutual constraint visible: at every time step, inputs minus outputs must equal the sum of storage variations, and a figure of the budget should let the reader see whether this closure holds, column by column.

One step in this direction is the production of five Jupyter Notebooks, obtained with the help of Claude (Anthropic's AI assistant) and certainly improvable, which you find on OSF. As usual, this is a seed for improvements.

What the notebooks contain

The five parts share a single Python library (hydrobudget.py, accompanied by a test suite) and exchange data through feather files and JSON declarations, so that each notebook can be read — and modified — on its own.

Part 1 — Synthetic data. A minimal three-bucket model (snowpack, root zone, groundwater, with air temperature as forcing) generates the datasets used throughout: a one-year demonstration run, a four-year analysis run with an imposed dry year, and — this is a methodological point I care about — an independent thirty-year baseline from which all climatologies and drought thresholds are computed, so that the reference statistics are never contaminated by the period being analyzed. Everything is saved to feather, and every generated dataset is closure-checked at birth. This is shortcut since I was not having real data. 

Part 2 — Representing the budget. The central figure of the series: inputs drawn as bars hanging from the top of the panel (the hyetograph convention), outputs rising from the bottom, storage variations as signed stacked bars below, optionally the total storage in a third panel — all on one shared temporal abscissa, at any time step from hourly to yearly, with fluxes summed and states sampled at period end under aggregation. The budget check is built into the figure: a line marks in − out at every step, and each column of storage variations must reach it exactly. On the synthetic data the maximum column-wise closure error is of the order of 10⁻¹⁴ mm; on real data, the columns where the bars miss the line become a diagnostic of where measurements or model do not close. The bookkeeping is declared, not hard-coded — one learns, for instance, that snowfall is not a flux input to the soil but an input to the snowpack storage, and the figure enforces the distinction. 

Part 3 — Droughts. Once the budget is an object, droughts become excursions of that object: persistent negative anomalies of storages and accumulated fluxes with respect to their (baseline) climatology, identified by the classical threshold-level method of run theory (Yevjevich, 1967), with pooling, severities and intensities in explicit units. Because each drought type lives in a different variable of the budget, the notebook separates them — meteorological, snow, soil moisture, streamflow, groundwater — and displays their propagation chain on a timeline (in the spirit of Van Loon, 2015). Snow droughts are further classified as dry, warm, warm-and-dry or other, using precipitation ratios and temperature anomalies (Harpold et al., 2017): this is why the temperature is generated and stored in Part 1. There is certainly more to work out, in this part.

Part 4 — Representing the EPNs. The same systems drawn as Extended Petri Nets, following the graphical conventions of Bancheri, Serafin and Rigon (2019): colored circles for places, squares for transitions inheriting the color of the place they exit, framed triangles for controllers with dashed information arcs, diamonds for splitters with their partition fractions, everything laid on an integer grid with mathematical symbols as labels. Two details I find pleasing: arc routing can circumnavigate nodes through waypoints, and when a crossing between arcs is genuinely unavoidable — the notebook distinguishes by computation the avoidable from the necessary ones — it is denoted by a small open circle, the old circuit-diagram convention for "crossing without exchange". The basic idea is that giving as input the EPN representation of the model, the program knows which data to expect and consequently which graphs to produce, see Part 5 below. There are various improvements that can be envisioned also here. 

Part 5 — The EPN as budget analysis. The point of the whole exercise: the EPN declaration is not only a drawing, it is the bookkeeping. From the same JSON that draws the net, the code derives the whole-system budget and one budget per place, verifies the closure node by node, and plots the two-panel budget figure of every place of the net. The same JSON then acts as a data contract for simulation outputs: we propose a format that GEOframe-NewAGE results should obey — one tidy table per hydrological response unit, one column per EPN transition (internal fluxes included, which is an argument for making them first-class outputs of any simulator), one column per place state — so that if simulator and analysis share the same EPN, every node must close to numerical precision, and any residual is either a format violation or a bug. Coherence between the GEOframe outputs and the EPN scheme of the simulator, in other words, becomes something one can check mechanically.

A seed

The notebooks run top to bottom without errors, the figures are checked geometrically before they ship, and the library carries a few dozen tests — but none of this makes them finished. The synthetic model is deliberately minimal; the drought thresholds are heuristic; the energy budget is absent; the connection to real GEOframe runs is, for now, a contract waiting for its first signatures. As usual, this is a seed for improvements: take them, break them, and tell me where.

Files are here: https://osf.io/jt7z8/files/osfstorage

References

• Bancheri, M., Serafin, F., & Rigon, R. (2019). The representation of hydrological dynamical systems using Extended Petri Nets (EPN). Water Resources Research, 55(11), 8895–8921. https://doi.org/10.1029/2019WR025099
• Bancheri, M., Rigon, R., & Manfreda, S. (2020). The GEOframe-NewAge modelling system applied in a data-scarce environment. Water, 12(1), 86. https://doi.org/10.3390/w12010086
• David, O., Ascough II, J. C., Lloyd, W., Green, T. R., Rojas, K. W., Leavesley, G. H., & Ahuja, L. R. (2013). A software engineering perspective on environmental modeling framework design: The Object Modeling System. Environmental Modelling & Software, 39, 201–213. https://doi.org/10.1016/j.envsoft.2012.03.006
• Formetta, G., Antonello, A., Franceschi, S., David, O., & Rigon, R. (2014). Hydrological modelling with components: A GIS-based open-source framework. Environmental Modelling & Software, 55, 190–200. https://doi.org/10.1016/j.envsoft.2014.01.019
• Harpold, A. A., Dettinger, M., & Rajagopal, S. (2017). Defining snow drought and why it matters. Eos, 98. https://doi.org/10.1029/2017EO068775
• Rigon, R., Bertoldi, G., & Over, T. M. (2006). GEOtop: A distributed hydrological model with coupled water and energy budgets. Journal of Hydrometeorology, 7(3), 371–388. https://doi.org/10.1175/JHM497.1
• Rigon, R., Formetta, G., Bancheri, M., Tubini, N., D'Amato, C., David, O., & Massari, C. (2022). HESS Opinions: Participatory Digital eARth Twin Hydrology systems (DARTHs) for everyone. Hydrology and Earth System Sciences, 26, 4773–4800. https://doi.org/10.5194/hess-26-4773-2022
• Van Loon, A. F. (2015). Hydrological drought explained. WIREs Water, 2(4), 359–392. https://doi.org/10.1002/wat2.1085
• Yevjevich, V. (1967). An objective approach to definitions and investigations of continental hydrologic droughts. Hydrology Papers 23, Colorado State University.