Six Notebooks to represent water budgets from compartment models (with EPNs inside)

Since the paper on GEOtop (Rigon et al., 2006), which claimed that a hydrological model should look at the water and energy budgets, rather than at single fluxes, in order to gain real knowledge of the system, I have been struggling to build models that respond to this effort. Estimating a discharge, an evaporation, a snow water equivalent one at a time is not the same thing as closing a budget: it is the budget — the simultaneous accounting of inputs, outputs and storage variations — that constrains the pieces to be mutually coherent, and it is from that coherence that understanding comes.

Detail of a Giacomo Balla artwork

The effort has grown in various directions since then. On the conceptual and strategic side, it evolved into the vision of Digital eARth Twin Hydrology systems, the DARTHs (Rigon et al., 2022), discussed in several posts on this blog (see the posts on DARTHs). On the software side, it required a complete rebuilding of the informatics of these systems, deployed in the GEOframe system: a component-based infrastructure built on the OMS3 framework (David et al., 2013; Formetta et al., 2014), whose NewAGE branch (Bancheri et al., 2020) is the one we routinely use for operational and research modelling (see the GEOframe posts). On the formal side, it produced the Extended Petri Nets (EPN) formalism (Bancheri, Serafin and Rigon, 2019), which gives a graphical and mathematical grammar for writing budget-based models: places for storages, transitions for fluxes, controllers for the variables that regulate rates without exchanging mass, splitters for the partition of fluxes.

In all of this effort, one persistent issue has been how to represent the water budget in a compact and coherent way (on the energy budget we are still working). Compact, because a budget involves many time series at once — inputs, outputs, several storages — and they must be readable on a single temporal abscissa. Coherent, because the representation should not merely juxtapose the series but make their mutual constraint visible: at every time step, inputs minus outputs must equal the sum of storage variations, and a figure of the budget should let the reader see whether this closure holds, column by column.

One step in this direction is the production of six Jupyter Notebooks, obtained with the help of Claude (Anthropic's AI assistant) and certainly improvable, which you find on OSF. As usual, this is a seed for improvements.

What the notebooks contain

The six parts share a single Python library (hydrobudget.py, accompanied by a test suite) — with a second module, epn_registry.py, for the model dictionary of Part 6 — and exchange data through feather files and JSON declarations, so that each notebook can be read — and modified — on its own.

Part 1 — Synthetic data. A minimal three-bucket model (snowpack, root zone, groundwater, with air temperature as forcing) generates the datasets used throughout: a one-year demonstration run, a four-year analysis run with an imposed dry year, and — this is a methodological point I care about — an independent thirty-year baseline from which all climatologies and drought thresholds are computed, so that the reference statistics are never contaminated by the period being analyzed. Everything is saved to feather, and every generated dataset is closure-checked at birth — including, now, a small hillslope-and-riparian system, which is the net whose drawing in Part 4 has a crossing: it could previously be drawn but never verified, for want of a table. This is shortcut since I was not having real data. 

Part 2 — Representing the budget. The central figure of the series: inputs drawn as bars hanging from the top of the panel (the hyetograph convention), outputs rising from the bottom, storage variations as signed stacked bars below, optionally the total storage in a third panel — all on one shared temporal abscissa, at any time step from hourly to yearly, with fluxes summed and states sampled at period end under aggregation. The budget check is built into the figure: a line marks in − out at every step, and each column of storage variations must reach it exactly. On the synthetic data the maximum column-wise closure error is of the order of 10⁻¹⁴ mm; on real data, the columns where the bars miss the line become a diagnostic of where measurements or model do not close. The bookkeeping is declared, not hard-coded — one learns, for instance, that snowfall is not a flux input to the soil but an input to the snowpack storage, and the figure enforces the distinction. What makes the residual informative, rather than merely small, is how it behaves when the control volume is declared wrongly: the notebook now tabulates four candidate volumes — the soil view as drawn, the same view with Δ(snowpack) added, the whole system, and the whole system with Δ(snowpack) removed. The two consistent pairings close to about 10⁻¹³ mm; both inconsistent ones fail by the same 116.32 mm, which is the largest monthly change in snowpack storage. Equal failure at the scale of exactly one term is the signature of a term placed on the wrong side of a boundary, not of an accumulating error.

Part 3 — Droughts. Once the budget is an object, droughts become excursions of that object: persistent negative anomalies of storages and accumulated fluxes with respect to their (baseline) climatology, identified by the classical threshold-level method of run theory (Yevjevich, 1967), with pooling, severities and intensities in explicit units. Because each drought type lives in a different variable of the budget, the notebook separates them — meteorological, snow, soil moisture, streamflow, groundwater — and displays their propagation chain on a timeline (in the spirit of Van Loon, 2015). Snow droughts are further classified as dry, warm, warm-and-dry or other, using precipitation ratios and temperature anomalies (Harpold et al., 2017): this is why the temperature is generated and stored in Part 1. The reference is now computed by day of year rather than by calendar month, and the notebook shows both: the step discontinuities of a monthly reference are visible in the anomaly it produces, and therefore in the spells the threshold method extracts from it — which is a way of seeing how much the choice of reference, and not only the choice of threshold, is doing. There is certainly more to work out, in this part.

Part 4 — Representing the EPNs. The same systems drawn as Extended Petri Nets, following the graphical conventions of Bancheri, Serafin and Rigon (2019): colored circles for places, squares for transitions inheriting the color of the place they exit, framed triangles for controllers with dashed information arcs, diamonds for splitters with their partition fractions, everything laid on an integer grid with mathematical symbols as labels. Two details I find pleasing: arc routing can circumnavigate nodes through waypoints, and when a crossing between arcs is genuinely unavoidable — the notebook distinguishes by computation the avoidable from the necessary ones — it is denoted by a small open circle, the old circuit-diagram convention for "crossing without exchange". The rules that place the nodes are worth stating, because they are computable and not aesthetic preferences: places and transitions stay on grid nodes, arrows are straight lines or compositions of straight lines except for the controllers, a multi-part arrow is permitted precisely in order to avoid a crossing, and among the admissible configurations the more compact is chosen. The justification for not trading one against the other is that a net which obeys the no-crossing rule in some layout can always be deformed onto the grid, so compactness and crossing-freedom are not in competition. Part 6 gives these rules a test they cannot have on a hand-made example: applied blind to 47 published nets, none of which declares a single coordinate, 34 come out with no crossing at all. The basic idea is that giving as input the EPN representation of the model, the program knows which data to expect and consequently which graphs to produce, see Part 5 below. There are various improvements that can be envisioned also here. 

Part 5 — The EPN as budget analysis. The point of the whole exercise: the EPN declaration is not only a drawing, it is the bookkeeping. From the same JSON that draws the net, the code derives the whole-system budget and one budget per place, verifies the closure node by node, and plots the two-panel budget figure of every place of the net. The same JSON then acts as a data contract for simulation outputs: we propose a format that GEOframe-NewAGE results should obey — one tidy table per hydrological response unit, one column per EPN transition (internal fluxes included, which is an argument for making them first-class outputs of any simulator), one column per place state — so that if simulator and analysis share the same EPN, every node must close to numerical precision, and any residual is either a format violation or a bug. Coherence between the GEOframe outputs and the EPN scheme of the simulator, in other words, becomes something one can check mechanically. One lesson from making the contract strict: the closure check used to skip the missing steps, so a table that was 95 % gaps reported a better residual than an honest one — precisely backwards. It now reports the sample size beside the error (on a deliberately broken demo table, n_steps = 4 of_possible = 364), so a good number obtained from four steps cannot pass for a good number, and a declared column that is present but mostly empty now fails the contract outright.

Part 6 — The EPN dictionary. If an EPN is a grammar, then it should be possible to write a vocabulary in it, and the vocabulary is worth more than any single sentence. Part 6 transcribes the models of the MARRMoT collection (Knoben et al., 2019) as EPNs — 47 nets over 46 model identifiers, since one model is drawn twice in the source with two different structures — and stores each as a JSON entry carrying the net, the governing equations as they are printed beside it, the constitutive closures, the parameters, and the page it was read from. The collection then becomes something one can query rather than leaf through: find the models with a snowpack, the ones with an IUH transition, the ones with at least four storages. Sizes run from one storage to nine, with a median of four; 43 of the nets have a junction, 33 a splitter, one an IUH.

The reason for storing the equations next to the drawing is that it makes the transcription falsifiable. The deck prints the ODE of every store beside its net, so dS/dt = Ps − Es − Perc is an independent statement of which fluxes touch that store and with which sign, and the code compares the two readings: 173 of the 187 places carry a printed equation, and of those 161 agree with the arcs drawn around them. Twelve places in ten models do not. I have not guessed at those: each is recorded in a table with the arc missing on one side or the other, because deciding which of the two readings is right means going back to the original paper, not to my transcription. Fourteen places print no equation at all in the deck, so they cannot be checked — a gap in the source, and it seemed better to count it than to hide it.

The second half of the notebook is about adding an entry, since a dictionary nobody can extend is a catalogue. A small two-bucket model is walked through the three gates an entry must pass — the schema, the equation cross-check, and a redraw — and each gate is broken first, so that the reader sees the error message rather than a description of it.

Two things I did not expect to learn from doing this. The first is that transcribing a drawing is the weak link in the whole exercise: the errors the cross-check caught in my own transcriptions were exactly the kind a careful reader makes — a flux attached to the wrong store, a splitter branch that should have been an evaporative loss, a melt term turned into a self-loop. The second is that the disagreements which survive are interesting in themselves. A published net and a published equation that do not say the same thing are a fact about the literature, not about my code.

A seed

The notebooks run top to bottom without errors, the figures are checked geometrically before they ship, and the library carries 112 tests — the dictionary's own invariants among them: that an entry's stored counts match its net, that its status flag is the computed verdict of the cross-check rather than an opinion, that the search index is recomputable from the entries it summarises — but none of this makes them finished. The synthetic model is deliberately minimal; the drought thresholds are heuristic; the energy budget is absent; the connection to real GEOframe runs is, for now, a contract waiting for its first signatures. As usual, this is a seed for improvements: take them, break them, and tell me where.

One practical warning for anyone who does. The .ipynb files are generated, by the scripts build_01.py … build_06.py: an edit made in Jupyter is discarded by the next build. This is deliberate — it is what makes the series reproducible, and the build is now byte-identical, so two builds of unchanged sources give identical files and a real change shows up as a real difference — but it means that the natural thing to do, which is to open a notebook and fix something in place, is the one thing that will quietly lose the fix. Edit the builder.

Files are here: https://osf.io/jt7z8/files/osfstorage

References

• Bancheri, M., Serafin, F., & Rigon, R. (2019). The representation of hydrological dynamical systems using Extended Petri Nets (EPN). Water Resources Research, 55(11), 8895–8921. https://doi.org/10.1029/2019WR025099
• Bancheri, M., Rigon, R., & Manfreda, S. (2020). The GEOframe-NewAge modelling system applied in a data-scarce environment. Water, 12(1), 86. https://doi.org/10.3390/w12010086
• David, O., Ascough II, J. C., Lloyd, W., Green, T. R., Rojas, K. W., Leavesley, G. H., & Ahuja, L. R. (2013). A software engineering perspective on environmental modeling framework design: The Object Modeling System. Environmental Modelling & Software, 39, 201–213. https://doi.org/10.1016/j.envsoft.2012.03.006
• Formetta, G., Antonello, A., Franceschi, S., David, O., & Rigon, R. (2014). Hydrological modelling with components: A GIS-based open-source framework. Environmental Modelling & Software, 55, 190–200. https://doi.org/10.1016/j.envsoft.2014.01.019
• Harpold, A. A., Dettinger, M., & Rajagopal, S. (2017). Defining snow drought and why it matters. Eos, 98. https://doi.org/10.1029/2017EO068775
• Knoben, W. J. M., Freer, J. E., Fowler, K. J. A., Peel, M. C., & Woods, R. A. (2019). Modular Assessment of Rainfall–Runoff Models Toolbox (MARRMoT) v1.2: an open-source, extendable framework providing implementations of 46 conceptual hydrologic models as continuous state-space formulations. Geoscientific Model Development, 12(6), 2463–2480. https://doi.org/10.5194/gmd-12-2463-2019
• Rigon, R., Bertoldi, G., & Over, T. M. (2006). GEOtop: A distributed hydrological model with coupled water and energy budgets. Journal of Hydrometeorology, 7(3), 371–388. https://doi.org/10.1175/JHM497.1
• Rigon, R., Formetta, G., Bancheri, M., Tubini, N., D'Amato, C., David, O., & Massari, C. (2022). HESS Opinions: Participatory Digital eARth Twin Hydrology systems (DARTHs) for everyone. Hydrology and Earth System Sciences, 26, 4773–4800. https://doi.org/10.5194/hess-26-4773-2022
• Van Loon, A. F. (2015). Hydrological drought explained. WIREs Water, 2(4), 359–392. https://doi.org/10.1002/wat2.1085
• Yevjevich, V. (1967). An objective approach to definitions and investigations of continental hydrologic droughts. Hydrology Papers 23, Colorado State University.

The implementation of the GEOframe system in the Po river district – analysis of water availability and scarcity

In recent years, the frequency of extreme events like floods and droughts, which can cause severe environmental, social, and economic damage, has increased due to climate change and environmental alterations. In response to these challenges, the Po River Basin District Authority (AdBPo) initiated the implementation of the GEOframe modelling system across the entire district in 2021, in collaboration with the GCU-M (Gruppo di Coordinamento Unificato-Magre). The goal was to enhance the existing numerical models for water resource management, providing more accurate quantification and forecasting of spatial and temporal water availability across the Po River Basin, thereby improving overall planning and decision-making processes.

Additionally, a historical analysis of water availability was conducted in Valle d’Aosta and Piemonte, showcasing GEOframe's ability to simulate all key components of the water cycle, including evapotranspiration, water storage, snow accumulation, and water discharge. The implementation of GEOframe in these mountainous regions also underscored the critical role of snow and glaciers in determining water availability, particularly in the context of rising temperatures due to climate change. As a result, future developments of GEOframe will prioritize improving the modelling of these elements to better capture their influence on water resources in a warming climate. The short presentation given at IDRA24 can be obtained by clicking on the above figure.  The poster is available here. . 

Estimating water budgets with JGrass-NewAGE

We already talked about water budgets, and the papers of ours that deals with it (see below). Because in this Fall AGU meeting there was a dedicated session, we presented an abstract:

Recently we presented two papers one dedicated to the estimation of the water budget components in a small, basin, the Posina catchment [Abera et al., 2017], and the other in a large basin, the Blue Nile [Abera et al., 2017b]. Closing the budget in the two cases was different. Worth to say, it was much more difficult to close the budget at Posina, since at the large scale satellite platform can reasonably help to validate the results. At the smallest scale ground measurements usually available do not guarantee the closure of the budget without making additional hypothesis and remote sensing data cannot give very much help.  The hypothesis that we made is that the groundwater storage comes back to the initial level after a certain time, that we called Budyko time, TB. This time can be fixed arbitrarily, for instance, to five years and then varied to assess, through these trials the uncertainty of the budget. The large scale case was largely supported by remote sensing data, instead, either for calibration and/or validation. This contribution explains how we actually did, clarifies some aspects of the informatics necessary to obtain it and openly discusses the issues risen in our work. We also consider varying configuration of the water budget schemes at the subbasin level, and how this affects the estimates.

Finally we analyse the problem of travel times [e.g. Rigon et  al., 2016a, Rigon et al, 2016b]  as it comes out from considering the multiple fluxes and storages and discuss how much they can be realistic. All considerations and  simulations are based on the JGrass-NewAGE system [Formetta et al., 2014] and its evolution presented in Bancheri [2017].

As we say in the presentation, we could not talk about the travel times. However there are several other places where you can find about, here. 

Clicking on the above Figure, you will see the presentation that was used in NewOrleas. However, on Youtube, we uploaded an extended version with comments. 

References

Abera, W., Formetta, G., Borga, M., & Rigon, R. (2017). Estimating the water budget components and their variability in a pre-alpine basin with JGrass-NewAGE. Advances in Water Resources, 104, 1–18. http://doi.org/10.1016/j.advwatres.2017.03.010

Abera, W., Formetta, Brocca, L., & Rigon, R. (2017), Modelling the water budget of the Upper Blue Nile basin using the JGrass-NewAge model system and satellite data. Hydrol. Earth Syst. Sci., 21, 3145–3165, 2017

Rigon, R., Bancheri, M., & Green, T. R. (2016). Age-ranked hydrological budgets and a travel time description of catchment hydrology. Hydrology and Earth System Sciences, 20(12), 4929–4947. http://doi.org/10.5194/hess-20-4929-2016

Modelling the water budget of the Upper Blue Nile basin using the JGrass-NewAge model system and satellite data

This paper must be read after its companion on rainfall published in Atmospheric Research. There we were concerned with rainfall estimates over the large areas of Upper Blue Nile (UBN). Here we move on to estimate all the other components of the water budget. A similar goal was searched at small scales and with different tools, in this other paper about Posina catchment. So the paper can be considered sort of complimentary and covering a range of possibilities allowed by the JGrass-NewAGE system.

The paper abstracts reads: 

"The Upper Blue Nile basin is one of the most data-scarce regions in developing countries, hence, the hydrological information required for informed decision making in water resources management is limited. The hydrological complexity of the basin, tied with the lack of hydrometerological data, means that most hydrological studies in the region are either restricted to small subbasins where there are relatively better hydrometeorological data available, or at the whole basin scale but at very coarse time scales and spatial resolutions. In this study we develop a methodology that can improve the state-of-art by using the available, but sparse, hydrometerological data and satellite products to obtain the estimates of all the components of the hydrological cycle (precipitation, evapotranspiration, discharge, and storage). To this scope, we use the JGrass-NewAge system and various remote sensing products. The satellite products SM2R-CCI is used for obtaining the rainfall inputs; SAF EUMETSAT for cloud cover fraction for proper net radiation estimation; GLEAM for comparison with estimated ET; and GRACE gravimetry data for comparison of the total water storage amounts available. Results are obtained at daily time-steps for the period 1994-2009 (16 years), and they can be used as a reference for any water resource development activities in the region. The overall long term mean budget analysis shows that precipitation of the basin is 1360 ±230 mm per year. Evapotranspiration covers 56% of the yearly budget, runoff is 33%. Storage varies from minus 10% to plus 17% of the budget. "

The manuscript was submitted to HESS and went trough  a first round of revision (see the Discussion page). A revised manuscript was submitted. Please find below (on Zenodo):

• The answers to reviewers
• The new manuscript
• The complimentary material, including the shapefile of the UBN and the original figures.

The paper is now accepted and available at the HESS site.
   

A travel time model for estimating the water budget of complex catchments

This is the presentation given by Marialaura Bancheri for her admission to the final exam to achieve a Ph.D. in Environmental Engineering. It contains a synthesis of her studies about spatially integrated models of the water budget, and about travel time theory. A model structure is also presented preliminarily containing five reservoirs.

These reservoirs model the hydrology  of a  Hydrologic Response Unit (HRU) of a basin  which are connected together to treat a river catchment (as shown in Rigon et al. 2016). The figure above is a Petri net representation of the set od ordinary differential equations that  constitute the mathematical models of a HRU. The model uses the river network structure to organise the components execution, a work made conjointly with Francesco Serafin.

By clicking on the Figure, you will see Marialaura's presentation.

Reservoirology #2

Because, as we already noticed, often modelling the hydrological cycle is studying the mutual interactions of "reservoirs" of something (which I named "reservoirology"), I tried with my students to have a clean way to represent reservoirs. As we said in the slides below, we try to represent them in a way that, if it is not in a one to one correspondence with the equations that can be derived from the graphs, we are as close as possible to it. This post, left for saving the history, is, however, superseded by the new post reservoirology #3.

Please find above this graphic collection and help us to improve it. One thing to notice is that our representation is mappable, with different expressivity of the concepts to the Petri Networks algebra, which puts in our hands several studies in mathematics, computer sciences, biology and other sciences.