Tuesday, December 28, 2021

Edward Wilson (1929-2021) - A storm in the Amazon

From "The diversity of life" 

"I sorted the memories this way and that in hope of stumbling on some pattern not obedient to abstract theory of textbooks. I would have been happy with any pattern. The best of science doesn't consist of mathematical models and experiments, as textbooks make it seem. Those come later. It springs fresh from a more primitive mode of thought, wherein the hunter's mind weaves ideas from old facts and fresh metaphors and the scrambled crazy images of things recently seen. ...


The storm grew until sheet lightning spread across the western sky. The thunderhead reared up like a top-heavy monster in slow motion, tilted forward, blotting out the stars. The forest erupted in a simulation of violent life. Lightning bolts broke to the front and then closer, to the right and left, 10,000 volts dropping along an ionizing path at 800 kilometers an hour, kicking a countersurge skyward ten times faster, back and forth in a split second, the whole perceived as a single flash and crack of sound. The wind freshened, and rain came stalking through the forest. In the midst of chaos something to the side caught my attention. The lightning bolts were acting like strobe flashes to illuminate the wall of the rain forest. At intervals I glimpsed the storied structure: top canopy 30 meters off the ground, middle trees spread raggedly below that, and a lowermost scattering of shrubs and small trees. The forest was framed for a few moments in this theatrical setting. Its image turned surreal, projected into the unbounded wildness of the human imagination, thrown back in time 10,000 years. Somewhere close I knew spear-nosed bats flew through the tree crowns in search of fruit, palm vipers coiled in ambush in the roots of orchids, jaguars walked the river's edge; around them eight hundred species of trees stood, more than are native to all of North America; and a thousand species of butterflies, 6 percent of the entire world fauna, waited for the dawn.

…..

The storm arrived, racing from the forest's edge, turning from scattered splashing drops into sheets of water driven by gusts of wind. It forced me back to the shelter of the corrugated iron roof of the open-air living quarters, where I sat and waited with the mateiros. The men stripped off their clothing and walked out into the open, soaping and rinsing themselves in the torrential rain, laughing and singing. In bizarre counterpoint, leptodactylid frogs struck up a loud and monotonous honking on the forest floor close by. They were all around us. I wondered where they had been during the day. I had never encountered a single one while sifting through the vegetation and rotting debris on sunny days, in habitats they are supposed to prefer. "

Edward O. Wilson, The Diversity of life, Penguin

Monday, December 27, 2021

DARTH4MED - A Digital eARth Twin of Hydrology for the prediction of water scarcity in the Mediterranean area


The DARTH4MED, D4M for short, project aims to be a high resolution twin of the hydrology and carbon cycle of the Italian peninsula. It is based on Po, WATZON and WATERSTEM projects, making treasure of previous modelling efforts like GEOtop and the GEOframe system, and GIS tools implementations like Jgrass and the Horton Machine toolbox. It builds upon state-of-art hydrological modelling case studies of various catchment sizes, from hillslope to Po and Blue Nile. It also draws on experiences in IT applied to hydrology with developments of the object modelling system, OMS.

D4M gives substance, both technical and scientific, to the Digital Earth metaphor and exploits it to improve the work of scientists and professionals, and to support open science. It aims to provide a shared infrastructure usable by scientists and users to investigate the processes involved in the water, energy and carbon budgets, WB, EB and CB, at a very fine spatial and temporal scale, 1 km2, hourly.

The GEOframe system already contains a sophisticated and complete set of modelling components, constituting a solid basis of comparison for innovative developments. Open API and training will be offered to anyone to advance the mathematical, statistical and numerical descriptions of hydrological and eco-hydrological processes with little programming effort. From this perspective, the project will be an experiment in participatory science, since the tools developed could be improved and given back by collaborative researchers. The method of multiple hypothesis testing will be the rule of scientific endeavour.

The core of the system will manage the interactions of groundwater, vadose zone, surface water, snow, vegetation, atmosphere, usually analyzed separately, and join them seamlessly in the continuum containing the feedbacks among the parts. On these bases researchers will be able to evaluate climate, hydrologic, pedological, ecological droughts.




D4M has several primary objectives, listed below:
  • To provide the core of a DE, defined as a Digital eARth Twin Hydrology system (a DARTH), to do hydrology by computer, with an infrastructure that allows partecipative hydrology and makes Earth system science practice easier for all the Italian Peninsula.
  • To improve the modelling of the water budget, WB, energy budget, EB, Vegetation and Carbon Cycle.
  • To provide forecasts for several variables, as detailed in the Synopsis.
  • To resolve some research questions, as presented in the Synopsis.
  • To give researchers sound tools on which to base their analysis of climate, hydrologic, pedological, ecological and agronomic droughts.
  • To provide a high level of abstraction and encapsulation for modelling services, so to allow improvements to parts of the DARTHs by anyone without disrupting the whole.
  • To give API and web services to final users, researchers, technical professionals, programmers, to connect their studies and products to the whole D4M, thus combatting the fragmentation of hydrological modelling through a participatory open platform.
Besides efficient algorithms, the effort will require the smart implementation of parallel computing infrastructures, which will remain mostly invisible to the users. All the infrastructure will be open source, built with open source tools and provided with open data.

The project was just submitted for the FIS call. Here below you find the proposal and the relevant annexes.
Compressing all the ideas in such a few words was quite difficult and the platform on which we had to upload the material with some issues (non accepting, for instance "()[]-/" and other characters. Some requirements quite stupid. The selection will be great. I obviously think that the gain for the country with such a project really great. Finger crossed and, if there are better projects, hope they'll win. 

Wednesday, December 15, 2021

DARTHs (Digital eARth Twin Hydrology systems)

There is a great hype about Digital Earth Twins (DETs) and EU, ESA, NASA and other institutions issue calls for building such IT infrastructures (ITI). This paper face the topic from a point of view of hydrologists who are concerned with the science content of these ITI. The Authors see in DETs an opportunity to make easier the work of scientists and professionals. However they claim that some aspect of making science should be respected. Mainly they are the hypothesis testing and estimation of errors in hindcasting or forecasting. Beside, the Authors claim that building a DET for Hydrology (called DARTH) is an enterprise that implies some choices about the implementation of models and of the infrastructure. DARTHs are not in fact just "models" and have requirements that need to be satisfied. Finally the Authors support the opinion of an open science oriented implementation of these ITI that also allows the participatory action of all the scientists that like to contribute (and look with suspect to science processing where just a few contribute to the core science). In turn also this options has requirements that should be reflected in the implementation.
To sum up, the Authors think that this is the right moment to push these ideas and desire to open a discussion with other colleagues. 

Final published paper: 



You can find the manuscript submitted to HESS discussions HESSD at the moment in our OSF repository, here.  

UPDATE: The paper had a positive first round of reviews that you can see here.  Below, please find the revised text with the supplemental material. 

Final published paper: 

Thursday, November 11, 2021

How to write a paper on a new hydrological model component

 Let’s try to keep the matter simple.  General rules apply:



Analyzing back the general scheme, in the case of software presenting, you need a specific part dedicated to the availability and delivery of the software. The main parts required here were already illustrated in explaining the Zero Notebook contents.

Because you are talking about scientific software your methodology has two parts. One related to the science you have to produce and one related to the science of writing good software.

Taking the example of Evapotranspiration. The science could be the one included in the sub-models you are implementing. Meaning, what is the science behind Priestley-Taylor, which the one behind FAO approach, and which the one one behind, for instance, our Prospero model ? Here the material is very large so you have to work usually by extracting the essentials and citing the literature. Part of it can easily fit actually inside the introduction. The informatics has to do with the way your system is built. Which is the framework you use, in our case, OMS3, and why you use it, instead of others. It also the system you are working with, like in our case GEOframe that provides ancillary tools. Finally the informatics can boil down to the algorithms and their organization in classes. Algorithms can be new or old and irrelevant. Just in the first case it is important to mention them with details, otherwise just a a little note can be done. Classes, assuming we are talking of some OO programming, have two scopes, one is to contain the algorithms, the other is to orchestrate the software relations in order to make easy the reuse of the softwares and their expansion. This part will be routine in future, maybe, but now it is not part of the common knowledge of hydrologist, and therefore it is worth to be explained if well engineered. In explaining classes and the overall working of the software using of UML diagrams is mandatory.

In a software paper, it is debatable what is the test of the contents. Let’s say that, because we are hydrologists, we need to test both the software running, and the models’ physics.

The software running test for who is programming in Java, like we do, is obtained through the appropriate Unit Tests and this part is commented, in case, inside the section which inherit from the Notebook Zero. For the physics we have, in turn, two modes. If we are solving problems, i.e. equations,  that have an analytical solution, then we have to reproduce the analytical results. Secondly the nasty reviewer, would also see that the model reproduces measurement. Getting some measurements to reproduce is then important. A third case is also ideally possible, which is that, no measurements are available and therefore eventually the model provide a possibility to test something that was never tried before. In this case it must be emphasized that the model makes possible something that before was not not, and we have to rely to some virtual, behavioural, experiment.

If measurements are involved, new methodological steps come in: explaining the case study, first. Secondly, not differently from other cases, we have to say if parameters to calibrate and to mention the techniques we use for doing it. Explaining how we assess the goodness of the results, and finally commenting the results are the rest of the story. An exceptionally good software that does not reproduce reality is simply not useful from the hydrology point of view, even if its implementation can still provide novelties worth to be explained. The physical test, however, should not extend to be very complicate but just functional to convince that the software is doing what it is designed to do. In the mentioned case of evapotranspiration, another issue is relevant, which is the comparison among sub-models or models alternative. It is clear that different models produce different results, so assessing in which case they work or work netter is important. However, that can be pursued with moderation in a “software presentation” paper, because this is clearly an argument which requires a paper by itself. For example , in a recent paper, Clark et al., 2021, they talk of “laugh test” for emphasizing this aspect.

At the end of the post, you have some ingredients and an idea of the procedure. To cook them together for a nice result is a little of art.  In general, a good example to follow is the WHETGEO paper.

Monday, November 8, 2021

GEOframe Winter School 2022 (GWS2022) - Save the dates Dec 20-22, 2021; January 10-14, 2022

December 20-22, 2021 - Installations and Informatics - Online

January 10 - 14 2022 - Onsite and Online

Scientific Committee: Prof. Riccardo Rigon, Ph.D.; Prof. Giuseppe Formetta, Ph.D; Ing. Niccolò Tubini, Ph.D., Ing. Marialaura Bancheri, Ph.D.

Organizing Committee: Ing. Concetta d’Amato, Eng. Shima Azimi, Ms. Sc. Martin Morlot, Ing. Daniele Andreis, Ing. Gaia Roati, Ing. Riccardo Busti (the fantastic group of our Ph.D. students)

Organizing Institutions:
  • Department of Civil, Environmental and Mechanical Engineering, University of Trento
  • Center Agriculture Food Environment, University of Trento
  • Institute for Agricultural and Forest Systems in the Mediterranean, National Research Council, Ercolano NA, Italy

GENERALITIES
GEOframe is a system for doing hydrology by computer. By saying that it is a system, we emphasize that it is not a model but an infrastructure that can contain many differentiated modelling solutions (some tens of that) that are built upon models components. This is because GEOframe leverage on the Object Modelling system-framework (v3) that allows to connect modelling components to solve a specific hydrological issue together and having many alternative for its mathematical/numerical description. This infrastructure allows adapting the tools to the problems and not viceversa. GEOframe has been applied to hydrological simulations from the point scale to large catchments as the Blue Nile, and among those is being deployed to the Po river (the largest in Italy) with great detail. GEOframe is open source and built with open source tools.

CONTENTS OF THE SCHOOL

GEOframe contains tens of components that cover rainfall-runoff, evaporation, transpiration, infiltration, terrain analysis tools, interpolation models, calibrations tools, and so on.
The Winter school is about using some of these tools to perform the hydrological budget of catchments. The core rainfall-runoff model are dynamical systems (systems of ordinary differential equations) and the school mainly treats their theory and their use in a contemporary way as summarized in these7 steps.
Besides the lectures and the hands-on sessions, the Summer School is the occasion for discussion and experience exchange among senior scholars and young researchers.

And Pizza party every night ! Just Kidding


PARTICIPANTS' BACKGROUND

Admissions are reserved to up to 30, PhD students and postdoctoral students, young researchers willing to learn the use of the GEOframe tools envisioned for the study of infiltration, energy budget, vegetation transpiration, water budget with process-based models
All students are asked to upload a CV and a motivation letter when applying.


WORKLOAD AND CREDITS

The Winter School which is to be held in English, consists of 8 hours/day of activities for 8 days. The first three days, 20-22 of December will be dedicate to the the installation of the new version of GEOframe-OMS system tools. Lectures will be brief, dedicated to informatics and the exploiting of the concepts of modeling by components, digital twin Earth and most of the time will be used for supporting participants’ installations.

The other five days (10-14 January) will cover:
  • Catchment and Hydrologic Response Unit delineation
  • Meteorological variables interpolation with Kriging techniques
  • Simple evapotranspiration methods
  • Rainfall-Runoff modelling (as explained in these 7 steps)

LOCATION & TIMING

University of Trento Polo Mesiano, H1 Room and Online. The three days on informatics and installations will be online. The others online and onsite. The time schedule will be 9-13 and 14-18 CET each of the days. Lectures and workout will be recorded and immediately post on the VIMEO Channel of the School and therefore they could be followed off line. Special agreement will be arranged for supporting abroad students with fuse issues.

PARTICIPATION COSTS

The cost is free for Students of the Hydrological Modelling Classes at the University of Trento, for Ph.D. students of the University of Trento DICAM and C3A programs, for the participants of the WATZON PRIN project and for all who wants to participate without having a certificate of GEOframe proficiency. Subscription to the class is necessary for anyone to receive the information to participate. For those who want the certificate, the Course costs 180 Euros. In any case the certificate is issued after the presentation of a small project of simulations for which appropriate tutoring will be given during and after the School.


CONTACTS

For further information write to: abouthydrology@google.com or to the Secretary of the Class dott. Lorena Galante, lorena.galante@unitn.it

OTHER INFORMATION

The GSS2022 talks and labs will be recorded and made publicly available during the School for self-training through the GEOframe blog (http://geoframe.blogspot.com). Information about past Schools can be found here.

GEOframe Soil Plants Atmosphere Continuum and hydrology Estimator (GEO-SPACE) essentials

GEO-SPACE (formerly known also as Lys-GEO in its its 1D implementation)  is intended to collect the growing set of GEOframe tools developed on the base of process-based philosophy. This can be found envisioned first in  Freeze and Harlan, 1969, and, fo instance well documented recently in Fatichi et al., 2016 and Paniconi and Putti, 2016. From a different point of view, it can be considered the upgrade of the GEOtop model, that still efficient and up-to-date, and more advanced than other similar models, was considered to be improvable from the algorithmic and informatics structure.  GEO-SPACE make leverage on the various common tools (components) shared with GEOframe-NewAGE and is made up  specifically of two main groups components, WHETGEO (mainly due, so far to the work of Niccolò Tubini) and the evaporation and transpiration, as follows from the work by Michele Bottazzi and Concetta D'Amato (ET-GEO). 



At present the development of GEO-SPACE (to become GEOtop 4.0) has still to achieve some goals, including the connection of plants treatment on WHETGEO 2D, the implementation of WHETGEO-3D, and so on. The current status of the project can be well described looking at the material presented at the Summer Schools on GEOframe that started in 2021 and will be held usually in week in middle June every year.  The most recent School addresses the more recent material. 

We remind here below, the general declaratory about GEOframe:

GEOframe is a system for doing hydrology by computer. By saying that it is a system, we emphasize that it is not a model but an infrastructure that can contain many differentiated modelling solutions (some tens of that) that are built upon models components. This is because GEOframe leverage on the Object Modelling system-framework (v3) that allows to connect modelling components to solve a specific hydrological issue together and having many alternative for its mathematical/numerical description. This infrastructure allows adapting the tools to the problems and not viceversa. In GEOframe particular attention has been dedicated to allow enhancements and additions writing the less code possible. The core code has been designed to open to addition and closed to modifications, thus allowing stability over time.  The systems contains tens of components that cover rainfall-runoff, evaporation, transpiration, infiltration, terrain analysis tools, interpolation models, calibrations tools, and so on. Every modelling paradigm is included, as, for instance process based modelling, lumped modelling, machine learning, or can be included. Spatially disjoint catchments can be modelled separately and joined together in a bigger model. GEOframe has been applied to hydrological simulations from the point scale to large catchments as the Blue Nile, and among those is being deployed to the Po river. GEOframe is open source and built with open source tools.

References

Fatichi, Simone, Enrique R. Vivoni, Fred L. Ogden, Valeriy Y. Ivanov, Benjamin Mirus, David Gochis, Charles W. Downer, et al. 2016. “An Overview of Current Applications, Challenges, and Future Trends in Distributed Process-Based Models in Hydrology.” Journal of Hydrology 537 (C): 45–60.

Freeze, R. Allan, and R. L. Harlan. 1969. “Blueprint for a Physically-Based, Digitally-Simulated Hydrologic Response Model.” Journal of Hydrology 9 (3): 237–58.

Paniconi, Claudio, and Mario Putti. 2015. “Physically Based Modeling in Catchment Hydrology at 50: Survey and Outlook.” Water Resources Research 51 (9): 7090–7129.

Tubini, Niccolò, and Riccardo Rigon. “Implementing the Water, HEat and Transport Model in GEOframe (WHETGEO): Algorithms, Informatics, Design Patterns, Open Science Features and 1D Deployment.Geoscientific Models Development Discussions.

Thursday, November 4, 2021

Seven Steps Into Catchments analysis by Hydrological models

 First step: Overall


Define the overall scope of the analysis. (e.g. Blöschl et al., 2019)
Document about the literature existing on the catchments. Including papers appeared in any type of studies.  Choose a large set of  performances indicators (depending on the model use and objectives) (Addor et al., 2017).  Set a strategy for assessing the results uncertainty and variability. (e.g. Clark et al., )

Second step: Geomorphology

Extract the catchment from DEM by state-of-art  analysis tools (as TauDEM or the Horton Machine Toolbox). Investigate if the surface catchment corresponds to the effective catchment (i.e. if including/excluding karst is an option). Control the surface water network extension (talk at the GSS2021). Analyze the presence and the number of lakes/reservoirs. Pay attention to endorheic catchments. Discuss the catchment connectivity.

Third step: Hydrological Data

Analyze the time series of available data and make a comparison  between rainfall and runoff amounts. Analyze any other time series or map time series, like  map of snow and evapotranspiration (Abera et al., 2017). Define the calibration set and the validation set.


Fourth step: Modelling Setup

Setup the space partitioning in hydrologic response units  (e.g. Dal Molin, 2021). Discuss the data density required (or possible) to give robust results or, viceversa, reduce your objectives to something achievable with the data available.   Choose the model among existing ones for adequacy not for legacy (Addor, 2019).  Choose a model structure as hypothesis zero (Fenicia and Kavetski, 2021, VimeoVideo).*  Setup the modelling solution (MS). Choose the better modelling structure (Clark et al., 2011).  Analyze the MS  parameters and discuss their variability.  Plan the model runs thinking to open science protocols (Hall et al., 2021). 

Fifth step: Modelling Execution

Executing the model, including discharges, snow and ET. Annotate the model performances issues. 

Sixth step: Results Delivery

Show the results appropriately (discharge analysis is not the only one quantity to watch at).  Analyze the performances of indicators. Add comments and discussion. Assess results confidence.

Seventh step: Deployment

Deploy the results for open science and public discussion (e.g. Hall et al., 2021). 

Notes

*With regards to the modelling structure, if you are using ODEs for modelling, consider a standard way to visualize and describe the model structure. As many know, the Extended Petri Net can be a sound way to do it. 

Some Slides and Videos about the above topics:

References

Abera, W., G. Formetta, and L. Brocca. 2017. “Modeling the Water Budget of the Upper Blue Nile Basin Using the JGrass-NewAge Model System and Satellite Data.” Hydrology and Earth System Sciences. http://nora.nerc.ac.uk/id/eprint/517346/.

Addor, Nans, Andrew J. Newman, Naoki Mizukami, and Martyn P. Clark. 2017. “The CAMELS Data Set: Catchment Attributes and Meteorology for Large-Sample Studies.” Hydrology and Earth System Sciences 21 (10): 5293–5313.

Addor, N., and L. A. Melsen. 2019. “Legacy, Rather Than Adequacy, Drives the Selection of Hydrological Models.” Water Resources Research 55 (1): 378–90.

Blöschl, Günter, Marc F. P. Bierkens, Antonio Chambel, Christophe Cudennec, Georgia Destouni, Aldo Fiori, James W. Kirchner, et al. 2019. “Twenty-Three Unsolved Problems in Hydrology (UPH) – a Community Perspective.” Hydrological Sciences Journal 64 (10): 1141–58.

Clark, Martyn P., Andrew G. Slater, David E. Rupp, Ross A. Woods, Jasper A. Vrugt, Hoshin V. Gupta, Thorsten Wagener, and Lauren E. Hay. 2008. “Framework for Understanding Structural Errors (FUSE): A Modular Framework to Diagnose Differences between Hydrological Models.” Water Resources Research, Water Sci. Appl., 44 (12): 2135.

Clark, Martyn P., Dmitri Kavetski, and Fabrizio Fenicia. 2011. “Pursuing the Method of Multiple Working Hypotheses for Hydrological Modeling: HYPOTHESIS TESTING IN HYDROLOGY.” Water Resources Research 47 (9). https://doi.org/10.1029/2010wr009827.

Dal Molin, Marco. 2021. “Improvement and Application of Flexible Frameworks for Modelling Regional Streamflow Variability.” Edited by Marco Schirmer Fabrizio Fenicia. Ph.D., Université de Neuchâtel.

Fenicia, Fabrizio, and Dmitri Kavetski. 2021. “Behind Every Robust Result Is a Robust Method: Perspectives from a Case Study and Publication Process in Hydrological Modelling.” Hydrological Processes 35 (8). https://doi.org/10.1002/hyp.14266.

Hall, Caitlyn A., Sheila M. Saia, Andrea L. Popp, Nilay Dogulu, Stanislaus J. Schymanski, Niels Drost, Tim van Emmerik, and Rolf Hut. 2021. “A Hydrologist’s Guide to Open Science.” Hydrol. Earth Syst. Sci. https://doi.org/10.5194/hess-2021-392.