Tuesday, February 22, 2022

Is your modelling system adapt to be part of a digital Twin for Hydrology ?

Recently we submitted a paper to HESSD for discussing how Digital Twin can serve a better doing of Hydrology.  As you can see below, we named the Digital twins for hydrology DARTHs. The discussion paper is: 

Rigon, Riccardo, Giuseppe Formetta, Marialaura Bancheri, Niccolò Tubini, Concetta D’Amato, Olaf David, and Christian Massari. 2022. “HESS Opinions: Participatory Digital Earth Twin Hydrology Systems (DARTHs) for Everyone: A Blueprint for Hydrologists.” Hydrology and Earth System Sciences Discussions, 1–38.

and if you want to contribute with your opinion, the discussion is open until March 14.  If you want to grasp quickly what DARTHs are, you can take five minutes to read to the previous post here (DARTHs cheat sheet). 

As part of the review process, one of the reviewers, dr. Marc Hrachowitz (GS) that we thanks for his comments and suggestions, asked for giving examples of exiting modelling platform or systems that could satisfy part of the requirements that DARTHs, according to us, must have. However, the operation of judging elements of modelling systems of other researchers is always a slippery matter because one usually know just superficially what the systems provided by others actually do.  Therefore  I and co-authors have decided to issue a questionnaire where part of the questions are on characteristics that DARTHs should have.  Developes and users are asked to answer about their systems or models so to have a first hand information.  It could be noticed that we are not asking which processes the models aim to simulate because the hydrology (the physics, chemistry, biology) of the model is not actually the topic here. In the questionnaire the main topic is more about the infrastructure capability and some additional features that should be present.

As Authors of the paper, we know that our own systems do not satisfy the DARTHs' requirements and therefore don't be worry about if your system does not accomplish all. With the questionnaire we try to put a first brick towards them by trying to make researchers reflect on the steps they should do in future developments to converge to build a DARTH component. 

The results of the questionnaire which will remain upon until March 14 2022 will be made public as supplemental material of the paper.  The survey can be found by clicking on the image above.

You can view the partial results of the survey here.

Posted by at No comments:
Labels: ,

Wednesday, February 16, 2022

DARTHs programming cheat sheet

We recently wrote about DARTHs.  Here it is a summary of those more extensive documents. DARTHs fall under the category of thematic Digital Twins.

DARTHs are composable infrastructures whose parts are loosely connected, communicate with standard protocols and can easily be substituted. There is not such a thing like a DARTH but DARTH solutions. They constitute an ecosystem of tools whose parts can be separated and recomposed in new solutions, exactly as it happens for Linux distributions (distros).  Distros need usually an integrator  of resources which is a further component of DARTHs. Below we subdivide the main requirements necessary to make the DARTHs effectively working. 


Data
  • Must be Open
  • Provided on demand over the cloud
  • Discoverable on the web
  • Provided in standardized self-explanatory formats
  • Retrievable on the base on open APIs on the most common computer languages
DARTHs IT Architecture requirements

DARTHs serve different users and roles
Therefore they are agnostic:
  •  with respect to the science
  • the programming language
  • The operative system
  • Modelling styles and paradigm
  • They are lightweight with respect to programming habits, meaning they should be minimalist in adding programming rules and aims to maintain the code short
  • Variables names should be mapped into standard names that provide an unique identification

in Open DARTHs,
  • Programs must be open source and developable with open tools
  • Simulations and operations must be traceable and replicable by construction
  • Chunks of data, codes and modelling solutions should be organized in standard ways and be deployable over the web for third parties testing and reuse

Hydrological models for DARTHs

  • need to be able to easy any flavor of modelling effort through like can be models based on ODEs (systems of ordinary differential equations), PDEs (system of partial differential equations), MS (Statistical Modelling), ML (Machine Learning) and any other tool that science will invent.
  • should be deployed in reusable and interoperable components (A component model is a model that is used to encapsulate equations or specific tasks into a reusable form. Components can be connected at runtime and communicate exchanging data in RAM memory)
  • reusable components should obey to standard rules for inputs and outputs appropriate identification
  • need to be deployable along the web on all the available platforms.
  • model-to-model (components) communication should be allowed through API

DARTHs programming practices

DARTHs must
  • Adopt software quality testing for any component separately
  • Adopting good programming practices
  • Have literate computing tools 
  • Promet clean code and literate programming

DARTHs and Computing facilities
  • They use various form  of parallelism but overall, parallelism should be provided as a service without having to change programming habits of models developers.
  • Workflows of tasks that can be schematize on graphs should be internally parallelized using piping methods.
  • Models on grids should use middleware that separates the parallelization from the "physics" of implementation and should be as less invasive as possible.
  • They can use multicores, multiprocessor, high end computers, distributed computing without necessarily direct intervention of the programmer
  • They should use the natural partition of Earth surface in river catchments and subcatchments to split the computational work.

DARTHs and EO
  • The complex retrieval chain from observed quantities to hydrological quantities has to become explicit and integrated in modeling
  • Data Assimilation of any type must become part of the modelling chain

DARTHs and Science Reliability and practices

  • DARTH must make very smooth hypothesis testing (in the sense of allowing alternative modelling structures)
  • The quantification of uncertainty must become  inherently part of an open DARTH. 
  • DARTHs must support open science practices by construction
Overall
  • To accomplish all the above task DARTHs requires some appropriate, lightweight, non invasive  infrastructure (framework) 
    • that supports the features required
    • allows the development the use and the continuous evolving of the DARTHs
    • separates the programming  and the use of the tools from the connection to data resources (EO, IoT, more traditional datasets),  
    • connects to web services, 
    • provides parallelism of computation and 
    • accesses to High Performance Computing or  HPC cloud services, 
    • allows communication between components, 
    • and components dispatching over the web, 
    • manages the several security issues, 
just to mention a few. 
Posted by at No comments:
Labels:

Tuesday, February 15, 2022

The Hydrological Modelling Class 2022 - The Lab

 Go to:

2021-02-24 - Installations - For the Installation go to this GEOframe page and install all what required.
We are also going to install the Horton Machine toolbox. 

For Italian speakers, they can also give a look do not miss the occasion to listen to Concetta D'Amato introducing installations in Italian:
OMS and GEOframe:

Basic information for who has never seen OMS and GEOframe: an introduction to the GEOframe/OMS3 system
 QGIS & Other Info 
2022-03-03 - Starting to work with the Object Modelling System
  • Material of the class and presentations can be found here. Please download them before the class
  • Running the Hello World program as an Object Modelling System Component (Vimeo 2022)
  • Reading an OMS file (Vimeo 2022)
  • Running a simple example (a tank model) that uses more than one component 
2022-3-04
2022-3-10
  • The extraction of river network with the Horton Machine Tools (Vimeo2022)          
Michele Vettorazzi, Environmental Engineer and Photographer

2022-03-11 - GEOmorphology part III - The delineation of river basins (Vimeo2022)

Optional/Alternative Material - Some elaboration of DEMs with with the Horton Machine Toolbox
The Horton Machine Toolbox offers an alternative interface to the same tools seen previously. For some purposes it can be more simple than using the .sim files. 
2022-03-24 - Rainfall Analysis with PANDAS (Vimeo2022)

2022-03-31/2022-04-01 - Preparing time series for GEOframe and analyzing them
  • Preparing a time series in OMS3 csv format (Vimeo2022)
  • Analyzing discharges with Jupyter and Python  (Vimeo2022)
  • Analyzing temperatures with Jupyter and Python  (Vimeo2022)
2022-05-19/20 - Kriging
2022-05-25 - Kriging - II
  After all radiation moves it all

2022-06-08 - Running  GEOframe-NewAGE 
  • Setup of GEOframe-NewAGE (Vimeo2022)
  • How works the workflow in GEOframe-NewAGE, Net3 (Vimeo2022)
  • Evapotraspiration for GEOframe-NewAGE (Vimeo2022)
  • How to prepare a run with a simple reservoir for any HRU with GEOframe (Vimeo2022)
  • The Notebook (Vimeo2022)
Posted by at No comments:
Labels: ,

The Hydrological Modelling Class 2022 - The Schedule

 Foreseen Schedule (up to Easter)

(boldface dates are those with definitive material, I or directly written in Italian  in for Italian,  unspecified for English. All slides are in English)

To understand better what is below: 
  • A storyboard is a summary, usually in Italian, of the lecture
  • A whiteboard is an explanation of a particular topic made on the whiteboard (using Notability on the iPad) usually in Italian
  • Slides are commented in English (since 2021)
  • Additional  information (only for the brave or the curious) and references are in italics

2022-02-24 - I  - Syllabus - Introduction 2 Hydrological Modelling 

Here  I introduced the class. Its learning by doing philosophy (altered by the necessity due to COVID-19 times that impose to do first the all the theoretical parts and subsequently all the practical parts hoping that they can be done in presence). 
2022-02-25 and 2022-03-03- Geomorphometry - I  - Discussion of previous lesson topics. Summary of the lecture in Italian. (This will be always done, each lesson, but for now on omitted). The rational of introducing these concepts  is that catchments are spatially extended and in this course we are interested to deal with catchments hydrology. 

In this first part we deal with the geometrical (differential) characteristics of the topography. Elevations, slopes, curvatures. They will be necessary later to extract the river network and the parts of a catchment.
In this class we define also what the drainage directions are and how they are computed in the case of DEMs (a topography discretized over a regular grid).  From drainage directions are determined the total contributing areas in each point of  a DEM. These two characteristics are eventually used to determine  the channels head and extract the river network. In turn, the extraction of the channel network allows for the extraction of hillslope and a first definition of  the Hydrologic Response Units (HRU). 


    2022-03-10 -  Interpolations 
    This lecture, assuming that now you have at least the concepts of what a catchment is and theoretically you know how to extract it and subdivide it in parts, deals with the data to feed catchments hydrology models. Because catchments have a spatial distribution, then also the driving data must be distributed. We need therefore methods of interpolation. 

    2022-03-17 -  Interpolations part II. 
    In this class we try to understand how to estimate the errors over the estimates. Besides we introduce a method (the Normal Score) to avoid to obtain negative values when positive interpolated values are required.

    2022-03-17 - Hydrological Models. This is a class about hydrological models, so what are they ?

    The title is self-explanatory. A theoretical approach to modelling is necessary because we have to frame properly our action when we jump from the laws of physics to the laws of  hydrology. Making hydrology we do not have to forget physics but for getting usable models we have to do appropriate simplifications and distorsions. The type of model we will use in the course are those in the tradition are called lumped models. Here we also introduce a graphical tool to represent these models.
    2022-03-17 - Hydrological Models
    2022-03-18 - Hydrological Models - II
    2022-03-24 - Linear Models for HRUs

    Once we have grasped the main general (and generic) ideas, we try to draw the simplest systems. They turn out to be analytically solvable, and we derive their solutions carefully. From the group of linear systems springs out the Nash model, whose derivation is performed.  Obviously, it remains the problem to understand how much the models can describe "reality". However, this an issue we leave for future investigations.
    2022-03-25 - 
     A little more on the IUH and looking at the variety of HDSys models

    We introduced previously without very much digging into it the concept of Instantaneous Unit Hydrograph. Here we explain more deeply its properties, Then we observe that there are issues related to the partition of fluxes and we discuss some simple models for obtaining them. Not rocket science here. The concept that we need those tools is more important than the tools themselves. We also observe that linearity is not satisfactory and we give a reference to many non linear models. Finally we discuss an implementation of some of the discussed concepts in the System GEOframe. 
    2022-03-31
    2022-04-07

     Travel Time, Residence Time and Response Time
    • Here below we started a little series of lectures about a statistical way of seeing water movements in catchments. This view has a long history but recently had a closure with the work of Rinaldo, Botter and coworkers. Here it is presented an alternative vie to their concepts. Some passages could be of some difficulty but the gain in understanding the processes of fluxes formation at catchment scale is, in my view, of great value and deserves some effort.  The way of thinking is the following: a) the overall catchments fluxes are the sum of the movements of many small water volumes (molecules); b) the water of molecules can be seen through 3 distributions: the travel time distribution, the residence time distribution and the response time distributions; c) the relationships between these distributions are revealed; d) the relation of these distributions with the the treatment of the catchments made through ordinary differential equations is obtained through the definition of age ranked distributions; e) The theory this developed is a generalizations of the unit hydrograph theory. 
    • The view of the catchment as the statistics of elementary water volumes moving stochastically, a storyboard
    • Travel Time, Residence Times (Vimeo2022)
    2021-04 -08
    Some References (advanced)


    2022-04-08 -  After all radiation moves it all.
    So what ? - Steps in hydrological modelling with integrated distributed models
    2022-05-19 - Seminar by Ing. Mar co Bezzi, Ph.D., Safe Waters 4 Agricolture (Vimeo2022)
    2022-06-08 - Seminar by Ing. Matteo Dall'Amico, Ph.D., How to work with Hydrology and Earth Observations (Vimeo 2022)
    Posted by at No comments:
    Labels:

    The Hydrological Modelling Class 2022 - Index

     Index

    Go to:

    Introduction

     The Hydrological Modeling course aims to teach to simulate the hydrological cycle at various spatial scales in order to be able to adequately manage the water resource and to prevent the risk of floods. The importance of these two issues is widely covered by the EU 2000/60 framework directives or "Water directive" and EU 2006/60, "Flood directive". Based on the hydrological knowledge acquired in the course of Hydrology at the Bachelor of Engineering for the Environment and the Territory, the hydrological processes, analyzed as punctual phenomena are extended to the water catchment areas. 

    Precipitation is analyzed as a measured statistical data, both from ground stations and from remote sensing; the other processes are suitably modeled, as briefly described below. At the end of the course, a student must be able to independently model the flow rates, evaporation and transpiration in a river basin of various sizes, after having delineated it starting from digital terrain data. Of course, the student will have to demonstrate that he has critically understood the concepts that underlie the hydrological modeling presented. 
    The knowledge acquired may be used in the River Engineering course for the design of defense works. Hydrological modeling also introduces concepts that are used in the course of Aqueducts and Sewers for the calculation of stormwater networks. The course is partly useful for the Hydraulic Protection of the Territory course.  A more condensed part of the version of the course can be found @GWS2021.
    An overview of the topics (in Italian) can be found on the seminar done for the District Authority of river Po (here). 

    By Michele Vettorazzi, Environmental Engineer and Photographer


    The lectures of the course will be held in English, according to the methods already followed in the Numerical Modeling course (i.e. with summary in Italian at the beginning of the lesson, lessons in English, questions and explanations in Italian). The first part of the course, until on April 3, will be dedicated to the presentation and discussion of theoretical concepts. The lectures will be recorded and uploaded on the course's YouTube channel (or Vimeo). The lessons will cover 4 of the five hours per week. The fifth hour will be dedicated to the preparation of the data necessary for the projects to be completed in the second part of the course. 

    Students must take care to understand the hydrological concepts and discuss them with the lecturer. The first twenty minutes of each lesson will be devoted to the discussion of the topics covered in the previous lesson and the problems that arose in the preparation of the data (in Italian). Every group had to prepare an appropriate question or comment to which the lecturer will replay. A summary, again in Italian, of the lesson and then the actual lesson will follow. 
    The second part of the course will use the theoretical themes of the first part and using the tools made available by the GEOframe system (https://abouthydrology.blogspot.com/2015/03/jgrass-newage-essentials.html). Students, in groups of two or three, will have to estimate hydrological flows and quantities over a significant period of time and with an hourly time step using a time series of hydro-meteorological data in inputs for period long enough to allow adequate calibration of the models. With the help of the tutor and the reader, students will face problems of missing data, validate the models, discuss and implement an adequate configuration of the GEOframe hydrological system in order to get the hydrological water balance of the basin. 

    The following works by Abera can be taken as an example of the outcomes expected :
    References 

    The lessons will be video recorded and made available. Each lesson will be given through slides in English which will be delivered to students in advance. When necessary, the lessons will be accompanied by appropriate in-depth papers. There is no real text because the course, even in the hydrological tradition, elaborates the concepts in a contemporary way and uses innovative tools.
    As general reference texts we recommend:

    • Beven, K. - Raifall-runoff, the primer, ISBN 10: 047071459X, ISBN 13: 978047071459, Second Edition, Wiley-Blackwell, 2012
    • Dingmann, L., Physical Hydrology, ISBN-13: 978-1478611189, ISBN-10: 1478611189, Third Edition, Waveland Press, 2015
    • Lu, N. and Godt, J.W., Hillslope Hydrology and Stability, Cambridge University Press, ISBN-13: 978-1107021068, ISBN-10: 11070210652010, 2013
    • Bonan, G., Ecological Climatology, concepts and applications, ISBN-13: 978-1107619050, ISBN-10: 110761905X, 2016

    These books represent a shareable review of the phenomena and hydrological modeling but the methods they present are not necessarily those used in the course. The course, cause of time constraints, presents a selected and limited perspective of the subject that the texts cited dissect from various points of view often complementary to the one of the course.

    Posted by at No comments:

    The Hydrology Class 2022 - The Lab Page

    Go to the Introduction/References

    Go to the Foreseen Schedule

    Go to the Software Installation page


    Photo by Michele Vettorazzi (Environmental Engineer and Photographer)


    For the installations Go to the Software installation page, anyway do not miss the occasion to listen 2 Concetta D'Amato introducing it to you in Italian on 2022-03-21:
    2021-02-28 Introduction to working with Jupyter and Notebooks
    2022-03-07 
    2022-03- 22
    2022-03-29
        • Counting the events and producing their empirical statistics (Vimeo2022)
    2022-05-02 - Estimation of  the intensity-duration-frequency curves with Python
    With this class we start the determination of the rainfall extremes in a point. For obtaining this we make use of the PANDAS and the Matplotlib libraries of Python (3.*). The determination of the extremes is a necessary step in any engineering project that involves water flows. Constructions like sewage systems, dams, levees, all require the knowledge of the expected precipitation with an assigned return period.
    Do not forget to have the data of  Pluviometria Paperopoli ready.

    2022-05-09 - Introduction to the GEOframe/OMS3 system
    Geoframe is a system for doing hydrology by computer. It provides the components to model all the compartments of the hydrological cycle with tools that follow various modelling philosophies. The main idea it deploys is that these components can be joined at run time with a scripting language to provide "modelling solutions". During this course we explore some possibilities it offers.

    2022-05-09/10 - Modelling Evapotranspiration with GEOET
    The second part of the lab regards the simulation of evaporation and transpiration. Three models are used, Priestley-Taylor, Penman-Monteith-FAO and the Prospero model in various combination. The estimation of radiation is an important part of the estimate. 

    2022-05-23/24 - Estimating the infiltration with WHETGEO 1D
    To model infiltration we use a 1D implementation of the Richards equation called WHETGEO, an acronym which stands for Water, HEat and Transport in GEOframe. These classes introduces the simulations with WHETGEO 1D step by step
    • A series of Notebooks with planned simulations for your inspiration (please pay attention that the environment geoframe_verona already contains some of the scripts that needed to be uploaded. The Example Notebooks use an obsolete version of the simulations though)
    Posted by at No comments:
    Labels: ,

    Wednesday, February 9, 2022

    The Hydrology Class 2022 - The schedule

    This post contains the foreseen schedule of the course. Material uploaded is subject to modifications prior to the schedule date. All the lectures will be to students present in blended format. Meaning that students can be either present or attend remotely. All of them will be recorded and uploaded to web. 

    Go to the Introduction/References
    Go to the Software Installation page
    Go to the lab page

    Photo by Michele Vettorazzi


    Topics in bold are definitive. Topics in normal characters are still subject to modifications.  All the 2022 videos will be available at this Vimeo Showcase.

    2022-02-21 - Introduction to the course and to hydrology

    Complementary References

    2022-02-22  - Ground based Precipitations and their statistics Separation snow-rainfall - measure of precipitation

    In this part of the class we describe where it rains and how much it rains using statistical concept. One important objective is to understand what are the extreme precipitations for their importance in engineering. 
    2022-02-28 - 2022-03-01  - Statistics of extreme precipitations

     Some reviews on statistics - Return Period
    Extreme precipitations  (Storyboard2020)
    Distributions Storyboard2020

    Determination of Gumbel's parameters
    Extreme precipitations  II
    Beyond Gumbel
    A summary about the extreme precipitation estimations (Whiteboard)

    2022-03- 07 - Water in soil and aquifers. Darcy-Buckingham. Hydraulic conductivity. Soil water retention curves (Storyboard2020)

    Once precipitations arrive to the ground surface they either infiltrate or generate runoff. We first state how they infiltrate and, actually how water behave in the soil and in the ground. We talk about the complexity of the Earth surface that contains life and call it, the Critical Zone. To study infiltration we introduce the Darcy and Richards equations of which we explain the characteristics. 
    2022-03-08
    2022-03-14

    2022 - 03- 15      - The Richards equation  (Storyboard 2020)
    2022-03-21
      2021-03-28 - Runoff Generation and propagation (Summary 2020)

      Once the rainfall gains the terrain, it can infiltrate or producing runoff. In the next we discuss the main mechanisms that produce runoff.
      Q&A - Runoff - Runoff 2022

      2020-03-29 - The surface water propagation (a brief Storyboard 2021)

      Runoff moves on the surface of the terrain according to the de Saint-Venant equation. In the following the equation is derived in the 1D case.
      2022-04-04/05

      Evaporation generalities       (Storyboard2020)

      A consistent part of root zone and surface water evaporates and returns to the atmosphere to eventually form clouds and precipitation again. The process follows quite complicate routes and is different when happening from liquid surfaces, soil or vegetation (and BTW animals).  In this group of lectures we try to figure out the physical mechanisms that act in the process and give some hint on methods to estimate evaporation and transpiration with physically based models. 
       Evaporation and Transpiration Formulas
      2022-04-12 -  After all radiation moves it all
      Posted by at No comments:
      Labels: ,
      Subscribe to: Posts (Atom)