Friday, September 30, 2016

EPAs for water

EPA, in this case, stands for Environmental Protection Agency (of the United States).
We use a couple of models developed there, especially in my class of Hydraulic Constructions.
These products are:

and they are available as open source.

There is  an organization of people gathering who are interested in working on them.  You can join them at:

I do not endorse them as the best models ever. But they are there, opens source, and have a community. I see them as a starting point for injecting new ideas and extensions.

Thursday, September 29, 2016

The hydrologist's toolbox

One is doing hydrology, s/he is a hydrologist, and asks to me how to increase his/her knowledge.
I answered that s/he should be proficient in:

Being specific, and following the above points: 

A hydrologist should then look for classes and courses on the above topics. This blog has a lot of information about the above topics. Please keep an eye on it.  My lectures on hydrology (in Italian) are available here.

Friday, September 23, 2016

The Adige database or the database NewAGE

This is to introduce the Adige database that collects as many possible data about the Adige river.  It is the result  of many years of work (involving Alberto Bellin (GS, RG), myself (GS, RG), Bruno Majone (GS), Francesca Villa, Hydrologis and many others), across various projects the latter of which are the CLIMAWARE project and GLOBAQUA ones . It contains geometries (it is a spatial database), time series, intakes and outakes data. It is a gold mine for whom wants to dig into it.
The presentation is due to Stefano Tasin, one of my master students, and it is in Italian.  However the slides could be understandable also by those who have a knowledge of SQL or want to get it, after some little effort. Clicking on the figure above you access his presentation pdf. Tables in the database are documented here
The database is in sqlite and spatialite, expandable and, I hope to find someone who can legally maintain it and offer it as open data.  The whole DB is contained in a ~1.5 GB file called NewAgeDB.sqlite.

Thursday, September 22, 2016

Cool R - Tydyverse

More or less there was R before Hadley Wickham and after him. Before him it was a solid programming for doing statistical analysis. After him it was also cool.
Reasons are that probably he was able to intercept the best of functional programming^1 and introduce it in R.

His new book R for Data Science promises to be a breakthrough. You can buy it or find its contents by clicking on the image.  From Revolutions:

" In the Tidy Tools Manifesto, he proposes four basic principles for any computer interface for handling data:
  • Reuse existing data structures. 
  • Compose simple functions with the pipe. 
  • Embrace functional programming. 
  • Design for humans. 

Those principles are realized in a new collection of his R packages: the tidyverse. Now, with a simple call to library(tidyverse) (after installing the package from CRAN), you can load a suite of tools to make managing data easier into your R session:
readr, for importing data from files
tibble, a modern iteration on data frames
tidyr, functions to rearrange data for analysis
dplyr, functions to filter, arrange, subset, modify and aggregate data frames

The tidyverse also loads purrr, for functional programming with data, and ggplot2, for data visualization using the grammar of graphics.
Installing the tidyverse package also installs for you (but doesn't automatically load) a raft of other packages to help you work with dates/time, strings, factors (with the new forcats package), and statistical models. It also provides various packages for connecting to remote data sources and data file formats.
Simply put, tidyverse puts a complete suite of modern data-handling tools into your R session, and provides an essential toolbox for any data scientist using R. (Also, it's a lot easier to simply add library(tidyverse) to the top of your script rather than the dozen or solibrary(...) calls previously required!) Hadley regularly updates these packages, and you can easily update them in your R installation using the provided tidyverse_update() function.

For more on tidyverse, check out Hadley's post on the RStudio blog, linked here."

^1 - For a course in functional programming, see here.

Wednesday, September 21, 2016

Italian Hydrology 2016

I am summarising here what I saw that rised my interest in the Italian biennial meeting of Hydraulics, Hydrology and Hydraulic constructions. Obviously I followed just the hydrology section and missed the rest. Therefore I could have not seen some very fundamental in one of the other subdisciplines. Do not blame me !
As a general observation, I have to say that few are still really producing models. Many are using products from others notably: tRibs, WRF-Hydro, SWAT, among the foreign. Cathy, Topkapi-X and GEOtop among genuine Italian products were presented at the Conference. Many young people also work on remote and proximate sensing research, where they exploit the capabilities of the new tools (especially UAVs but also on traditional remote sensing). Some work on statistical and probability models.  Some on eco-hydrology. Many we know they work on groundwater, but almost no abstract was presented on the topic that dominated last century literature. Remote sensing is certainly a great topic but today I will not talk about.
In doing my choices I keep an eye on the three step conceptualisation of learning processes represented in figure. There is stuff that is completely mainstream and has the maximum of attention in these years, other on which interest is growing, and other that foresees the future of the discipline.

The Italian hydrological community, especially if we include the numerous those who live abroad, is pretty alive (at present the WRR, HESS, ADWR, PNAS have Italian Editors). However, did I see material changing the paradigms ? Let's see in my comments below.

So, my favourite (if you click on figures, you are redirected to the slides):


Marco Marani (GS) and coworkers rethink the estreme value concepts, observinfg that Pearson's distributions are obtained as a limit of an infinite number of events. He proposed intermediate distributions, when the number of observations is limited. He, they, called these distributions "metastistical". This is, I think, good and pretty much necessary too. Authors assume that extremes are sampled from iid variables, while others, Jim Smith for instance, think that extreme events are sampled out from a separate population. Is there any method to infer it from data ?

Elena Volpi and coworkers discussed the idea of return period. Her statement is that statistical independence is not a requirement for obtaining the classical equation of return period (post coming soon).
This is inscribed in the old story of stochastic processes used as representation of phenomena that appear highly variable. The field needs some refreshment after the discovery of climate change. Volpi's et al. Has the merit to bring in some novelty. I was intrigued by the separation and relation between return period and waiting time.

In the same subfield I registered attention to copulas, as a means to move from a univariate dominated applications to multivariate.

Strangely not application in machine learning or pattern recognition which could be os some interest when coupled with complex time and spatially varying signals.


Gabriele Manoli (GS) uses a simplified ABL theory to study the effects of vegetation on precipitations, and, in particular he sorted out the effects of vegetation ages. Some about the theory of ABL came from an evolution of thet good old model by John Albertson (GS) but the novelty here is that eco-hydrology enters in the merit of phenology-plants evolution. Conditions in which conclusions are drawn are pretty uniform (probably Durham forest can be considered a nice approximation of it). the problem of heterogeneity calls for a treatment made with a process-based model.

Nadia Ursino and Chiara Callegaro explore the formation of some vegetational patterns made possible by water availability in water-limited environments. 


Tuscia’s guys (Salvatore Grimaldi et al., GS) make a lot f interesting things, but here, I have chose their 100 square meters pluviometer. Details on on it, the paper, slides and poster are below.

Process-based modelling

We’ve got some good work by Giacomo Bertoldi (GS), but he is too close to me for me being neutral. I mention two works. One by Alfonso Senatore (GS) and coworkers. He uses WRF coupled with WRF-Hydro. I am not sure of the contents of the latter. The good and the new could be that there is one model that treats with very detail the interactions with the atmosphere (thanks to WRF) or should. If it is a real thing, I am envy, because it is one thing I believe we have to add to GEOtop.

Monica Piras showed various comparison between process based models. She and co-workers used the model “blindly” without giving direct judgement about the performances of each one. Anyway, it is apparent that models behave differently, and someone should be wrong. Interesting part of her presentaion was the mention to downscaling techniques, necessary to couple climate projections and hydrological models. Below, please find her presentation.

I did not mention travel time theories. We already talk about it extensively. Nothing especially new was presented that it is not already in my previous posts, just some incremental advancement.

Saturday, September 17, 2016

Urban Hydrology and Models

Urban hydrology does not clearly indicate which the topic is. Has it to do with the rainfall that falls with different schedule on cities than elsewhere  or viceversa ? Has it to do with different canopies ? Different partition of fluxes ? Or all of it ?
Cities gave a lot to sciences, they are, partially, the product of science an technology, but technical sciences does not study very much cities, … maybe.  Fletcher et al., (2013) can be a short review of the topic for beginners. 

When hydrologists think to cities often they do not directly think to the built environment, and the way the models has to be adapted to simulate the urban water cycle. 
Fact checking shows that just a few models are present around the world that are concerned with the topic. At least recently and in major journal. An exception is Joshua Cantone and Arthur Schmidt work, who have an approach very similar to mine, and Schmidt is steadily adding material to their first ideas. 

Actually we, as engineers, we are also concerned on how these infrastructures can be designed (for a literature review, see my presentation here). As my students knows, I try to use a geomorphological approach to the problem, and I developed a model called Trento_p to design culverts and the relates sewage.  I also wrote a couple of paper on it (in Italian). However, I grow a sort of  dissatisfaction or not being able to deal with the whole set of measures that constitute the center of the modern view that aims to develop sustainable cities
In this new context, new tools are necessary. The more used tool in recent context (almost the only one which seems referenced) is an old tool, called SWMM (its site here) written and promoted by EPA researchers. It is open source (but better see EPA's Github site or the Open Water Analytics Initiative), and exists from long time. Therefore it is pretty well documented (see below), and supported by many video tutorials.

First impression is that its hydrology is really outdated. However, since it has many features that I like, in a pragmatic perspective, I could  use it with students, while my groups, on the ashes of the old Trento_p, builds a new model.

An update (2018-01-18)

Recently I came across the work of Aditi Bhaskar (GS), which addressed the urban hydrology issues "from below", meaning considering the effects on urban groundwater. The perspective is very interesting and complementary to the surface waters one.

Another update(2019-10-10)

Interesting is the approach used by this project, called Virtual Water, which deals with the applications of virtual reality to Urban Drainage Systems.


Bhaskar, A.S. and C. Welty (2012), Water Balances along an Urban-to-Rural Gradient of Metropolitan Baltimore, 2001-2009, Environmental and Engineering Geoscience. 18(1), 37-50. doi: 10.2113/gseegeosci.18.1.37.

Bhaskar, A.S., C. Welty, R.M. Maxwell, A.J. Miller (2015), Untangling the effects of urban development on subsurface storage in Baltimore, Water Resources Research, doi: 10.1002/2014WR016039.

Bhaskar, A.S. and C. Welty (2015), Analysis of subsurface storage and streamflow generation in urban watersheds, Water Resources Research, doi: 10.1002/2014WR015607.

Bhaskar, A.S., L. Beesley, M.J. Burns, T.D. Fletcher, P. Hamel, C.E. Oldham, and A.H. Roy (2016), Will it rise or will it fall? Managing the diverse effects of urbanization on base flow, Freshwater Science, 35(1), 293-310, doi: 10.1086/685084.

AS Bhaskar, DM Hogan, SA Archfield, Urban base flow with low impact development, Hydrological Processes 30 (18), 3156-3171

D. Tamanini, A.B. Esmail, F. Zanotti, S. Simoni, P. Bertola, R. Rigon (2009). Trento_p : un modello geomorfologico per lo studio del drenaggio urbano. L'ACQUA, vol. 2009, p. 73-74, ISSN: 1125-1255

Fletcher T.D., Andrieu  H., Hamel P., Understanding, management and modelling of urban hydrology and its consequences for receiving waters: A state of the art, Advances in Water Resources 51 (2013) 261–279

Kexuan Wang (advisor, A.Schmidt) , Hydrologic response of sustainable urban drainage to different climate scenario, M.Sc. Thesis, 2015

Morales, V. M., Quijano, J. C., Schmidt, A., & Garcia, M. H. (2016). Innovative framework to simulate the fate and transport of non‐conservative constituents in urban combined‐sewer catchments. Water Resources Res., 1–53.

Pathirana, A,  Introduction to EPA-SWMM, presentation

Rigon, R., Bertola, P. - La progettazione con un metodo geomorfologico delle reti di drenaggio urbane, II Conferenza Nazionale sul Drenaggio Urbano, Palermo, 10-12 maggio 2000

Sanzana, P., Gironás, J., Braud, I., Branger, F., Rodriguez, F., Vargas, X., et al. (2017). A GIS-based urban and peri-urban landscape representation toolbox for hydrological distributed modeling. Environmental Modelling and Software, 91, 168–185.

Wang, A., Park, S., Huang, S., and Schmidt, A. (2015) Hydrologic Response of Sustainable Urban Drainage to Different Climate Scenarios. World Environmental and Water Resources Congress 2015: pp. 312-321.doi: 10.1061/9780784479162.030

Wednesday, September 7, 2016

On " How to make our models more physically-based"

These reflections came after having read the discussion paper of the same title by Savenije and Hrachowitz (S&H) on HESSD, and I offer them to your own thinking (their paper was very successful in rising my interest, then). So first, read the opinion paper.

The paper has some (a ?) very good point:

In brief, our hydrological system is alive and has a strong capacity to adjust itself to prevailing and changing environmental conditions. Although most physically based models take Newtonian  theory at heart, as best they can, what they generally miss is Darwinian theory on how an ecosystem evolves and adjusts its environment to maintain crucial hydrological functions. If this active agent is not reflected in our models, then they miss essential physics.

However, let me divagate on their concepts and ideas, and debate first, the concept of what Physics is (related to modern Hydrology). In the wide, general sense, Physics is the study of nature, however, it has codified during the last centuries, since Galilei, as a science that uses experiments to validate (I know the danger in using this verb) some theoretical issue about the behavior of some phenomena. One key aspect of Physics is the the word experiment, which means that we have something to measure (a physical quantity) and tools to do it (instruments). Repetition of experiments and confirmation of outcomes, and establishing (mathematical) relations among quantities, brings to laws, Physical laws. 
I am aware that each one of the words in the paragraph above would require a book to be dissected, analysed in its historical development, and in fact this was done.
Physics and or “Physical Sciences” have evolved to specialise people. Someones are inclined to work on the theory, others to design experiments. Theory, in turn, means that there is some formal (meaning following unambiguous rules to process statements and precise definitions) language that expresses relations among things, quantities, the latter being closely related to what is measurable (i.e. to the ability to build tools to detect something). The entanglement between theory and measures (or the possibility to do some measurement, even in a “virtual” or “thought” way, or with tools that are not existing but can be conceived) is inextricable. 

So when Peter Eagleson  claim for hydrology as a (separate) physical science, I believe he meant that  there was the technology for implementing measures and a “corpus” of “mathematics” to be able to process and forecast hydrological facts, and doing it properly. 

During its history, Physics has changed and enormously expanded its field of interest. Galilei started with the motion laws, and continued with planets, attitude that Newton brought to a first completion. Optics came in, then electricity, electromagnetism, quantum mechanics, quantum electrodynamics, chemistry, thermodynamics, to name a few areas. 

The lighthouse, were always the “mathematical” approach in describing the world, and repeatable experiments to confirm the findings, and development of mathematics and of measurements techniques went along with it.  Besides,  another reference were conservation laws: mass, momenta, and energy (charge, and so on). 

So we can say that hydrology is a physical science, for instance,  also because it uses mass conservation law. In this sense, any of models presented in the paper by S&H are physical.  However,  a judgment on their assertions can be obtained by asking which is their “mathematics” and what do they measure.  

To be fully respectable, we would expect that authors of a physical model were concerned also with momentum conservation and energy conservation, but actually, if we would add this requirement, almost no hydrological model would be a physical model^1.  

If we accept the beginning phrase, according to its Authors thinking, we should become more physical being more “Darwinian” too. However, Darwin was, as everybody knows,  a biologist^2, or better, maybe, a "natural scientist".  I believe the Authors are right when they claim it but Schroedinger, in his “What is life”, was more concretely a physicist facing biology and his claiming that the approach of Physics could be applied also to life offers a strong counterexample that the Physics approach can be applied to Natural sciences, maybe pushing beyond the present limits the actual science. His work was inspirational for many and can be inspirational even now for doing research in hydrology. With its obsolescence, it is a gigantic conceptual contribution, and I suggest my students to read it, to get the fundamentals of what arguing about Physics is.

Going to some detail, S&H seem to claim that the hydrosphere obeys a homeostatic behavior, which seems to self-regulate and affect earth as much as possible, to maintain the conditions that sustain life. This is reminiscent of the Gaia hypotheses^3.  The difference in scale from hillslope hydrology to the global earth should be taken into account though.  
Furthermore, I think we have some realistic hints that the system (Gaia) can be broken, and therefore, I believe, that the evolutionary conditions that the many claim to be a possible guide to build new paradigms of models are valid under the assumption of a certain degree of equilibrium, that is far from being evident in this climate change era. In other words, the hydrological cycle and the ecosystems could be out of balance.
In any case, homeostasis can exist if feedbacks, which physical models should be able to capture, exist. Where is the mathematics for doing it ? I am not sure that arguing by adding reservoirs could be enough to capture intertwined behaviors, but, I admit, they are a starting point (I already kind of wrote it).  Frankly, I think the way inaugurated by Ruddel and Kumar is much more visionary and we should push on that side of research, instead that sticking only with a trial and error (remove and put) guided by uncertain measurements and weak modeling abilities. IMO, future is in network mathematics not reservoirs assemblage.  

S&H cite Aristotele, and they have good reason for doing it. Our hydrology is, maybe,  a physical science but often we can just observe the phenomena, not control them, as Galileian experiments would require. So we are lame in our trials, and immersed in dim light, not exposed to the full splendor of the Knowledge. Statistics is necessary to to disentangle measures and observations. Causal relations are often less than obvious, and as all we know, "correlation does not mean causation". All this fuzzines makes the matter prone to exciting but ineffective narrative  that usually starts with the word “holistic” (BTW a word I like too) and continues by saying that “the whole is more than its parts” but  most of the time does not continue with a proper fomalisation of what  "holistic" is and how this damn “whole” can happen. So maybe, he was very wise Galilei when he said to his Aristotelian antagonists “Io stimo più il trovar un vero, benché di cosa leggiera, che 'l disputar lungamente delle massime questioni senza conseguir verità nissuna. ” (I like more to find a truth in a small subject than discuss long time of general subjects without obtaining anything”)

I think that an interesting working hypothesis is that "the whole is the sum of its parts and the interactions among the parts", and that part of the quality of the system, seen as a whole, derives from parts' interactions and feedbacks. A system is itself a quite unidentified entity, and its definition is certainly recursive, meaning that, most of the time, a system is a system of systems, and reality is “stratified”.  But having a "basic system" at some scale should be feasible.

Finally, where is falsification ? Falsification is certainly a characteristic of a scientific enterprise, and therefore of Physics. Certain theories seem to me missing of the precision needed to obtain a proper falsification, and S&H should be more convincing on that side. On the contrary, Pete Eagleason book, Dynamic Hydrology, probably the best book ever in Hydrology, after fourty five years is “all wrong”, the right sense of wrongness. That’s science, that’s Physics !

P.S. - Another attitude I would not spread (S&H are affected) is to see remote sensing as an “oracle” which gives the right answer without paying debts to uncertainty and unknownness. I already wrote about. 


^ GEOtop is one of the few notable exceptions.

^2 - I remind an ironic slides by Per Bak with written: "is biology too difficult for biologists ?" Here the worth of retaliation hits.

^3 About the Gaia hypothesis an eminent colleague said “ … Are they testable ? Are they useful ?“ (IMHO:  useful, they were …)