Thursday, August 30, 2012

Sandro Marani 1936-2012

Sandro passed away last sunday for an heart attack. He was 76. As many knows, I due to him my being here in this field.
 After my master in Physics I had a grant for working with him to "Statistical Models of the Quality of the Atmosphere of Venice Lagoon". Apparently far from Hydrology. But not in the mind of Sandro who looked at the diffusion processes from a geometrical point of view. He believed,  after reading Mandelbrot works, that diffusion patterns could be understood better with fractal geometry. However, to understand these patterns we had to visualise them: and what better than rivers networks ? In fact,  river networks could have been thought, in his view, as a "reverse diffusion" pattern from which we could learn a lot. From diffusion patterns therefore we moved to study hydro-geomorphology, and published together with Andrea Rinaldo the paper: A note on fractal channel networks. Others arrived a little before us on the subject (e.g. Tarboton et al., 1988, and La Barbera and Rosso, 1989)  but Sandro path was quite independent, and the "discover" that the flow-path distances (i.e. the width function) have a multifractal statistical structure was ours.
He was always intrigued with geometry being the inner explanation of many phenomena and therefore his interested in the geomorphic unit hydrograph (Rodriguez-Iturbe and Valdes, 1979, Gupta et al., 1980) was natural. He saw in it both the mathematical way to fit geometry into equations, and as a theory suitable to generalisations for coupling water flow and nutrients transport (and diffusion). His work on the Mass Response Function with Andrea Rinaldo (Rinaldo et al,1988) was decades in advance with respect to the interest that eventually was raised on the topic.  I believe that also Geomorphological dispersion was quite an achievement that can be listed in the "gemoetrical" effort. That work reflected the idea that, at catchment scale, geometry is as much or more important than flow dynamics to produce the form of the hydrograph.

He was a man of innumerables ideas and initiatives. Always in advance of times (maybe too in advance). The School of Environmental Dynamics at IVSLA founded with Andrea Rinaldo was the field where many of us met with science not simply with hydrology. Any edition had an eye to new insights and paradigms (the recent editions were organized by his son Marco, and maintain the same standards and vision). How much I miss those times!
In his effort to promote modelling, he  around the end of the eighties organized a "modelling connection" among environmental scientists, hydrologist , geophysicists, economists, urban planners. The most exciting guys of the Universities of Venice who met for talking about quantitative modeling.
Some colleagues choose the particulars or the details of a discipline: he chose to look at the whole ! Ideas, ideas, ideas. That he was.

As everybody can realize, my recent work has been strongly influenced by working with him. What else was building JGrass, if not giving body to Sandro's vision about processes representation and  the idea that spatial explicit modeling  was necessary for any environmental problem? What else is my commitment with modeling frameworks like OMS3?  What else is my recent involvement with thermodynamics?

He used to say: "Models are wrong ? (Sbaiemo coi modei ?) We do mistake by doing models. (Sbaiemo.) But let's imagine without! (Ma figuresemose sensa!). We should then rely just on qualitative arguments, of ignorant people, based on unformalised belief ?"

Sandro I'll miss you.

Thursday, August 23, 2012

Free Italian Cartographic Data on the Web

I received from Madi Di Leo  the news that a lot of Italian cartographic data are available at the geoSDI web site. The data that can be found at:

http://www.geosdi.org/index.php/it/blog/item/geoportale-dati-cartografici

Data contain at the moment 2500 layers of information and are free. There are also a couple of videos to let people to know how to use the web site. Unfortunately, for foreign,  all information is in Italian, so far.

Prendo da JGrassTechTips il link del Repertorio Nazionale dei Dati Territoriali (RNDT). Attualmente questo strumento sta muovendo i primi passi, leggi l'articolo su TANTO.

Wednesday, August 22, 2012

R resources for Hydrologists

R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, many of them became more proficient than me in the field. Now it seems natural to use it for everything, but this was not always the case. A list of introductory material is here.

A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.

Recently many resources were added for hydrologists, which I list below with a little comment.

New:  a second paper by distinguished colleagues is also on HESSD: Hydrology modelling R packages: a unified analysis of models and practicalities from a user perspective, by Astagnau et al.

New: Now we also have a very informative paper at HESSD: "Using R in hydrology: a review of recent developments and future directions" by Slate et al., 2019. It is a pity they do not cite this blog, but we cannot pretend too much!

  • OCNet  OCNet enables the creation and analysis of Optimal Channel Networks (OCNs). These are oriented spanning trees (built on rectangular lattices made up of square pixels) that reproduce all scaling features characteristic of real, natural river networks. 
  • DonwscaleR is an R package for empirical-statistical downscaling focusing on daily data and covering the most popular approaches and techniques (quantile mapping, analogs, regression, generalized regression, neural networks). This package has been conceived to work in the framework of both seasonal forecasting and climate change studies and is part of the climate4R framework, formed byloadeRtransformeRdownscaleR and visualizeR.
  • bigleaf (version 0.6.5), an open source toolset for the derivation of meteorological, aerodynamic, and physiological ecosystem properties from eddy covariance (EC) flux observations and concurrent meteorological measurements.
  • rtop is a package for geostatistical interpolation of data with irregular spatial support such as runoff related data or data from administrative units.
  • The boussinesq package is a collection of functions implementing the one-dimensional Boussinesq Equation (ground-water).
  • dynatopmodel is a native R implementation and enhancement of the Dynamic TOPMODEL, Beven and Freers’ (2001) extension to the semi-distributed hydrological model TOPMODEL (Beven and Kirkby, 1979).
    RclimTool. It was designed with the aim of facilitating users in statistical analysis for quality control, filling of missing data, homogeneity analysis and calculation of indicators for daily weather series for temperature (maximum and minimum) and precipitation.
  • LumpR a tool for HRU delineations (the reference paper is here).
  • Cropwat FAO model (it is a decision support system developed by the Land and Water Development Division of FAO for planning and management of irrigation) translated into R.
  •  Two socio-hydrological models of human-flood interactions that were recently developed. The two models are similar, but based on different socio-economic variables (wealth and human proximity to rivers VS relative population density). A full description of the two models can be found in the following two papers: HESS (2013), https://www.hydrol-earth-syst-sci.net/17/3295/2013/hess-17-3295-2013.pdf; WRR (2015), http://onlinelibrary.wiley.com/doi/10.1002/2014WR016416/full 
  • RWater: A Cyber-enabled Data-driven Tool for Enhancing Hydrology Education
  • airGR: Hydrological modelling tools developed at Irstea-Antony (HBAN Research Unit, France). The package includes several conceptual rainfall-runoff models , a snowmelt module and the associated functions for their calibration and evaluation
  • airGRteaching:“It is an add-on package to the 'airGR' package that simplifies its use and is aimed at being used for teaching hydrology. The package provides 1) three functions that allow to complete very simply a hydrological modelling exercise 2) plotting functions to help students to explore observed data and to interpret the results of calibration and simulation of the GR ('Génie rural') models 3) a 'Shiny' graphical interface that allows for displaying the impact of model parameters on hydrographs and models internal variables.“
  • airGRdatassim: it allows performing streamflow data assimilation in the GR rainfall-runoff models with EnKF and particle filter. 
  • lumpR. A tool facilitating landscape discretisation for hillslope-based hydrological models. It is described in a paper on GMDD
  • HydroGOF and HydroTSM by Mauricio Zambrano-Bigiarini.  The first provides functions implementing both statistical and graphical goodnes-of-fit measures between observed and simulated values, mainly oriented to be used during the calibration, validation, and application of hydrological models. The second provides functions for management, analysis, interpolation and plotting of time series used in hydrology and related environmental sciences.  Mauricio also had a poster at EGU 2010 general assembly on the topic.
  • Jasper Vrugt's DREAM calibration method
  • RMWAGEN by Emanuele Cordano which is a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
  • Other stochastic generators of precipitation can be found here. Do not forget to explore the links in that page, and particularly the presentations given at the Roscoff's Workshop on stochastic generators, where many examples are in R
  • The RHydro which included TOPMODEL (apparently not anymore supported), tools for DEM analysis (this last type of tools however are also available through the work by R. Bivand, E.J. Pebesma and V. Gomez-Rubio ), an implementation of the FUSE by Clark et al (2008) methodology, and many other tools for hydrological analysis. These were initially promoted by Wouter Buytaert and Dominik Reusser who also gave a nice tutorial at EGU a few years ago.
  • Hydromad: It provides a modelling framework for environmental hydrology: water balance accounting and flow routing in spatially aggregated catchments. It supports simulation, estimation, assessment and visualisation of flow response to time series of rainfall and other drivers
  • TUWmodel  is a lumped conceptual rainfall-runoff model, following the structure of the HBV model. The model runs on a daily time step and consists of a snow routine, a soil moisture routine and a flow routing routine. See Parajka, J., R. Merz, G. Bloeschl (2007) Uncertainty and multiple objective calibration in regional water balance modelling: case study in 320 Austrian catchments, Hydrological Processes, 21, 435-446
  • Sean Turner's and Stefano Galelli's, reservoir package, Tools for Analysis, Design, and Operation of Water Supply Storages. Sean Turners's also buildes Scenario 
  • Code: Handy routines for Hydrologists by Dan Moore and others.
  • Hydrosanity: It provides a graphical user interface for exploring hydrological time series. It is designed to work with catchment surface hydrology data (mainly rainfall and streamflow time series at a set of locations). There are functions to import from a database or files; summarise and visualise the dataset in various ways; estimate areal rainfall; fill gaps in rainfall data; and estimate the rainfall-runoff relationship. Probably the most useful features are the interactive graphical displays of a spatial set of time series. (This project seems actually being abandoned). 
  • aqp: Algorithms for quantitative pedology. A collection of algorithms related to modeling of soil resources, soil classification, soil profile aggregation, and visualization by Dylan Beaudette and Pierre Roudier. A paper talking about it is given here. And a presentation is not missing.
  • A package for plotting soil water retention curves and hydraulic conductivity by Emanuele Cordano, Fabio Zottele and Daniele Andreis is soilwater.
  • soilDB, of the same authors of aqp, is useful to access some soil databases.
  • soiltexture: Functions for soil texture plot, classification and transformation by Jules Moeys
  • Hydrome: This package estimates the parameters in infiltration and water retention models by curve-fitting method.
  • SoilWater address to a couple of packages for estimating Soil Water Retention Curves and some Pedotransfer Functions
  • hydropso: This package implements a state-of-the-art version of the Particle Swarm Optimisation (PSO) algorithm, with a special focus on the calibration of environmental models.
  • Evapotranpiration: by Dan Lu Guo and Seth Westra. This package estimates Potential and Actual Evapotranspiration with multiple models (see also the paper here).
  • EcoHydRology developed by DR. Fuka, MT WalterJA Archibald,  TS Steenhuis, and ZM Easton which presents a community modeling foundation for Eco-Hydrology. 
  • Claudia Vitolo's Curve Number (Curve Number!) and other R stuff, including some tools for data discovery. Claudia also manages a Google+ group, R4Hydrology.
  • nsRFA:  this is collection of statistical tools for objective (non-supervised) applications of the Regional Frequency Analysis methods in hydrology made by Alberto Viglione. The package refers to the index-value method and, more precisely, helps the hydrologist to: (1) regionalize the index-value; (2) form homogeneous regions with similar growth curves; (3) fit distribution functions to the empirical regional growth curves. 
  • Wasim: Helpful tools for data processing and visualisation of results of the hydrological model WASIM-ETH.
  • Geotopbricks by Emanuele Cordano, analyses raster maps and other information as input/output files from the Hydrological Distributed Model GEOtop
  • hddtools by Claudia Vitolo is a tool for hydrological data discovery.
  • waterData is a USGS Package for Retrieval, Analysis, and Anomaly Calculation of Daily Hydrologic Time Series Data
  • Lmoments and Lmomco are two packages for the estimation of the L-moments of a distribution.
  • The SPEI R Package by Santiago Begueria which includes a set of functions for computing potential evapotranspiration and several widely used drought indices including the Standardized Precipitation-Evapotranspiration Index (SPEI). 
  • The USGS-R packages at github
  • Alessio Pugliese and Attilio Castellarin pREC: a package for the regionalisation of some hydrological variables.
  • Alberto Montanari version of Hymod: here.
  • Emanuele Cordano work in connecting R with JGrasstools, here, to do geomorphological analysis (slides in Italian here, and in English here) within R.
  • meteo package by Kilibarda, Sekulic, Hengl, Pebesma and Graeler, A package for spatio-temporal geostatistical mapping of meteorological data. Global spatio-temporal models calculated using publicly available data are stored in package.
  • The reservoir package by Sean Turner [aut, cre], Jia Yi Ng [aut], Stefano Galelli [aut]. It measures single-storage water supply system performance using resilience, reliability, and vulnerability metrics; assess storage-yield-reliability relationships; determine no-fail storage with sequent peak analysis; optimize release decisions for water supply, hydropower, and multi-objective reservoirs using deterministic and stochastic dynamic programming; evaluate inflow persistence using the Hurst coefficient. A companion paper for this tool is available.
  • The scripts and data I use in my short mini-classes on R inside my Hydrology and Hydraulic Constructions classes. The include a quick introduction to R, plotting a treating a time serie of rainfall data read from a file, reading and plotting discharges data from a file, estimating the idf curves of rainfall from a standard (Italian standard) set of maxima of annual precipitation, with interpolation of Gumbel probabilities with various methods.
  • Anne Van Loon Drougths R code
Blogs about R and Hydrology
Time series Analysis (among the many others):
  • The TSA package contains R functions and datasets detailed in the book "Time Series Analysis with Applications in R (second edition)" by Jonathan Cryer and Kung-Sik Chan
A miscellany of topics:
In R there are also many toolkit for the analysis of extreme events. The basic is:
but also:
can be useful.
Frankly I did not test them all: but usually CRAN packages are really good.

Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:

  • Geostatistics: gstat, automap, geoR, fields, RandomFields
  • GIS: spgrass6, RSAGA, rgdal, sp, proj4, raster, mapproj, maptools, RGoogleMaps, RArcInfo, RpyGeo,
  • Flood frequency: POT, evd, nsRFA, extremes, lmomco
  • Optimization: pso, DEoptim,
  • High Performance Computing: parallel, snowfall, multicore, jit, nws, Rmpi, snow,  taskPR
  • Spreadsheets & DB: RPostgreSQL, RMySQL, RSQLite, RNetCDF, RexcelInstaller, xlsReadWrite
  • Bayesian statistics: BAS, BLR, ensembleBMA, evdbayes, LearnBayes,
  • ramps, spBayes,...
  • Latex: xtable, Sweave
  • Wavelets: wavelets; wavethresh, wmtsa, Rwave
  • Data Mining: Rweka, rattle, party, RandomForest, ...
  • Machine Learning tools (in Java with R connection)
R for hydrology is a blog dedicated to activities related to hydrology performed by using R.

If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development. 

Wednesday, August 8, 2012

GEOtop Fishing Days

With the aim to help people to run GEOtop and understand a little more its code, and having a preview of what its port to C++ means, I and Mountain-eering organized a three days workshop with a few people interested. This is possible a seed for workshops for a larger audience.
The speaker was Matteo Dall'Amico, a former Ph.D. student of mine, and at present one of the owners of Mountain-eering.
The workshop  provided to participants the key information necessary to disentangle the complexity of GEOtop's use from the scratch: from its installation on the different platforms (Linux, Windows, Mac), including its compilation with the gnu compiler,  understanding its discretization scheme and the input data,  running some template example, working with SVN.  There was also some little theory, especially about snow (but you can refer also to an older post for the theory and some is also contained in the GEOtop History- Part I and Part II).
The slides of the three days can be found here.  However, they do not contain the full information, and should be used as an index of the thing to look for becoming really GEOtop proficient.

Monday, August 6, 2012

CUAHSI youtube and the recent conferences at the biennial meeting

CUAHSI is the consortium of Universities for the advancement of hydrological sciences (my university is a member). Every two years they have a very interesting meeting. I was invited to participate to the 2008 meeting, and I could not attend in 2010 and 2012. However, the plenary sessions are now on-line thanks to CUAHSIyoutube.  This year the main talks were by
Roger Pielke (Policy Relevant Science)
Tom Dunne  (The Evolution of Floodplain complexity)
Soroosh Sorooshian (Hydrological Forecasting over time and space scales)

Have a nice listening to.

Saturday, August 4, 2012

Utilizing online resources for climatic research

Mamhud, continues his excellent job here in presenting free data. Moreover he used his findings for driving the model SWAT in this other post. I use different models but the indications are anyway quite useful.
This intervetion is the second one. The first can be found here.