R resources for Hydrologists

R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, many of them became more proficient than me in the field. Now it seems natural to use it for everything, but this was not always the case. A list of introductory material is here.

A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.

Recently many resources were added for hydrologists, which I list below with a little comment.

New:  a second paper by distinguished colleagues is also on HESSD: Hydrology modelling R packages: a unified analysis of models and practicalities from a user perspective, by Astagnau et al.

New: Now we also have a very informative paper at HESSD: "Using R in hydrology: a review of recent developments and future directions" by Slate et al., 2019. It is a pity they do not cite this blog, but we cannot pretend too much!

• OCNet  OCNet enables the creation and analysis of Optimal Channel Networks (OCNs). These are oriented spanning trees (built on rectangular lattices made up of square pixels) that reproduce all scaling features characteristic of real, natural river networks. 
• DonwscaleR is an R package for empirical-statistical downscaling focusing on daily data and covering the most popular approaches and techniques (quantile mapping, analogs, regression, generalized regression, neural networks). This package has been conceived to work in the framework of both seasonal forecasting and climate change studies and is part of the climate4R framework, formed byloadeR, transformeR, downscaleR and visualizeR.
• bigleaf (version 0.6.5), an open source toolset for the derivation of meteorological, aerodynamic, and physiological ecosystem properties from eddy covariance (EC) flux observations and concurrent meteorological measurements.
• rtop is a package for geostatistical interpolation of data with irregular spatial support such as runoff related data or data from administrative units.
• The boussinesq package is a collection of functions implementing the one-dimensional Boussinesq Equation (ground-water).
• dynatopmodel is a native R implementation and enhancement of the Dynamic TOPMODEL, Beven and Freers’ (2001) extension to the semi-distributed hydrological model TOPMODEL (Beven and Kirkby, 1979).
  RclimTool. It was designed with the aim of facilitating users in statistical analysis for quality control, filling of missing data, homogeneity analysis and calculation of indicators for daily weather series for temperature (maximum and minimum) and precipitation.
• LumpR a tool for HRU delineations (the reference paper is here).
• Cropwat FAO model (it is a decision support system developed by the Land and Water Development Division of FAO for planning and management of irrigation) translated into R.
•  Two socio-hydrological models of human-flood interactions that were recently developed. The two models are similar, but based on different socio-economic variables (wealth and human proximity to rivers VS relative population density). A full description of the two models can be found in the following two papers: HESS (2013), https://www.hydrol-earth-syst-sci.net/17/3295/2013/hess-17-3295-2013.pdf; WRR (2015), http://onlinelibrary.wiley.com/doi/10.1002/2014WR016416/full 
• RWater: A Cyber-enabled Data-driven Tool for Enhancing Hydrology Education
• airGR: Hydrological modelling tools developed at Irstea-Antony (HBAN Research Unit, France). The package includes several conceptual rainfall-runoff models , a snowmelt module and the associated functions for their calibration and evaluation
• airGRteaching:“It is an add-on package to the 'airGR' package that simplifies its use and is aimed at being used for teaching hydrology. The package provides 1) three functions that allow to complete very simply a hydrological modelling exercise 2) plotting functions to help students to explore observed data and to interpret the results of calibration and simulation of the GR ('Génie rural') models 3) a 'Shiny' graphical interface that allows for displaying the impact of model parameters on hydrographs and models internal variables.“
• airGRdatassim: it allows performing streamflow data assimilation in the GR rainfall-runoff models with EnKF and particle filter. 
• lumpR. A tool facilitating landscape discretisation for hillslope-based hydrological models. It is described in a paper on GMDD
• HydroGOF and HydroTSM by Mauricio Zambrano-Bigiarini.  The first provides functions implementing both statistical and graphical goodnes-of-fit measures between observed and simulated values, mainly oriented to be used during the calibration, validation, and application of hydrological models. The second provides functions for management, analysis, interpolation and plotting of time series used in hydrology and related environmental sciences.  Mauricio also had a poster at EGU 2010 general assembly on the topic.
• Jasper Vrugt's DREAM calibration method
• RMWAGEN by Emanuele Cordano which is a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
• Other stochastic generators of precipitation can be found here. Do not forget to explore the links in that page, and particularly the presentations given at the Roscoff's Workshop on stochastic generators, where many examples are in R
• The RHydro which included TOPMODEL (apparently not anymore supported), tools for DEM analysis (this last type of tools however are also available through the work by R. Bivand, E.J. Pebesma and V. Gomez-Rubio ), an implementation of the FUSE by Clark et al (2008) methodology, and many other tools for hydrological analysis. These were initially promoted by Wouter Buytaert and Dominik Reusser who also gave a nice tutorial at EGU a few years ago.
• Hydromad: It provides a modelling framework for environmental hydrology: water balance accounting and flow routing in spatially aggregated catchments. It supports simulation, estimation, assessment and visualisation of flow response to time series of rainfall and other drivers
• TUWmodel  is a lumped conceptual rainfall-runoff model, following the structure of the HBV model. The model runs on a daily time step and consists of a snow routine, a soil moisture routine and a flow routing routine. See Parajka, J., R. Merz, G. Bloeschl (2007) Uncertainty and multiple objective calibration in regional water balance modelling: case study in 320 Austrian catchments, Hydrological Processes, 21, 435-446
• Sean Turner's and Stefano Galelli's, reservoir package, Tools for Analysis, Design, and Operation of Water Supply Storages. Sean Turners's also buildes Scenario 

• R Code: Handy routines for Hydrologists by Dan Moore and others.
• Hydrosanity: It provides a graphical user interface for exploring hydrological time series. It is designed to work with catchment surface hydrology data (mainly rainfall and streamflow time series at a set of locations). There are functions to import from a database or files; summarise and visualise the dataset in various ways; estimate areal rainfall; fill gaps in rainfall data; and estimate the rainfall-runoff relationship. Probably the most useful features are the interactive graphical displays of a spatial set of time series. (This project seems actually being abandoned). 
• aqp: Algorithms for quantitative pedology. A collection of algorithms related to modeling of soil resources, soil classification, soil profile aggregation, and visualization by Dylan Beaudette and Pierre Roudier. A paper talking about it is given here. And a presentation is not missing.
• A package for plotting soil water retention curves and hydraulic conductivity by Emanuele Cordano, Fabio Zottele and Daniele Andreis is soilwater.
• soilDB, of the same authors of aqp, is useful to access some soil databases.
• soiltexture: Functions for soil texture plot, classification and transformation by Jules Moeys
• Hydrome: This package estimates the parameters in infiltration and water retention models by curve-fitting method.
• SoilWater address to a couple of packages for estimating Soil Water Retention Curves and some Pedotransfer Functions
• hydropso: This package implements a state-of-the-art version of the Particle Swarm Optimisation (PSO) algorithm, with a special focus on the calibration of environmental models.
• Evapotranpiration: by Dan Lu Guo and Seth Westra. This package estimates Potential and Actual Evapotranspiration with multiple models (see also the paper here).
• EcoHydRology developed by DR. Fuka, MT Walter, JA Archibald,  TS Steenhuis, and ZM Easton which presents a community modeling foundation for Eco-Hydrology. 
• Claudia Vitolo's Curve Number (Curve Number!) and other R stuff, including some tools for data discovery. Claudia also manages a Google+ group, R4Hydrology.
• nsRFA:  this is collection of statistical tools for objective (non-supervised) applications of the Regional Frequency Analysis methods in hydrology made by Alberto Viglione. The package refers to the index-value method and, more precisely, helps the hydrologist to: (1) regionalize the index-value; (2) form homogeneous regions with similar growth curves; (3) fit distribution functions to the empirical regional growth curves. 
• Wasim: Helpful tools for data processing and visualisation of results of the hydrological model WASIM-ETH.
• Geotopbricks by Emanuele Cordano, analyses raster maps and other information as input/output files from the Hydrological Distributed Model GEOtop
• hddtools by Claudia Vitolo is a tool for hydrological data discovery.
• waterData is a USGS Package for Retrieval, Analysis, and Anomaly Calculation of Daily Hydrologic Time Series Data
• Lmoments and Lmomco are two packages for the estimation of the L-moments of a distribution.
• The SPEI R Package by Santiago Begueria which includes a set of functions for computing potential evapotranspiration and several widely used drought indices including the Standardized Precipitation-Evapotranspiration Index (SPEI). 
• The USGS-R packages at github
• Alessio Pugliese and Attilio Castellarin pREC: a package for the regionalisation of some hydrological variables.
• Alberto Montanari version of Hymod: here.
• Emanuele Cordano work in connecting R with JGrasstools, here, to do geomorphological analysis (slides in Italian here, and in English here) within R.
• meteo package by Kilibarda, Sekulic, Hengl, Pebesma and Graeler, A package for spatio-temporal geostatistical mapping of meteorological data. Global spatio-temporal models calculated using publicly available data are stored in package.
• The reservoir package by Sean Turner [aut, cre], Jia Yi Ng [aut], Stefano Galelli [aut]. It measures single-storage water supply system performance using resilience, reliability, and vulnerability metrics; assess storage-yield-reliability relationships; determine no-fail storage with sequent peak analysis; optimize release decisions for water supply, hydropower, and multi-objective reservoirs using deterministic and stochastic dynamic programming; evaluate inflow persistence using the Hurst coefficient. A companion paper for this tool is available.
• The scripts and data I use in my short mini-classes on R inside my Hydrology and Hydraulic Constructions classes. The include a quick introduction to R, plotting a treating a time serie of rainfall data read from a file, reading and plotting discharges data from a file, estimating the idf curves of rainfall from a standard (Italian standard) set of maxima of annual precipitation, with interpolation of Gumbel probabilities with various methods.
• Anne Van Loon Drougths R code

Blogs about R and Hydrology

• Tony Ladson's fresh and informative blog, and its links

Time series Analysis (among the many others):

• The TSA package contains R functions and datasets detailed in the book "Time Series Analysis with Applications in R (second edition)" by Jonathan Cryer and Kung-Sik Chan

A miscellany of topics:

• Here James B. Elsner and  Thomas H. Jagger wrote a tutorial for using R for Climate Research.
• Here tools for visualising California snow cover data are provided.
• A visualisation of the Bayesian search of a distribution stimulated by Climate Research.
• Simulating wind speed with R.
• Visualising droughts.
• Visualising NCEP global data
• Resources for Spatial Analysis
• Analysis of Dutch Rainfall data 

In R there are also many toolkit for the analysis of extreme events. The basic is:

• evd

but also:

• evir
• evdbayes
• ismev
• extRemes [for Windows users (I do not like the packages linked to a platform!!!)]
• SpatialExtremes

can be useful.
Frankly I did not test them all: but usually CRAN packages are really good.

Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:

• Geostatistics: gstat, automap, geoR, fields, RandomFields
• GIS: spgrass6, RSAGA, rgdal, sp, proj4, raster, mapproj, maptools, RGoogleMaps, RArcInfo, RpyGeo,
• Flood frequency: POT, evd, nsRFA, extremes, lmomco
• Optimization: pso, DEoptim,
• High Performance Computing: parallel, snowfall, multicore, jit, nws, Rmpi, snow,  taskPR
• Spreadsheets & DB: RPostgreSQL, RMySQL, RSQLite, RNetCDF, RexcelInstaller, xlsReadWrite
• Bayesian statistics: BAS, BLR, ensembleBMA, evdbayes, LearnBayes,
• ramps, spBayes,...
• Latex: xtable, Sweave
• Wavelets: wavelets; wavethresh, wmtsa, Rwave
• Data Mining: Rweka, rattle, party, RandomForest, ...
• Machine Learning tools (in Java with R connection)

R for hydrology is a blog dedicated to activities related to hydrology performed by using R.

If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development. 

GeoPaparazzi is now for free in Android Market

It was announced in the cousin blog about Jgrass. No excuse then to avoid to use it !

This version, the 2.4 had also a Japanese porting by the people of the Disaster Management Information System of the city of Osaka which is visible here .

Characteristics of good modeling software

Making good models is just one part of the whole job of a hydrologist. It is a tradition in our research field to make good research with not very good computer codes. Please do not misunderstand me. I do not mean that the algorithms used are wrong: I mean that the overall simulations machinery is usually not very well engineered, and using the code produced by researchers is not as easy as it could be (and actually is for many   industrial programs). This eventually makes scientists (and users too) loose a lot of time in redoing the same things, even when the original codes are available, simply because these codes are not well documented or do not provide those  functionalities that make them usable. The following paper (that was addressed to me by the Author of the Csparse library, T. Davis) covers some of the topics of making a good and usable code, and is a must to read for who does modeling.

Please follow the link below for getting the paper (last accessed February 3rd, 2011)
Characteristics of Industrial strength software.

The main conclusions by the Authors are summarized, and a little edited below for the laziest.

" … It is important to design a ... software to be easy to use and robust. Often it is better to assume that the user is not an expert in … algorithms, but someone who has a problem to solve and wishes to solve it accurately and efficiently with minimal effort. After all, even experienced users were once novices and a user’s initial experiences of using a solver are likely to determine whether he or she goes on to become an expert user. Based on our experiences …., in addition to the requirements of good performance (in terms of memory and speed) and the availability of comprehensive well-written documentation, in our opinion the following features characterize an ideal …. solver.

• Simplicity: the interface should be simple and enable the user to be shielded from algorithmic details (note: this is called in OO information hiding). The code should be easy to build and install, with no compiler warning messages. During the building of the software from supplied source, minimum effort and intervention by the user should be required. …. dynamic memory allocation should be used so that the user need not preallocate memory. In fact, the software developer needs very good reasons for not selecting a language that includes dynamic memory allocation.

The software developer should consider providing interfaces to popular high-level programming environments, such as Matlab, Mathematica, and Maple (note: and I add R, because Open Source is an add value.. Besides offering an appropriate interface is also behind the whole JGrass Project).

• Clarity: …. Furthermore, to allow repeated solves and iterative refinement there should be a clear distinction between (note by RR:) preprocessing and solve phases. … Developers should consider offering simple (all-in-one) interface as well as an interface with the greater flexibility of access to the different phases of modeling.

• Smartness: good choices for the default parameters and of the algorithms to be used should be automatically made without the user having to understand the algorithms and to read a large amount of detailed technical documentation. There should be an option to check the user-supplied input data, particularly for any assumptions that the code relies on. (Note by RR:) Parameters of the models should be as much as possible explained in documentation and code.

• Flexibility: for more experienced users and those with specific applications in mind, the solver should offer a wide range of options, ….. There should also be options for the user to specify the information that he or she requires …. The software should …. support 64-bit architectures, (note by RR) and be platform independent.

• Persistence: the solver should be able to recover from failure. For example, if it is found that there is not enough memory, a code that contains both in-core and out-of-core algorithms should automatically switch to out-of-core mode. Reverse communication should be designed to allow corrections to the input data.

…

• Threadsafety: The code should be threadsafe to enable the user to safely run multiple instances of the package simultaneously in different threads or on different processors.

"