Wednesday, August 22, 2012

R resources for Hydrologists

R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, many of them became more proficient than me in the field. Now it seems natural to use it for everything, but this was not always the case. A list of introductory material is here.

A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.

Recently many resources were added for hydrologists, which I list below with a little comment.

  • RWater: A Cyber-enabled Data-driven Tool for Enhancing Hydrology Education
  • airGR: Hydrological modelling tools developed at Irstea-Antony (HBAN Research Unit, France). The package includes several conceptual rainfall-runoff models , a snowmelt module and the associated functions for their calibration and evaluation
  • lumpR. A tool facilitating landscape discretisation for hillslope-based hydrological models. It is described in a paper on GMDD
  • HydroGOF and HydroTSM by Mauricio Zambrano-Bigiarini.  The first provides functions implementing both statistical and graphical goodnes-of-fit measures between observed and simulated values, mainly oriented to be used during the calibration, validation, and application of hydrological models. The second provides functions for management, analysis, interpolation and plotting of time series used in hydrology and related environmental sciences.  Mauricio also had a poster at EGU 2010 general assembly on the topic.
  • Jasper Vrugt's DREAM calibration method
  • RMWAGEN by Emanuele Cordano which is a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
  • Other stochastic generators of precipitation can be found here. Do not forget to explore the links in that page, and particularly the presentations given at the Roscoff's Workshop on stochastic generators, where many examples are in R
  • The RHydro which included TOPMODEL (apparently not anymore supported), tools for DEM analysis (this last type of tools however are also available through the work by R. Bivand, E.J. Pebesma and V. Gomez-Rubio ), an implementation of the FUSE by Clark et al (2008) methodology, and many other tools for hydrological analysis. These were initially promoted by Wouter Buytaert and Dominik Reusser who also gave a nice tutorial at EGU a few years ago.
  • Hydromad: It provides a modelling framework for environmental hydrology: water balance accounting and flow routing in spatially aggregated catchments. It supports simulation, estimation, assessment and visualisation of flow response to time series of rainfall and other drivers
  • TUWmodel  is a lumped conceptual rainfall-runoff model, following the structure of the HBV model. The model runs on a daily time step and consists of a snow routine, a soil moisture routine and a flow routing routine. See Parajka, J., R. Merz, G. Bloeschl (2007) Uncertainty and multiple objective calibration in regional water balance modelling: case study in 320 Austrian catchments, Hydrological Processes, 21, 435-446
  • Sean Turner's and Stefano Galelli's, reservoir package, Tools for Analysis, Design, and Operation of Water Supply Storages
  • Code: Handy routines for Hydrologists by Dan Moore and others.
  • Hydrosanity: It provides a graphical user interface for exploring hydrological time series. It is designed to work with catchment surface hydrology data (mainly rainfall and streamflow time series at a set of locations). There are functions to import from a database or files; summarise and visualise the dataset in various ways; estimate areal rainfall; fill gaps in rainfall data; and estimate the rainfall-runoff relationship. Probably the most useful features are the interactive graphical displays of a spatial set of time series. (This project seems actually being abandoned). 
  • aqp: Algorithms for quantitative pedology. A collection of algorithms related to modeling of soil resources, soil classification, soil profile aggregation, and visualization by Dylan Beaudette and Pierre Roudier. A paper talking about it is given here. And a presentation is not missing.
  • A package for plotting soil water retention curves and hydraulic conductivity by Emanuele Cordano, Fabio Zottele and Daniele Andreis is soilwater.
  • soilDB, of the same authors of aqp, is useful to access some soil databases.
  • soiltexture: Functions for soil texture plot, classification and transformation by Jules Moeys
  • Hydrome: This package estimates the parameters in infiltration and water retention models by curve-fitting method.
  • SoilWater address to a couple of packages for estimating Soil Water Retention Curves and some Pedotransfer Functions
  • hydropso: This package implements a state-of-the-art version of the Particle Swarm Optimisation (PSO) algorithm, with a special focus on the calibration of environmental models.
  • Evapotranpiration: by Dan Lu Guo and Seth Westra. This package estimates Potential and Actual Evapotranspiration with multiple models (see also the paper here).
  • EcoHydRology developed by DR. Fuka, MT WalterJA Archibald,  TS Steenhuis, and ZM Easton which presents a community modeling foundation for Eco-Hydrology. 
  • Claudia Vitolo's Curve Number (Curve Number!) and other R stuff, including some tools for data discovery. Claudia also manages a Google+ group, R4Hydrology.
  • nsRFA:  this is collection of statistical tools for objective (non-supervised) applications of the Regional Frequency Analysis methods in hydrology made by Alberto Viglione. The package refers to the index-value method and, more precisely, helps the hydrologist to: (1) regionalize the index-value; (2) form homogeneous regions with similar growth curves; (3) fit distribution functions to the empirical regional growth curves. 
  • Wasim: Helpful tools for data processing and visualisation of results of the hydrological model WASIM-ETH.
  • Geotopbricks by Emanuele Cordano, analyses raster maps and other information as input/output files from the Hydrological Distributed Model GEOtop
  • hddtools by Claudia Vitolo is a tool for hydrological data discovery.
  • waterData is a USGS Package for Retrieval, Analysis, and Anomaly Calculation of Daily Hydrologic Time Series Data
  • Lmoments and Lmomco are two packages for the estimation of the L-moments of a distribution.
  • The SPEI R Package by Santiago Begueria which includes a set of functions for computing potential evapotranspiration and several widely used drought indices including the Standardized Precipitation-Evapotranspiration Index (SPEI). 
  • The USGS-R packages at github
  • Alessio Pugliese and Attilio Castellarin pREC: a package for the regionalisation of some hydrological variables.
  • Alberto Montanari version of Hymod: here.
  • Emanuele Cordano work in connecting R with JGrasstools, here, to do geomorphological analysis (slides in Italian here, and in English here) within R.
  • meteo package by Kilibarda, Sekulic, Hengl, Pebesma and Graeler, A package for spatio-temporal geostatistical mapping of meteorological data. Global spatio-temporal models calculated using publicly available data are stored in package.
  • The reservoir package by Sean Turner [aut, cre], Jia Yi Ng [aut], Stefano Galelli [aut]. It measures single-storage water supply system performance using resilience, reliability, and vulnerability metrics; assess storage-yield-reliability relationships; determine no-fail storage with sequent peak analysis; optimize release decisions for water supply, hydropower, and multi-objective reservoirs using deterministic and stochastic dynamic programming; evaluate inflow persistence using the Hurst coefficient. A companion paper for this tool is available.
  • The scripts and data I use in my short mini-classes on R inside my Hydrology and Hydraulic Constructions classes. The include a quick introduction to R, plotting a treating a time serie of rainfall data read from a file, reading and plotting discharges data from a file, estimating the idf curves of rainfall from a standard (Italian standard) set of maxima of annual precipitation, with interpolation of Gumbel probabilities with various methods.
Blogs about R and Hydrology
Time series Analysis (among the many others):
  • The TSA package contains R functions and datasets detailed in the book "Time Series Analysis with Applications in R (second edition)" by Jonathan Cryer and Kung-Sik Chan
A miscellany of topics:
In R there are also many toolkit for the analysis of extreme events. The basic is:
but also:
can be useful.
Frankly I did not test them all: but usually CRAN packages are really good.

Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:

  • Geostatistics: gstat, automap, geoR, fields, RandomFields
  • GIS: spgrass6, RSAGA, rgdal, sp, proj4, raster, mapproj, maptools, RGoogleMaps, RArcInfo, RpyGeo,
  • Flood frequency: POT, evd, nsRFA, extremes, lmomco
  • Optimization: pso, DEoptim,
  • High Performance Computing: parallel, snowfall, multicore, jit, nws, Rmpi, snow,  taskPR
  • Spreadsheets & DB: RPostgreSQL, RMySQL, RSQLite, RNetCDF, RexcelInstaller, xlsReadWrite
  • Bayesian statistics: BAS, BLR, ensembleBMA, evdbayes, LearnBayes,
  • ramps, spBayes,...
  • Latex: xtable, Sweave
  • Wavelets: wavelets; wavethresh, wmtsa, Rwave
  • Data Mining: Rweka, rattle, party, RandomForest, ...
  • Machine Learning tools (in Java with R connection)
R for hydrology is a blog dedicated to activities related to hydrology performed by using R.

If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development. 

25 comments:

  1. From Shaun Harrigan:

    Thank you Riccardo for your helpful post.

    I have recently started a PhD in Hydrology and have chosen R to perform the majority of my statistical analyses and to produce publication quality graphics. I also use it to write code (Making good progress but still learning) for managing and cleaning up hydrological and climate data (data aggregation, missing data, extracting indicators for testing etc.).

    After spending quite some time deciding whether to concentrate my time learning R or Matlab I chose R as I can see a real future for it in hydrologists’ day to day tasks. Matlab is also very good, however, the fact that R is free is a big help for students and researchers. Cost aside, R has a rapid growing user group that are continually adding really nice libraries. It means often you don’t have to re-invent the wheel for every task.

    Some helpful libraries I use are:

    - HydroGOF and HydroTSM (as discussed in Riccardo’s blog)
    - Kendall ( for Mann-kendall trend test)
    - zyp (Sen’s slope method)
    - wq (water quality monitoring, also has Mann-Kendall and Sen slope included)
    - boot (functions for bootstrapping)

    The more hydrologists use R the more libraries that will be created. It would be great if a workshop/conference was organised to get an international community of R using hydrologists together. I am not aware of any? Be great to here from other hydrologists using R.

    Shaun Harrigan,
    National University of Ireland Maynooth (NUIM)

    ReplyDelete
    Replies
    1. This comment has been removed by a blog administrator.

      Delete
  2. I am not aware of any specialised R-for-Hydrologists conferences. I think that Wouter Buytaert and Dominik Reusser, as well as Mauricio Zambrano could be interested in organizing something.

    ReplyDelete
    Replies
    1. I just came across this page - what a great resource, thank you Riccardo! In fact Dan Fuka and I tried to organize a session for R in Hydrology last year (2011) at the AGU conference in San Francisco. We didn't publicize it enough, and unfortunately it was too small and had to be combined with another environmental modeling session, so it lost its R focus. If there is interest, perhaps we could try again for a R-Hydrology session at AGU next year (2013).

      Delete
  3. Interesting page. I am trying to get acquainted with it.

    ReplyDelete
  4. Dear colleagues.
    You could find two more R packages for hydrologists in itia's research team website here
    http://itia.ntua.gr/en/softinfo/3/
    and here
    http://itia.ntua.gr/en/softinfo/29/

    The first is "Hyetos-R". It is a package for the temporal stochastic simulation of rainfall process at fine time scales based on Bartlett-Lewis rectangular pulses rainfall model.

    The second is "eas" which is an evolutionary annealing-simplex algorithm. The first package uses it to estimate the Bartlett-Lewis model's parameters.

    Hristos Tyralis
    National Technical University of Athens

    ReplyDelete
    Replies
    1. Dear Hristos,

      thank you for your information. However it would be great if you could upload your packages on the CRAN web site.

      All the best,

      riccardo

      Delete
  5. Please feel free to add to your list my blog and github sites on hydrology and stochastic precipitation.

    http://gopigoteti.tumblr.com/
    https://github.com/RationShop

    Thanks
    Gopi Goteti

    ReplyDelete
  6. Dear Ricardo, thank you for sharing these links. I am just starting my period as visiting scholar in Sydney University. My background is geology (which is in very qualitative approach), whereas now I am working in the Environmental Dept (which is very quantitative). And I am a very beginner in R, but I have fun though, learning R from very start, and gathering scattered information about scripts etc. The interesting thing was, we could get help from anyone on the net. Thank you once again.

    ReplyDelete
  7. Dea Desapta, thank you for your kind words. Please do not hesitate if you come to know new pieces of R software that can be added to this list.

    riccardo

    ReplyDelete
  8. Thanks for this post, I'm writing a paper and this was usefull. Soon me and my collegues will we post a new package with a hydrologic model much used in Brazil.

    All the best.

    ReplyDelete
  9. I got admitted in PhD on this jan and am intersted to do water quality analysis, its prediction and bayesian analysis of water quality. please suggest me the packages and the books available for beginners like me.
    thank you

    ReplyDelete
  10. Dear Sahoo,

    I will not be able to give you suggestions on the topic you like. You should be also more specific. What do you mean for water quality ? Analysis of chemicals in water ? Advection/dispersion of pollutants and nutrients in groundwater or surface water ? Sanitation ?

    ReplyDelete
  11. R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, ... mitfernbedienung.blogspot.de

    ReplyDelete
  12. I have started a PhD in Hydrology and I have chosen R to write code (about disaggregation streamflow from annual to daily by using K-nearest neighbour( K-NN) model) if you have a code ,please help me .

    ReplyDelete
  13. R is great ... i used it in my M.Sc. thesis and i'm still learning new things

    as an advice for new learners ... there is an R package to learn R "Sounds funny but it is really efficient"
    you can see this website http://swirlstats.com/
    it is "probably" the same used for coursera : https://www.coursera.org/course/rprog
    and it starts in 3 days if you want to join.

    ReplyDelete
  14. We wanted people to be aware of two new R packages from USGS. Both are now available on CRAN.

    dataRetrieval - retrieves all types of USGS water data and enters it directly into R data frames. Also retrieves from the water quality data portal that contains USGS, EPA-STORET, and USDA-STEWARDS data base.

    EGRET (Exploration and Graphics for RivEr Trends) does exploratory data analysis dealing with long term streamflow statistics, surface water quality loads (fluxes) and water quality trends. Strongly oriented towards graphical methods. Contains its own data retrieval routines that ingest data from USGS NWIS, Water Quality Portal, or spreadsheets.

    For more information see: https://github.com/USGS-R/EGRET/wiki

    ReplyDelete
  15. Thank you for the information you shared in your blog.
    I think there is a need to have a specific youtube channel for hydrologists to help them in using R.

    ReplyDelete
  16. Dear Morteza, thank you for the suggestion. However, in the open source community, everything is based on volunteers. So you could be the first to give a good example.

    ReplyDelete
  17. This comment has been removed by a blog administrator.

    ReplyDelete
  18. Thanks for posting this, Riccardo, I will be looking into some of those packages asap. As Desapta says, it is sometimes hard to bridge the gap between the qualitative and quantitative approaches in eco-hydrological work. I am a biologist doing a PhD on eco-hydrology in temporary ponds, and I want to link precipitation to groundwater patterns and flood episodes (using met records and diver data, so lots of data!). A lot of the focus in the time series analysis methods seems to be on rivers and surface waters - are you aware of any packages that look at groundwater? I hope this is not a stupid question - there is just so much information out there it is hard to know where to look.

    ReplyDelete
    Replies
    1. I do not inow about specific packages, but there should be no difference in doing correlations between rainfall and groundwater levels and doing any other correlation.
      Modern techniques could actually involve Machine Learning (I have a post or two on it) which is certainly covered in some R package.

      Delete
  19. Hi,

    I am a hydro geologist.

    Is there R package relating discharge,Q, of a well, specific capacity, hydraulic conductivity, and drawdown?

    ReplyDelete
  20. I wanted people looking at this site to know about a new blog from the USGS Office of Water Information. It is at https://owi.usgs.gov/blog/. On that blog there are a couple of posts about new developments in the EGRET software (Exploration and Graphics for RivEr Trends) and EGRETci ("ci" for confidence intervals). There are many new R tools for hydrologists coming out of USGS these days. Bob Hirsch, Research Hydrologist, USGS

    ReplyDelete
  21. Hi there - I am wondering if there are any known R packages available that will allow the user to develop and update a stage-discharge rating curve using stream discharge measurement records? Thanks -- Nick Harrison

    ReplyDelete