- GEOstatistics (course held by Alberto Bellin)
- Statistics (course held by Stefano Siboni)
My reflections and notes about hydrology and being a hydrologist in academia. The daily evolution of my work. Especially for my students, but also for anyone with the patience to read them.
Showing posts with label R software. Show all posts
Showing posts with label R software. Show all posts
Tuesday, February 19, 2019
Ph.D. Miscellanea - Jupyter Notebook with R or Python on Statistics and Hydrology
This blog post is to share some of the notebooks provided by my Ph.D. students on the topics they follow in their Ph.D. classes. Please observe that some of the material are lecture notes by some of my colleagues. You can use them but you should cite the source when you do it.
Thursday, September 22, 2016
Cool R - Tydyverse
More or less there was R before Hadley Wickham and after him. Before him it was a solid programming for doing statistical analysis. After him it was also cool.
Reasons are that probably he was able to intercept the best of functional programming^1 and introduce it in R.
" In the Tidy Tools Manifesto, he proposes four basic principles for any computer interface for handling data:
Those principles are realized in a new collection of his R packages: the tidyverse. Now, with a simple call to library(tidyverse) (after installing the package from CRAN), you can load a suite of tools to make managing data easier into your R session:
Reasons are that probably he was able to intercept the best of functional programming^1 and introduce it in R.
His new book R for Data Science promises to be a breakthrough. You can buy it or find its contents by clicking on the image. From Revolutions:
- Reuse existing data structures.
- Compose simple functions with the pipe.
- Embrace functional programming.
- Design for humans.
Those principles are realized in a new collection of his R packages: the tidyverse. Now, with a simple call to library(tidyverse) (after installing the package from CRAN), you can load a suite of tools to make managing data easier into your R session:
readr, for importing data from files
tibble, a modern iteration on data frames
tidyr, functions to rearrange data for analysis
dplyr, functions to filter, arrange, subset, modify and aggregate data frames
The tidyverse also loads purrr, for functional programming with data, and ggplot2, for data visualization using the grammar of graphics.
Installing the tidyverse package also installs for you (but doesn't automatically load) a raft of other packages to help you work with dates/time, strings, factors (with the new forcats package), and statistical models. It also provides various packages for connecting to remote data sources and data file formats.
Simply put, tidyverse puts a complete suite of modern data-handling tools into your R session, and provides an essential toolbox for any data scientist using R. (Also, it's a lot easier to simply add library(tidyverse) to the top of your script rather than the dozen or solibrary(...) calls previously required!) Hadley regularly updates these packages, and you can easily update them in your R installation using the provided tidyverse_update() function.
For more on tidyverse, check out Hadley's post on the RStudio blog, linked here."
^1 - For a course in functional programming, see here.
tibble, a modern iteration on data frames
tidyr, functions to rearrange data for analysis
dplyr, functions to filter, arrange, subset, modify and aggregate data frames
The tidyverse also loads purrr, for functional programming with data, and ggplot2, for data visualization using the grammar of graphics.
Installing the tidyverse package also installs for you (but doesn't automatically load) a raft of other packages to help you work with dates/time, strings, factors (with the new forcats package), and statistical models. It also provides various packages for connecting to remote data sources and data file formats.
Simply put, tidyverse puts a complete suite of modern data-handling tools into your R session, and provides an essential toolbox for any data scientist using R. (Also, it's a lot easier to simply add library(tidyverse) to the top of your script rather than the dozen or solibrary(...) calls previously required!) Hadley regularly updates these packages, and you can easily update them in your R installation using the provided tidyverse_update() function.
For more on tidyverse, check out Hadley's post on the RStudio blog, linked here."
^1 - For a course in functional programming, see here.
Saturday, April 2, 2016
A few R scripts useful for hydrologists
These are the script I use as templates for my short (very short) classes (Hydrology and Hydraulic Constructions/Urban Hydrology) when I introduce R. Reading a good R textbook or manual would be important, indeed to understand what I am doing. These scripts have to be "studied", meaning that operations are commented but for really understand them, any command should be studied. Since data comes in various formats, it is shown how to deal with them.
Knowledge of what R Studio is is given for granted.
- A quick introduction to R (Knitr pdf)
This assume almost no priori knowledge of R. However you have to know how to load and execute a script in Rstudio. I suggest that, once loaded the script, you execute one command at time. You will know what vector and data frame are, and how to access their data.
- Simple plot in R (Knitr pdf)
It generates some random data (normally and uniformly) distributed. Plots them as a time series, as an histogram as a empirical cumulative distribution function.
- Reading rainfall data and plotting them (Knitr pdf, the dataset, the Readme of the data)
It reads data in cvs format, which come in many columns, each for one year; does some data treatment, melts the columns to produce a time series object, plots them, does the statistics of non-raining days and raining days. Estimates the autocorrelation function.
- Reading the data of discharges of River Adige (Knitr pdf, the dataset, Readme file)
It reads the data from a very rudimental file. Transforms the data in a reasonable time series, and plots it. It introduces also the use of boxplots.
- Estimating depth duration frequency (ddf) curves of Duckburg with R the dataset, Readme file)
Starting from the annual maxima of given duration (as given in Italy) it interpolates the data with a Gumbel pdf using a couple of standard techniques. Uses Pearson ranking for getting the best estimator. After obtaining the probabilities for the various durations, it uses linear interpolation for getting the parameters of the idf curves. I admit, it is not a great statistics exercise. Let say, it is the standard we use in Italy. All is done step by step. The smart reader can shorten the procedures a lot. In principle is not difficult to automate the whole procedure, producing from the initial data table the final product.
Knowledge of what R Studio is is given for granted.
- A quick introduction to R (Knitr pdf)
This assume almost no priori knowledge of R. However you have to know how to load and execute a script in Rstudio. I suggest that, once loaded the script, you execute one command at time. You will know what vector and data frame are, and how to access their data.
- Simple plot in R (Knitr pdf)
It generates some random data (normally and uniformly) distributed. Plots them as a time series, as an histogram as a empirical cumulative distribution function.
- Reading rainfall data and plotting them (Knitr pdf, the dataset, the Readme of the data)
It reads data in cvs format, which come in many columns, each for one year; does some data treatment, melts the columns to produce a time series object, plots them, does the statistics of non-raining days and raining days. Estimates the autocorrelation function.
- Reading the data of discharges of River Adige (Knitr pdf, the dataset, Readme file)
It reads the data from a very rudimental file. Transforms the data in a reasonable time series, and plots it. It introduces also the use of boxplots.
- Estimating depth duration frequency (ddf) curves of Duckburg with R the dataset, Readme file)
Starting from the annual maxima of given duration (as given in Italy) it interpolates the data with a Gumbel pdf using a couple of standard techniques. Uses Pearson ranking for getting the best estimator. After obtaining the probabilities for the various durations, it uses linear interpolation for getting the parameters of the idf curves. I admit, it is not a great statistics exercise. Let say, it is the standard we use in Italy. All is done step by step. The smart reader can shorten the procedures a lot. In principle is not difficult to automate the whole procedure, producing from the initial data table the final product.
- Elaboration of 1 hour rainfall (Knitr)
- Elaboration of 3 hours (knitr), 6 hours (knitr), 12 hours (knitr), 24 hours (knitr)
- Summary of elaborations (data)
Thursday, March 17, 2016
Renjin
Renjin is a JVM-based interpreter for the R language. For Java developers of scientific application, I think it is a must (but still I need to test it). It can be used in various ways and offers itself a simple interface to be used. The fraction of packages ported into it is here.
For an introduction you can watch this webinar. But obviously you can have more by browsing Rnjin website.
For an introduction you can watch this webinar. But obviously you can have more by browsing Rnjin website.
Among others, gvSIG uses Renjin for making calculations. The source code is available on GIThub.
Click on the Figure to go to Renjin website.
Monday, July 20, 2015
True random numbers
Generating random numbers is probably impossible. The substitute is generating pseudorandom (see a video here) numbers according to some deterministic rule well explained, for instance, in Knuth, (1997). But the doubt that pseudorandom numbers is not enough usually sticks with you. So where having real random number, if they exist ?
You can try hardware generated random numbers. One particular set of these random numbers generators come from Dirk EddelBuettel who actually uses Mads Haar work for bringing true random numbers to R. Dirk points also to a nice introduction to random number generator, Maads Haar (1999), and an R package called random.
References
Dirk Eddelbuettel, random: An R package for true random numbers, 2006
Mads Haar, Introduction to random numbers (see also here), 1999
Knuth D.E. The Art of Computer Programming, Volume 2: Seminumerical Algorithms, Third Edition. Addison-Wesley, 1997. ISBN 0-201-89684-2. Chapter 3. [Extensive coverage of statistical tests for non-randomness.]
You can try hardware generated random numbers. One particular set of these random numbers generators come from Dirk EddelBuettel who actually uses Mads Haar work for bringing true random numbers to R. Dirk points also to a nice introduction to random number generator, Maads Haar (1999), and an R package called random.
References
Dirk Eddelbuettel, random: An R package for true random numbers, 2006
Mads Haar, Introduction to random numbers (see also here), 1999
Knuth D.E. The Art of Computer Programming, Volume 2: Seminumerical Algorithms, Third Edition. Addison-Wesley, 1997. ISBN 0-201-89684-2. Chapter 3. [Extensive coverage of statistical tests for non-randomness.]
Friday, August 22, 2014
A little bridge between JGrasstools and R
I receive from Emanuele Cordano, and I publish, knowing that is of interest to many:
"Dear all,
some months ago I developed a R package on github which allows to execute some classes of jgrasstools from R. I did it because I needed to do hydro-geormorphological analysis with R rasters maps. It is quite trivial. The package creates a groovy script file from a R S3 object and then executes the script.
The R code is here on Github with GPL license:
It also contains a jar of jgrasstools or they can be complied and downloaded from github. It needs that groovy is previously installed.
The code is now experimental and needs more testing. The examples are limited to geomorphological analysis and basin extraction.
In the next months I'm going to continue the development and to perform the documentation and the examples.
I would like to share this experimental R package with those who are potentially interested.
Any feedback is appreciated and please let me know if you know something similar already existing.
Regards
Emanuele Cordano
"
Who wants more information about the JGrasstools, could browse this link, and for knowing more about to geomorphological analysis (he can get a quite comprehensive set of slides in Italian here, and, a less comprehensive introduction in English here).
The link is also added to the main R hydrological resources post.
Wednesday, May 21, 2014
R General References
You can start from here:
- R reference card
- Quick Graphics and Graphics and Murrel's Graphics
- Quick R
- R-Spatial and Spatial R
- California Soil Resources Lab
- Manuals
- Videos and video tutorials and the Google R Videos
- Packages
- Search engines
- R Site search
- The Guerilla guide to R
- R Inferno give a stimulating alternative vision of R.
- Here you can download the Matlofs book, The Art of R Programming.
- And here you can download a book on ggplot2 by its own author Hadley Wickham, and other ggplot resources.
- Three-Toed Sloth's book is a fresh-air statistical book which uses R.
- Here you will find the book on "Modeling data with functional programming in R" by Bryan Lee Yung Row
- A curated list of awesome packages and R-related stuff
- A guide to R graphics by Joachim Schork
- Last, but not least, Hadley Wickham book' and Advanced R.
If you have further questions, ask Stackoverflow
If you managed to learn how to write function, please decide to make a package. Emanuele Cordano found this nice post on github explaining why to do (thinking to statistics).
An alternative pathway of instructions is here.
Monday, December 16, 2013
USGS r-packages for hydrology
As I learn AGU Fall meeting, USGS is committed to develop various R Packages to help analysis of hydrological data.
These packages include:
These packages include:
- GLRItcl
- usgs-r.github.com: Splash page for USGS-R organization
- EflowStats: Calculates Hydrologic Indicator stats and fundamental properties of daily streamflow for a given set of data
- EGRET: An R-package for the analysis of long-term changes in water quality and streamflow, including the water-quality method Weighted Regressions on Time, Discharge, and Season (WRTDS)
- EflowStats: Calculates Hydrologic Indicator stats and fundamental properties of daily streamflow for a given set of data. A related poster, presented at AGU Fall meeting 2013, can be found here.
- USGSwsBase: Base USGS water science R functions.
- surragateRegression:
- dataRetrieval: This R package is designed to obtain its water quality sample data, streamflow data, and metadata directly from either the USGS NWIS (National Water Information System), but it allows for user-supplied text files as inputs. The program is designed to ingest the data directly into R and structure them into file structures suited for EGRET analysis.
- USGSwsQW: Water quality USGS water science R functions.
- USGSwsStats:Statistic USGS water science R functions.
- USGSwsData: Data sets as data.frames and as text files for examples in the USGS core pacakges
- USGSwsGraphs: Graphical USGS water science R functions
- restrend
- rloadest
- DVstats
- NWCCompare
- rGDP
- functionCollection
All of them can be retrieved from the USGS-R Github repository. Link to other R packages useful for hydrology can be found at the main R post.
Sunday, November 17, 2013
A Java Implementation of R !
Since I am becoming a Java programmer, and an R evangelist, I found very interesting the FastR project whose scope is to provide a Java implementation of the R language. This would be really interesting, especially thinking to a possible integration among OMS and R, which would be very helpful for data management and I/O for model components.
Jan Vitek, one of the authors of the code, gave a nice presentation about, that can be found below the image.
Any comment is left to the readings and to the trial of the software. Eventually, other material and code itself, can be found at github
Jan Vitek, one of the authors of the code, gave a nice presentation about, that can be found below the image.
Any comment is left to the readings and to the trial of the software. Eventually, other material and code itself, can be found at github
Tuesday, September 24, 2013
Clarifications about a formula
I reading the TRIGRS manual I fond this:
[ J28] - Cordano E. R. Rigon, A perturbative view on the subsurface water pressure response at hillslope scale, Water Resour. Res., Vol. 44, No. 5, W05407- W05407, doi:10.1029/2006WR005740, 2008
RMWAGEN, a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
Soilwater, A package for plotting soil water retention curves and hydraulic conductivity written by Emanuele Cordano, Fabio Zottele and Daniele Andreis.
Comparison of Computed Results with Published Analyses
We tested TRIGRS against Iverson’s (2000) examples from the Minor Creek landslide and from an experiment at the USGS debris-flow flume and were able to reproduce his results for pore pressure and factor of safety profiles. Thus, we were able to verify that the computer code accurately implements Iverson’s (2000) formulas for pore pressure and factor of safety. After publishing the original version of TRIGRS, students of Ricardo Rigon (University of Trento, Trento, Italy) pointed out an error in the coordinate transformation in Iverson’s (2000) original paper. Correction of that simple error causes our numerical results to differ somewhat from Iverson’s (2000) published values, though preserving their basic form. Folders named
25“MinorCreek” and “flume” contain files with necessary data to reproduce these results. By varying elapsed time, t , in the initialization file, the user can compute pressure head and factor of safety profiles like those of figures 7, 8, 10, and 11 of Iverson (2000). Note that the computed profiles conform to the limit imposed by equation 3. We have also verified that the new version of TRIGRS described in this report reproduces the published results of Srivastava and Yeh (1991). Data in the folder “sy91” produce profiles that allow the user to reproduce the curves plotted in figure 3 of Srivastava and Yeh (1991).
A part the misspelling of my name (two "r"s, not one like in Spanish), I need to precise that the error was discovered by Emanuele Cordano, a.k.a. the authors of at least couple of papers with me, and some R packages useful for hydrologists. His papers, maybe difficult to read, and so far not kissed by the goddess of citations, remain among my own favorite.
References
[J38] - Cordano, E., and R. Rigon (2013), A mass-conservative method for the integration of the two-dimensional groundwater (Boussinesq) equation, Water Resour. Res., 49, doi:10.1002/wrcr.20072.
Boussinesq code is available on Github.
R- Software
RMWAGEN, a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
Soilwater, A package for plotting soil water retention curves and hydraulic conductivity written by Emanuele Cordano, Fabio Zottele and Daniele Andreis.
Sunday, August 11, 2013
A proposal of an R package for investigating some simple soil properties
In analysing the soil hydrological properties the first step is certainly to be able to draw the Soil Water Retention Curves (SWRC), at least according to the two major parameterisations: the van Genuchten and the Brooks and Corey ones. Possibly some outsider parameterisation could be also covered, as the double porosity type envisioned by Romano et al., and the (deprecated) Gardner ones (but used, for instance in TRIGRS to have analytical solution for the vertical infiltration). So why do not face this task to implement all of them in R (see also here, a previous post) ?
These parameterisations, in turn, could be used inside the Mualem scheme to obtain the hydraulic conductivity (K, here: slides no 127), and after plotting them (with ggplot2) their properties could be investigated. Particularly important it has been found to be the delay of increase/decrease with suction of K with respect SWRC (here, from slides no 72), so plotting them one above the other can be useful to understand qualitatively how the subsurface flow dynamics (excluding macropores) works at hillslope scale. Plotting hydraulic capacity, C (slide 26), a.k.a. the derivative of the SWRC with respect to suction, could be nice to see. Also the plot of the hydraulic diffusivity, D = K/C, can be useful. So we have at least four functions for any parameterisation to plot.
One variation on this scheme can be considering the extension of the SWRC from the negative pressures (suction) domain to the positive ones. Since many phenomena occurs just at the edge of saturation a proper extension of the SWRC in this domain can help to understand. This paper by Schaap and van Genuchten can be used as a basis for any investigation.
To solve Richards equation analytically either adopt the Gardner parameterisation of SWRC or over-simplifiy it. Assuming constant hydraulic diffusivity (as a product of constant K and C) it is possible to obtain well know analytical solutions of the 1-D Richards equation derived from well-known heat transfer solution (here, from slides no 179 or here, in the original paper). These solution, not only offer an opportunity to plot another, more complicate function, but also allows for calculating an approximation of the 1-D infiltration phenomena induced by a variable rainfall, as a convolution of these solutions with the variable rainfall inputs (which requires a little more of R programming).
Once the whole machinery would implement, would be a piece of cake to evaluate the errors that can be made with these solution by appropriately varying the parameters.
References
Baum R.L., Savage W.S., and Godt J.W., TRIGRS—A Fortran Program for Transient Rainfall Infiltration and Grid-Based Regional Slope-Stability Analysis, Version 2.0
D'Odorico, P., Fagherazzi, S., & Rigon, R. (2005). Potential for landsliding: Dependence on hyetograph characteristics. Journal of Geophysical Research, 110(F1), 1–10. doi:10.1029/2004JF000127
Rigon R. - Slides about Soil Hydrology (here in Italian, here in English), 2011
Rigon R. - An Overview of hillslope hydrology, 2013
Romano, N., Nasta, P., Severino, G., & Hopmans, J. W. (2011). Using Bimodal Lognormal Functions to Describe Soil Hydraulic Properties. Soil Science Society of America Journal, 75(2), 468. doi:10.2136/sssaj2010.0084
Schaap, M. G., and van Genuchten, M. T. (2006). A Modified Mualem–van Genuchten Formulation for Improved Description of the Hydraulic Conductivity Near Saturation. Vadose Zone Journal, 5, 27–34. doi:10.2136/vzj2005.0005
These parameterisations, in turn, could be used inside the Mualem scheme to obtain the hydraulic conductivity (K, here: slides no 127), and after plotting them (with ggplot2) their properties could be investigated. Particularly important it has been found to be the delay of increase/decrease with suction of K with respect SWRC (here, from slides no 72), so plotting them one above the other can be useful to understand qualitatively how the subsurface flow dynamics (excluding macropores) works at hillslope scale. Plotting hydraulic capacity, C (slide 26), a.k.a. the derivative of the SWRC with respect to suction, could be nice to see. Also the plot of the hydraulic diffusivity, D = K/C, can be useful. So we have at least four functions for any parameterisation to plot.
One variation on this scheme can be considering the extension of the SWRC from the negative pressures (suction) domain to the positive ones. Since many phenomena occurs just at the edge of saturation a proper extension of the SWRC in this domain can help to understand. This paper by Schaap and van Genuchten can be used as a basis for any investigation.
To solve Richards equation analytically either adopt the Gardner parameterisation of SWRC or over-simplifiy it. Assuming constant hydraulic diffusivity (as a product of constant K and C) it is possible to obtain well know analytical solutions of the 1-D Richards equation derived from well-known heat transfer solution (here, from slides no 179 or here, in the original paper). These solution, not only offer an opportunity to plot another, more complicate function, but also allows for calculating an approximation of the 1-D infiltration phenomena induced by a variable rainfall, as a convolution of these solutions with the variable rainfall inputs (which requires a little more of R programming).
Once the whole machinery would implement, would be a piece of cake to evaluate the errors that can be made with these solution by appropriately varying the parameters.
References
Baum R.L., Savage W.S., and Godt J.W., TRIGRS—A Fortran Program for Transient Rainfall Infiltration and Grid-Based Regional Slope-Stability Analysis, Version 2.0
D'Odorico, P., Fagherazzi, S., & Rigon, R. (2005). Potential for landsliding: Dependence on hyetograph characteristics. Journal of Geophysical Research, 110(F1), 1–10. doi:10.1029/2004JF000127
Rigon R. - Slides about Soil Hydrology (here in Italian, here in English), 2011
Rigon R. - An Overview of hillslope hydrology, 2013
Romano, N., Nasta, P., Severino, G., & Hopmans, J. W. (2011). Using Bimodal Lognormal Functions to Describe Soil Hydraulic Properties. Soil Science Society of America Journal, 75(2), 468. doi:10.2136/sssaj2010.0084
Schaap, M. G., and van Genuchten, M. T. (2006). A Modified Mualem–van Genuchten Formulation for Improved Description of the Hydraulic Conductivity Near Saturation. Vadose Zone Journal, 5, 27–34. doi:10.2136/vzj2005.0005
Wednesday, July 24, 2013
GB ET Potential Dataset for United Kingdom
In this post I am presenting another R exercise. Not that I believe very much on the physical content of the what Scottischsnow does here. (Actually this bold way of treating data and empirical formulas at large scale time and spatial is something that I usually blame).
However the information gathered in this way could be useful as a reference step for doing physically based things, at a more detailed scale (for instance by using our JGrass-NewAGE).
Moreover I like it because it uses R all the way for extracting the data and visualising them, and this can be equally likely instructive.
However the information gathered in this way could be useful as a reference step for doing physically based things, at a more detailed scale (for instance by using our JGrass-NewAGE).
Moreover I like it because it uses R all the way for extracting the data and visualising them, and this can be equally likely instructive.
Saturday, May 18, 2013
Using ggplot2 for Hydrology
Leonardo Perathoner prepared this for helping my bachelor students:
"Since some student asked to me some information about ggplot, I write it here for everybody.
If you are interested to plots and a graphic quality superior to the standard R, you can utilize the R package ggplot2 !!
Here they are some resources that I want to I would advise to use in your learning process:
1) First a good manual
and then, these other sites about ggplot I often use:
2) the official site:
3) a quick cookbook with examples and code snippets
Leonardo "
PS: many other resources here:
Wednesday, August 22, 2012
R resources for Hydrologists
R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, many of them became more proficient than me in the field. Now it seems natural to use it for everything, but this was not always the case. A list of introductory material is here.
A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.
Recently many resources were added for hydrologists, which I list below with a little comment.
New: a second paper by distinguished colleagues is also on HESSD: Hydrology modelling R packages: a unified analysis of models and practicalities from a user perspective, by Astagnau et al.
New: Now we also have a very informative paper at HESSD: "Using R in hydrology: a review of recent developments and future directions" by Slate et al., 2019. It is a pity they do not cite this blog, but we cannot pretend too much!
but also:
Frankly I did not test them all: but usually CRAN packages are really good.
Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:
If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development.
A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.
Recently many resources were added for hydrologists, which I list below with a little comment.
New: a second paper by distinguished colleagues is also on HESSD: Hydrology modelling R packages: a unified analysis of models and practicalities from a user perspective, by Astagnau et al.
New: Now we also have a very informative paper at HESSD: "Using R in hydrology: a review of recent developments and future directions" by Slate et al., 2019. It is a pity they do not cite this blog, but we cannot pretend too much!
- OCNet OCNet enables the creation and analysis of Optimal Channel Networks (OCNs). These are oriented spanning trees (built on rectangular lattices made up of square pixels) that reproduce all scaling features characteristic of real, natural river networks.
- DonwscaleR is an R package for empirical-statistical downscaling focusing on daily data and covering the most popular approaches and techniques (quantile mapping, analogs, regression, generalized regression, neural networks). This package has been conceived to work in the framework of both seasonal forecasting and climate change studies and is part of the climate4R framework, formed by
loadeR,transformeR,downscaleRandvisualizeR. - bigleaf (version 0.6.5), an open source toolset for the derivation of meteorological, aerodynamic, and physiological ecosystem properties from eddy covariance (EC) flux observations and concurrent meteorological measurements.
- rtop is a package for geostatistical interpolation of data with irregular spatial support such as runoff related data or data from administrative units.
- The boussinesq package is a collection of functions implementing the one-dimensional Boussinesq Equation (ground-water).
- dynatopmodel is a native R implementation and enhancement of the Dynamic TOPMODEL, Beven and Freers’ (2001) extension to the semi-distributed hydrological model TOPMODEL (Beven and Kirkby, 1979).
RclimTool. It was designed with the aim of facilitating users in statistical analysis for quality control, filling of missing data, homogeneity analysis and calculation of indicators for daily weather series for temperature (maximum and minimum) and precipitation. - LumpR a tool for HRU delineations (the reference paper is here).
- Cropwat FAO model (it is a decision support system developed by the Land and Water Development Division of FAO for planning and management of irrigation) translated into R.
- Two socio-hydrological models of human-flood interactions that were recently developed. The two models are similar, but based on different socio-economic variables (wealth and human proximity to rivers VS relative population density). A full description of the two models can be found in the following two papers: HESS (2013), https://www.hydrol-earth-syst-sci.net/17/3295/2013/hess-17-3295-2013.pdf; WRR (2015), http://onlinelibrary.wiley.com/doi/10.1002/2014WR016416/full
- RWater: A Cyber-enabled Data-driven Tool for Enhancing Hydrology Education
- airGR: Hydrological modelling tools developed at Irstea-Antony (HBAN Research Unit, France). The package includes several conceptual rainfall-runoff models , a snowmelt module and the associated functions for their calibration and evaluation
- airGRteaching:“It is an add-on package to the 'airGR' package that simplifies its use and is aimed at being used for teaching hydrology. The package provides 1) three functions that allow to complete very simply a hydrological modelling exercise 2) plotting functions to help students to explore observed data and to interpret the results of calibration and simulation of the GR ('Génie rural') models 3) a 'Shiny' graphical interface that allows for displaying the impact of model parameters on hydrographs and models internal variables.“
- airGRdatassim: it allows performing streamflow data assimilation in the GR rainfall-runoff models with EnKF and particle filter.
- lumpR. A tool facilitating landscape discretisation for hillslope-based hydrological models. It is described in a paper on GMDD
- HydroGOF and HydroTSM by Mauricio Zambrano-Bigiarini. The first provides functions implementing both statistical and graphical goodnes-of-fit measures between observed and simulated values, mainly oriented to be used during the calibration, validation, and application of hydrological models. The second provides functions for management, analysis, interpolation and plotting of time series used in hydrology and related environmental sciences. Mauricio also had a poster at EGU 2010 general assembly on the topic.
- Jasper Vrugt's DREAM calibration method
- RMWAGEN by Emanuele Cordano which is a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
- Other stochastic generators of precipitation can be found here. Do not forget to explore the links in that page, and particularly the presentations given at the Roscoff's Workshop on stochastic generators, where many examples are in R
- The RHydro which included TOPMODEL (apparently not anymore supported), tools for DEM analysis (this last type of tools however are also available through the work by R. Bivand, E.J. Pebesma and V. Gomez-Rubio ), an implementation of the FUSE by Clark et al (2008) methodology, and many other tools for hydrological analysis. These were initially promoted by Wouter Buytaert and Dominik Reusser who also gave a nice tutorial at EGU a few years ago.
- Hydromad: It provides a modelling framework for environmental hydrology: water balance accounting and flow routing in spatially aggregated catchments. It supports simulation, estimation, assessment and visualisation of flow response to time series of rainfall and other drivers
- TUWmodel is a lumped conceptual rainfall-runoff model, following the structure of the HBV model. The model runs on a daily time step and consists of a snow routine, a soil moisture routine and a flow routing routine. See Parajka, J., R. Merz, G. Bloeschl (2007) Uncertainty and multiple objective calibration in regional water balance modelling: case study in 320 Austrian catchments, Hydrological Processes, 21, 435-446
- Sean Turner's and Stefano Galelli's, reservoir package, Tools for Analysis, Design, and Operation of Water Supply Storages. Sean Turners's also buildes Scenario
- R Code: Handy routines for Hydrologists by Dan Moore and others.
- Hydrosanity: It provides a graphical user interface for exploring hydrological time series. It is designed to work with catchment surface hydrology data (mainly rainfall and streamflow time series at a set of locations). There are functions to import from a database or files; summarise and visualise the dataset in various ways; estimate areal rainfall; fill gaps in rainfall data; and estimate the rainfall-runoff relationship. Probably the most useful features are the interactive graphical displays of a spatial set of time series. (This project seems actually being abandoned).
- aqp: Algorithms for quantitative pedology. A collection of algorithms related to modeling of soil resources, soil classification, soil profile aggregation, and visualization by Dylan Beaudette and Pierre Roudier. A paper talking about it is given here. And a presentation is not missing.
- A package for plotting soil water retention curves and hydraulic conductivity by Emanuele Cordano, Fabio Zottele and Daniele Andreis is soilwater.
- soilDB, of the same authors of aqp, is useful to access some soil databases.
- soiltexture: Functions for soil texture plot, classification and transformation by Jules Moeys
- Hydrome: This package estimates the parameters in infiltration and water retention models by curve-fitting method.
- SoilWater address to a couple of packages for estimating Soil Water Retention Curves and some Pedotransfer Functions
- hydropso: This package implements a state-of-the-art version of the Particle Swarm Optimisation (PSO) algorithm, with a special focus on the calibration of environmental models.
- Evapotranpiration: by Dan Lu Guo and Seth Westra. This package estimates Potential and Actual Evapotranspiration with multiple models (see also the paper here).
- EcoHydRology developed by DR. Fuka, MT Walter, JA Archibald, TS Steenhuis, and ZM Easton which presents a community modeling foundation for Eco-Hydrology.
- Claudia Vitolo's Curve Number (Curve Number!) and other R stuff, including some tools for data discovery. Claudia also manages a Google+ group, R4Hydrology.
- nsRFA: this is collection of statistical tools for objective (non-supervised) applications of the Regional Frequency Analysis methods in hydrology made by Alberto Viglione. The package refers to the index-value method and, more precisely, helps the hydrologist to: (1) regionalize the index-value; (2) form homogeneous regions with similar growth curves; (3) fit distribution functions to the empirical regional growth curves.
- Wasim: Helpful tools for data processing and visualisation of results of the hydrological model WASIM-ETH.
- Geotopbricks by Emanuele Cordano, analyses raster maps and other information as input/output files from the Hydrological Distributed Model GEOtop
- hddtools by Claudia Vitolo is a tool for hydrological data discovery.
- waterData is a USGS Package for Retrieval, Analysis, and Anomaly Calculation of Daily Hydrologic Time Series Data
- Lmoments and Lmomco are two packages for the estimation of the L-moments of a distribution.
- The SPEI R Package by Santiago Begueria which includes a set of functions for computing potential evapotranspiration and several widely used drought indices including the Standardized Precipitation-Evapotranspiration Index (SPEI).
- The USGS-R packages at github
- Alessio Pugliese and Attilio Castellarin pREC: a package for the regionalisation of some hydrological variables.
- Alberto Montanari version of Hymod: here.
- Emanuele Cordano work in connecting R with JGrasstools, here, to do geomorphological analysis (slides in Italian here, and in English here) within R.
- meteo package by Kilibarda, Sekulic, Hengl, Pebesma and Graeler, A package for spatio-temporal geostatistical mapping of meteorological data. Global spatio-temporal models calculated using publicly available data are stored in package.
- The reservoir package by Sean Turner [aut, cre], Jia Yi Ng [aut], Stefano Galelli [aut]. It measures single-storage water supply system performance using resilience, reliability, and vulnerability metrics; assess storage-yield-reliability relationships; determine no-fail storage with sequent peak analysis; optimize release decisions for water supply, hydropower, and multi-objective reservoirs using deterministic and stochastic dynamic programming; evaluate inflow persistence using the Hurst coefficient. A companion paper for this tool is available.
- The scripts and data I use in my short mini-classes on R inside my Hydrology and Hydraulic Constructions classes. The include a quick introduction to R, plotting a treating a time serie of rainfall data read from a file, reading and plotting discharges data from a file, estimating the idf curves of rainfall from a standard (Italian standard) set of maxima of annual precipitation, with interpolation of Gumbel probabilities with various methods.
- Anne Van Loon Drougths R code
- Tony Ladson's fresh and informative blog, and its links
- The TSA package contains R functions and datasets detailed in the book "Time Series Analysis with Applications in R (second edition)" by Jonathan Cryer and Kung-Sik Chan
- Here James B. Elsner and Thomas H. Jagger wrote a tutorial for using R for Climate Research.
- Here tools for visualising California snow cover data are provided.
- A visualisation of the Bayesian search of a distribution stimulated by Climate Research.
- Simulating wind speed with R.
- Visualising droughts.
- Visualising NCEP global data
- Resources for Spatial Analysis
- Analysis of Dutch Rainfall data
but also:
- evir
- evdbayes
- ismev
- extRemes [for Windows users (I do not like the packages linked to a platform!!!)]
- SpatialExtremes
Frankly I did not test them all: but usually CRAN packages are really good.
Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:
- Geostatistics: gstat, automap, geoR, fields, RandomFields
- GIS: spgrass6, RSAGA, rgdal, sp, proj4, raster, mapproj, maptools, RGoogleMaps, RArcInfo, RpyGeo,
- Flood frequency: POT, evd, nsRFA, extremes, lmomco
- Optimization: pso, DEoptim,
- High Performance Computing: parallel, snowfall, multicore, jit, nws, Rmpi, snow, taskPR
- Spreadsheets & DB: RPostgreSQL, RMySQL, RSQLite, RNetCDF, RexcelInstaller, xlsReadWrite
- Bayesian statistics: BAS, BLR, ensembleBMA, evdbayes, LearnBayes,
- ramps, spBayes,...
- Latex: xtable, Sweave
- Wavelets: wavelets; wavethresh, wmtsa, Rwave
- Data Mining: Rweka, rattle, party, RandomForest, ...
- Machine Learning tools (in Java with R connection)
If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development.
Sunday, June 10, 2012
Thursday, February 16, 2012
GGPlot: a way to do good graphics and understand data
Making good graphics is very important to understand data and to appropriately promote research achievements. ggplot2 is a recent R package of which there are several good tutorial and introductions. But recently a youtube webinar was posted by the very same author of the package, Hadley Wickham. Spending a little time to listen to him is really worthwhile. Click on the picture to go on the page where the video, the slides, and other information are.
Hadley kindly provided the following resources in his slides:
ggplot2 mailing list: http://groups.google.com/group/ggplot2
stackoverflow: http://stackoverflow.com/tags/ggplot2
Lattice to ggplot2 conversion: http://learnr.wordpress.com/s=lattice
Cookbook for common graphics: http://wiki.stdout.org/rcookbook/Graphs/
ggplot2 book: http://amzn.com/0387981403
Hadley kindly provided the following resources in his slides:
ggplot2 mailing list: http://groups.google.com/group/ggplot2
stackoverflow: http://stackoverflow.com/tags/ggplot2
Lattice to ggplot2 conversion: http://learnr.wordpress.com/s=lattice
Cookbook for common graphics: http://wiki.stdout.org/rcookbook/Graphs/
ggplot2 book: http://amzn.com/0387981403
Subscribe to:
Posts (Atom)















