Rainfall statistics for the Netherlands

One of my most successful postings is the one of R resources for hydrologists.   Sometimes contributions arrive from non -hydrologist. This is the case that  of two posts from Wiekvoet whose authors download and analyse rainfall data from the Netherlands  Royal Institute. The first post mainly performs an exploratory analysis of the data. The second one looks for rainfall changes in the past 100 years. Enjoy the reading.

As a byproduct one learn also how to use ggplot2 through some examples. 

Using ggplot2 for Hydrology

Leonardo Perathoner prepared this for helping my bachelor students:

"Since some student asked to me some information about ggplot, I write it here for everybody.

If you are interested to plots and a graphic quality superior to the standard R, you can utilize the R package ggplot2 !!

Here they are some resources that I want to I would advise to use in your learning process:

1) First a good manual

http://www.bioinformaticslaboratory.nl/twikidata/pub/Education/ComputinginR/ggplot2-book.pdf 

and then, these other sites about ggplot I often use:

2) the official site:

http://docs.ggplot2.org/current/ 

3) a quick cookbook with examples and code snippets

http://www.cookbook-r.com/Graphs/ 

Leonardo "

PS: many other resources here:

http://abouthydrology.blogspot.it/2012/08/r-resources-for-hydrologists.html
http://abouthydrology.blogspot.it/2016/09/cool-r-tydyverse.html

R resources for Hydrologists

R is my statistical software of election. I had hard time to convince my Ph.D. students to adopt it, but finally they did, and, as usually happens, many of them became more proficient than me in the field. Now it seems natural to use it for everything, but this was not always the case. A list of introductory material is here.

A nice series of video tutorial on R was just posted by the Google Developers Group, on youTube, here.

Recently many resources were added for hydrologists, which I list below with a little comment.

New:  a second paper by distinguished colleagues is also on HESSD: Hydrology modelling R packages: a unified analysis of models and practicalities from a user perspective, by Astagnau et al.

New: Now we also have a very informative paper at HESSD: "Using R in hydrology: a review of recent developments and future directions" by Slate et al., 2019. It is a pity they do not cite this blog, but we cannot pretend too much!

• OCNet  OCNet enables the creation and analysis of Optimal Channel Networks (OCNs). These are oriented spanning trees (built on rectangular lattices made up of square pixels) that reproduce all scaling features characteristic of real, natural river networks. 
• DonwscaleR is an R package for empirical-statistical downscaling focusing on daily data and covering the most popular approaches and techniques (quantile mapping, analogs, regression, generalized regression, neural networks). This package has been conceived to work in the framework of both seasonal forecasting and climate change studies and is part of the climate4R framework, formed byloadeR, transformeR, downscaleR and visualizeR.
• bigleaf (version 0.6.5), an open source toolset for the derivation of meteorological, aerodynamic, and physiological ecosystem properties from eddy covariance (EC) flux observations and concurrent meteorological measurements.
• rtop is a package for geostatistical interpolation of data with irregular spatial support such as runoff related data or data from administrative units.
• The boussinesq package is a collection of functions implementing the one-dimensional Boussinesq Equation (ground-water).
• dynatopmodel is a native R implementation and enhancement of the Dynamic TOPMODEL, Beven and Freers’ (2001) extension to the semi-distributed hydrological model TOPMODEL (Beven and Kirkby, 1979).
  RclimTool. It was designed with the aim of facilitating users in statistical analysis for quality control, filling of missing data, homogeneity analysis and calculation of indicators for daily weather series for temperature (maximum and minimum) and precipitation.
• LumpR a tool for HRU delineations (the reference paper is here).
• Cropwat FAO model (it is a decision support system developed by the Land and Water Development Division of FAO for planning and management of irrigation) translated into R.
•  Two socio-hydrological models of human-flood interactions that were recently developed. The two models are similar, but based on different socio-economic variables (wealth and human proximity to rivers VS relative population density). A full description of the two models can be found in the following two papers: HESS (2013), https://www.hydrol-earth-syst-sci.net/17/3295/2013/hess-17-3295-2013.pdf; WRR (2015), http://onlinelibrary.wiley.com/doi/10.1002/2014WR016416/full 
• RWater: A Cyber-enabled Data-driven Tool for Enhancing Hydrology Education
• airGR: Hydrological modelling tools developed at Irstea-Antony (HBAN Research Unit, France). The package includes several conceptual rainfall-runoff models , a snowmelt module and the associated functions for their calibration and evaluation
• airGRteaching:“It is an add-on package to the 'airGR' package that simplifies its use and is aimed at being used for teaching hydrology. The package provides 1) three functions that allow to complete very simply a hydrological modelling exercise 2) plotting functions to help students to explore observed data and to interpret the results of calibration and simulation of the GR ('Génie rural') models 3) a 'Shiny' graphical interface that allows for displaying the impact of model parameters on hydrographs and models internal variables.“
• airGRdatassim: it allows performing streamflow data assimilation in the GR rainfall-runoff models with EnKF and particle filter. 
• lumpR. A tool facilitating landscape discretisation for hillslope-based hydrological models. It is described in a paper on GMDD
• HydroGOF and HydroTSM by Mauricio Zambrano-Bigiarini.  The first provides functions implementing both statistical and graphical goodnes-of-fit measures between observed and simulated values, mainly oriented to be used during the calibration, validation, and application of hydrological models. The second provides functions for management, analysis, interpolation and plotting of time series used in hydrology and related environmental sciences.  Mauricio also had a poster at EGU 2010 general assembly on the topic.
• Jasper Vrugt's DREAM calibration method
• RMWAGEN by Emanuele Cordano which is a weather generator, a package that contains functions for spatial multi-site stochastic generation of daily timeseries of temperature and precipitation. A presentation can be found here.
• Other stochastic generators of precipitation can be found here. Do not forget to explore the links in that page, and particularly the presentations given at the Roscoff's Workshop on stochastic generators, where many examples are in R
• The RHydro which included TOPMODEL (apparently not anymore supported), tools for DEM analysis (this last type of tools however are also available through the work by R. Bivand, E.J. Pebesma and V. Gomez-Rubio ), an implementation of the FUSE by Clark et al (2008) methodology, and many other tools for hydrological analysis. These were initially promoted by Wouter Buytaert and Dominik Reusser who also gave a nice tutorial at EGU a few years ago.
• Hydromad: It provides a modelling framework for environmental hydrology: water balance accounting and flow routing in spatially aggregated catchments. It supports simulation, estimation, assessment and visualisation of flow response to time series of rainfall and other drivers
• TUWmodel  is a lumped conceptual rainfall-runoff model, following the structure of the HBV model. The model runs on a daily time step and consists of a snow routine, a soil moisture routine and a flow routing routine. See Parajka, J., R. Merz, G. Bloeschl (2007) Uncertainty and multiple objective calibration in regional water balance modelling: case study in 320 Austrian catchments, Hydrological Processes, 21, 435-446
• Sean Turner's and Stefano Galelli's, reservoir package, Tools for Analysis, Design, and Operation of Water Supply Storages. Sean Turners's also buildes Scenario 

• R Code: Handy routines for Hydrologists by Dan Moore and others.
• Hydrosanity: It provides a graphical user interface for exploring hydrological time series. It is designed to work with catchment surface hydrology data (mainly rainfall and streamflow time series at a set of locations). There are functions to import from a database or files; summarise and visualise the dataset in various ways; estimate areal rainfall; fill gaps in rainfall data; and estimate the rainfall-runoff relationship. Probably the most useful features are the interactive graphical displays of a spatial set of time series. (This project seems actually being abandoned). 
• aqp: Algorithms for quantitative pedology. A collection of algorithms related to modeling of soil resources, soil classification, soil profile aggregation, and visualization by Dylan Beaudette and Pierre Roudier. A paper talking about it is given here. And a presentation is not missing.
• A package for plotting soil water retention curves and hydraulic conductivity by Emanuele Cordano, Fabio Zottele and Daniele Andreis is soilwater.
• soilDB, of the same authors of aqp, is useful to access some soil databases.
• soiltexture: Functions for soil texture plot, classification and transformation by Jules Moeys
• Hydrome: This package estimates the parameters in infiltration and water retention models by curve-fitting method.
• SoilWater address to a couple of packages for estimating Soil Water Retention Curves and some Pedotransfer Functions
• hydropso: This package implements a state-of-the-art version of the Particle Swarm Optimisation (PSO) algorithm, with a special focus on the calibration of environmental models.
• Evapotranpiration: by Dan Lu Guo and Seth Westra. This package estimates Potential and Actual Evapotranspiration with multiple models (see also the paper here).
• EcoHydRology developed by DR. Fuka, MT Walter, JA Archibald,  TS Steenhuis, and ZM Easton which presents a community modeling foundation for Eco-Hydrology. 
• Claudia Vitolo's Curve Number (Curve Number!) and other R stuff, including some tools for data discovery. Claudia also manages a Google+ group, R4Hydrology.
• nsRFA:  this is collection of statistical tools for objective (non-supervised) applications of the Regional Frequency Analysis methods in hydrology made by Alberto Viglione. The package refers to the index-value method and, more precisely, helps the hydrologist to: (1) regionalize the index-value; (2) form homogeneous regions with similar growth curves; (3) fit distribution functions to the empirical regional growth curves. 
• Wasim: Helpful tools for data processing and visualisation of results of the hydrological model WASIM-ETH.
• Geotopbricks by Emanuele Cordano, analyses raster maps and other information as input/output files from the Hydrological Distributed Model GEOtop
• hddtools by Claudia Vitolo is a tool for hydrological data discovery.
• waterData is a USGS Package for Retrieval, Analysis, and Anomaly Calculation of Daily Hydrologic Time Series Data
• Lmoments and Lmomco are two packages for the estimation of the L-moments of a distribution.
• The SPEI R Package by Santiago Begueria which includes a set of functions for computing potential evapotranspiration and several widely used drought indices including the Standardized Precipitation-Evapotranspiration Index (SPEI). 
• The USGS-R packages at github
• Alessio Pugliese and Attilio Castellarin pREC: a package for the regionalisation of some hydrological variables.
• Alberto Montanari version of Hymod: here.
• Emanuele Cordano work in connecting R with JGrasstools, here, to do geomorphological analysis (slides in Italian here, and in English here) within R.
• meteo package by Kilibarda, Sekulic, Hengl, Pebesma and Graeler, A package for spatio-temporal geostatistical mapping of meteorological data. Global spatio-temporal models calculated using publicly available data are stored in package.
• The reservoir package by Sean Turner [aut, cre], Jia Yi Ng [aut], Stefano Galelli [aut]. It measures single-storage water supply system performance using resilience, reliability, and vulnerability metrics; assess storage-yield-reliability relationships; determine no-fail storage with sequent peak analysis; optimize release decisions for water supply, hydropower, and multi-objective reservoirs using deterministic and stochastic dynamic programming; evaluate inflow persistence using the Hurst coefficient. A companion paper for this tool is available.
• The scripts and data I use in my short mini-classes on R inside my Hydrology and Hydraulic Constructions classes. The include a quick introduction to R, plotting a treating a time serie of rainfall data read from a file, reading and plotting discharges data from a file, estimating the idf curves of rainfall from a standard (Italian standard) set of maxima of annual precipitation, with interpolation of Gumbel probabilities with various methods.
• Anne Van Loon Drougths R code

Blogs about R and Hydrology

• Tony Ladson's fresh and informative blog, and its links

Time series Analysis (among the many others):

• The TSA package contains R functions and datasets detailed in the book "Time Series Analysis with Applications in R (second edition)" by Jonathan Cryer and Kung-Sik Chan

A miscellany of topics:

• Here James B. Elsner and  Thomas H. Jagger wrote a tutorial for using R for Climate Research.
• Here tools for visualising California snow cover data are provided.
• A visualisation of the Bayesian search of a distribution stimulated by Climate Research.
• Simulating wind speed with R.
• Visualising droughts.
• Visualising NCEP global data
• Resources for Spatial Analysis
• Analysis of Dutch Rainfall data 

In R there are also many toolkit for the analysis of extreme events. The basic is:

• evd

but also:

• evir
• evdbayes
• ismev
• extRemes [for Windows users (I do not like the packages linked to a platform!!!)]
• SpatialExtremes

can be useful.
Frankly I did not test them all: but usually CRAN packages are really good.

Finally, Mauricio Zambrano suggested also a series of other CRAN R packages that could be useful:

• Geostatistics: gstat, automap, geoR, fields, RandomFields
• GIS: spgrass6, RSAGA, rgdal, sp, proj4, raster, mapproj, maptools, RGoogleMaps, RArcInfo, RpyGeo,
• Flood frequency: POT, evd, nsRFA, extremes, lmomco
• Optimization: pso, DEoptim,
• High Performance Computing: parallel, snowfall, multicore, jit, nws, Rmpi, snow,  taskPR
• Spreadsheets & DB: RPostgreSQL, RMySQL, RSQLite, RNetCDF, RexcelInstaller, xlsReadWrite
• Bayesian statistics: BAS, BLR, ensembleBMA, evdbayes, LearnBayes,
• ramps, spBayes,...
• Latex: xtable, Sweave
• Wavelets: wavelets; wavethresh, wmtsa, Rwave
• Data Mining: Rweka, rattle, party, RandomForest, ...
• Machine Learning tools (in Java with R connection)

R for hydrology is a blog dedicated to activities related to hydrology performed by using R.

If you arrived here and you never used R, you can start from here.
Finally, you also would like to know why it is so hard to learn R. This blogpost clarify it, and also why I am using other languages for real model development. 

GGPlot: a way to do good graphics and understand data

Making good graphics is very important to understand data and to appropriately promote research achievements. ggplot2 is a recent R package of which there are several good tutorial and introductions. But recently a youtube webinar was posted by the very same author of the package, Hadley Wickham. Spending a little time to listen to him is really worthwhile. Click on the picture to go on the page where the video, the slides, and other information are.

Hadley kindly provided the following resources in his slides:
ggplot2 mailing list: http://groups.google.com/group/ggplot2
stackoverflow: http://stackoverflow.com/tags/ggplot2
Lattice to ggplot2 conversion: http://learnr.wordpress.com/s=lattice
Cookbook for common graphics: http://wiki.stdout.org/rcookbook/Graphs/
ggplot2 book: http://amzn.com/0387981403

40 fascinating blogs on statistics

All the following is from:

http://www.bschool.com/blog/2011/40-fascinating-blogs-for-the-ultimate-statistics-geek/

There is no need of further comments:

At some point in nearly every (but not all) major and career path, statistics will be encountered. Even tangentially, they still define and describe everything from the efficacy of cancer medications to earned run averages, and everything in between. Understandably, the near-universality of the subject attracts scads of admirers, and the internet provides them with a lovely public platform to trade ideas and insights through blogs, forums, communities, podcasts and more. While not comprehensive, the following list offers up an eclectic mix of blogs to pique the interest of many different breeds of the statistics geek.

1- Probability and statistics blog: Matt Asher finds statistics and probability everywhere, and his blog offers up some amazing, educational and provocative content.


2 -Xi'an's Og: Not all of the content presented here revolves around statistics, but this Universite Paris Dauphine professor sticks mainly with research, papers, events, computations and Bayesian subject matter.


3- Three-Toed Sloth: Math and statistics geeks wanting to see the disciplines applied to current events, science and plenty more will have plenty to love about Three-Toed Sloth.


4- Statisfaction: PhD students affiliated with the CREST research institute blog extensively about statistics, economics and plenty more relevant, often overlapping topics.


5- Freakonometrics: Anyone with a particular affinity for statistics as it relates to business, economics, finances and more seriously needs to give Arthur Charpentier's blog a visit. Those who do not speak French need only run it through Google Translate to read what he has to say.


6- R-bloggers: 140 contributors discuss the latest news and views from across the R community, which focuses on computational software for statisticians.


7- Statistical modeling, Causal Inference, and Social Science: From Dartmouth comes a frequently-updated, highly intelligent resource packed with discussions about statistics, art, political science and plenty of models.


8-The Endeavour: John D. Cook offers math, probability and statistics geeks plenty of intelligent commentary on a wonderful array of related subjects.


9- Revolutions: Give Revolution Analytics' blog a look for detailed discussions about their R programs, open source and — of course — statistics.


10- Numbers Rule Your World: Statistics are absolutely everywhere, and this blog peers into how they permeate daily life as well as marketing, business and economics.


11-The Numbers Guy: Many statistics buffs turn to this Wall Street Journal offering glimpses in how the media and the public use and abuse numbers every day.


12- Observational Epidemiology: Epidemiology, higher education and statistics collide in one frequently updated, insightful resource about the practical applications of the maths and sciences.


13 -Understanding Uncertainty: University of Cambridge's Statistical Lab and the Winton Programme it hosts hope to educate the public about the mathematical risks and uncertainties associated with numerous important topics and current events.


14 - Five Thirty Eight: Nate Silver with The New York Times blogs about the numbers that drive politics, with statistics and calculus forming the site's core.


15 -The Stats Blog: While this resource may not update as often as some of the others listed here, both The Stats Blog and its accompanying website deserve plenty of attention.


16 - Social Science Statistics Blog: Statistics fans who enjoy applying the associated formulas to psychology, sociology and other "soft" sciences would do well to follow Harvard's myriad takes on the matters at hand.


17 - Significance: As both a blog and a magazine, Significance brings statistics to the masses, making it a valuable resource for those new to the discipline.


18 - Realizations in Biostatistics: Check out this resource by a Bayesian biostatistician for some interesting, educational research on the numerical world of medicine and more.


19 - Blog about Stats: Everything readers need to know about this multi-author reference can be found right there in this title — it's a place where anyone who enjoys the subject can come and talk with like-minded individuals.


20 - R-statistics blog: It may update comparatively infrequently to some of the others, but readers curious about R and open source will still enjoy the content here.


21 - R-Chart: Take a look at R from the perspective of a web and database developer, who illustrates main points with some excellent, effective visuals.


22 - The Numbers: Gary Langer explains the statistical and mathematical findings behind recent polls and political undertakings.


23 - Lies, damned lies and statistics: Another seriously cool read merging daily life and statistics, this time focusing on social media, PR, branding and other topics of interest to businesspeople.


24 - Empirical Legal Studies: Several different writers dissect legal issues using statistical models and other empirical means in this enlightening resource.


25 - Stats Make Me Cry: In spite of its glacial schedule, Stats Make Me Cry is still worthwhile reading for those who enjoy learning as much as they can about the eponymous discipline.


26 - CoolData blog: Kevin MacDonnell researches and data mines in order to find interesting tidbits for anyone needing to raise funds for higher education. Statistics, naturally, finds its way into more than a few postings.


27 - Stats with Cats blog: Along with comforting pictures of feline friends, Charlie Kufs' accompaniment to his book makes statistics palatable for eager novices.


28 - LoveStats: For the statistician who embrace social media and market research, LoveStats delivers plenty to pique their interest.


29 - StatChat: Stay on top of news and views from around the worlds of applied statistics, data analysis and their intersections.


30 - Sabermetric Research: Baseball greatly appeals to plenty of statistics geeks out there, but visitors who prefer other sports still see some favorites represented here sometimes.


31 - Statistical Sage Blog: Four different writers keep a general statistics blog that plenty of students and professionals needing to collect data will find a valuable reference.


32 - Fishing in the Bay: Almost everything statistical eventually shows up here at Fishing in the Bay, which looks at things through an Antipodean lens.


33 - Blogs at AnalyticBridge: Scroll down to the "Blog Posts" section on the main page to read a broad number of perspectives from across the statistics and data analysis communities.


34 - Beyond the Box Score: For sabermetrics buffs, this highly engaging forum and blog delves deeply into the numbers that drive baseball.


35 - Mathematics & Statistics at Williams College: Another multi-author resource, offering statistics fans a chance to seek out different opinions and ideas regarding math, education, research and more.


36 - Mr. C's AP Statistics Blog: Visitors hoping to perhaps teach high school statistics someday may find this resource useful, though it does not update as often as some of the others on this list.


37 - dataists: Discover helpful methods the make gathering and analyzing statistical data as easy and painless as possible.

38 - FlowingData: Nathan Yau creatively blends the technical with the creative in his innovative visuals incorporating various statistical findings.


39 - Byte Mining: A statistician-turned-computer scientist talks about both his fields, along with the expected intersections and overlaps.


40 -information aesthetics: Another seriously cool blog blending statistics and data analysis with the latest trends in graphic design and art.

How Extremes Change Over Time

From EOS, vol 92, No 2, 11 January 2011:

" Average behavior is often studied, with well developed techniques from the field of statistics allowing for inferences to be readily made. However, in many atmospheric, hydrologic, and other Geophysical problems extremes are often of great interest. .... "

Here a link to the complete article:

from EOS







Here below a link to:

Richard Katz's work