Tuesday, November 29, 2016

Water, energy and carbon balance of a peatland catchment in the Alps

When I started the enterprise called GEOtop, it was at this kind of studies that I was thinking, where experimenters meet theoreticians and/or modeler. In this case, we are studying a small peatland on a plateu on top of Monte Bondone, one the mountains close to Trento. It is usually overlooked in favour of the great dolomites we have all around here. However, it has its beauty, and I can tell you that it is a pleasure to walk over there and see its flora, in summer, or skiing there in winter.
This paper deals with the water budget of the biotope of the plateau, and its carbon cycle. The paper used measurement campaign, laboratory analysis and accurate modelling that push to the limit of GEOtop capabilities.

The paper, entitled: "Water, energy and carbon balance of a peatland catchment in the Alps "by Jeroen Pullens et al.,  has this abstract: " Over millennia, peatlands have stored around 30% of the global soil organic carbon.  Peat is formed and accumulated due to the slow decomposition rate of organic matter in the waterlogged, anaerobic soils. Therefore, the understanding of the water cycle of peatlands is important in evaluating the functioning of peatlands. To be able to study  these dynamics, the process-based hydrological model GEOtop and an appropriate set of in situ measurements were used. They were functional to simulate 4 years (2012- 2015) of the water and energy dynamics of an alpine catchment in Italy, which included a peatland. The modelled energy fluxes are comparable to the fluxes  measured by the on-site eddy covariance tower. The modelled water cycle was used  to quantify the loss of dissolved organic carbon (DOC) and to calculate the carbon balance of the peatland. The model outcomes showed an overall good fit with the  measurements during the snow-free period. During snow cover, the model had difficulties simulating the soil temperature due to insulation by the snow. Based on the measured DOC and the modelled discharge, the DOC adds another source of carbon, since the presented peatland is already acting as a carbon source based on  carbon fluxes. The total amount of loss of DOC (10.2 (± 3.8) g C m-2 yr-1) is comparable to other peatlands. In total, the peatland had a carbon balance (CO2, CH4 fluxes and DOC losses combined) of 112.3, 273.8, 190.8 and 95.3 g C m-2 yr-1 for 2012, 2013, 2014 and 2015 respectively. 

You can find the pdf of the manuscript by clicking on the figure above.

Friday, November 25, 2016

Python resources for Hydrologists

Python is a modern object oriented language. Occasionally I wrote about it in my posts, also for remarking that I went in a different direction. However, I cannot deny the evidence that more and more people are choosing it, and there are good reasons, as their language of choice for doing research and hydrological applications. In fact since 2017, I am using it in place of R for scripting and data managing. Below you will find a list of resources. Please do not hesitate to bring my attention to yours or others' contributions which I have not yet in my group.

Motivations for Python use, over other choices, can be found in this blog post or in this paper.

To understand how to start you can follow Python programming for hydrology students that starts with indicating how to install it.

For who wants to start with Python (for hydrologists), I suggest to give a look to my blog post Python general resources. For others, please give a look below.

Python is especially use as a glue for existing program, either written in C or FORTRAN. We have the cases of
  • CFM is a programming library to create hydrological models. Although written in C++, it has a Python interface 
  • ESMF regridding has been interfaced with Python ESMPy
  • GRASS GIS has been interfaced with Python
  • Python is also interfaced to gvSIG, as you can see here
  • HPGL a High Performance Geostatistics Library. Written in C++ is glued together by Python 
  • MODFLOW the groundwater model is interfaced by FloPy. Documentation and other information is here
  • PcRaster - Is a collection of software targeted at the development and deployment of spatio-temporal environmental models. It has a python interface which is constantly being enhanced. 
  • OpenHydrology is a library of open source hydrological software written in Python to operate as packages under an umbrella interface 
  • PyHSPF Python extensions to the Hydrological Simulation Program in Fortran (HSPF
  • PyQGIS: A Python interface to QGIS 
  • RhessysWorkflow  RHESSysWorkflows provides Python scripts for building RHESSys models. Other Pythonic material on RHESSys can be found here.
  • UWHydro tools for connecting University of Washington hydrological models, and, in particolar, the VIC driver PythonDriver

In the reign of hydrologic applications entirely written in Python, we remind:

  • ANUGA 2 - package for modelling dam breaks, riverine flooding, storm-surge or tsunamis. In Python and C. 
  • EcoHydrolib provides a series of Python scripts for performing ecohydrology data preparation workflows. 
  • evaplib: Python library containing functions for calculation of evaporation rates. Functions include Penman open water evaporation, Makkink reference evaporation, Priestley Taylor evaporation Penman Monteith (1965) evaporation and FAO's Penman Monteith ET0 reference evaporation for short, well-watered grass. In addition there is a function to calculate the sensible heat flux from temperature fluctuation measurements. View documentation of evaplib module functions. Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
  • A GLUE, Generalised Likelihood Uncertainty Estimation (GLUE) developed by Framework Joost Delsman, at Deltares, 2011 
  • Groundwater flow modelling manual for Python written by Vincent post 
  • Hydro-conductor: A set of Python scripts and modules written to couple a hydrologic model with a regional glacier model
  • ODMToolsPython and ODMTools ODMTools is a python application for managing observational data using the Observations Data Model. ODMTools allows you to query, visualize, and edit data stored in an Observations Data Model (ODM) database.ODMTools was originally developed as part of the CUAHSI Hydrologic Information System. YOu can find a presentation about here
  • PyETo is a package for calculating reference/potential evapotranspiration (ETo). 
  • Python script for rectangular Piper plot (version December 2014): Python script for plotting chemical data in a rectangular python plot (see image) according to Ray and Mukherjee (2008) Groundwater 46(6): 893-896. Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
  • Python script for multiple Stiff plots (version June 2011): Python script for preparing multiple Stiff diagrammes (see image). Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
  • Haran Kiruba tools for hydrology Not really clear what he does. 
  • USEPA site contains various python (and other languages) tools, including an interface to Epanet and SWMM (a connection to swmmtoolbox is also avilable here). 
  • Also USGS has its python tools
  • sMAP 2.0 is a tutorial will cover how to retrieve data from a sMAP archiver using Python. 
  • ulmo clean, simple and fast access to public hydrology and climatology data 

Specific hydrological Hydrological Models are enumerated below.

  • EXP-HYDRO Model is a catchment scale hydrological model that operates at a daily time-step. 
  • Landlab Landlab is a python-based modeling environment that allows scientists and students to build numerical landscape models. Designed for disciplines that quantify earth surface dynamics such as geomorphology, hydrology, glaciology, and stratigraphy, it can also be used in related fields. 
  • LHMP - lumped hydrological models playground - tiny docker container with complete environment for predictions.
  • PyCatch is a component based hydrological model of catchments built within the PCRaster Python framework. The code is here. A related paper, here
  • PyTOPKAPI is a BSD licensed Python library implementing the TOPKAPI Hydrological model (Liu and Todini, 2002). The model is a physically-based and fully distributed hydrological model, which has already been successfully applied in several countries around the world 
  • SPHY. See for details on model and publications (HESS, Nature, etc) here. Just recently a new paper on climate change and mountain hydrology in PloS came out, using SPHY model, more info here.
  • Topoflow a python hydrologic model by Scott Peckham 
  • WOFpy is an implementation of CUAHSI's Water One Flow service stack in python 
  • wflow is a distributed hydrological model platform that currently includes two models: the wflow_sbm model (derived from the topog_sbm soil concept) and the wflow_hbv model which is a distributed version of the HBV model. This is actully part of a larger Deltares project called OpenStream

GIS capabilities are also present:

Also tools to deal with Meteorology:

  • meteolib: Python library containing meteorological functions for calculation of atmospheric vapour pressures, air density, latent heat of vapourisation, heat capacity at constant pressure, psychrometric constant, day length, extraterrestrial radiation input, potential temperature and wind vector. The documentation for this module is presented at here (meteolib module functions web site). Functions to convert event-based data records to equidistant time-spaced records (event2time) and to convert date values to day-of-year values (date2doy) are now in a separate meteo_util module. Documentation is presented here (meteo_util module functions web site). Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
  • MetPy is An Open Source Python Toolkit for Meteorology 
  • Melodist (MEteoroLOgical observation time series DISaggregation Tool) is an open-source software package written in Python for temporally downscaling (disaggregating) daily meteorological time series to hourly data. It is documented in a GMD paper by Forster et al., 2016. 
  • Various resources for meteorology can be found in the pyaos blog
Statistical and data analysis tools are abundant
  • CUAHSI time series viewer
  • The basic cheatshit
  • NetCDF file operations are available here. However, there is also txt2netcdf which containsvarious Python functions for importing text into NetCDF data files (creating files, adding variables, listing structure, etc.), developed by Ko van Huissteden. 
  • Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. (A short tutorial is also here
  • Extreme distribution (from scipy.stats) is here
  • An example of use of Pandas for analysing time series 
Visualisation is well served
  • ggplot is a plotting system for Python based on R's ggplot2 and the Grammar of Graphics. It is built for making profressional looking, plots quickly with minimal code. 
  • An impressive tour of Python possibilities in this field is given by “Regress to Impress” 
  • VisTrails an open-source scientific workflow and provenance management system that supports data exploration and visualization. Its website is here
  • uvcmetrics metrics aka diagnostics for comparing models with observations or each other. This is part of the Uv-CDAT website which contains also other visualisation tools. 
Tools for dealing with uncertainty and sensitivity analysis
Yet another repository of Python models and resources
A final comment

I am actually impressed by the quality of the contributions in Python. I think there is not anymore reason to use commercial program like Matlab in Universities (see this review here). What Matlab does, also Python does. Compared to R, it has a much more clear sintax and is, certainly, a better language. So I would suggest to use it with students (or R). I like R but I never built program on it, because its object orientation is really poor. Python is better and, as it is known its syntax is clean. Python is great to link FORTRAN and C/C++ native libraries, so actually many uses it to assembled libraries they wrote in those more performant languages.
As you know, however, my group uses Java as its principal programming language. Java, in comparison with Python, is less immediate and more verbose, but it allows to build many framework to work with, and is usually faster than Python. Probably because, Java is supported by magnificent building tools (Maven, Gradle) that allow to manage large projects in a way that probably cannot be done in Python.

Monday, November 21, 2016

Zenodo and Joss

Here I am robbing from Living in an Ivory Basement blog. Titus Brown (GS) introduces two interesting tools. Maybe the word “tools” is an inappropriateas description.

Zenodo is a way to archive software products, under the idea that GitHUB is not an archival:

“A common point of concern, however, is that GitHub repositories are not archival. That is, you can modify, rewrite, delete, or otherwise irreversibly mess with the contents of a git repository. And, of course, GitHub could go the way of Sourceforge and Google Code at any point. … But! Never fear! The folk at Zenodo and Mozilla Science Lab (in collaboration with Figshare) have solutions for you! … Zenodo is an open-access archive that is recommended by Peter Suber (as is Figshare). … Items will be retained for the lifetime of the repository. This is currently the lifetime of the host laboratory CERN, which currently has an experimental programme defined for the next 20 years at least.“

For more information, nothing better than reading the original blog post.

JOSS stands for Journal of Open Source Software and it promises to be a vehicle to publish software, not what software is about, or a description of the software and its peculiarities, but of the software, which seems to go through a process of peer review, extremely useful and new. You can find the Joss post here.

Sunday, November 13, 2016

The Soil Water Retention Curves

When dealing with soils you are forced to implement mass conservation dependent on two variables, the dimensionless water content, usually named $\theta$ and suction, $\psi$, i.e. The energy contained in a volume of soil per unit mass. Therefore, to solve the budget, you need (at least) to get a new relationship which connects them.  This relation is called soil water retention curve. The plural in the title means that there are many. At least one for any soil type. 

In fact,  the  relationship, and precisely $\theta(\psi)$, is dependent on soil types and structure (and some other factor probably, like temperature, organic content etc). It is a statistical quantity, which averages the behavior of many pores, and an ensamble of water injecting/extracting possibilities.  
The figure below from Lu (GS) and Godt (GS) book (2013) is a clear visualisation of the problem.

The same Ning Lu, in a recent paper (2015) tried to disentangle the various forces acting on water when in pores, and obtained what is shown below.

As expected, the forces acting are not all of the same type, at varying suction values. At very high suction, adsorption forces act in which single water molecules adhere to soils. When more layers of water molecule add, water constitute  thermodynamics compounds, whose equilibrium is globally determined in between adhesion forces, bulk water weights, surface of water and air gas interactions, and which is usually known as capillarity.  
Laws governing capillarity are described by Young-Laplace and  Kelvin laws.  Some insight of the therodynamics of these phenomena (an excellent explanation, indeed) can be found in the first pages of Steudle (2001) review about plant-root suction. 
At this stage liquid water seems to, constitute a disconnected phase, while air gas is continuous inside the medium pores.  
Increasing the water content water becomes a continuos medium and usual hydrodynamics laws become valid.  A recent review of parameterizations of the soil water retention curves (not particularly deep or brilliant though) is given by Too et al. (2014) that cites other older reviews.

When pressure increase, however, we can have two effect which partially depends on how wetting happens. If wetting happens through some sort of flooding then air can stay trapped in pores and decrease the space available for water. The net effect is  associable to a decrease of porosity. However, when water fills all the space (i.e. the soil is saturated) the soil matrix cannot be considered anymore rigid. 

Assume it would be rigid. Then water content could not increase, any pressure applied to the saturated soil would transmit instantaneously through the water volume and water would be expelled where pressure is not applied or there is less pressure in a sort of piston flow.  
Instead, because the medium is not rigid, any pressure is transmitted with a certain speed, and pressure waves can be measured. This fact implies that after saturation, the system behaves as porosity increases and, at the same time pressure varies.  

From a practical point of view, soil water retention curves can be extended to positive pressure (negative suctions) adding a term which is well known in groundwater literature and is called specific specific storage
These qualitative descriptions do not end the complex phenomenology of water retention curves.  

As Nunzio Romano (GS) and coworkers noticed, and Kosugi (1994) before them, soil water retention curves shape depend directly on the pores' distribution. This, however, is not necessarily a unimodal distribution but can be multimodal because of soil structure and soil "disturbances" in form of macropores due to animal or roots decay. In this case soil water retention curves (their integral) can be more complex than expected, as shown in Figure below.

This opens to a series of generalisation, but it would be the topic of some other post (and actually was already the topic of several posts on soil freezing).


Kosugi, K. 1994. Three-parameter log-normal distribution model for soil water retention. Water Resour. Res. 30:891–901. 

Lu N, Godt JW. Hillslope Hydrology and Stability. Cambridge: Cambridge University Press; 2013. 

Lu, N. (2016). Generalized Soil Water Retention Equation for Adsorption and Capillarity. Journal of Geotechnical and Geoenvironmental Engineering, 142(10), 04016051–15. http://doi.org/10.1061/(ASCE)GT.1943-5606.0001524

Romano, N., Nasta, P., Severino, G., & Hopmans, J. W. (2011). Using Bimodal Lognormal Functions to Describe Soil Hydraulic Properties. Soil Science Society of America Journal, 75(2), 468. http://doi.org/10.2136/sssaj2010.0084

Steudle, E. (2001). The Cohesion-Tension Mechanism and the Acquisition of Water by Plant Roots. Annual Review of Plant Physiology-Plant Molecular Biology, 847–877.

Too, V. K., Omuto, C. T., Biamah, E. K., & Obiero, J. P. (2014). Review of Soil Water Retention Characteristic (SWRC) Models between Saturation and Oven Dryness. Open Journal of Modern Hydrology, 04(04), 173–182. http://doi.org/10.4236/ojmh.2014.44017

Monday, November 7, 2016

Reservoirology #3

This is a revision of the previous post on the same topic. There I tried to develop my own algebra of symbols to represent coarse grained (spatially integrated) hydrological system. Later on I understood that Petri networks were already there and useful to obtain the same result. The graphs obtained in such a way where, besides, studied in several places, and many contributes of literature convergent from other disciplines, can be used for hydrological scopes.
The presentation (click on the figure) completely substitutes the old one. Who liked it, will like better this.  When you will be done with this post, consider that there is also Reservoirology #4.


Sunday, November 6, 2016

About graphs, DSL and replicable research in Francesco Serafin's work

This is the summary of what Francesco Serafin (his blog) did in its first year of doctoral studies, defending for his admission to his second year Ph.D. Undoubtedly he did a lot of work and he programs to do even more. Three are the lines of his research:

  • implementing a new flexible structure based on graphs for commanding simulations of interacting systems; 
  • implementing a domain specific language for doing environmental models (and particularly to solve ordinary and partial differential equations); 
  • deploying a system that makes easier to do replicable science. 
Clicking on the Figure above you will find his presentation. Obviously the committee admitted him to the second year of his Ph.D. studies.

Thursday, November 3, 2016

Adige River Research in GLOBAQUA and CLIMAWARE projects

This contains the presentation for the Diffuse seminar held November the 4th in 20 italian cities. This is to do memory of the events of November 1966, fixty years ago, when many Italian cities were flooded.  The presentation is not actually about flooding but the general management of water in medium and large (well someone could consider it small, but it is the second largest in Italy, and its complexity is quite overwhelming) catchments, like River Adige, which is the basin where I live (actually my house is in a prone to flooding area so, all my interest is that it will not happen anymore).
Possibly I will upload also an English version of it for the interested reader. Click on the figure above to see the presentation.