Lectures on probability and second order random fields

This is a gift from twenty five years ago. These are lectures from a short course held in Padua at the Department of Applied Mathematics. We were younger then ! I remember the lectures by Diego Bricio Hernandez as exciting and interesting. Looking back at the nineties, one of the dominant topics, were random fields and the interplay of randomness with hydrological phenomena. The work of Gedeon Dagan was one of the growing paradigms. But random fields and techniques (see Bras and Rodriguez-Iturbe book) were ubiquitous in Hydrology. Here they come the Lectures by Diego Bricio Hernandez, a Mexican scholar in sabbatical at Padua University.

You can find the same lectures also on Google Books, but clicking on the figure above, you will have the pdf.  Diego also wrote this "On some guiding principles in mathematical modelling with special emphasis on determinism". Oldies but goodies.

A travel time model for estimating the water budget of complex catchments

This is the presentation given by Marialaura Bancheri for her admission to the final exam to achieve a Ph.D. in Environmental Engineering. It contains a synthesis of her studies about spatially integrated models of the water budget, and about travel time theory. A model structure is also presented preliminarily containing five reservoirs.

These reservoirs model the hydrology  of a  Hydrologic Response Unit (HRU) of a basin  which are connected together to treat a river catchment (as shown in Rigon et al. 2016). The figure above is a Petri net representation of the set od ordinary differential equations that  constitute the mathematical models of a HRU. The model uses the river network structure to organise the components execution, a work made conjointly with Francesco Serafin.

By clicking on the Figure, you will see Marialaura's presentation.

A list of papers estimating the water budget at various scales

Since almost couple of decades I am trying to develop tools that evaluate the surface water budget components, and I look at the closure of the budget equation.  The outcomes of this research are our models GEOtop and our system JGrass-NewAGE, and some applications are listed below. My impression was that many researchers are talking of the water budget closure, since many actually have the knowledge and the tools for estimating the budget, but less are really doing it. We wrote  (more or less)  it in the introduction of one of our paper and we were asked by the reviewers to be more precise. The list below, certainly to be cleaned, improved and enriched, says that there are effectively many papers (out of around a hundred we inspected) that do it. They are distributed according to four  threads. 

• The one of “global and continental hydrology” where the water budget of the whole earth or of the largest river basins is studied. The methods used are remote sensing data, global circulation models, large-scale hydrological models.
• One is (but the focus is more often on evaporation) based on the use of Budyko curves, at various scales. 
• The  use models plus in-situ data, with various levels of simplification, usually from few kilometers to thousands of kilometers scales. Models are process-based (like  CATHY,  GEOtop  or ParFlow, to cite three of them)  or more conceptualised (as our JGrass-NewAGE). Data are the most various, depending on the spatial scale of the application and the type of model. Process-based models use more data (which is a richness or a weakness, depending on the point of view), while conceptual models use less data. Larger scale applications require a coarse graining of the data set and, obviously, a limitation in the description of spatial heterogeneity. 
• Finally there are fully experimental papers, especially in forest and agricultural areas, with accurate measurements, for some specific plant stand, or even single trees.

In the selection of the paper below, I searched for the water budget equation, with all of the terms, its minimal expression being:

$\frac{\Delta S}{\Delta t} = P - ET - R$

where $S$ is the soil/groundwater storage, $P$ precipitation rate, $ET$ evapotranspiration, $R$ runoff. Various papers present a more articulated baudget, but certainly I did not listed the paper that not deals with the equation. Many papers, having “water budget” in the title, actually deal with evapotranspiration and were excluded. As Praveen Kumar (GS) argued to me, all good models preserve mass: but they often deal only with a part of the budget, and/or their authors are  concerned with other specific topic. Also these papers (and some really very  interesting were excluded).  Finally, please find below the list. A different version of the same list (and its LaTeX editable version)  with some comments about the spatial and temporal scales of the budget and some further information can be found here ( where references can be sorted).

P.S. - Another list (to review, just received from Roger Moussa) of water budget studies is here.

References

Adelana, S. M., Dresel, P. E., Hekmeijer, P., Zydor, H., Webb, J. A., Reynolds, M., & Ryan, M. (2014). A comparison of streamflow, salt and water balances in adjacent farmland and forest catchments in south-western Victoria, Australia. Hydrological Processes, 29(6), 1630–1643. http://doi.org/10.1002/hyp.10281

Arnold, J. C., & Allen, P. M. (2016). Estimating hydrologic budgets for three Illinois watersheds. Journal of Hydrology, 176, 57–77.

Azarderakhsh M,  Rossow WB, Papa F,  Norouzi H, Khanbilvardi R. Diagnosing water variations within the Amazon basin using satellite data. Journal of Geophysical Research: Atmospheres 116 (2011).

Batelaan, O., & De Smedt, F. (2007). GIS-based recharge estimation by coupling surface–subsurface water balances. Journal of Hydrology, 337(3-4), 337–355. http://doi.org/10.1016/j.jhydrol.2007.02.001

Bertoldi, G., Rigon, R., & OVER, T. M. (2005). Impact of watershed geomorphic characteristics on the energy and water budgets. Journal of Hydrometeorology, 1–29.

Brye, K. R., Norman, J. M., Bundy, L. G., & Gower, S. T. (2000). Water-Budget evaluation of Prairie and Maize Ecosystems, 64, 715–724.

Chen J,  Lee C, Tian-Chyi Yeh J, Yu J. A Water Budget Model for the Yun-Lin Plain, Taiwan. Water Resources Management 19, 483–504 (2005).

Claessens, L., Hopkinson, C., Rastetter, E., & Vallino, J. (2006). Effect of historical changes in land use and climate on the water budget of an urbanizing watershed. Water Resources Research, 42(3), n/a–n/a. http://doi.org/10.1029/2005WR004131

Cook, P. G., Hatton, T. J., Pidsley, D., Herczeg, A. L., Held, A., O'Grady, A., & Eamus, D. (2016). Water balance of a tropical woodland ecosystem, Northern Australia: a combination of micro-meteorological, soil physical and groundwater chemical approaches. Journal of Hydrology, 210, 161–177.

Dages C, Voltz M,  Bsaibes A,  Prévot L,  Huttel O,  Louchart X, Garnier F, S Negro. Estimating the role of a ditch network in groundwater recharge in a Mediterranean catchment using a water balance approach. Journal of Hydrology 375, 498–512 (2009).

Dean, J. F., Webb, J. A., Jacobsen, G. E., Chisari, R., & Dresel, P. E. (2015). A groundwater recharge perspective on locating tree plantations within low-rainfall catchments to limit water resource losses. Hydrology and Earth System Sciences, 19(2), 1107–1123. http://doi.org/10.5194/hess-19-1107-2015

Fang, Z., H.R. Bogena, S. Kollet, J. Koch and H. Vereecken (2015): Spatio-temporal validation of long-term 3D hydrological simulations of a forested catchment using empirical orthogonal functions and wavelet coherence analysis. J. Hydrol. 529: 1754-1767, doi:10.1016/j.jhydrol.2015.08.011.

Fleischbein, K., Wilcke, W., Valarezo, C., Zech, W., & Knoblich, K. (2006). Water budgets of three small catchments under montane forest in Ecuador: experimental and modelling approach. Hydrological Processes, 20(12), 2491–2507. http://doi.org/10.1002/hyp.6212

Graf, A., Bogena, H. R., Drüe, C., Hardelauf, H., Pütz, T., Heinemann, G., & Vereecken, H. (2014). Spatiotemporal relations between water budget components and soil water content in a forested tributary catchment. Water Resources Research, 50(6), 4837–4857. http://doi.org/10.1002/2013WR014516

Harder, S. V., Amatya, D. M., Callahan, T. J., Trettin, C. C., & Hakkila, J. (2007). Hydrology and water budget for a Forested atlantic coastal plain watershed, South Carolina. Journal of the American Water Resources Association, 43(7), 563–575.

Hentschel, R., Bittner, S., Janott, M., Biernath, C., Holst, J., Ferrio, J. P., et al. (2013). Simulation of stand transpiration based on a xylem water flow model for individual trees. Agricultural and Forest Meteorology, 182-183, 31–42. http://doi.org/10.1016/j.agrformet.2013.08.002

Herron, N., & Wilson, C. (2001). A water balance approach to assessing the hydrologic buffering potential of an alluvial fan. Water Resources Res., 37(2), 341–351.

Hingerl L, Kunstmann H, Wagner S, Mauder M, Bliefernicht J, Rigon R. Spatiotemporal variability of water and energy fluxes - A case study for a meso-scale catchment in pre-alpine environment. Hydrological Processes 1–20 (2016).

Högström, U. (1968). Studies on the water balance of a small natural catchment area in southern Sweden, XX(4), 623–631.


Huntington, J.L., and Niswonger, R.G., 2012, Role of surface-water and groundwater interactions on projected summertime streamflow in snow dominated regions: An integrated modeling approach : Water Resources Research, vol. 48, W11524, doi: 10.1029/2012WR012319.

Hutley, L. B., Doley, D., Yates, D. J., & Boonsaner, A. (1997). Water Balance of an Australian Subtropical Rainforest at Altitude: the Ecological and Physiological Significance of Intercepted Cloud and Fog. Australian Journal of Botany, 45(2), 311–20. http://doi.org/10.1071/BT96014

Jothityangkoon, C., Sivapalan, M., & Farmer, D. L. (2001). Process controls of water balance variability in a large semi-arid catchment: downward approach to hydrological model development. Journal of Hydrology, 254, 174–198.

Kochendorfer, J. P., & Ramirez, J. A. (2010). Modeling the monthly mean soil-water balance with a statistical-dynamical ecohydrology model as coupled to a two-component canopy model. Hydrology and Earth System Sciences, 14(10), 2099–2120. http://doi.org/10.5194/hess-14-2099-2010

Landerer, F. W., Dickey, J. O., & Güntner, A. (2010a). Terrestrial water budget of the Eurasian pan-Arctic from GRACE satellite measurements during 2003–2009. Journal of Geophysical Research, 115(D23), D23115–14. http://doi.org/10.1029/2010JD014584

Lewis, C., Albertson, J., Zi, T., Xu, X., & Kiely, G. (2012). How does afforestation affect the hydrology of a blanket peatland? A modelling study. Hydrological Processes, 27(25), 3577–3588. http://doi.org/10.1002/hyp.9486

Lewis D, Singer MJ, Dahlgren RA, Tate KW. Hydrology in a California oak woodland watershed: a 17-year study. Journal of Hydrology 240, 106–117 (2000).

Lorenz, C., & Kunstmann, H. (2012). The Hydrological Cycle in Three State-of-the-Art Reanalyses: Intercomparison and Performance Analysis. Journal of Hydrometeorology, 13(5), 1397–1420. http://doi.org/10.1175/JHM-D-11-088.1

Luxmoore, R. J. (1983). Water Budget of an Eastern Deciduous Forest Stand. Soil Science Soc. Am. J., 47, 785–791. 

Marengo, J. A. (2004). Characteristics and spatio-temporal variability of the Amazon River Basin Water Budget. Climate Dynamics, 24(1), 11–22. http://doi.org/10.1007/s00382-004-0461-6

Maxwell, R. M., & Condon, L. (2016). Connections between groundwater flow and transpiration partitioning. Science, 353(6297), 377–379. http://doi.org/10.1126/science.aaf8589

Mitchell, V. G., McMahon, T. A., & Mein, R. G. (2003). Components of the total Water Balance of an urban Catchment. Environmental Management, 32(6), 735–746.

Munier S, Aires F, Schlaffer S, Prigent C, Papa F, Maisongrande P, Pan M. Combining data sets of satellite-retrieved products for basin-scale water balance study: 2. Evaluation on the Mississippi Basin and closure correction model. Journal of Geophysical Research: Atmospheres 119 (2014).


Niedzialek, J.M., and F.L. Ogden, 2012, First-order catchment mass balance during the wet season in the Panama Canal watershed, J. Hydrol. doi: 10.1016/j.jhydrol.2010.07.044.

Obojes, N., Bahn, M., Tasser, E., Walde, J., Inauen, N., Hiltbrunner, E., et al. (2014). Vegetation effects on the water balance of mountain grasslands depend on climatic conditions. Ecohydrology, 8(4), 552–569. http://doi.org/10.1002/eco.1524

Ogden, F.L., T.D. Crouch, R.F. Stallard, and J.S. Hall, 2013. Effect of land cover and use on dry season river runoff and peak runoff in the seasonal tropics of central Panama, Water Resour. Res. 49(12):8443-8462, doi:10.1002/2013WR013956.

Oliveira PTS, Nearing MA, Moran MS, Goodrich DC, Wendland E, Gupta HV. Trends in water balance components across the Brazilian Cerrado. Water Resources Research 50, 7100–7114 (2014).

Pan, M., & Wood, E. F. (2006). Data Assimilation for Estimating the Terrestrial Water Budget Using a Constrained Ensemble Kalman Filter. Journal of Hydrometeorology, 7, 534–547.

Pan X, Helgason W, Ireson A, Wheater H. Field-scale water balance closure in seasonally frozen conditions. Hydrology and Earth System Sciences Discussions 2016, 1–37 (2016)

Qu, W., H. R. Bogena, J. A. Huisman, M. Schmidt, R. Kunkel, A. Weuthen, B. Schilling, J. Sorg and H. Vereecken (2016): The integrated water balance and soil data set of the Rollesbroich hydrological observatory. Earth Syst. Sci. Data, 8: 517–529, doi:10.5194/essd-8-1-2016.

Sahoo, A. K., Pan, M., Troy, T. J., Vinukollu, R. K., Sheffield, J., & Wood, E. F. (2011). Reconciling the global terrestrial water budget using satellite remote sensing. Remote Sensing of Environment, 115(8), 1850–1865. http://doi.org/10.1016/j.rse.2011.03.009

Scarascia Mugnozza, G., Valentini, R., Spinelli, R., & Ervedo, G. (1988). Osservazioni sul ciclo dell'acqua in un bosco ceduo di Quercus cerris L. Annali Accademia Italiana Scienze Forestali, 3–22.

Schaake, J., Koren, V., Duan, Q., Mitchell, K., & Chen, F. (2007). Simple water balance model for estimating runoff at different spatial and temporal scales. Journal of Geophysical Research, 101(D3), 7461–7475.

Schreiner-McGraw, A. P., Vivoni, E. R., Mascaro, G., & Franz, T. E. (2016). Closing the water balance with cosmic-ray soil moisture measurements and assessing their relation to evapotranspiration in two semiarid watersheds. Hydrology and Earth System Sciences, 20(1), 329–345. http://doi.org/10.5194/hess-20-329-2016

Scott, R. L. (2010). Using watershed water balance to evaluate the accuracy of eddy covariance evaporation measurements for three semiarid ecosystems. Agricultural and Forest Meteorology, 150(2), 219–225. http://doi.org/10.1016/j.agrformet.2009.11.002

Sheffield, J., Ferguson, C. R., Troy, T. J., Wood, E. F., & McCabe, M. F. (2009). Closing the terrestrial water budget from satellite remote sensing. Geophysical Research Letters, 36(7), n/a–n/a. http://doi.org/10.1029/2009GL037338

Silberstein, R. P., & Sivapalan, M. (1995). MODELLING VEGETATION HETEROGENEITY EFFECTS ON TERRESTRIAL WATER AND ENERGY BALANCES. Environmental International, 21(5), 477–484.

Sottocornola, M. (2007, July). Four years of observations of carbon dioxide fluxes, water and energy budgets, and vegetation patterns in an Irish Atlantic blanket bog. Ph.D. Thesis (Chapter 6), (G. Kiely, Ed.).

Su, F., & Lettenmaier, D. P. (2009). Estimation of the Surface Water Budget of the La Plata Basin. Journal of Hydrometeorology, 10(4), 981–998. http://doi.org/10.1175/2009JHM1100.1

Tomasella, J., Hodnett, M. G., Cuartas, L. A., Nobre, A. D., Waterloo, M. J., & Oliveira, S. M. (2008). The water balance of an Amazonian micro-catchment: the effect of interannual variability of rainfall on hydrological behaviour. Hydrological Processes, 22(13), 2133–2147. http://doi.org/10.1002/hyp.6813

Vertessy, R. A., Watson, F. G. R., & Sullivan, S. K. (2001). Factors determining relations between stand age and catchment water balance in mountain ash forests. Forest Ecology and Management, 143, 13–26.

Wagner, S., Kunstmann, H., Bárdossy, A., Conrad, C., & Colditz, R. R. (2009). Water balance estimation of a poorly gauged catchment in West Africa using dynamically downscaled meteorological fields and remote sensing information. Physics and Chemistry of the Earth, Parts a/B/C, 34(4-5), 225–235. http://doi.org/10.1016/j.pce.2008.04.002

Wang H, Guan H, Gutiérrez-Jurado HA, Simmons CT. Examination of water budget using satellite products over Australia. Journal of Hydrology 511, 546–554 (2014).

Whitehead, D., & Kelliher, F. M. (1991). Modeling the water balancevof a small Pinus radiuta catchment. Tree Physiology, 9, 17–33.

Wilson, K. B., Hanson, P. J., Mulholland, P. J., Baldocchi, D. D., & Wullschleger, S. D. (2001). A comparison of methods for determining forest evapotranspiration and its components: sap-flow, soil water budget, eddy covariance and catchment water balance. Agricoltural and Forest Meteorology, 106, 153–168.

Yang, Dawen, Sun, F., Liu, Z., Cong, Z., Ni, G., & Lei, Z. (2007). Analyzing spatial and temporal variability of annual water-energy balance in nonhumid regions of China using the Budyko hypothesis. Water Resources Research, 43(4), n/a–n/a. http://doi.org/10.1029/2006WR005224

Yao, Y., Liang, S., Xie, X., Cheng, J., Jia, K., Li, Y., & Liu, R. (2014). Estimation of the terrestrial water budget over northern China by merging multiple datasets. Journal of Hydrology, 519, 50–68. http://doi.org/10.1016/j.jhydrol.2014.06.046

Yoshiyukiishii YK, Nakamura R., Water balance of a snowy watershed in Hokkaido, Japan. Northern Research Basins Water Balance 13 (2004).

Zhang, L., Potter, N., Hickel, K., Zhang, Y., & Shao, Q. (2008a). Water balance modeling over variable time scales based on the Budyko framework – Model development and testing. Journal of Hydrology, 360(1-4), 117–131. http://doi.org/10.1016/j.jhydrol.2008.07.021

Water, energy and carbon balance of a peatland catchment in the Alps

When I started the enterprise called GEOtop, it was at this kind of studies that I was thinking, where experimenters meet theoreticians and/or modeler. In this case, we are studying a small peatland on a plateu on top of Monte Bondone, one the mountains close to Trento. It is usually overlooked in favour of the great dolomites we have all around here. However, it has its beauty, and I can tell you that it is a pleasure to walk over there and see its flora, in summer, or skiing there in winter.

This paper deals with the water budget of the biotope of the plateau, and its carbon cycle. The paper used measurement campaign, laboratory analysis and accurate modelling that push to the limit of GEOtop capabilities.

The paper, entitled: "Water, energy and carbon balance of a peatland catchment in the Alps "by Jeroen Pullens et al.,  has this abstract: " Over millennia, peatlands have stored around 30% of the global soil organic carbon.  Peat is formed and accumulated due to the slow decomposition rate of organic matter in the waterlogged, anaerobic soils. Therefore, the understanding of the water cycle of peatlands is important in evaluating the functioning of peatlands. To be able to study  these dynamics, the process-based hydrological model GEOtop and an appropriate set of in situ measurements were used. They were functional to simulate 4 years (2012- 2015) of the water and energy dynamics of an alpine catchment in Italy, which included a peatland. The modelled energy fluxes are comparable to the fluxes  measured by the on-site eddy covariance tower. The modelled water cycle was used  to quantify the loss of dissolved organic carbon (DOC) and to calculate the carbon balance of the peatland. The model outcomes showed an overall good fit with the  measurements during the snow-free period. During snow cover, the model had difficulties simulating the soil temperature due to insulation by the snow. Based on the measured DOC and the modelled discharge, the DOC adds another source of carbon, since the presented peatland is already acting as a carbon source based on  carbon fluxes. The total amount of loss of DOC (10.2 (± 3.8) g C m-2 yr-1) is comparable to other peatlands. In total, the peatland had a carbon balance (CO2, CH4 fluxes and DOC losses combined) of 112.3, 273.8, 190.8 and 95.3 g C m-2 yr-1 for 2012, 2013, 2014 and 2015 respectively. 
"

You can find the pdf of the manuscript by clicking on the figure above.

Python resources for Hydrologists

Python is a modern object oriented language. Occasionally I wrote about it in my posts, also for remarking that I went in a different direction. However, I cannot deny the evidence that more and more people are choosing it, and there are good reasons, as their language of choice for doing research and hydrological applications. In fact since 2017, I am using it in place of R for scripting and data managing. Below you will find a list of resources. Please do not hesitate to bring my attention to yours or others' contributions which I have not yet in my group.

Motivations for Python use, over other choices, can be found in this blog post or in this paper.

To understand how to start you can follow Python programming for hydrology students that starts with indicating how to install it.

For who wants to start with Python (for hydrologists), I suggest to give a look to my blog post Python general resources. For others, please give a look below.

Python is especially use as a glue for existing program, either written in C or FORTRAN. We have the cases of

• CFM is a programming library to create hydrological models. Although written in C++, it has a Python interface 
• ESMF regridding has been interfaced with Python ESMPy
• GRASS GIS has been interfaced with Python
• Python is also interfaced to gvSIG, as you can see here. 
• HPGL a High Performance Geostatistics Library. Written in C++ is glued together by Python 
• MODFLOW the groundwater model is interfaced by FloPy. Documentation and other information is here. 
• PcRaster - Is a collection of software targeted at the development and deployment of spatio-temporal environmental models. It has a python interface which is constantly being enhanced. 
• OpenHydrology is a library of open source hydrological software written in Python to operate as packages under an umbrella interface 
• PyHSPF Python extensions to the Hydrological Simulation Program in Fortran (HSPF) 
• PyQGIS: A Python interface to QGIS 
• RhessysWorkflow  RHESSysWorkflows provides Python scripts for building RHESSys models. Other Pythonic material on RHESSys can be found here.
• UWHydro tools for connecting University of Washington hydrological models, and, in particolar, the VIC driver PythonDriver

In the reign of hydrologic applications entirely written in Python, we remind:

• Yet another repository of Python models and resources by Raoul Collenteur 
• AMBHAS - hydrological library in Python 

• ANUGA 2 - package for modelling dam breaks, riverine flooding, storm-surge or tsunamis. In Python and C. 
• EcoHydrolib provides a series of Python scripts for performing ecohydrology data preparation workflows. 
• evaplib: Python library containing functions for calculation of evaporation rates. Functions include Penman open water evaporation, Makkink reference evaporation, Priestley Taylor evaporation Penman Monteith (1965) evaporation and FAO's Penman Monteith ET0 reference evaporation for short, well-watered grass. In addition there is a function to calculate the sensible heat flux from temperature fluctuation measurements. View documentation of evaplib module functions. Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
• A GLUE, Generalised Likelihood Uncertainty Estimation (GLUE) developed by Framework Joost Delsman, at Deltares, 2011 
• Groundwater flow modelling manual for Python written by Vincent post 
• Hydro-conductor: A set of Python scripts and modules written to couple a hydrologic model with a regional glacier model
• ODMToolsPython and ODMTools ODMTools is a python application for managing observational data using the Observations Data Model. ODMTools allows you to query, visualize, and edit data stored in an Observations Data Model (ODM) database.ODMTools was originally developed as part of the CUAHSI Hydrologic Information System. YOu can find a presentation about here. 
• PyETo is a package for calculating reference/potential evapotranspiration (ETo). 
• Python script for rectangular Piper plot (version December 2014): Python script for plotting chemical data in a rectangular python plot (see image) according to Ray and Mukherjee (2008) Groundwater 46(6): 893-896. Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
• Python script for multiple Stiff plots (version June 2011): Python script for preparing multiple Stiff diagrammes (see image). Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
• Haran Kiruba tools for hydrology Not really clear what he does. 
• USEPA site contains various python (and other languages) tools, including an interface to Epanet and SWMM (a connection to swmmtoolbox is also avilable here). 
• Also USGS has its python tools
• sMAP 2.0 is a tutorial will cover how to retrieve data from a sMAP archiver using Python. 
• ulmo clean, simple and fast access to public hydrology and climatology data 

Specific hydrological Hydrological Models are enumerated below.

• EXP-HYDRO Model is a catchment scale hydrological model that operates at a daily time-step. 
• Landlab Landlab is a python-based modeling environment that allows scientists and students to build numerical landscape models. Designed for disciplines that quantify earth surface dynamics such as geomorphology, hydrology, glaciology, and stratigraphy, it can also be used in related fields. 
• LHMP - lumped hydrological models playground - tiny docker container with complete environment for predictions.
• PyCatch is a component based hydrological model of catchments built within the PCRaster Python framework. The code is here. A related paper, here. 
• PyTOPKAPI is a BSD licensed Python library implementing the TOPKAPI Hydrological model (Liu and Todini, 2002). The model is a physically-based and fully distributed hydrological model, which has already been successfully applied in several countries around the world 
• SPHY. See for details on model and publications (HESS, Nature, etc) here. Just recently a new paper on climate change and mountain hydrology in PloS came out, using SPHY model, more info here.
• Topoflow a python hydrologic model by Scott Peckham 
• WOFpy is an implementation of CUAHSI's Water One Flow service stack in python 
• wflow is a distributed hydrological model platform that currently includes two models: the wflow_sbm model (derived from the topog_sbm soil concept) and the wflow_hbv model which is a distributed version of the HBV model. This is actully part of a larger Deltares project called OpenStream

GIS capabilities are also present:

• To select Hydrological Response Units (HRUs) using PyWPS + GRASS + QGIS
• pyDEM: A phyton digital elevation model analysis package. Note that pyDEM depends on TauDEM for certain steps (e.g., pitfilling) and it also makes extensive use of the GDAL library for working with geospatial rasters. (a small video about) 
• PyGeoprocessing is a Python/Cython based library that provides a set of commonly used raster, vector, and hydrological operations for GIS processing. 

Also tools to deal with Meteorology:

• meteolib: Python library containing meteorological functions for calculation of atmospheric vapour pressures, air density, latent heat of vapourisation, heat capacity at constant pressure, psychrometric constant, day length, extraterrestrial radiation input, potential temperature and wind vector. The documentation for this module is presented at here (meteolib module functions web site). Functions to convert event-based data records to equidistant time-spaced records (event2time) and to convert date values to day-of-year values (date2doy) are now in a separate meteo_util module. Documentation is presented here (meteo_util module functions web site). Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
• MetPy is An Open Source Python Toolkit for Meteorology 
• Melodist (MEteoroLOgical observation time series DISaggregation Tool) is an open-source software package written in Python for temporally downscaling (disaggregating) daily meteorological time series to hourly data. It is documented in a GMD paper by Forster et al., 2016. 
• Various resources for meteorology can be found in the pyaos blog

Statistical and data analysis tools are abundant

• CUAHSI time series viewer
• The basic cheatshit
• NetCDF file operations are available here. However, there is also txt2netcdf which containsvarious Python functions for importing text into NetCDF data files (creating files, adding variables, listing structure, etc.), developed by Ko van Huissteden. 
• Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. (A short tutorial is also here) 
• Extreme distribution (from scipy.stats) is here
• An example of use of Pandas for analysing time series 

Visualisation is well served

• ggplot is a plotting system for Python based on R's ggplot2 and the Grammar of Graphics. It is built for making profressional looking, plots quickly with minimal code. 
• An impressive tour of Python possibilities in this field is given by “Regress to Impress” 
• VisTrails an open-source scientific workflow and provenance management system that supports data exploration and visualization. Its website is here. 
• uvcmetrics metrics aka diagnostics for comparing models with observations or each other. This is part of the Uv-CDAT website which contains also other visualisation tools. 

Tools for dealing with uncertainty and sensitivity analysis

• Stijn Van Hoey tools

Yet another repository of Python models and resources

• by Raoul Collenteur 

A final comment

I am actually impressed by the quality of the contributions in Python. I think there is not anymore reason to use commercial program like Matlab in Universities (see this review here). What Matlab does, also Python does. Compared to R, it has a much more clear sintax and is, certainly, a better language. So I would suggest to use it with students (or R). I like R but I never built program on it, because its object orientation is really poor. Python is better and, as it is known its syntax is clean. Python is great to link FORTRAN and C/C++ native libraries, so actually many uses it to assembled libraries they wrote in those more performant languages.
As you know, however, my group uses Java as its principal programming language. Java, in comparison with Python, is less immediate and more verbose, but it allows to build many framework to work with, and is usually faster than Python. Probably because, Java is supported by magnificent building tools (Maven, Gradle) that allow to manage large projects in a way that probably cannot be done in Python.

Zenodo and Joss

Here I am robbing from Living in an Ivory Basement blog. Titus Brown (GS) introduces two interesting tools. Maybe the word “tools” is an inappropriateas description.

Zenodo is a way to archive software products, under the idea that GitHUB is not an archival:

“A common point of concern, however, is that GitHub repositories are not archival. That is, you can modify, rewrite, delete, or otherwise irreversibly mess with the contents of a git repository. And, of course, GitHub could go the way of Sourceforge and Google Code at any point. … But! Never fear! The folk at Zenodo and Mozilla Science Lab (in collaboration with Figshare) have solutions for you! … Zenodo is an open-access archive that is recommended by Peter Suber (as is Figshare). … Items will be retained for the lifetime of the repository. This is currently the lifetime of the host laboratory CERN, which currently has an experimental programme defined for the next 20 years at least.“

For more information, nothing better than reading the original blog post.

JOSS stands for Journal of Open Source Software and it promises to be a vehicle to publish software, not what software is about, or a description of the software and its peculiarities, but of the software, which seems to go through a process of peer review, extremely useful and new. You can find the Joss post here.

The Soil Water Retention Curves

When dealing with soils you are forced to implement mass conservation dependent on two variables, the dimensionless water content, usually named $\theta$ and suction, $\psi$, i.e. The energy contained in a volume of soil per unit mass. Therefore, to solve the budget, you need (at least) to get a new relationship which connects them.  This relation is called soil water retention curve. The plural in the title means that there are many. At least one for any soil type. 

In fact,  the  relationship, and precisely $\theta(\psi)$, is dependent on soil types and structure (and some other factor probably, like temperature, organic content etc). It is a statistical quantity, which averages the behavior of many pores, and an ensamble of water injecting/extracting possibilities.  

The figure below from Lu (GS) and Godt (GS) book (2013) is a clear visualisation of the problem.

The same Ning Lu, in a recent paper (2015) tried to disentangle the various forces acting on water when in pores, and obtained what is shown below.

As expected, the forces acting are not all of the same type, at varying suction values. At very high suction, adsorption forces act in which single water molecules adhere to soils. When more layers of water molecule add, water constitute  thermodynamics compounds, whose equilibrium is globally determined in between adhesion forces, bulk water weights, surface of water and air gas interactions, and which is usually known as capillarity.  

Laws governing capillarity are described by Young-Laplace and  Kelvin laws.  Some insight of the therodynamics of these phenomena (an excellent explanation, indeed) can be found in the first pages of Steudle (2001) review about plant-root suction. 

At this stage liquid water seems to, constitute a disconnected phase, while air gas is continuous inside the medium pores.  

Increasing the water content water becomes a continuos medium and usual hydrodynamics laws become valid.  A recent review of parameterizations of the soil water retention curves (not particularly deep or brilliant though) is given by Too et al. (2014) that cites other older reviews.

When pressure increase, however, we can have two effect which partially depends on how wetting happens. If wetting happens through some sort of flooding then air can stay trapped in pores and decrease the space available for water. The net effect is  associable to a decrease of porosity. However, when water fills all the space (i.e. the soil is saturated) the soil matrix cannot be considered anymore rigid. 

Assume it would be rigid. Then water content could not increase, any pressure applied to the saturated soil would transmit instantaneously through the water volume and water would be expelled where pressure is not applied or there is less pressure in a sort of piston flow.  

Instead, because the medium is not rigid, any pressure is transmitted with a certain speed, and pressure waves can be measured. This fact implies that after saturation, the system behaves as porosity increases and, at the same time pressure varies.  

From a practical point of view, soil water retention curves can be extended to positive pressure (negative suctions) adding a term which is well known in groundwater literature and is called specific specific storage. 

These qualitative descriptions do not end the complex phenomenology of water retention curves.  

As Nunzio Romano (GS) and coworkers noticed, and Kosugi (1994) before them, soil water retention curves shape depend directly on the pores' distribution. This, however, is not necessarily a unimodal distribution but can be multimodal because of soil structure and soil "disturbances" in form of macropores due to animal or roots decay. In this case soil water retention curves (their integral) can be more complex than expected, as shown in Figure below.

This opens to a series of generalisation, but it would be the topic of some other post (and actually was already the topic of several posts on soil freezing).

References

Kosugi, K. 1994. Three-parameter log-normal distribution model for soil water retention. Water Resour. Res. 30:891–901. 

Lu N, Godt JW. Hillslope Hydrology and Stability. Cambridge: Cambridge University Press; 2013. 

Lu, N. (2016). Generalized Soil Water Retention Equation for Adsorption and Capillarity. Journal of Geotechnical and Geoenvironmental Engineering, 142(10), 04016051–15. http://doi.org/10.1061/(ASCE)GT.1943-5606.0001524

Romano, N., Nasta, P., Severino, G., & Hopmans, J. W. (2011). Using Bimodal Lognormal Functions to Describe Soil Hydraulic Properties. Soil Science Society of America Journal, 75(2), 468. http://doi.org/10.2136/sssaj2010.0084

Steudle, E. (2001). The Cohesion-Tension Mechanism and the Acquisition of Water by Plant Roots. Annual Review of Plant Physiology-Plant Molecular Biology, 847–877.

Too, V. K., Omuto, C. T., Biamah, E. K., & Obiero, J. P. (2014). Review of Soil Water Retention Characteristic (SWRC) Models between Saturation and Oven Dryness. Open Journal of Modern Hydrology, 04(04), 173–182. http://doi.org/10.4236/ojmh.2014.44017

Reservoirology #3

This is a revision of the previous post on the same topic. There I tried to develop my own algebra of symbols to represent coarse grained (spatially integrated) hydrological system. Later on I understood that Petri networks were already there and useful to obtain the same result. The graphs obtained in such a way where, besides, studied in several places, and many contributes of literature convergent from other disciplines, can be used for hydrological scopes.

The presentation (click on the figure) completely substitutes the old one. Who liked it, will like better this.  When you will be done with this post, consider that there is also Reservoirology #4.

THIS MATERIAL IS NOW REFINED AND PUBLISHED IN WATER RESOURCES RESEARCH. YOU CAN FIND THE PAPER HERE.

About graphs, DSL and replicable research in Francesco Serafin's work

This is the summary of what Francesco Serafin (his blog) did in its first year of doctoral studies, defending for his admission to his second year Ph.D. Undoubtedly he did a lot of work and he programs to do even more. Three are the lines of his research:

• implementing a new flexible structure based on graphs for commanding simulations of interacting systems; 
• implementing a domain specific language for doing environmental models (and particularly to solve ordinary and partial differential equations); 
• deploying a system that makes easier to do replicable science. 

Clicking on the Figure above you will find his presentation. Obviously the committee admitted him to the second year of his Ph.D. studies.

Adige River Research in GLOBAQUA and CLIMAWARE projects

This contains the presentation for the Diffuse seminar held November the 4th in 20 italian cities. This is to do memory of the events of November 1966, fixty years ago, when many Italian cities were flooded.  The presentation is not actually about flooding but the general management of water in medium and large (well someone could consider it small, but it is the second largest in Italy, and its complexity is quite overwhelming) catchments, like River Adige, which is the basin where I live (actually my house is in a prone to flooding area so, all my interest is that it will not happen anymore).

Possibly I will upload also an English version of it for the interested reader. Click on the figure above to see the presentation.

Site and Regional Flood Frequency at USGS

I asked to my Friend Tom M. Over (GS), to contribute something on Flood Frequency. Here it is is enjoable answer:

....

Sure, but that's a very big topic. How to narrow it down? As I've been with the USGS now for 15 years, you'll get very much a USGS-tinged view of things, which is focused on at-site flood frequency and regional flood frequency (under stationary conditions).

For at-site, as you may know, the general US federal government recommendation is to fit the log-Pearson type III distribution. Along those lines, the long-awaited update to Bulletin17B, called Bulletin17C, has been out for public comment, and has been revised in response to those comments. See:

http://acwi.gov/hydrology/Frequency/b17c/index.html

One big innovation in Bulletin 17C is "Expected Moments Analysis" (EMA) which allows moments for historic or other threshold measurement-based information to be included:

Cohn, T.A., Lane, W.L., and Baier, W.G., 1997, An algorithm for computing moments-based flood quantile estimates when historical flood information is available: Water Resources Research, v. 33, no. 9, p. 2089–2096.

Cohn, T.A., Lane, W.L., and Stedinger, J.R., 2001, Confidence intervals for expected moments algorithm flood quantile estimates: Water Resources Research, v. 37, no. 6, p. 1695–1706.

EMA is already implemented in PeakFQ, at:

http://water.usgs.gov/software/PeakFQ/

On regional flood frequency, the standard technology for USGS has been for ~30 years GLS regression on basin characteristics, computed one quantile at a time, going back to papers by Jery Stedinger and Gary Tasker in the 1980s, now implemented in WREG:

http://water.usgs.gov/software/WREG/

You could start with the manual for the program to see the theoretical background: http://pubs.usgs.gov/tm/tm4a8/.

A typical product of this type of study might be:

http://il.water.usgs.gov/projects/2004_flood_freq/

Bulletin 17B-based regional flood frequency work also brings you to the topic of regional skew. A former student of Jery Stedinger, Andrea Veilleux, has been working to implement a new method in the U.S. based on her Ph.D. work using Bayesian regional regression; see, e.g.,

http://pubs.usgs.gov/sir/2010/5260/

We're not totally ignorant of non-stationarity; you are probably aware of several important descriptive papers from recent years by USGS scientists such as:

http://onlinelibrary.wiley.com/doi/10.1029/2005GL024476/full ("Nature's style: naturally trendy")

and

http://science.sciencemag.org/content/319/5863/573 ("Stationarity Is Dead: Whither Water Management?"),

and we do have a new workgroup on the topic that is looking at when and how to address the problem. Since it's so new I can't predict very much about what we'll come up with.

But along the lines of non-stationarity, from a personal perspective, recently a pair of reports from a project that I have been working on for several years came out in which we used regional quantile regression to assess and adjust for the effect of urbanization on flood frequency. I think quantile regression holds a lot of promise for regional flood frequency because it allows you estimate the exceedance probability of a given flood event at a given site based on regional information, even in the presence of non-stationarity (though climatic non-stationary would require a modification of the particular approach we used), without going through the whole at-site flood frequency thing and THEN doing a regionalization analysis. These reports are at:

https://pubs.er.usgs.gov/publication/sir20165050

and

https://pubs.er.usgs.gov/publication/sir20165049


One other personal perspective is that I am still interested in scaling of floods (and streamflow statistics in general), a la my Ph.D. advisor, Vijay Gupta, (see, e.g, http://onlinelibrary.wiley.com/doi/10.1029/94WR01791/pdf).

A couple colleagues and I published a paper last year trying to adapt that thinking to scaling of flow-duration curves, taking into account the effects of omitted variable bias, and trying to make an approach to the unification of quantile versus moment analysis (not very successfully on the last point but perhaps the problem is better demonstrated than in the past): http://onlinelibrary.wiley.com/doi/10.1002/2014WR015924/pdf

So I guess that's what I know about, i.e., what the USGS does and what I have been working on recently. Hope it helps! Let me know if you have questions.

Best,

Tom

Sssh! GEOtop 2.0 video tutorial in Italian

GEOtop is our process based hydrological model. At the beginning of the year, Mountain-eering, paid by CAMILAB produced a series of video lectures in Italian to train people to use it. Unfortunately this big work is closed in some informatics drawer and nobody can use it. I came to have one copy, and I am temporarily posting it here. Untill CAMILAB will provide the right links and make it available.

To access the video lectures, you have just to understand where to click.

Working with us

After the scaring version, the practical version.  To incoming Ph.D. students

Dear *,

working with us means using our models

•   GEOtop 
•  JGrass-NewAGE

Both of them have a consistent history that involves also quite a group of publications. Their main information can be found following the links.

• GEOtop essentials 
• NewAGE essentials 

GEOtop,  to say the complete thruth, has a group of video tutorial (in Italian) that you can find here. University of Calabria paid Mountain-eering for doing them, so please use them with confidentiality.

Looking in perspective, I am working to a new incarnation of GEOtop in components. As I already explained here.  There will be then, a convegence of tools towards OMS3 and its evolutions.

So what I suggest ? First start to study the models at the links above. Willing to learn a computer language, start with Java. To start, read here.  To continue, go here.

In this language I invested quite a lot during the years. Why I choose Java can be found here.
I wrote it four years ago, but the concepts are still valid.  Recently I become more moderate, and opened to other languages. Here my opinion.

Every breath you take

I found these photographies by Robert Dash of stomata and I could not avoid to share them in this blog. The collections of photo are, at least two:

"Show me the carbon" 

"Every Breath you take"

They were taken with an electronic microscope but preserve artistic values. Clicking on the figures you are redirected to the whole collection.

Hillslope stability tools

Here I am on landslides. I gave some contributions to this topics, and I wrote also something about, however I never tried to put a list of models that can be used.

As a general reading I would suggest certainly 

Lu (GS) and Godt, Hillslope hydrology and stability, 2012

It is one of the few books that have a modern approach to quantitative hillslope hydrology. Who starts from it, is already a few years behind others.  Fortunately, Ning Lu covered some of the book chapters in the summer school on landslides held in 2013, and you can also learn directly from his voice and video.

If you have red the book, you can then understand that having at least a 2D tool for assessing hillslope stability is a necessity that you cannot avoid. 

So here they are my favorite tools:

Jgrasstools (see also here) - They contain a SHALSTAB implementation that can be used for comparison. They also contains the necessary tools for terrain analysis. 

CISLAM -  model was originally implemented in R by Cristiano Lanni (GS), but it was ported to JGrasstools in a Google Summer of Code by Marco Foi. I cannot guarantee its quality, since I never used it, but it is built on the theory I co-developed with Cristiano that you can find addressed here. (Jgrasstools are migrated to the gvSIG 2.3 now or are available trough S.T.A.G.E.).

Boussinesq - This is not directly a tool for hillslope stability estimation. However, it serves to estimate the water content (neglecting at the moment, the vadose zone). There are two version of it:  a C version by Emanuele Cordano (stable and working) and a Java version by Francesco Serafin (that is in Java, for being inserted into OMS3, and still a project under construction).

RiDI. - This was developed by Fabio Ciervo in his Ph.D.  it has the peculiarity that it implements a double porosity soil water retention curve proposed by Nunzio Romano (GS) and coworkers.

GEOtop - It was used a lot to this scope, conjointly with simple and less simple hillslope stability analysis (we did some papers with it).

At present, all the tools require to become part of a consistent framework. But we (Giuseppe Formetta -GS-, Francesco Serafin and I) are working on it, looking forward to the next EGU General Assembly in Wien (April 2017). Stay tuned.