A good (hydrological) model according to Thorsten Wagener

This is what Thorsten Wagener (GS) thinks is a good model. Almost no comment is necessary.

JGrass-NewAGE history - Version zero and version one

Jgrass-NewAGE (from now on, simply NewAGE) was conceived after the Adige River Authority was requesting a model for t he river Adige to help the managements of droughts. We decide to name the project Nuovo Adige. “Nuovo” since we already implemented a model, almost twenty years before for the same river, and that was the Adige model. Because the English translation of “Nuovo Adige” sounds very much like “new age”, this became soon the name of the model. The first model was implemented mostly by the group head by Alberto Bellin for hydrology and Aronne Armanini for the hydraulic part (I had some part in it, in designing the file organisation required by the model and suggesting some about the geomorphological unit hydrograph approach). The new model, however, had another ancestors: the real-time model operational at the Civil Protection of Province of Trento, also implemented by Alberto Bellin and collaborators, and including a snow model by some of my former students (Hydrologis, and Stefano Endrizzi). 

When I got the project, the first thought I did, was that, I had to abandon my beloved GIUH models for a more general one. We needed to estimate the discharges in multiple points of the river network, while GIUH gives the discharge just at the outlet of a basin, and this claimed, at least, for a generalisation of the usual GIUH for a system where multiple GIUH where used, each for any subbasin. (Well GIUH has also some limitations, but I will go back on this in a future post). 

If one browse the slides presented at the beginning of the project (In Italian) one could also notice that the emphasis of the project was not only on the models of physical processes, either conceptual or physical, but on the whole infrastructure of modelling, including a data base for storing the data (even the geographic data, of input as well as of output), and a visualisation system based on a GIS system (uDig) aimed to grow into a Decision Support System (DSS). 

Since the early nineties, I was in fact intrigued by the image of  a DSS system, found in a lost book of proceedings, which included a database, a system of visualisation, and models, and where all of the needed  concepts where already developed. Twenty years or so later I am still asking myself why the people who envisioned the picture, did not ever actually realised it in practice: but probably it was because a lot of tools (which I spent year to create consistently) were missing. 

Another key in the presentation was the inclusion in modelling of infrastructures like  water intakes, withdraw, dams and any other devices. An explicit treatment of flow in channel through a solid algorithm [Casulli, 1990] solving 1D de Saint Venant equation was also implemented.  

These features were known to be necessary to account for human action which, potentially, during low flows, could withdraw all river water for for irrigation and other uses [a clarification of the concept of Anthropocene at this local scale]. 

After the experience made with GEOtop, it was clear that modelling of such  systems could not anymore be implemented in the classical way as a monolithic program. We therefore look at OpenMI as a system to pursue a strategy of modelling by components.  However, presentation given at CUASHI 2008 biennial meeting (a must read including also some considerations about GEOtop) clarifies these and other design issues. 

From the point of view of the process mathematical descriptions, the plan was to “recycle” the snow model of the real time Adige model, to absorb the GEOtransf model [e.g. Majone et al., 2010] into the picture; to implement the estimation of Penman-Monteith (see also here the lectures by Dara Entekhabi) scheme for evapotranspiration, to recover all the tools of the Horton Machine for the geomorphometric analyses required for basin delineation. An ambitious idea was to include directly in modelling, through the use of GEOtools, the formats suitable for a direct GIS representation. 

Unfortunately, GEOtransf model was not available, due mainly to licensing issues, and we had to change the direction of our efforts, and we grabbed the Chris Duffy model [Duffy, 1996] as implemented in the Mantilla’s CUENCAS model. A couple of spatial interpolator for data measured at hydro-meteo stations were implemented, i.e. an ordinary Kriging, and Just Another Model Interpolator (JAMI), a simple bare-bone robust (which never break) method for making measure available in any catchment point. Finally an original description of the river networks, describing the topology, and governing the order of execution of the various modules, was implemented (here an early view of the watershed description, given as a Poster at EGU-Wien 2008, and here a more mature paper, just on the generalised Pfafstetter numbering in NewAGE).

We can call this above the zero version of NewAGE, which actually did not became never operational. It needed a testing calibration phase that, for various reasons could not be pursued. The financial support from the River Basin Authority terminated, and data base, model components, and whatever developed was closed in a drawer. A big lost occasion to have a new type of model working on River Adige!  What actually remains of version zero has to be archeologically retrieved. But we are doing it.

However, I (we) did not give up. Eventually we abandoned OpenMI in favour of OMS, and the reasons were explained in a previous post. Porting to OMS was done as well by Hydrologis. We also reduced our scope and concentrated just on the model components to improve them and verify their respondence to reality. 

To obtain this goal Giuseppe Formetta, in his start of Ph.D. implemented goodness of fit methods (GoF) and the calibration methods  DREAM and Particle Swarm that became two new OMS components. With them, a systematic analysis of the other components started.

We soon realised that Duffy’s model was not easy to calibrate (or, at that time, we were maybe not experienced in using the tools), and Giuseppe decided to implement a new entire module, based on Hymod. This was successfully accomplished, and the results are reported in Formetta et al, 2011.

Functional to this work was the incremental improvement of the Kriging (.) components, now including, besides Ordinary Kriging, Detrended Kriging, their local versions (accounting not for all the measurements points but just on the next neighbours), and five variograms models that can be fitted automatically at any time step having available data (also made by Giuseppe).

The refocused development of NewAGE was described in a concept paper actually published this year 2014.

The NewAGE zero had radiation simulation, however, the components was entirely rewritten for the short wave radiation on  the basis of Javier Corripio’s work.  The implementation was  initiated by Daniele Andreis, and enhanced, cleaned,  and completed with various contributions on how to estimate radiation attenuation by atmosphere and clouds by  Giuseppe Formetta. All of this work is summarised in this GMD paper (for some rehearsal on solar radiation, you can look at the slides referred here).

On the basis of the work on radiation, components for snow accumulation and melting were built and documented in another paper where the study was concerned about the Cache La Poudre basins close to Fort Collins, Co, another piece of Giuseppe Formetta's Ph.D. thesis. 

The actual state-of-art of NewAGE  includes also the implementation of a new version of the Penman-Monteith, its FAO counterpart,  and Priestley-Taylor formulas for the estimation of evapotranspiration (also implemented by Giuseppe F.). The potentialities in the latter components were not yet exploited as they could. But we will do it.

At present, therefore, NewAGE is constituted by a a set of components that can be used to simulated the whole hydrological cycle for any basin, from a few square kilometres to continental scale rivers. We can call the actual version JGrass-NewAGE version 1, but, actually is a set of components that can be arranged in various modeling solutions for various analyses that could not be easily obtainable with more traditional models.

The story is continuing and soon other components will be made available to the public, those available now are listed in a previous post, here.  A different view on the same concepts presented here, can be found here (seen from the Visualisation and Informatics perspective) and here (the idea of building a physico-statistical model).  If you resist and look the recent presentation given at Fort Collins you can see a rework of the ideas developed at CUASHI in 2008 and complete your overview.


The source code of the system can be found on Github.

References

V Casulli, Semi-implicit finite difference methods for the two-dimensional shallow water equations, Journal of Computational Physics 86 (1), 56-74, 1999

CJ Duffy, A Two‐State Integral‐Balance Model for Soil Moisture and Groundwater Dynamics in Complex Terrain, Water resources research 32 (8), 2421-2434, 1996

Formetta, G.; Mantilla, R.; Franceschi, S., Antonello A., Rigon R., The JGrass- NewAge system for forecasting and managing the hydrological budgets at the basin scale: models of flow generation and propagation/routing, Geoscientific Model Development Volume: 4 Issue: 4 Pages: 943-955, DOI: 10.5194/gmd-4- 943-201, 2011 

Formetta G., Antonello A., Franceschi S., David O. and Rigon R., The informatics of the hydrological modelling system JGrass-NewAge, 2012 International Congress on Environmental Modelling and Software Managing Resources of a Limited Planet, Sixth Biennial Meeting, Leipzig, Germany R. Seppelt, A.A. Voinov, S. Lange, D. Bankamp (Eds.) http://www.iemss.org/society/index.php/iemss- 2012-proceedings, 2012

Formetta G., Rigon R., Chavez J.L., David O., The short wave radiation model in JGrass-NewAge System, Geosci. Model Dev., 6, 915-928, 2013, www.geosci-model-dev.net/6/915/2013/

doi:10.5194/gmd-6-915-2013

Formetta G., Antonello A., Franceschi S., David O., and Rigon R., Hydrological modelling with components: A GIS-based open-source framework, Environmental Modelling & Software, 5 (2014), 190-200

B Majone, A Bertagnoli, A Bellin, A non-linear runoff generation model in small Alpine catchments,  Journal of hydrology 385 (1), 300-312, 2010

Vrugt, J. A., ter Braak, C. J. F., Diks, C. G. H., Robinson, B. A., Hyman, J. M., Higdon, D., 2009. Accelerating Markov chain Monte Carlo simulation by differential evolution with self-adaptive randomized subspace sampling. International Journal of Nonlinear Sciences and Numerical Simulation 10 (3), 273-290. DOI:10.1515/IJNSNS.2009.10.3.273

Earth System Modelling Framework and Related Resources

Well, we pushed on OMS and Java. However, for FORTRANists (or FORTRANers ?) a very interesting and modern resources is the Earth Science Modelling Framework or (ESMF in short). This was due to a large effort by NOAA, UCAR and others. Learning it has an apparent steep learning curve.  In any case, either you use it or do not use it, it is certainly a term of reference. Especially for the tools and the concepts developed, independently from their actual implementation. Click on the image to get the sites.

I obviously knew the program since a lot of time. However, It was brought to my attention again in these days because I was looking for the models' run metadata implementation, which is now part of the Earth System Documentation effort. The general goal of the initiative can be found in this 2012 Fall AGU Meeting  presentation.

The publication of geoscientific model developments v1.0

Geoscientific Model Development (a.k.a. GMD) has become for many, including us, a Journal of reference. Its open access policy with the fact that it covers a gap in scientific literature, has made of it a journal that quickly gained an impact factor of 5 (which is the higher among the journals in which I publish). Its success made to increase  the papers submitted exponentially, and, at this point, the Editors decided to restrict the policy for publication of the papers. The new editorial for publication is here (or clicking under the image).

I suggest that any modeller reads it. 

A school to learn how to model with the Object Modelling System version 3

It will be held in Trento (Italy)  between the 14 and 18 (both included) of october 2013.
The school is the first of a cycle of high educational courses focusing on the hydrologic and environmental modeling by using Object Modeling System (OMS). OMS, a pure Java, object-oriented modeling framework., allows model construction and model application based on components.
Information about the system can be found at the OMS site, and in previous postings of this blog.

The school is dedicated to young Professors, Researchers, Post Docs, PhD students, and environmental modellers (but gifted Master Students are also welcome to apply) and it is organised in 5 days of lessons and working groups.

Lecturers of the School will be:

Dr. Olaf David (Colorado State University, US)
Dr. Giuseppe Formetta (Università della Calabria)
Dr. Timothy Green (USDA Agricultural Research Service, US)
Dr. Gabriella Turek (NIWA, New Zealand

The number of participants will be limited to twenty (20) due to the intention of the organisers that the school will  be effective (our scope is that the participants could be able, at the end of the school, to program their own OMS component). 

Software Prerequisites: JDK 1.6, gcc, installed on personal notebooks (we also provide lab computers with the software pre-installed): no time will be lost for installations.

The course is not intended to teach Java (neither C or FORTRAN, indeed). Therefore the applicants should specify in their CV their level (at least basic) of knowledge of the language (by saying what they actually did).  Furthermore, application should include a simple statement declaring why the applicant is interested in being considered for admission to the Summer School. The interest for porting a hydrological model or a modeling component (either in Java, C/C++ or FORTRAN) to OMS will be considered as a title of merit for the admission.  Every applicant must provide his/her contact information: email address, telephone, and mailing address. 

NEW: the material of the school can be found  here. And here you can find the videos.

Application should be sent to:

summerschooloms2013<at>gmail.com

Deadline for applications is September 13.

The schematic schedule of the school is:

The Flyer of the school can be downloaded here. The material of the school can be found instead here.

(Almost) a perfect Answer

I read this in a LinkedIn discussion, and I found it the perfect complement of my previous post "Essential for Hydrologists"

Question:

Hello,

I have a question specified in the following and hope some of you can guide me a

bit:

There is a lake which is the source of water for an industrial project. The project is known for being water intensive, approximately 20 tons fresh water is needed for production of 1 ton final product. According to satellite images, the lake area has been shrinking for approximately 62% since the project started operation about 5 years ago. Local farmers also complained that they had to dig water as deep as 100 meters in order to get water, indicating a severe drop of ground water level. We would like to know quantitatively how water extraction has caused local water scarcity problem. I wonder whether we need a hydrological model to do such assessment; if a hydrological model is not available for that specific lake, is that possible use time series data to run a regression model where lake area is seen as a function of water extraction, precipitation, temperature, etc.

I wonder how analysis like is done typically, if any of you is aware of any exisiting studies, please do let me know. Many thanks in advance.

Best wishes

Sissi

Answer:

Dear Sissi,

Five year data is not adequate for detailed analysis. The reason of decline is due to inadequate rainfall in these five years or increase of draft for different purposes during the five year.  Recently I have done a similar study in one of the lakes in India, though not for the same purpose. I would like to suggest the following points.

1. Study the rainfall / inflow pattern for at least for 50 years

2. Use the Soil and Water Assessment Tool (SWAT)and estimate the water balance taking the rainfall and temperature and other data for at least 15 year daily data.

3. If you have actual inflow data, compare the results with the actual inflow data.

4. If you do not have actual inflow data, asses the inflow by SCS-CN Method (Soil Conservation services (SCS), HEC_HMS  and compare the data with SWAT.

5. Use HEC-RAS and HEC-GEORAS for flood modelling if required to delineate the areas of flood in the lake catchment.

.........

With best wishes,

**

My comment:

The answer is certainly correct. HEC* and SWAT constitute very well engineered standard models used in operational (and operative) hydrology. SCS-CN method, was already commented. Certainly a term of reference. However, if you look at the physics under the hood, you can recognise the existence of many of those "old-style" parameterisations and simplifications that I tried to avoid in favour of more physically sound choices in GEOtop, more modernity, both in the way to implement the model solutions and the way to do simplifications, in JGrass-NewAGE and  my lectures in hydrology. 

Therefore the answer is perfect if you have to do (possibly is also perfect if you, at the end are an engineer) but I feel uncomfortable with it. Can't we do it better ?  

What is the minimal geomorphology-based hydrological model?

This is one of the objects of the research Alban de Lavenne, a Ph.D. student of Christophe Cuddenec pursued here in Trento, during his three months stay the last Fall. Similar approaches can be found in the work by Fenicia, and especially in Fenicia et al., 2008. Fenicia 2008, is a must-read paper, since it is well written, and smart. However there, the  Authors did not use any geomorphic information as guideline in their modeling but just a scheme based on reservoirs, which is, in my view, out-of-date. Which is kind of a pity, considering that we know very much of the morphology of river, even when they are ungauged. Thanks to SRTM, ASTER, and other topographic data, the topology and geometry of river networks, and Earth's elevation is known with unsurpassed precision all over the World, and therefore, the first step to predict what happens in ungauged basins would be to use their geo-morphology, which, in fact, was shown to be important in many papers.

Alban used in his investigation a few simple geomorphological unit hydrograph schemes based on the width function:

• 1F1U1P: 1 velocity applied to entire flow path length
• 1F2U2P: 2 velocities (respectively on hillslope and channelized length) -
• 2F2U3P: 1 velocity for surface flow, 1 velocity for subsurface flow
• 2F3U4P: 2 velocities for surface flow, 1 velocity for subsurface flow

In this context, "velocity" can be read also as "mean travel time" since the relation between the two quantities is given by the appropriate lengths measured along the flow paths.

In any case, the poster you can retrieve below the figure (click on the figure) tells it all. 

Beyond and side by side with numerics

The 23rd of april was quite busy for me. Early in the morning I gave a tak of acouple of hours to the students of Numerical Analysis.  My goal was actually to try to capture their interest about the topics I cover in my research, which I considere complementary to good numerics.

For supporting this idea, I divided my presentation in two parts. The first under the motto: the right numerics, for the right equations. There, I showed how Richards equation can be modified to account for transition to saturated conditions and  for freezing soil. This last part has been largely derived from the work of Matteo Dall'Amico (here his Ph.D. thesis) and the subsequent paper on the Cryosphere journal. With the benefit of hindsight, I can tell that I could have been much more clear on the physics of the problem, but it was just doing the presentation that I realised it.

The second part is dedicated to justify the rational of what I called "The Geoframe project", which is an integrated system for doing hydrology by computer, completely open source, and which has a deployment in the JGrass-NewAGE system. Click on the painting above  to see the presentation. 

Open Java resources for Hydrologists

Next question about Java, after having introduced it, and talked about options for doing numerics, is to see if there are effective resources on which hydrologists can build something useful, incrementally, and as a part of a community.

• The first recource that comes to my mind, is obviously the Object Modelling System version 3 (or OMS3). This is the framework we use for our modelling. We wrote some models and papers using it, and OMS3 is also at the core of the uDig Spatial Toolbox

Also OpenMI uses Java, but, at the moment, Java there is just the second choice, well behind their C#. A second framework which we are looking to is:

• Openda. It is a framework for data assimilation and calibration of models promoted by Deltares. 

There are out there probably other Java frameworks, and I will add them to the list, whenever they will be brought to my attention.
The field of more intense Java activity is actually the GIS one. There are at least three/four major open source efforts in Java:

• uDig, the GIS of our choice, with its field companion BeeGIS, and its mobile younger brother, Geopaparazzi
• GvSig, a good product indeed, which also has a companion for mobiles
• OpenJump, on which the Italian cartographic project is based

In fact, it is not very clear to me, but also the

• North52 

initiative seems having parts based on Java.

They are based on various type of resources among those in Java:

• GeoTools
• Sextante, a library for data geoprocessing by Victor Olaya

Both GeoTools and Sextante use 

• JTS, the Java topology suite for several elementary operations.

An interesting,  old but not outdated initiative to give a third dimension to data visualisation is 

• Visad, a library to visualise scientific datasets

from which sprout

• the IDV  virtual globes family to visualize 3D geophysical data

IDV is not the unique Java Virtual globe. Another one is:

• the Nasa WorldWind which gives (as open source) what Google Earth gives to the general public. Various prototype implementations of Nasa WorldWind in uDig (and in OMS3) were already made, and possibly sooner or later we will have a full mainstream implementation of it.

Last but not least an important resources is 

• the NetCDF data format (of which IDV, but also ncBrowse,  is a viewer) has also an implementation in Java.

Even if my post is more concerned about the development of Hydrological resources and models, the above frameworks/programs/applications/models or part of them can just be used (and just not developed) with profit.  Starting from OMS3 there are at least three major modelling efforts that use it:

• PRMS
• AgES
• JGrass-NewAGE

Not to forget is the whole Jgrasstools available for DEM manipulation (and further modelling) inside (and outside) uDig. All of them have more resources in a post dedicated to OMS 3 resources.
The Sextante library itself comes with more than 300 tools for manipulation of DEM and images. Also IDV and NasaWorldWind can be just used for the purposes of any research.
Another full-fledged Java Catchment model is

• Cuencas

by Ricardo Mantilla. Its source code can be obtained following the instructions in the user manual.

A recent addition to the resources is

• H2O, an engine for doing Machine Learning and Data Science

Why use Java, instead of  R ?

In one of my previous post, I talked about R software, and in fact, my group of people use both.  R is much more for some higher level operations, and is much less customisable than the resources developed in Java, and,  usually, less efficient.  My former students, using Java capabilities were able to built professional/industrial applications on the basis of Java (the GIS are a proof), that using just R would have been impossible.
However, the new incoming version of OMS3, should include the way to call R from the OMS console. Please contact the OMS version 3 developers for more information.

Why do not use Python ?

Python (or many other languages) could have been a good choice. Never say never, however I chose Java and to be successful one has to consistently invest his/her own limited resources in one direction.

There exist material/documentation on the above material ?

One introduction to programming GIS and essentially all I would require to a collaborator to know  can be found here by Andrea Antonello

My Past Research on Process Based Physical Modelling on the Hydrological Cycle

The first studies in this field took inspiration from analysis of moisture distribution in the soil, [J14, A8] where it was shown that, during relatively “dry periods, the moisture distribution in the soil can be understood and described with fractal analysis techniques. From these studies, and the desire to model evapotranspiration, eco-hydrological phenomena, hydrology and slope stability, and the evolution of the snowpack, there arose the need to develop an instrument capable of modelling the water cycle and soil moisture dynamics continually over time. These goals were the founding reasons for the implementation of the GEOtop Model [J24, A22]. GEOtop is “terrain-based (it is based on the use of digital terrain models and uses the knowledge of interac- tion between morphology and process) distributed (all the simulated variables are calculated for each pixel of the basin) model of “the water cycle (it simulates all the components of the water cycle, taking account of both the mass budget and the energy budget, the two budget equations being coupled through the temperature of the soil, which controls evaporation, hydraulic conductivity, and accumulation of the snowpack [J22]). A complete description of the model can be found in [J24, A22], articles that present the model system and a practical application to the Little Washita basin in Oklahoma, and, obviously in the manual [eb-05].

The GEOtop model was also applied during the study of the water cycle of Lake Serraia (Trentino, Italy) [A34]. [J25] demonstrates the effects of complex topography and morphology on the water cycle. In particular, one can observe that a more extensive channel network (as might arise in presence of greater slopes or more erodible soil) causes greater surface runoff and less evapotranspiration, which, in the energy budget, causes an increase in latent heat exchange with atmospheric boundary layer. The paper demonstrates, therefore, that topographic effects cannot be neglected in formulating the energy budget of the soil, as most global climate models normally do. Among the more theoretical studies, but essential to the distributed modelling of flows in unsaturated media, are [J27, A29]. In [J28] Richards Equation was perturbatively decomposed into a vertical component and a lateral one. The first dominates the initial phases of infiltration, the second the long-term redistribution of water volumes. With the work reported in [J30] the model was expanded with a soil freezing and thawing module, that allowed the analyses of the PermaNET project to be executed, and other studies performed by other researchers.

Recently, the GEOtop model has been used to estimate the impact of climate change on mountain catchments [rep05]. Ancillary studies have been dedicated to parameter calibration and uncertainties in hydrologic model forecasting [A37, A40].
The article [J38] envisages the restructuring of the GEOtop model with new numeric methods, developed together with Prof Vincenzo Casulli, and the adoption of a non-structured grid for the modeling. However, [J43] represents the state-of art of  GEOtop version 2.0, a milestone in the model history which contains Richards 3D integration, permafrost modelling, a multilayer snow model, renewed options for the treatment of meteo-data and radiation.
[J44] embrace the use of CLM and face the problem of data assimilation complemented by the use of Kriging techniques for filling the missing data.

A small community of users and developers has developed around GEOtop which is steadily growing.

The article [j38] envisages the restructuring of the GEOtop model with new numeric methods, developed together with Prof Vincenzo Casulli, and the adoption of a non-structured grid for the modeling.   [J43] represents the state-of art of  GEOtop version 2.0, a milestone in the model history which contains Richards 3D integration, permafrost modelling, a multilayer snow model, renewed options for the treatment of meteo-data and radiation.
[J44] embraces the use of CLM model, and faces the problem of data assimilation complemented by the use of Kriging techniques for filling the missing data.

References

In English:

[J14] - Rodriguez-Iturbe, I, Gregor K. Vogel, R. Rigon, D. Entekhabi, F. Castelli and A. Rinaldo, On the spatial organization of soil moisture fields, Geoph. Res. Letters, 22(20), 2757-2760, 1995.

[ J22] - Zanotti, F., Endrizzi, S, Bertoldi, G. e R. Rigon, The GEOTOP snow module, Hydrol. Proc., 18, 3667-3679 (2004), DOI 10.1002/hyp.5794

[ J24] - Rigon R., Bertoldi G e T. M. Over, GEOtop: A distributed hydrological model with coupled water and energy budgets, Vol. 7, No. 3, pages 371-388

[ J25] Bertoldi G. R. Rigon e T. M. Over, Impact of watershed geomorphic char- acteristics on the energy and water budgets, Vol. 7, No. 3, pages 389-394, 2006

[ J28] - Cordano E. R. Rigon, A perturbative view on the subsurface water pressure response at hillslope scale, Water Resour. Res., Vol. 44, No. 5, W05407- W05407, doi:10.1029/2006WR005740, 2008 

[j30]- M. Dall’Amico, S. Endrizzi, S. Gruber, and R. Rigon, An energy-conserving model of freezing variably-saturated soil, The Cryosphere Discussion, The Cryosphere Discuss., 4, 1243-1276, doi:10.5194/tcd-4-1243-2010, 2010

[J38] Cordano E., Rigon R., A mass-conservative method for the integration of the two-dimensional groundwater (Boussinesq) equation, submitted to Water Resour. Res., 2012  

[J43] - Endrizzi S., Gruber, S, Dall'Amico M., and Rigon R., GEOtop 2.0: simulating the combined energy and water balance at and below the land surface accounting for soil freezing, snow cover and terrain effects, Geosci. Model Dev.,  7, 2831–2857, 2014, www.geosci-model-dev.net/7/2831/2014/ doi:10.5194/gmd-7-2831-2014

[J44] -  Han X., Lin, X, Rigon R., Jin R., Endrizzi S., Local Analysis of L-Band Microwave Brightness Temperature Assimilation With Geostatistics for Soil Moisture Estimation, PLOSONE, 2015

[A8] - Rodriguez-Iturbe, I., G. K. Vogel, R. Rigon, D. Entekhabi, F. Castelli and A. Rinaldo, Scaling properties of soil moisture, Proceeding of the workshop on climate change and hydrometereological hazards in the mediterranean area, Perugia, 1995 

[A34] - Bertola P., Bertoldi G.,Grisenti P., Piva G., Ragazzi M., Righetti M., Rigon R., Salvaterra M., Soppelsa G., Tomazzolli V., Integrated research on eutrophication processes on Caldonazzo lake (Trento, Italy). Atti del Convegno Simposio Internazionale di Ingegneria Sanitaria e Ambientale, Taormina, 23-26 Giugno, 2004

[A37] - Entezarolmahdi R., G. Bertoldi e R. Rigon, An Automatic “Watershed Model Calibration process, 7th International Congress of Civil Engineering, Teheran, 2006

[A31] - E. Cordano, P., Bartolini, Rigon R. A flexible numerical approach to solving a generalized Richards’ equation problem and some applications, 2004

[A40] Entezarolmahdi R., Rigon R., Bertoldi G., Assessment of parameter uncertainty for physically based hydrologic model, using automatic optimization approach, XXX Convegno di Idraulica e Costruzioni Idrauliche, Roma 2006 

[eb05] Dall’Amico, A., Endrizzi, E., Gruber, S., Rigon R., The GEOtop Manual, Università di Trento, A Draft here, in press, 2012

[rep05] - R. Rigon, A. Bellin, L. Forlin, H. Fowler, S. Blenkinsop, Testing of climate change scenarios on a case-study catchment using different methodologies, Deliverable C2.4 AQUATERRA EU Project, 2005

In Italian:

[A22] - Bertoldi, G., Rigon R., Overt T.M. Un indagine sugli effetti della topografia sul ciclo idrologico con il modello GEOtop. Atti del XXVIII Convegno di Idraulica e Costruzioni Idrauliche, Potenza, pp.313-324, 2002

[A32] - Cordano, E., Panciera R., Rigon R., Bartolini P. Sulla soluzione diffusiva dell’equazione di Richards. Atti del XXIX Convegno di Idraulica e Costruzioni Idrauliche, Settembre 2004

A paper in Nature on Scientific Software

The news was brought to me by Martin Davis who had from Stefan Steiner.  The paper is:  The case for open computer programs and was published in Nature.

Here it is what Martin says:

"The paper raises the argument for open source software to a higher plane, that of being a necessary component of scientific proof. It points out that the increasing use of computational science as a basis for scientific discovery implies that open source must become a standard requirement for documentation. Apparently some journals such as Science already require source code to be supplied along with submissions of articles. Amongst other advantages, access to source code is an essential element of peer review.
An interesting example they mention is the infamous HadCRUT and CRUTEM3 meteorological datasets. One of the (few) salient criticisms levelled at this information during Climategate was the inability to reproduce the results by re-running the software. (Mind you, the software was probably a pile of crufty old Fortran programs mashed up by Perl scripts, so maybe it's just as well)"

This clearly reflect what I already wrote in some of my posts:

http://abouthydrology.blogspot.it/2011/03/going-beyond-present-stato-of-art-in.html
http://abouthydrology.blogspot.it/2012/02/reproducible-research-and-papers.html

Modelling Environments for Biophysical Modelling in Hydrology and Agriculture

Modelling Environments for Biophysical Modelling in Hydrology and Agriculture: Object Modeling System 3 (OMS3) and Biophysical Model Applications (BioMA)

July 9-13, 2012

Joint Research Centre, European Commission, Ispra, Italy

The need for integrated analysis, and the multiplicity of possible goals in analysis which require biophysical modelling, necessitates more than ever the capability of composing modelling solutions of known quality which are transparent to users and consist of reusable model components.

There is a variety of modelling platforms and the number of model components and tools is growing; however, there is little interaction among developers of modelling platforms and the potential community of researchers who may benefit from new techniques and technologies in their work. This workshop is meant to provide this opportunity by illustrating the concepts at the base of two modelling frameworks, and by having a hands-on approach to both. The two modelling environments which will be presented are the Object Modeling System (OMS3) and the Biophysical Model Application (BioMA), briefly described below. Both are operational and allow running complex analyses in the biophysical domains of hydrology and agricultural production. The closing session will include a critical discussion on the concepts and realizations experienced during the week.

More Information on the website of the course.  OMS3 as you can verify from many post:

• uDig got a spatial toolbox
• Going Beyond the Present Stato-of-Art in hydrological modeling
• The JGrass-NewAge system for forecasting and managing the hydrological budgets at the basin scale: the models of flow generation, propagation, and aggregation

is at the core of my modeling strategy. I believe is one of the best way to have a modern approach to do models. So the school will be the occasion to learn about it (for who is already a good programmer)

Going Beyond the Present Stato-of-Art in hydrological Modeling. My point of view.

In the recent years there was a bloom of initiatives to support the modelling-by-component paradigms. A relevant European experience is OpenMI (http://www.openmi.org) which CUDAM and Hydrologis endorsed, and applied to a large modelling project, which was expected to produce a modeling system prototype of the river Adige. The system was intended to provide also the informatics for a decision support system to be built eventually. The OpenMI system, a product of the EU project HarmonIT, promised to improve collaboration among scientists, and to embed previous modelers' knowledge in a new framework. The choice of OpenMI came after an accurate analysis of competing frameworks, developed with hydrology and other science in mind. This analysis, which was partially borrowed from Rizzoli et al. [2005] included economic modeling, the agricultural domain (APSIM, STICS, CROPSYST, APES, and in general the results of the ongoing SEAMLESS EU project), the hydrology-water management milieu (TIME, OpenMI, IMT, OMS 2.2, and the Jupiter APIs). Some integrated environments such as those presented by SME, TARSIER, MODCOM, SWAT [for references, please see Argent and Rizzoli, 2004] and Wesselung at al. [1998] were also studied for comparison as well as products coming from the atmospheric sciences (i.e ESMF), and from the platform for high performance computing of the CCA Forum. A recent review of some of them was provided by Jagers [2010]. They were asked to respond to the requests exposed at the second biennial meeting of CUASHI in Boulder 2008 (http://www.slideshare.net/GEOFRAMEcafe/geoframe-a-system-for-doing-hydrology-by- computer).
These questions were essentially:
• Bringing to hydrological modelers the best of modern software technologies in terms of software architecture, design and engineering, code reusability, and collaborative information sharing
• Allowing third party revision of models and codes, thus improving the inspection of science embedded
in modeling. (Clearly in order to achieve this state, we needed to set a new model, where people can submit their code for review. In chain this caused the requirement - not yet fulfilled - to set of guidelines and rules to use as criteria for code quality)
• Creating models that could be used and perused by various users, for a variety of requirements (e.g. Rizzoli et al., 2005)
• Providing a collaboratively produced set of tools for calibration, data assimilation and statistical analysis, which are not by themselves the core physical science we pursue, but are necessary to exploit the possibilities open by the modern measurement systems, to supporting remote sensing and web-distributed resources, and therefore let scientists to concentrate on their science more than on tools. These tools which should however support all of the requirements for executing modern, cutting-edge science.
• Accessing data seamlessly and transparently, providing data marts, independent of the database layer
• Combining modules which work entirely in memory and step through time sequentially, making it possible to estimate non-linear environmental feedback systems.
• Integrating of models with the appropriate tools for gathering and formatting data deriving from heterogeneous sources, whilst promoting the use of standards such as those requested by INSPIRE, S@NY, the Open Geospatial Consortium, W3C and others. To accomplish this, further additions to the design requirements were deemed necessary, which entered the design requirements:
• the source code must be made available under open source license (to enable inspection of the internals of the models and progress of research)
• the platform should be language neutral (connecting code written in Java, C/C++, and FORTRAN) and platform neutral (i.e. working on MS Windows, Linux and MacOS)
• the platform should be built entirely by open source tools (as a desired approach, however not mandatory)
• interfaces to OGC data types should be included
• the platform should be compliant of INSPIRE directives for data formats and protocols
The initial choice for realizing the system was the OpenMI system (http://www.openmi.org, Gregersen et al., 2007), and since a binding language was needed to connect model components, Java was the choice, being open source, multiplatform and able to obtain performances comparable to compiled codes as demonstrated by the Mines Java Toolkit (http://inside.mines.edu/~dhale/jtk/index.html) and COLT (http://acs.lbl.gov/software/colt/) libraries, and by other sources (e.g. Knoll and Mirzaei, 2010). However, after the adoption of OpenMI 1.4, and having made possibly a large implementation of codes out of the group of OpenMI developers (numbering more that sixty model components included in the distribution of the GIS JGrass), CUDAM and Hydrologis decided to compare OpenMI with the OMS3 system (http://www.javaforge.com/project/oms) before switching to OpenMI 2.0 to which hydrologis contributed [e.g. Gijsbers et al., 2010]. Following this comparison, the decision to move to OMS3 was made. There were several reasons for this decision:
• The OpenMI system which has been implemented is not completely component-oriented: it is a model coupling framework and legacy centered.
• OpenMI 2.0 will be an improved platform, but the overhead it introduces in terms of boiler plate code threatens the normal modelist. OMS3 is much simpler and offers a more intuitive and cleaner design. After several years of development, there are just a few models that can be considered OpenMI (and these include CUDAM's newAge model and the Horton Machine components). Most of the models "ported to OpenMI" are just wrapped with an OpenMI interface, and not built upon OpenMI. This only manifests the difficulty of the task. Experiments in teaching modelers to build OpenMi components were frustrating and unsuccessful. A recent study (Lloyd, 2010) shows the quantitative overhead introduced by traditional modeling frameworks for model development, compared to the non-invasive and lightweight design of OMS3 models.
• OMS 3 adopts the meta data annotations for describing and managing the components [David et al, 2010a,b], which make any port, from any platform, much more simpler. CUDAM and Hydrologis had more that 60 components written in OpenMI 1.4 and bringing them to OpenMi 2.0 turned out to be a major undertaking. OpenMi offers a xml based component description, which is not as effective as the annotations in OMS3. While in principle OpenMI can adopt this strategy for component description, OMS3 is already has it implemented.
• OMS3 is considered lightweight and non -invasive [Lloyd et al., 2009]. The version of Hydrologis and CUDAM's NewAGE model contains around one third of the code of its OpenMi 1.4 counterpart. A simple OMS3 component contains little meta data overhead complementing the core computational code.
• OMS3 offers a larger academic base of modelers than OpenMI [e.g. Ascough et al., 2010]. CoLab [https://colab.sc.egov.usda.gov/], could potentially offer a rare pool of existing model components for reuse within other modular models. As a result it incorporates methods such as SCE, LUCA, FAST, DDS, and other approaches reflecting its academic origin. In addition OMS3 also reflects operational needs by providing ensemble simulation support, such as ESP.
• While OpenMI is mostly a component linking framework, it does not offer features such as simulation management for calibration, parameter estimation, uncertainty, and sensitivity analysis. They are not within its scope.
• OMS3 automatically supports multithreading and Amazon's elastic cloud computing environment [David et al., 2010c]. OpenMI 2.0 does not support parallelization and multi-threading. OMS3 also supports some of the more standard calibration methods required by environmental modelers.

• OpenMI 1.4 supports a Java execution system and console through JGrass. Although this is a great system, it needs to be upgraded to support OpenMI 2.0. This work will likely only be carried out by Hydrologis and CUDAM. A small but solid community of Java developers surrounds OMS3, which whom we can talk and cooperate. OMS3 currently offers a Java execution system and console and this community will eventually integrate OMS3 with the J-Grass console.
• OMS3 is a program which will exist for many years, being supported by a long term financial support from USDA. OpenMi promises to be the same, but is only financially supported by its participants.
• OMS3 plans to be compatible with CSDMS, in the sense that modules that will work with OMS3 will be easily scaled up to CSDMS, and supercomputers. Related to this effort, a OpenMI 2.0 wrapper generator was developed as a part of the OMS3 toolchain, to offer the use of any OMS3 component/model as an OpenMI model.
• OMS3 already provides tools for visualizing time series and doing simple statistical analysis. All major statistical checks analyzing models performance are built in (e.g Nash Suttcliffe, TRMSE, and many others).
Many of the tools formerly present in OpenMI have now been ported to OMS3 (see http://www.jgrasstools.org). OMS3 is provides a comprehensive library of process components originating in hydrology and agricultural sciences. Component based model implementations are currently being adopted for water supply forecasting in the western US (national Water and Climate Center Portland Oregon), for conservation effects assessment (USDA) using fully distributed watershed models, and for USDA-NRCS erosion modeling. Ongoing efforts at USDA focus on the integration of OMS3 into SOA and Cloud environments allowing a cost effective and scalable use of OMS3 based simulation models. While the software architecture behind the APES simulator (http://www.apesimulator.it/) promises to be as lightweight as OMS3, its software still remains undisclosed to modelers. Despite the declaration to be "framework independent", in practice, APES requires a third party engine to be executed and its generality is more theoretical than practical.

References

- Argent, R.M., Rizzoli, A.E., 2004. Development of multi-framework model components. In: Pahl-Wostl, C., Schmidt, S., Rizzoli, A.E., Jakeman, A.J. (Eds.), Transactions of the Second Biennial Meeting of the International Environmental Modelling and Software Society, Osnabru ? ck, Germany, vol. 1, pp. 365e370.

-Ascough II J.C. et al. Integrated Agricultural System Modeling Using OMS 3: Component Driven Stream Flow and Nutrient Dynamics Simulations. iEMSs 2010 International Congress (2010) pp. 1-9

-David O. et al. Object Modeling System v3.0, Developer and User handbook. Manual (2010) pp. 1-111

-David O. et al. Rethinking Modeling Framework Design: Object Modeling System 3.0. iEMSs 2010 International Congress (2010) pp. 1-9

-David O. et al. Cloud Services Innovation Platform (CSIP). Internal Draft (2010) pp. 1-17

-Donatelli et al., 2009 M. Donatelli, G. Russell, A.E. Rizzoli, M. Acutis, M. Adam, I. Athanasiadis, M. Balderacchi, L. Bechini, H. Belhouchette, G. Bellocchi, J.E. Bergez, M. Botta, E. Braudeau, S. Bregaglio, L. Carlini, E. Casellas, F. Celette, E. Ceotto, M.E. Charron-Moirez, R. Confalonieri, M. Corbeels, L. Criscuolo, P. Cruz, A.D. Guardo, D. Ditto, C. Dupraz, M. Duru, D. Fiorani, A. Gentile, F. Ewert, C. Gary, E. Habyarimana, C. Jouany, K. Kansou, M.J.R. Knapen, G.L. Filippi, P. Leffelaar, L. Manici, G. Martin, P. Martin, E.C. Meuter, N. Mugueta, R. Mulia, M.V. Noordwijk, R. Oomen, A. Rosenmund, V. Rossi, F. Salinari, A. Serrano, A. Sorce, G. Vincent, J.P. Theau, O. Therond, M. Trevisan, P. Trevisiol, F.K. Van Evert, D. Wallach, J. Wery and A. Zerourou, APES: The Agricultural Production and Externalities Simulator. In: F. Brouwer and M.K. Van Ittersum, Editors, Environmental and agricultural modelling: integrated approaches for policy impact assessment, Springer Academic Publishing (2009).

-Gijsbers, P., S. Hummel, S. Vanec ?ek, J. Groos, A. Harper, R. Knapen, J. Gregersen, P. Schade,
A. Antonello, and G. Donchyts. From OpenMI 1.4 to 2.0. In Proceedings iEMSs 2010, 2010.
Gregersen, J. B., P. J. A. Gijsbers, and S. J. P. Westen (2007), OpenMI: Open modelling interface, J. Hydroinf., 9(3), 175–191, doi:10.2166/ hydro.2007.023.
Jager, B.. Linking Data, Models and Tools: An Overview. iEMSs 2010 International Congress (2010) pp. 1-8

-Knoll and Mirzaei. Scientific computing with Java. Comput. Appl. Eng. Educ. (2010) vol. 18 (3) pp. 495-501

-Krause, P. Das hydrologische Modellsystem J2000, Schrft. FZ-Jülich, Bd. 29, 2001

-Krause, P., Kralisch, S., Flügel, W.A., 2005. Preface: model integration and devel- opment of modular modelling systems. Advances in Geosciences 4.

-Lloyd, W., O. David, J. C. Ascough II, K. W. Rojas, J. R. Carlson, G. H. Leavesley, P. Krause, T. R. Green, and L. R. Ahuja. An exploratory investigation on the invasiveness of environmen- tal modeling frameworks. InAnderssen, R. S., R. D. Braddock, and L. T. H. Newham, editors, 18th World IMACS Congress and MODSIM09 International Congress on Modelling and Simu- lation, pages 909–915. Modelling and Simulation Society of Australia and New Zealand and International Association for Mathematics and Computers in Simulation, July 2009.

-Nativi, S., Caron J., Davis, E., Domenico, B, Design and implementation of netCDF markup language (NcML) and its GML-based extension (NcML-GML) , Computer and Geosciences, (2005), 31(9), 1104-118

-Rizzoli, A.E., Svensson, M.G.E., Rowe, E.C., Donatelli, M., Muetzelfeldt, R., van der Wal, T., van Evert, F.K., Villa, F.:Modelling Framework (SeamFrame) requirements. SEAMLESS report no. 6, Dec 2005

-Wesselung, C. G., Karessenberg, D.-J., Burrough, P. A. and Van Deursen, W. P. A. (1996), Integrating dynamic environmental models in GIS: The development of a Dynamic Modelling language. Transactions in GIS, 1: 40–48. doi: 10.1111/j.1467-9671.1996.tb00032.x

Characteristics of good modeling software

Making good models is just one part of the whole job of a hydrologist. It is a tradition in our research field to make good research with not very good computer codes. Please do not misunderstand me. I do not mean that the algorithms used are wrong: I mean that the overall simulations machinery is usually not very well engineered, and using the code produced by researchers is not as easy as it could be (and actually is for many   industrial programs). This eventually makes scientists (and users too) loose a lot of time in redoing the same things, even when the original codes are available, simply because these codes are not well documented or do not provide those  functionalities that make them usable. The following paper (that was addressed to me by the Author of the Csparse library, T. Davis) covers some of the topics of making a good and usable code, and is a must to read for who does modeling.

Please follow the link below for getting the paper (last accessed February 3rd, 2011)
Characteristics of Industrial strength software.

The main conclusions by the Authors are summarized, and a little edited below for the laziest.

" … It is important to design a ... software to be easy to use and robust. Often it is better to assume that the user is not an expert in … algorithms, but someone who has a problem to solve and wishes to solve it accurately and efficiently with minimal effort. After all, even experienced users were once novices and a user’s initial experiences of using a solver are likely to determine whether he or she goes on to become an expert user. Based on our experiences …., in addition to the requirements of good performance (in terms of memory and speed) and the availability of comprehensive well-written documentation, in our opinion the following features characterize an ideal …. solver.

• Simplicity: the interface should be simple and enable the user to be shielded from algorithmic details (note: this is called in OO information hiding). The code should be easy to build and install, with no compiler warning messages. During the building of the software from supplied source, minimum effort and intervention by the user should be required. …. dynamic memory allocation should be used so that the user need not preallocate memory. In fact, the software developer needs very good reasons for not selecting a language that includes dynamic memory allocation.

The software developer should consider providing interfaces to popular high-level programming environments, such as Matlab, Mathematica, and Maple (note: and I add R, because Open Source is an add value.. Besides offering an appropriate interface is also behind the whole JGrass Project).

• Clarity: …. Furthermore, to allow repeated solves and iterative refinement there should be a clear distinction between (note by RR:) preprocessing and solve phases. … Developers should consider offering simple (all-in-one) interface as well as an interface with the greater flexibility of access to the different phases of modeling.

• Smartness: good choices for the default parameters and of the algorithms to be used should be automatically made without the user having to understand the algorithms and to read a large amount of detailed technical documentation. There should be an option to check the user-supplied input data, particularly for any assumptions that the code relies on. (Note by RR:) Parameters of the models should be as much as possible explained in documentation and code.

• Flexibility: for more experienced users and those with specific applications in mind, the solver should offer a wide range of options, ….. There should also be options for the user to specify the information that he or she requires …. The software should …. support 64-bit architectures, (note by RR) and be platform independent.

• Persistence: the solver should be able to recover from failure. For example, if it is found that there is not enough memory, a code that contains both in-core and out-of-core algorithms should automatically switch to out-of-core mode. Reverse communication should be designed to allow corrections to the input data.

…

• Threadsafety: The code should be threadsafe to enable the user to safely run multiple instances of the package simultaneously in different threads or on different processors.

"