Material for the GEOframe Winter School - Getting Started with OMS and Jupyterlab

Day first of the Winter School on GEOframe was conceived to give people the taste of what OMS is and how to use it with Python lab. For the Installations, please refer to the Installation page. 

Not much information about Jupyterlab though. It will be given interactively using notebooks and explaining their contents.

• The philosophy behind OMS/GEOframe (YouTube video)
• The GEOframe deployment
• A brief introduction to the Object Modelling System (YouTube video)
• OMS User Manual - Introduction (pdf, epub) 
• The OMS working environment (YouTube video)
• OMS User Manual - The working Environment (pdf, epub)
• The structure of a .sim file (YouTube video)
• The existing manual excerpt (outdated but still useful) (pdf, epub)
• Examples (All the material in the .zip files. Notebooks in /docs folder)
• ex_00 (zip files)
• Examples-basic (zip files). Please find Jupyter notebook shown in video below in the doc directory of the zipped files.
• Example 001YouTube video
• Controlling Iteration (Ex 05) YouTube video.
• Going on with examples: Again on You Tube Video Iteration, Bulk connect, Feedbacks
• Exercises 1&2
• Watertank  Example (zip files). Documentation in the doc folder.
• Water Tank example - Part I YouTube video
• Exercises 3&4
• Creating a notebook from the scratch (Live YouTube video by Niccolò Tubini)
• Open directions

Previous Winter School page

All Winter School pages

Next Winter School page 

Material for the Winter School on the GEOframe System - Installations

The Winter School on the GEOframe system is approaching.  In this webpost you will find all the preparatory material of the school indexed.

Installations

You are not assumed to know Python or Java to participate to the School. However programs runs on Java.. We will not do any Java programming though.  Input/Output of models will be treated by using some scripting in Python. We will communicate the appropriate notions during the classes. However, for the interested there are plenty of courses on the web (for instance this comes from SciPy 2018. Other from SciPy here.)

• Java. GEOframe and OMS are written in Java and they require to have installed Java on your computer. Here you can find instructions to install Java on your computer.  OMS need Java 8 JDK.  Please note that you need the Java Development Toolkit (JDK) installed not the Java Runtime Environment (JRE).  The official Oracle's pages are here. You can watch several videos for Windows: here. Googling you can get videos for your platform.
• We are going to use Docker for our modelling. Therefore you need to install it (here for Linuxes) on your computer. To quickly understand what Docker is, you can see here. Or read this tutorial. This video could be useful too. The knowledge of Docker required will be extremely elementary. You will not be required to do Docker applications. You simply use one and, at the end, is just matter of executing a command. 
• During the School we will use Jupyter and Python 3 for data management and visualisation. It would be great if you could arrive
• The post at this link contains all the information needed. For installation of the software, we suggest Anaconda. For any problem contact us through the mailing list. 
• To understand what a Jupyter notebook is about, please see its manual. However, we will use Jupyterlab.  (You can see a YouTube video about here).  One can think that Jupyter was heavily based on the look-and-feel of Mathematica notebooks: but Jupyterlab is a step ahead to something different. Installing Anaconda, you also have already installed Jupyter notebook. So you have just to try it issuing the command: 
• Jupyter notebook
• You still do not know what to do with it, but we will lear step by step during the Winter School
• For Jupyter lab, you have to do installation  explictly: 
•  conda install -c conda-forge jupyterlab 
• The manual of Jupyterlab is here.
• Finally, at least for MacOs user, due to a bug either in the Docker or in GEOtools, it is necessary to use the OMS Console.  You can download the beta 3.5.62 release from here.

• If you want to further with Jupiter and Jupiterlab, you can  install BeakerX. For its installation, please follow verbatim the instructions here. The instructions given at the main page of BeakerX are incomplete ;-).

Previous Winter School page

All Winter School pages

Next Winter School page

Updated material on Horton Machine

The Horton Machine is our set of tools for terrain analysis actually maintained by Hydrologis. They since 2000 and something took care of keeping my tools alive on various platforms: GRASS, JGrass, uDig and now on gvSIG. We also have a porting of the Horton machine that works under GEOframe but is the same codebase.

Silvia Franceschi and Andrea Antonello gave a tutorial at this year gvSIG conference and here it is the material.

You can find the same material also at this osf project. Soon they will follow some theoretical lectures.

Life After Ph.D. (especially for Italians but of general interest)

This is part of the Ph.D. days of Department of Civil, Environmental and Mechanical Engineering of University of Trento. October 29, 30, 31 were used to check the advancement of our almost one hundred Ph.D. students. Who was moving to the first to the second year, who from the second to the third, who had to prove to be able to write the final Thesis after the third year.

I think professor Scardi offered a good overview, worth to be listened to.

Parallel Computing in GEOframe

Inside our University we have a group that works on parallel computing. My colleagues work in various directions, using standard technologies as OpenMP or MPI. I confess I am very ignorant on the details. However, starting from our hydrological problems, it happens that we embraced OMS that implements an intrinsic parallelism which acts between components. Besides, my PhD student Francesco Serafin, introduced a new way to parallelize our problems called Net3. All of these things are shortly presented in the slides below.

The link, here.

HPC school for PhD students at Department of Civil, Environmental and Mechanical Engineering

Among the initiative organized by our doctoral school there is this introduction to computing that can be od interest for beginners (with a little attitude).

Here below, please find the Software Carpentry lectures that were streamed on YouTube and are now present at our Department page.

Unix Shell introduction

Jupyter notebooks and a little of Python programming

Crocus and Snowpack in a nutshell

There is not an enormous choice of process based models that treat snow Among them, two that are a reference for us are Snowpack and Crocus. Here it is their essential references.

SNOWPACK

• Bartelt, P., & Lehning, M. (2002). A physical SNOWPACK model for the Swiss avalanche warning Part I. Numerical model. Cold Regions Science and Technology, 35(3), 123–145. https://doi.org/10.1016/S0165-232X(02)00073-3
• Lehning, M., Bartelt, P., Brown, B., Fierz, C., & Satyawali, P. (2002). A physical SNOWPACK model for the Swiss avalanche warning Part II. Snow microstructure. Cold Regions Science and Technology, 35(3), 147–167. Retrieved from www.elsevier.com/locate/coldregions
• Lehning, M., Bartelt, P., Brown, B., & Fierz, C. (2002). A physical SNOWPACK model for the Swiss avalanche warning Part III: meteorological forcing, thin layer formation and evaluation. Cold Regions Science and Technology, 35(3), 169–184. Retrieved from www.elsevier.com/locate/coldregions

CROCUS

• Brun, E., David, P., Sudul, M., & Brunot, G. (1992). A numerical model to simulate snow-cover stratigraphy for operational avalanche forecasting. Journal of Glaciology (Vol. 38).
• Brun, E., Martin, E., Simon, V., Gendre, C., & Coleou, C. (1989). AN ENERGY AND MASS MODEL OF SNOW COVER SUITABLE FOR OPERATIONAL AVALANCHE FORECASTING. Journal of Glaciology (Vol. 35)
• Vionnet, V., Brun, E., Morin, S., Boone, A., Faroux, S., Le Moigne, P., … Willemet, J. M. (2012). The detailed snowpack scheme Crocus and its implementation in SURFEX v7.2. Geoscientific Model Development. https://doi.org/10.5194/gmd-5-773-2012

The flood/L'alluvione by Eugenio Montale

The flood has drowned the clutter

of forniture,  papers, and paintings that crammed

the double-padlocked cellar. 

Maybe they fought back blindly-the books 

in red morocco. Du Bos’s endless dedications, 

the wax seal with Ezra’s beard, Alain’s 

Valéry, the manuscript

of the Orphic Songs, as well as a couple 

of shaving brushes, a thousand knicknacks, and all 

your brother Silvio’s compositions. 

Ten, twelve days in that savage maw 

of fuel-oil and shit. Clearly they suffered 

terribly before losing their identity. 

I’m deep in crud too, up to my neck, though 

my civil status was doubtful from the outset. 

It’s not muck that besets me, but the events 

of an unbelievable, and always unbelieved, reality. 

My courage in facing it was the first 

of your loans, and perhaps you never knew. 

Eugenio Montale (see also here), Xenia I, appeared in Agni 34 

(translation by William Arrowsmith) 

L’alluvione ha sommerso il pack dei mobili,

delle carte, dei quadri che stipavano

un sotterraneo chiuso a doppio lucchetto.

Forse hanno ciecamente lottato i marocchini

 rossi, le sterminate dediche di Du Bos,

il timbro a ceralacca con la barba di Ezra,

il Valéry di Alain, l’originale 

dei Canti Orfici – e poi qualche pennello

da barba, mille cianfrusaglie e tutte 

 le musiche di tuo fratello Silvio.

Dieci, dodici giorni sotto un’atroce morsura

di nafta e sterco. Certo hanno sofferto

tanto prima di perdere la loro identità.

Anch’io sono incrostato fino al collo se il mio

stato civile fu dubbio fin dall’inizio.

Non torba m’ha assediato, ma gli eventi

di una realtà incredibile e mai creduta.

Di fronte ad essi il mio coraggio fu il primo

dei tuoi prestiti e forse non l’hai mai saputo.

For a comment in Italian see here

Data over Space and Time course by Cosma Shalizi

I confess I am a fan of Cosma Shalizi (GS). I do not know him, I do not know if he is a nice person or bad ass. He dos not know me. For sure I like the way he approaches statistics and his book (under the previous link). I am also astonished by his blog and the flood and variety of his readings.  Now he is giving a class on Spatial and temporal statistic. Topics that I think are necessary to know by Hydrologists. His class can be found here.

I hope that he does not mind if I cross reference his material.

Controllability and Dynamical Systems

Wow! There is a lot of literature on dynamical systems. Originally, when i though to dynamical systems I identify them with the non linear dynamical systems which produced chaos (e.g. Strogatz, 1994) or to elegant geometric representations of mechanics (Arnold, 1983, 1989 ). Looking to lumped hydrological models as dynamical systems opened to me new perspectives. The more important one: learning dynamical systems from system dynamics. Said this way, it seems a joke or a tautology. However system dynamics is its own discipline. It can be think to originate from von Bertalanffy but it is conjugate as “System and Control” since Kalman (e.g. Kalman, 1968). Wilems (e.g. Polderman and Wilems, 1998) offers a little different (but interesting) perspective (and a terrible notation).

 
When one think to a systems of ordinary differential equations (ODEs), obviously the thoughts go to their solutions. Does exist a solution for such a system ? Is it unique ? A mantra that we learned since our first courses on calculus. However “System and Controls” steers towards different concepts (e.g. Luenberger, 1979 or Kalilath, 1980 or more mathematically, Sontag, 1998). In fact the concept of control implies to study the conditions under which the ODEs systems solution can be moved to a precise beahvior (Willems, 2007) by actuators, i.e., external inputs (fluxes ?) to the state variables. This condition was easily translated into the determination of the rank of a special matrix, first written by Kalman (e.g. Liu and Barabási, 2016). Other equivalent formulation include the Gramian integral. What exactly means this property said “controllability” ? It means no more that what is intended: that trough a number of external actions you can move the systems to any conceivable state. Let $x_1$ to $x_n$ these variables, then you have to be sure that that all the n-dimensional vector domain can be spanned under the action of actuators, and the Kalman result, just establish the condition under which this is possible. I did not mention however, that the above results are valid for linear systems: a great result but a little poor in scope, since most of the problem we have are nonlinear. Fortunately people piooneered studies on the controllability on non linear systems since the seventies (Heines and Hemes, 1970; Hermann and Kremer, 1977; Cornelius et al, 2013, Isidori, A., 2013).
The application field of controllability had its first application oh hydraulic and mechanical engineering (e.g. Polderman and Wilems, 1998) , but soon moved to other systems. Notably: ecology (Iglesias and Ingals, 2010), brain (Lynn and Basset, 2018) , internet connections ( Liu and Barabási, 2016). There is then hope that those studies also impact hydrological and earth system sciences.
I did not mentioned it but really entangled with the more recents developments of the matter, there is the study of graph theorethic methods applied to the dynamical system (Yamada and Fouls,1990). Barabasi and Liu (Liu and Barabási, 2016) wrote a recent review that practically starts from the illustration of the systems with graph.
Made the note that I believe that Petri Nets are more exppressive than those used by Barabasi, I found the graph methods more easy to grasp and intuitive. Possibly one can also see connections between the results here (in dynamical systems) and some of those in causality theory (Pearl et al., 2016). For instance the theorem regarding the structural controllability of systems described by graphs (see in Lu and Barabási, 2016) recall the theorems on correlations in causality theory. A system which is correlated, read as a causal graph, is not controllable (it seems). A-posteriori, this seems obvious, since correlation, means some type of limitation on the phase space explorability.


References

• Arnold, V. L. (1983). Geometrical methods in the theory of ordinary differential equations. Springer-Verlag. 
• Arnold, V. L. (1989). Mathematical Methods of Classical Mechanics 2nd ed, 1–536. 
• Cornelius, S. P., Kath, W. L. & Motter, A. E., Realistic control of network dynamics. Nat.Commun. 4, 1942 (2013). 
• Isidori, A., Nonlinear control systems (Springer Science & Business Media, 2013). 
• Haynes, G. & Hermes, H. Nonlinear controllability via lie theory. SIAM J. Control 8, 450–460 (1970). 
• Hermann, R. & Krener, A. Nonlinear controllability and observability. IEEE Trans. Automat.Contr. 22, 728–740 (1977). 
• Kailath, T. (1980). Linear systems (pp. 1–704). Prentice-Hall. 
• Kalman, R. E. (1968). Lectures on controllability and observability. Seminar on Controllability and Observability, Fondazione Guglielmo Marconi, Pontecchio Marconi, 1968. 
• Liu, Y.-Y., & Barabási, A.-L. (2016). Control principles of complex systems. Reviews of Modern Physics, 88(3), 247–58. http://doi.org/10.1103/RevModPhys.88.035006 
• Luenberger, D. G. (1979). Introduction to Dynamic Systems (pp. 1–231). John Wiley & Sons. 
• Lynn, C. W., & Basset, D. S. (2018). The physics of brain network structure, function, and control. arXiv.V, 1–75. 
• Pearl, J, Glymour, M. and Jewell, N.P., (2016). Causal inference in statistics: a primer, Wiley 
• Polderman, W. J., & Willems, J. (1998). Introduction to the mathematical theory of systems and control: a behavioural approach (pp. 1–458). Springer-Verlag. 
• Willems, J, The Behavioral Approach to Open and Interconnected Systems. (2007) IEEE Control Systems, 27(6), 46–99. http://doi.org/10.1109/MCS.2007.906923 
• Sontag, E. D. (1998). Mathematical Control theory: deterministic finite dimensional systems (pp. 1–544). Springer-Verlag. 
• Strogatz, S. (1994). Nonlinear dynamics and chaos (pp. 1–505). Readin, Massachusetts: Perseus books. 
• Sussmann, H. J. & Jurdjevic, V., Controllability of nonlinear systems. Differ. Equ. 12, 95–116 (1972). 
• Whalen, A. J., Brennan, S. N., Sauer, T. D. & Schiff, S. J. Observability and controllability of nonlinear networks: The role of symmetry. Phys. Rev. X 5, 011005 (2015). 
• Yamada, T., and L. R. Foulds, 1990, A graph‐theoretic approach to investigate structural and qualitative properties of systems: A survey, Networks 20, 427.

Peter Germann's open book on preferential flow

Peter German (GS) first raised the issue of how preferential flow in soils due to preferential flow in a famous paper with Keith Beven (GS) in 1982, Macropores flow in soils. A benchmark paper from which uncountable studies followed. Now retired, he wrote a book entitled "Preferential flow: Stokes approach to infiltration and drainage".

Since 1982 there was a great debate on preferential flow modelling which is not yet concluded. Germann's book is evidently  a contribution from a protagonist of this story and he had the kindness to make it open. I am not sure about the content. Stokesian flows are those that happens when all acceleration is dissipated, so the subtitle is not very informative on the content. More detailed comments will arrive when I will have read it. For now you can get it by clicking on the Figure.

Kirchner 2016 model

I open a new series of posts that analyze simple (or less simple) lumped (reservoir based) Hydrological Models. They are Dynamical Systems and they are well represented through Petri Nets introduced in Reservoirology #3. The present is the case of a simple two reservoirs model presented in Kirchner 2016 (from now on, K2016b).

Because of  its simplicity, K2016b is a nice case study to test and verify various aspect treated more abstractly in Rigon et al., 2016.  The presentation that contains K2016 is here.

Reference

Kirchner, J. W. (2016). Aggregation in environmental systems-Part 2. Catchment mean transit times and young water fractions under hydrologic non-stationarity. Hess, 20, 299–328. http://doi.org/10.5194/hess-20-299-2016

Richard Rotunno on the predictability of Storms

In these days we have here in Trento,  at my Department, Richard Rotunno (GS), a pleasant person and an outstanding scientist. His field of interest is meteorology and micrometeorology and, in particular, the evolution of storms.  Here in Trento he gave various seminars and in particular one  on the predictability of storms, which he permitted to share with you.

As usual, by clicking on the above image, you will be able to access the presentation. He cited some interesting papers that you will find below. On the butterfly effect, you can also see this my previous post, to which I think the literature cited by Richards add something important.

References (In the order in which they are cited)

• Charney, J.G; Fjörtoft, R and von Neumann, Numerical integration of the barotropic vorticity equation, Tellus, 2:4, 237-254, DOI: 10.3402/tellusa.v2i4.8607, 1950 
• Lynch, P., The origins of computer weather prediction and climate modeling” , J Comp Phys, doi:10.1016/j.jcp.2007.02.034, 2008 
• Thompson, P.D., “Uncertainty of Initial State as a Factor in the Predictability of Large Scale Atmospheric Flow Patterns”, Tellus IX, 1957 
• Lorenz, E.N., “The Predictability of a Flow which Possesses Many Scales of Motion” E. N. Lorenz, Tellus XX1, 1969 
• Palmer, T.N; Döring, A.; Seregin, G. The Real Butterfly Effect”. 27,9, R123, Nonlinearity 2014 
• Lilly, D.K., “Numerical Simulation Studies of Two-Dimensional Turbulence: II. Stability and Predictability Studies” Geophysical Fluid Dynamics, 4(1), 1972 
• Tennekes, H, Turbulent flow in two and three dimensions, Bull. Amer. Met. Soc., 59(1), 1978 
• Skamarock, W.C., Evaluating Mesoscale NWP Models Using Kinetic Energy Spectra, Monthly Weather Review, 2004 
• Patton, E.g. and Sullivan, P.P, Atmospheric stability influences on coupled boundary-layer-canopy turbulence, Journal of Atmos. Sci., https://doi.org/10.1175/JAS-D-15-0068.1, 2016 
• Falko, J., Insights into Atmospheric Predictability through Global Convection-Permitting Model Simulations, Journal of Atmos. Sci., https://doi.org/10.1175/JAS-D-17-0343.1, 2018

Floods in Alberto Viglione research

I had the honour to be in the committee that selected Alberto Viglione (GS) to become Associate Professor at Politecnico di Torino.  Alberto gave a nice seminar entitled "understanding and Estimating River Flood Hazards across Timescales" which covers some issues on flooding forecasting and variability in space-time.

I really appreciate his point of view which is different from mine and from which I can learn a lot, therefore I  asked him the permission to publish his presentation in my blog here. Please by clicking on the Figure above, find Alberto's presentation.

References

Archfield, S. A., R. M. Hirsch, A. Viglione, and G. Blöschl (2016), Fragmented patterns of flood change across the United States, Geophysical Research Letters, doi:10.1002/2016GL070590.

Barendrecht, M.H., A. Viglione and G. Blöschl (2017) A dynamic framework for
flood risk, Water Security, 1, 3-11, doi:10.1016/j.wasec.2017.02.001.

Blöschl, G., M. Sivapalan, T. Wagener, A. Viglione, and H. H. Savenije (2013b), Runoff Prediction in Ungauged Basins - Synthesis across Processes, Places and Scales, 484 pp., Cambridge University Press, ISBN:9781107028180.

Blöschl, G., et al. (2017), Changing climate shifts timing of European
floods, Science, 357 (6351), 588–590, doi:10.1126/science.aan2506.

Di Baldassarre, G., A. Viglione, G. Carr, L. Kuil, J. L. Salinas, and G. Blöschl (2013), Socio-hydrology: conceptualising
human-flood interactions, Hydrology and Earth System Sciences, 17, 3295–3303, doi:10.5194/hess-17-3295-2013.

Merz, R., and G. Blöschl (2008), Flood frequency hydrology: 1. Temporal, spatial, and causal expansion of information, Water Resources Research, 44, W08432, doi:10.1029/2007WR006744.

Salinas, J. L., A. Kiss, A. Viglione, R. Viertl, and G. Blöschl (2016), A fuzzy Bayesian approach to
flood frequency estimation with imprecise historical information, Water Resources Research, 52 (9), 6730–6750, doi:10.1002/2016WR019177.

Viglione, A., R. Merz, and G. Blöschl (2009), On the role of the runoff coefficient in the mapping of rainfall to flood  return periods, Hydrology and Earth System Sciences, 13 (5), 577–593, doi:10.5194/hess-13-577-2009.

Viglione, A., G. B. Chirico, R. A. Woods, and G. Blöschl (2010a), Generalised synthesis of space-time variability in flood response: An analytical framework, Journal of Hydrology, 394, 198–212, doi:10.1016/j.jhydrol.2010.05.047.

Viglione, A., G. B. Chirico, J. Komma, R. A. Woods, M. Borga, and G. Blöschl (2010b), Quantifying space-time dynamics of flood event types, Journal of Hydrology, 394, 213–229, doi:10.1016/j.jhydrol.2010.05.041.

Viglione, A., R. Merz, J. L. Salinas, and G. Blöschl (2013), Flood frequency hydrology: 3. A Bayesian analysis, Water Resources Research, 49 (2), 675–692, doi:10.1029/2011WR010782.

Viglione, A., G. D. Baldassarre, L. Brandimarte, L. Kuil, G. Carr, J. L. Salinas, A. Scolobig, and G. Blöschl (2014), Insights from socio-hydrology modelling on dealing with flood risk – Roles of collective memory, risk-taking attitude and trust, Journal of Hydrology, 518 (A), 71–82, doi:10.1016/j.jhydrol.2014.01.018.

Viglione, A. and M. Rogger (2015) Flood processes and hazards, Chapter 1.1 in: J.F. Schroder, P. Paron and G. Di Baldassarre (Eds.) HydroMeteorological Hazards and Disasters, Elsevier, pp. 3-33, doi:10.1016/
B978-0-12-394846-5.00001-1.

Viglione, A., B. Merz, N. V. Dung, J. Parajka, T. Nester, and G. Blöschl (2016), Attribution of regional flood changes based on scaling fingerprints, Water Resources Research, 52 (7), 5322–5340, doi:10.1002/2016WR019036

Concentrations

Some notes on concentration of solutes in Hydrological Dynamical Systems  (a.k.a. lumped, a.k.a. compartmental ) Hydrological models.

The slides can be seen by clicking on the Figure above.  These slides pretend to be be quite intuitive in content but, obviously to people  who have some familiarity with the topic treated. A good lecture for  their introduction is Rigon et al. 2016. For understanding the graphs the blog posts entitled “Reservoirology”, especially Reservoirology #3 are also a preparatory lecture.

Java 11 trap

As you know we use Java (R and python).  Version 11 is a major release because it has long-term support (LTS). But it comes with some traps. Here below, please find  "verbatim" a post from Stephen Colebourne's blog which inform about some problems that Oracles's new licensing strategy can cause. (From the same blog, also interesting the suggestions on how to move from Java 8 to Java 11).

" ... It is a But Oracle have also set it up to be a trap (either deliberately or accidentally).

The trap

For 23 years, developers have downloaded the JDK from Oracle and used it for $free. Type "JDK" into your favourite search engine, and the top link will be to an Oracle Java SE download page (I'm deliberately not providing a link). But that search and that link is now a trap.
Oracle JDK, the one all web searches take you to, is now commercial not $free.
The key part of the terms is as follows:

You may not: use the Programs for any data processing or any commercial, production, or internal business purposes other than developing, testing, prototyping, and demonstrating your Application;

The trap is as follows:

• Download Oracle JDK (because that is what you've always done, and it is what the web-search tells you). 
• Use it in production (because you didn't realise the license changed)
• Get a nasty phone call from Oracle's license enforcement teams demanding lots of money

In other words, Oracle can rely on inertia from Java developers to cause them to download the wrong (commercial) release of Java. Unless you read the text/warnings/legalese very carefully you might not even realise Oracle JDK is now commercial, and that you are therefore liable to pay Oracle for Java.
Is this trap malicious behaviour on the part of Oracle? Readers will have their own opinions. I do suggest bearing in mind that Oracle invests huge amounts in developing Java, so it is reasonable to have a commercial plan available for those that want it. And they do provide a $free alternative...
The solution

The solution is simple!

Use an OpenJDK build.
As well as their commercial JDK, Oracle produce an OpenJDK build. It is free, zero-cot and GPL licensed (with Classpath exception, so safe for commercial use).

Download $free Java here.

Right now, the OpenJDK build from Oracle is the only one available. But more will be available soon such as from AdoptOpenJDK (probably later this week). And the OpenJDK build from groups other than Oracle will the the ones to use for Java 11 longer-term support. See Couleborne's post on zero-cost Java and the wide variety of OpenJDK builds for more details.


Summary

Do NOT download or use the Oracle JDK unless you intend to pay for it.
For Java 11, download and use an OpenJDK build, currently from here. (No comments on this post. There are plenty of other places to express opinions.)"

Where I was during the last weeks

I already wrote that I am participating to the competition for becoming Head of my Department (with roles similar to a Dean, however).  My short presentation I gave last July can be found here and recently I was so busy to talk to people and write my program.

I think I did a good work, and the results (in Italian) can be found here.  Elections are tomorrow, a have a good competitor,  and I do not know what I will achieve. However, this has to be done; I did at my best and voters are always right.  As masters I had Ignacio Rodriguez-Iturbe (GS) and Andrea Rinaldo (GS). Both of them were, besides outstanding researchers, very good Department Heads in various important Universities.  If I will be elected, I will use the treasure of having seen them in action.

News: I’ve lost! Quite clearly. Nevertheless I believe that my profram remains a good example of a possible management of a Department.

GEOtop goes towards a new major version

GEOtop 3.* is going to be different from what I envisioned. It is not a mystery that I quite abandoned its development for looking to a new platform based on GEOframe. However GEOtop remains an innovative piece of code. No other hydrological model simulates so many processes together. Despite there are good models for integrating Richards equation, there are few that also treat freezing soil and snow properly Viceversa there are, maybe, better models for snow but they usually do not deal with  snow and freezing soil.

No one (to my knowledge) fully solves the energy budget. And so on. GEOtop has a small but active group of users and developers. Stefano Cozzini (GS) and Giacomo Bertoldi (GS) are now leading this group. Here you will find the talk and the poster the group gave at the XXVI Italian Meeting of Hydraulic Constructions and Hydraulics.

Winter School on the GEOframe system

The course for doctoral students, post docs and young researchers in Hydrology, Forestry, and related disciplines will cover the simulation of the hydrological cycle of catchments of various sizes with the GEOframe system. To know about GEOframe and GEOframe-NewAGE, please refer to here.

They say that all models are wrong but useful. However, with better tools you forecast and decide better.

The course will enroll at most thirty students and will be held at the Department of Civil, Environmental and Mechanical Engineering of Trento from January 8 to January 18 included.
The course will be of totally 68 hours (8 a day) of which 34 (4 each day) will be dedicated to laboratory and personal work under the supervision of tutors. The course includes as option to get an exam certification, upon the completion of an exercise, to have doctoral credits.

Subscription at: https://webmagazine.unitn.it/en/evento/dicam/44808/geoframe-newage-winter-school

Instructors

• Riccardo Rigon
• Michele Bottazzi
• Niccolò Tubini

with material prepared by

• Giovanna Dalpiaz
• Marialaura Bancheri

The  topics treated has been:

• Tuesday, January 8: Introduction to the course - What is OMS - What is GEOframe - Using Python and Jupyter for visualising Inputs and Outputs 
• Wednesday, January 9- Catchments and Hydrologic Response Units delineation 
• Thursday and Friday, January, 10 - 11 - Treatment of spatial data and Calibration  
• Monday January 14 - Estimation of radiation components 
• Tuesday, January 15 - Evaporation and Transpiration 
• Wednesday, January 16 - Rainfall-Runoff modelling with various NewAGE components 
• Thursday, January 17 - Rainfall-Runoff modelling with various NewAGE components 
• Friday, January 18 - Opinions on hydrological modelling. Exercises on assignments.
• Epilogue and thanks

Go to the first day

Why choosing GEOframe over other models/platforms ? I would say for:

• Flexibility: GEOframe is not a model but a system of components that interact at run-time. You can chose among various components options for any of the processes.
• Expandability: If you like to program, with a little investment in Java you can write your own component and make them to interact with the others without having to reinvent the wheel.
• Parallelism. Components work in parallel when their tasks do not interact, but this is transparent for you (we call it implicit parallelism). 
• Spatial discretisation. A catchment is subdivided in parts (HRU) which can be modeled separately and are computed in parallel. The network structure is used to achieve the spatial parallelism. Its spatial modularity can be used to add/cut part of the basins without having to redo the spatial analysis, for doing multisite calibration, to progress the analysis of a larger basin in parts that are assembled together eventually.
• Beyond-state-of art components.  Besides traditional approach to processes, we implemented a few new ideas for all the processes we covered.
• Reliability.  GEOframe is currently used for the flood forecasting in real time by Regione Basilicata. It is not just a system for research that does not work in real cases. 
• Tracers studies. Not treated in the school are present tools for doing tracers studies,
• Process based modelling.  Not treated in the school, we have tools for integrating Richards equation in 1D, and we are developing tools for integrating it in 2d and 3d coupling it with the energy budget. These components will be able to interact with the other. We also started new developments on freezing soil and snow modelling.

The cost of the course for early subscribers is 270 Euros which includes lunch and parsimonious coffee-breaks  Member of SII, The Italian hydrological Society have a discount of 20 Euros. Cost of late subscribers (after November 15, 2018) is 370 Euros. 

After November 15 some work will be required to participant in order to setup their tools for running GEOframe. Installation of Java (version 8), installation of the Object Modelling System console, Installation of Python and Python notebooks, testing the use of some file formats. After the accomplishment of the requirements, students will be allow to bring their own study cases at the School.

Who wants to have early information or clarifications can write to me: riccardo.rigon at unit.it. Subscription page at:

https://webmagazine.unitn.it/en/evento/dicam/44808/geoframe-newage-winter-school

Looking for financial support of a valuable research (to whom it may concern)

It happens that I browsed the presentation reporting about your flood initiative and asking for collaboration. I am a professor in hydrology and I am interested. 

During the last years, in collaboration with Dr Olaf David of Colorado State University, we worked to implement an infrastructure, called GEOframe upon the framework offered by the Object Modelling System (OMS) and the Cloud Service Integration platform ( CSIP) platform. GEOframe tools are  devoted to forecast floods, droughts, snow cover and any component of the hydrological cycle in a flexible, maintainable, expandable platform.

My efforts on flood forecasting (and this link too) brought to both theoretical and applicative achievements. The latter came through the implementation, since more than a decade ago of the various systems used by the Province of Trento for its purposes, and recently to the implementation of a system for the Civil protection of Regione Basilicata. 

Academic achievements include works on small catchments and large catchments with various uses of remote sensed data. My former student Giuseppe Formetta (GS), now working at CEH at Wallingfort, (actually on-leave at JRC for this year) is applying an early version of our system to the Khrisna river in India. 

The current system, using the analogy of the river network, schematises a catchments as a graph and implements such informatics that allows the run in parallel of subcatchments, separate calibration in preparatory phases and their assemblage a-posteriori  (Net3). 

My Ph.D. student, Francesco Serafin (graduating in Spring 2019), besides Net3, enhanced the OMS console by allowing binding OMS compliant components (written in Java, C++ or FORTRAN) with Python and R codes, getting, at the end, a very flexible system.

Therefore, I think I can be useful for the project and I would very appreciate if we can establish some collaboration. 

The Horton Machine (formerly known also as JGrasstools) is back

The Horton Machine, the set of tools I developed several years ago with Hydrologis, is now back and well documented in gvSIG. All the merit goes to Andrea Antonello and Silvia Franceschi, the "Hydrologissers'.  I could not support new work on GIS (after my decade of involvement) but they were so kind to make me co-author of their presentations. During the last FOSS4G in Guimãraes they presented what they ported into  gvSIG and prepared some illustrative material that we share with you.

You can find:

• The Introduction/presentation given in Guimãraes
• The material of the workshop given by Silvia and Andrea
• A quick introduction for the installation of the Horton Machine plugin
• The Tutorial
• The DEM the can be used as example

If you need more information about gvSIG, please go to its site or its YouTube channel (learning gvSIG in 30 minutes, here).

Categories of systems of equations in ODEs based hydrological systems

This is the continuation of the reservoirology topic saga.  Especially preparatory for understanding  is the Reservoirology #3 post. Here we show that the same topological structure of Petri Nets can address various physical concepts related to a hydrological system conceptualised as a set of reservoir and therefore solvable as a set of ODEs.
The main ODE system, however, does not reveal all of the system. A finer inspection can be obtained by investigating travel times and concentration of tracers (being the theoretical point of view, the first, the experimental one, the second).

To fix some concept we wrote the short presentations above. Or just click here. The pdf contains itself links to other posts and literature.

Anke Hildebrandt's recent research

If you are interested in the hydraulics of plants, you should give a look to the recent production of Anke Hildebrandt. Her production encounter my favor since she is able to put together experimental work and theoretical work on the thermodynamics of trees. A topic on which recently I became interested in. Relevant among her production is the work on hess: A thermodynamic fomulation of root water uptake which was awarded as one of the best paper of 2017 in HESS. No more comments. Just papers to read below. 

Recent papers

Van Stan, J. T., Norman, Z., Meghoo, A., Friesen, J., Hildebrandt, A., Côté, J.-F., et al. (2017). Edge-to-Stem Variability in Wet-Canopy Evaporation From an Urban Tree Row. Boundary-Layer Meteorology, 165(2), 295–310. http://doi.org/10.1007/s10546-017-0277-7

Guderle, M., Bachmann, D., Milcu, A., Gockele, A., Bechmann, M., Fischer, C., et al. (2017). Dynamic niche partitioning in root water uptake facilitates efficient water use in more diverse grassland plant communities. Functional Ecology, 32(1), 214–227. http://doi.org/10.1111/1365-2435.12948

Metzger, J. C., Wutzler, T., Dalla Valle, N., Filipzik, J., Grauer, C., Lehmann, R., et al. (2017). Vegetation impacts soil water content patterns by shaping canopy water fluxes and soil properties. Hydrological Processes, 31(22), 3783–3795. http://doi.org/10.1002/hyp.11274

Weisser, W.W., C. Roscher, S. Meyer, A. Ebeling, G. Luo, E. Allan, H. Beßler, R. Barnard, N. Buchmann, F. Buscot, C. Engels, C. Fischer, M. Fischer, A. Gessler, G. Gleixner, S. Halle, A. Hildebrandt, H. Hillebrand, H. de Kroon, M. Lange, S. Leimer, X. Le Roux, A. Milcu, L. Mommer, P. Niklaus, Y. Oelmann, R. Proulx, C. Scherber, M. Scherer-Lorenzen, S. Scheu, T. Tscharntke, M. Wachendorf, C. Wagg, A. Weigelt, W. Wilcke, E.-D. Schulze, B. Schmid, N. Eisenhauer. Biodiversity effects on ecosystem functioning in a 14-year grassland experiment: patterns, mechanisms, and open questions. Basic and Applied Ecology, doi: 10.1016/j.baae.2017.06.002. (link)

Zimmermann, A., Voss, S., Metzger, J. C., Hildebrandt, A., & Zimmermann, B. (2016). Capturing heterogeneity: The role of a study area’s extent for estimating mean throughfall. Journal of Hydrology, 542(C), 781–789. http://doi.org/10.1016/j.jhydrol.2016.09.047

Guderle, M., D. Bachmann, A. Milcu, A. Gockele, M. Bechmann, C. Fischer, C. Roscher, D. Landais, O. Ravel, S. Devidal, J. Roy, A. Gessler, N. Buchmann, A. Hildebrandt. Dynamic niche partitioning in root water uptake facilitates efficient water use in more diverse plant communities. Functional Ecology, doi: 10.1111/1365-2435.12948. (link)

Metzger, J. C., N. Dalla Valle, T. Wutzler, J. Filipzik, R. Lehmann, M. Roggenbuck, D. Schelhorn, J. Weckmüller, K. Küsel, K. U. Totsche, S. Trumbore, A. Hildebrandt. Tracing spatial variation of canopy water fluxes to the soil with high resolution data. Hydrological Processes. doi: 10.1002/hyp.11274 (link)

Arnold, S., Attinger, S., Frank, K., Hildebrandt, A. 2016. Assessing the structural adequacy of alternative ecohydrological models using a pattern-oriented approach. Ecological Modelling 316: 52-61. doi: doi:10.1016/j.ecolmodel.2015.08.003. (link)

Hildebrandt, A. A. Kleidon and M. Bechmann. A thermodynamic fomulation of root water uptake. Hydrology and Earth System Sciences. 20: 3441-3454, doi: 10.51947hess-20-3441-2016. (link)

Milcu, A., W. Eugster, D. Bachmann, M. Guderle, Ch. Roscher, D. Landais, O. Ravel, A. Gessler, M. Lange, A. Ebeling, W. Weisser, J. Roy, A. Hildebrandt, N. Buchmann. 2015. Plant species and functional diversity increase grassland productivity-related water vapour fluxes: a combined Ecotron and modeling approach. Ecology 97(8): 2044-2054. doi: 10.1890/15-1110.1 (link)


Renner, M., S. K. Hassler, T. Blume, M. Weiler, A. Hildebrandt, M. Guderle, S. J. Schymanski, and A. Kleidon. Dominant controls of transpiration along a hillslope transect inferred from ecohydrological measurements and thermodynamic limits, Hydrology and Earth System Sciences 20: 2063-2083. doi: 10.5194/hess-20-2063-2016. (link)

Fischer, C., Tischer, J., Roscher, C., Eisenhauer, N., Ravenek, J. M., Gleixner, G., Attinger, S., Jensen, B., de Kroon, H., Mommer, L., Scheu, S., Hildebrandt, A. 2015. Plant species diversity affects infiltration capacity in an experimental grassland through changes in soil properties. Plant and Soil. 397(1): 1-16, doi: 10.1007/s11104-014-2373-5 (link)

Guderle M. and Hildebrandt A. 2015. Using measured soil water contents to estimate evapotranspiration and root water uptake profiles - a comparative study. Hydrology and Earth System Sciences. 19: 409-425. doi: 10.5194/hess-19-409-2015 (link)

Bechmann, M., C. Schneider, A. Carminati, D. Vetterlein, S. Attinger, A. Hildebrandt. 2014. Parameterizing complex root water uptake models - the arrangement of root hydraulic properties within the root architecture affects dynamics and efficiency of root water uptake. Hydrology and Earth System Sciences. 18:4189-4206. doi: 10.5194/hess-18-4189-2014 (link)

Fischer, C., C. Roscher, Jensen, N. Eisenhauer, J. Baade, S. Attinger, S. Scheu, W.W. Weisser, A. Hildebrandt. 2014. How do earthworm, soil texture and plant composition affect infiltration in managed grasslands along a plant diversity gradient? PLoS ONE9(6): e98987. doi:10.1371/journal.pone.0098987. (link)

Leimer, S., Kreutziger, Y., Rosenkranz, S., Beßler, H., Hildebrandt, A., Oelmann, Y., Weisser, W., Wirth, C. Wilcke, W., 2014. Plant diversity effects on the water balance of an experimental grassland. Ecohydrology, doi: 10.1002/eco.1464. (link)

Carminati, A., C. L. Schneider, A. B. Moradi, M. Zarebanadkouki, D. Vetterlein, H.-J. Vogel, A. Hildebrandt, U. Weller, L. Schüler, and S. E. Oswald. 2011. How the Rhizosphere May Favor Water Availability to Roots. Vadose Zone Journal 10:988. doi: 10.2136/vzj2010.0113. (link)

Kalbacher, T., C. L. Schneider, W. Wang, A. Hildebrandt, S. Attinger, and O. Kolditz. 2011. Modeling Soil-Coupled Water Uptake of Multiple Root Systems with Automatic Time Stepping. Vadose Zone Journal 10:727. doi: 10.2136/vzj2010.0099. (link)

Alexandrov, G. a., D. Ames, G. Bellocchi, M. Bruen, N. Crout, M. Erechtchoukova, A. Hildebrandt, F. Hoffman, C. Jackisch, and P. Khaiter. 2010. Technical assessment and evaluation of environmental models and software: Letter to the Editor. Environmental Modelling & Software 26:328-336. doi: 10.1016/j.envsoft.2010.08.004. (link)

If I was the head of my Department

I manifested to my colleague my availability to be voted for Department Head. Not an easy task indeed, and a challenge if I want to continue to do some research. However, this is the right time for doing it. In six years I will be too old to plan to for ten years in advance and the baton has to be given to others, while I will give "good advise, if I will not be able to give bad examples".
Today we are asked, the so far three candidates, to give our vision for our mandate in front  our peers and electors.

Below the above figure, you find my talk (in Italian). Have to say my opponents are good guys, and I have not problems to give up if I recognize if their will and view is better than mine.  At the end, I think one of the quality of good academics is to recognize where excellence stands.  My complete program can be found here. 

Towards a new SWMM, JSWMM

You want to design a storm water management system. What you usually get is a rainfall-runoff model. In this specific subfield, the model is SWMM. EPA SWMM contains many features that were implemented to simulate urban storm water depletion network. Meaning that there are specific model's parameters set for that, and that a community gathered around this tool.

However SWMM is not a system for designing sewers. Designing requires that you repeat the modeling actions several times. At the outlet of any pipe, you have to:

• estimate the runoff under a "design rainfall" coming from some intensity-duration-frequency curves.
• get the maximum discharge with an assigned return period (say 10 years)
• Use simplified hydraulics for obtaining the size of the pipe apt to contain the maximum discharge
• repeat the operation for the pipes downhill, without leaving out uphill branches.

Operation 2 above requires a search algorithm to find the rainfall duration that is responsible for the maximum discharge. The complete theory is in Rigon et al., 2011.

The point is that SWMM does not do the sequence of operation above. This is one of the reasons we implemented JSWMM. To see what it does, click on the figure.

News: A ne presentation of the work was given at the 2020 iEMSs biennial Conference and can be found here

On complex networks computation of mountain catchments

From June 12 to June 14 2018 in Trento there is the 5th European IAHR conference. I am convening (and also presenting a contribution) which derives from our modelling within the GEOframe system.

The scope of the presentation is to inform about functionalities of GEOframe and something of what it is hidden under the hood. Clicking on the figure above, please access the pdf of the presentation.

Snow related Ph.D. position (the Stradivari snow project)

The Stradivari project research aims to build better tools for analysing the processes of the hydrological cycle. The project is more focused on the tools (building the violin) but it does not forget the music that has to be played. It is conceived to account for hydrological processes interactions and feedbacks, and develop new mathematics (equations) for their description. Use of appropriate contemporary numerics is also part of the project. The overall project builds on the foundations given by the GEOtop (http://abouthydrology.blogspot.com/2015/02/geotop-essentials.html) model and the GEOframe-NewAGE (http://geoframe.blogspot.com/) infrastructure. 

It is time to move on the generation of snow models. The computation abilities and the physics of snow are now much better known than thirty years ago. However, most of the snow model are based on parameterisations which should be updated. We start from the experience made in GEOtop (versions 0.*,1.* and 2.*) which is capable of reliable snow height, temperature and densities, at operational level over all the Alps, but we look also to the experiences made by CROCUS, SnowPack and Alpine 3D. In particular GEOtop use highlighted various issues that we plan to overcome with a new version of the model, which requires both deepening the thermodynamics of snow and its numerical implementation.

In synthesis we identify the following aspects to be improved:

• Actual GEOtop snow model is 1-d. It can improved including the vapor phase movements explicitly. Besides, we can add modules to have a better account for: density, viscosity and, adding the vapor phase, type of grain. 
• The physics. Analysing the thermodynamics to keep out of the future formulation empirical parameterisations of the processes which revealed obsolete; 
• Adding Richards equation for water percolation; 
• Analysing the separation rainfall, snowfall through a more physical modelling than actual; 
• the description of deposition on canopies, and subsequent effects of vegetation on sublimation; the effect of slope and topography characteristics on snow deposition and sublimation 
• blowing snow in complex terrain 
• assimilation of hydro-meteorological data and calibration 
• integration with remote sensing data 

This research will be implemented in strict collaboration with MobyGIS, EURAC Research (Giacomo Bertoldi, Ph.D), Stephan Gruber,  Professor at Carleton University and with Niccolò Tubini, the Ph.D. student who is actually developing new theory codes for Richards equation and freezing soil. Collaborative and unselfish spirit is required in this research group.
The project has also some practical outcomes that are related to:

• the avalanche triggering 
• the water availability due to snow and glaciers melting (both in the short and long terms) 
• the hydroelectric production 

All the code developed will be done in Github (or similar platform), inside the GEOframe community and will be Open Source according to the GPL v3 license.

The candidate will take care of implementing, besides the code, the appropriate procedures for continuous integration of the evolving source code, and s/he will be also asked to maintain a regular rate of commits to the common open platform. Despite these conditions, and being free and open source, the code will be intellectual property by the coder. This will be guaranteed also by the components-based infrastructure offered by OMS3, which allows to better define the contributions of anyone.
The implementation part will be followed, accompanied by testing activities, either for mathematical consistency, than for physical consistency with experiments and field measurements.
The Ph.D. student is intended to produce, besides working and tested codes, also at least three papers in major journals (VQR Class A), of which, at least one as first Author. Duration of the doctoral studies could be three or four years.

This project can enter either the curriculum C (Environmental Engineering) or the curriculum A (Modelling and Simulation) of our doctoral school.

Further information of the policies of the research group can be found:

• To my students
• Working with us
• The scaring version of “working with us"
• About doing a Ph.D.
• Why Open Source ?
• Going beyond the present state of Art Of Hydrological Modeling (since 2011 but still valid)

Evaporation and Transpiration

This contains the video related to my interpretation of evapotranspiration.

The thermodynamics of evaporation

Definitions

Transport of vapor in the atmospheric boundary layer

Evaporation from soils

Transpiration

The energy and mass budget 

The Penman-Monteith approach