Open Java resources for Hydrologists

Next question about Java, after having introduced it, and talked about options for doing numerics, is to see if there are effective resources on which hydrologists can build something useful, incrementally, and as a part of a community.

• The first recource that comes to my mind, is obviously the Object Modelling System version 3 (or OMS3). This is the framework we use for our modelling. We wrote some models and papers using it, and OMS3 is also at the core of the uDig Spatial Toolbox

Also OpenMI uses Java, but, at the moment, Java there is just the second choice, well behind their C#. A second framework which we are looking to is:

• Openda. It is a framework for data assimilation and calibration of models promoted by Deltares. 

There are out there probably other Java frameworks, and I will add them to the list, whenever they will be brought to my attention.
The field of more intense Java activity is actually the GIS one. There are at least three/four major open source efforts in Java:

• uDig, the GIS of our choice, with its field companion BeeGIS, and its mobile younger brother, Geopaparazzi
• GvSig, a good product indeed, which also has a companion for mobiles
• OpenJump, on which the Italian cartographic project is based

In fact, it is not very clear to me, but also the

• North52 

initiative seems having parts based on Java.

They are based on various type of resources among those in Java:

• GeoTools
• Sextante, a library for data geoprocessing by Victor Olaya

Both GeoTools and Sextante use 

• JTS, the Java topology suite for several elementary operations.

An interesting,  old but not outdated initiative to give a third dimension to data visualisation is 

• Visad, a library to visualise scientific datasets

from which sprout

• the IDV  virtual globes family to visualize 3D geophysical data

IDV is not the unique Java Virtual globe. Another one is:

• the Nasa WorldWind which gives (as open source) what Google Earth gives to the general public. Various prototype implementations of Nasa WorldWind in uDig (and in OMS3) were already made, and possibly sooner or later we will have a full mainstream implementation of it.

Last but not least an important resources is 

• the NetCDF data format (of which IDV, but also ncBrowse,  is a viewer) has also an implementation in Java.

Even if my post is more concerned about the development of Hydrological resources and models, the above frameworks/programs/applications/models or part of them can just be used (and just not developed) with profit.  Starting from OMS3 there are at least three major modelling efforts that use it:

• PRMS
• AgES
• JGrass-NewAGE

Not to forget is the whole Jgrasstools available for DEM manipulation (and further modelling) inside (and outside) uDig. All of them have more resources in a post dedicated to OMS 3 resources.
The Sextante library itself comes with more than 300 tools for manipulation of DEM and images. Also IDV and NasaWorldWind can be just used for the purposes of any research.
Another full-fledged Java Catchment model is

• Cuencas

by Ricardo Mantilla. Its source code can be obtained following the instructions in the user manual.

A recent addition to the resources is

• H2O, an engine for doing Machine Learning and Data Science

Why use Java, instead of  R ?

In one of my previous post, I talked about R software, and in fact, my group of people use both.  R is much more for some higher level operations, and is much less customisable than the resources developed in Java, and,  usually, less efficient.  My former students, using Java capabilities were able to built professional/industrial applications on the basis of Java (the GIS are a proof), that using just R would have been impossible.
However, the new incoming version of OMS3, should include the way to call R from the OMS console. Please contact the OMS version 3 developers for more information.

Why do not use Python ?

Python (or many other languages) could have been a good choice. Never say never, however I chose Java and to be successful one has to consistently invest his/her own limited resources in one direction.

There exist material/documentation on the above material ?

One introduction to programming GIS and essentially all I would require to a collaborator to know  can be found here by Andrea Antonello

A Java Library (of books) for beginners, and a little path to self instruction for geophysicists

It is quite a few years that I insist for students moving to program in Java. Java is not anymore Java 1, it has evolved towards a mature language and now it is at its version 7 (Java 8, now and Java 9 soon) Java  13. Times ago I talked about doing matrix algebra in Java. Most of that post is still valid.

However, many would realize that for learning Java the right way s/he has to start from the beginning, and possibly, start to learn also the concepts of object oriented programming (the real reason to use Java).

My personal training began with the videos (in Italian) of one of my Trento university colleague Marco Ronchetti. For others, MIT Courseware could be a good starting, but also these less academical clips can help.

Programming is more than just knowing a language. This concept is well covered in

Introduction to programming using Java by David J.Eck
Java Precisely by P. Sestoft (suggested by Joshua Bloch, see below)

and nothing can substitute the reading of a good book. More specialised classics on Java are also:

Thinking in Java 4th edition

Thinking in Java annotated solutions book

the two by Bruce Eckel

The design pattern Java companion, by J. W. Cooper

These books sum up to a lot of pages to read and working them out is not a thing you do in a day.
People who teach Java, usually rely on command line. However, I would prefer to learn how to use an IDE. Our choice is, since many years, Eclipse. Learning to use it is easy but is made easier by reading Lars Vogel tutorials and material.

Obviously Oracle itself has a series of very good tutorials.

Going to more specialised book, for scientists, I also recommend

Mak, R. (2010). Prentice Hall - Java Number Cruncher. The Java Programmer's Guide to Numerical Computing (pp. 1–493).

Besset, D.H., Object-Oriented Implementation of Numerical Methods: An Introduction with Java & Smalltalk , 2000. (~70 Mb) (His code can be found here).

Nikishkov, G. (2010). Programming Finite Elements in Java (pp. 1–394). The preface of his book can give a good rational for using Java instead of other languages.

A general collection of Algorithms in Java is Algorithms by Kevin Wayne and Robert Sedgewick (also addressed here).

You can also give a look to old C Numerical Recipes (for algorithms) and use this free Java project for their application. (Do not support NR license policies).

Having got a clear idea of what Java is, you probably wants to write clean code. The right book is Clean Code by Robert Martin.

Now, you are probably ready to understand the subtleties of the Java Matrix Libraries I indicated in my older post.

If one wants to move to geosciences and GIS, the best choice is to give a look to the GEOtools site and to al the documentation s/he can find there. At that point s/he will be probably also ready for programming the Jgrasstools.  Programming OMS3 components would be easier, indeed, if you do not care of geographic features.  For GIS and working with us, one should consider the introduction by Andrea Antonello.

Eventually you can probably go back and make reacher your library (of books) by also reading and applying

Martin Fowler's UML Distilled: A Brief Guide to the Standard Object Modeling Language

Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides: Design Patterns: Elements of Reusable Object-Oriented Software (but the easiest book, from the opinion of many is Head First Design Patterns by Eric Freeman and Elisabeth Robson.

and  the book that many consider the best one:

Effective Java Second Edition, by Joshua Bloch  by many considered the best book about Java.
Here it is the third Edition ;-).
Here there is also a video about.

For mastering Java generics and collections, I suggest:

Java Generics and collections by Maurcie Naftalin & Philip Wadler, by O'Reilly.

Obviously Java is one of the most used languages, and there is a lot of material on the web. Starting from the Oracles itself main site.

However, programming is about writing code, not just reading about it. Wishing I would have time to do it! 

The Boussinesq equation paper

Finally, after a long story (see the submission post) the paper on Boussinesq equation integration was accepted by Water Resources Research.  In the revision phase it was asked to add further comparison of the model with a model solving Richards equations.  We chose, for our comparison, and, a-posteriori , the choice was quite obvious, the model GEOtop.  Emanuele Cordano and I believe that the new added section could be quite interesting for understanding some characteristics of infiltration at hillslope scale.

To read the preprint, please follows the link. The code and documentation is available from the Boussinesq site. However, a more recent version, in Java and OMS compliant is also available at Github, after the joint work of Francesco Serafin (here) and Giuseppe Formetta.

Direct Solar Radiation Models by Formetta et al. 2012-2013

This paper presents two new modelling components based on the Object Modelling System v3 for the calculation of the shortwave incident radiation on complex topography settings, and the implementation of several ancillary tools. To understand it, it can be useful to give a look to the previous post on radiation of few weeks ago. The first component, NewAGE-SwRB, accounts for slope, aspect, shadow and the topographical information of the sites, and use suitable parametrisation for obtaining the cloudless irradiance. A second component, NewAGE-DEC-MOD's is implemented to estimate the irradiance reduction due to the presence of clouds, according to three parameterisations. To obtain a working modelling composition, suitable to be compared with ground data at measurement stations, the two components are connected to a Kriging component, and, with the use of a further component NewAGE-V (verification package), the performance of modeled is quantitatively evaluated. The two components (and the various parametrisations they contain) are tested using the data from three basins catchments, and some simple verification test is made to assess the goodness of the methods used. The components are part of a larger system, JGrass-NewAGE, their input and outputs are given as geometrical objects immediately visualisable in a GIS (for instance the companion uDig), and can be used seamlessly with the various modelling solutions available in JGrass-NewAGE for the estimation of long wave radiation, evapotranspiration, and snow melting, as well as stand-alone components to just estimate shortwave radiation for various uses. The modularity of the approach is shown to be extensible to more accurate physical-statistical studies aimed to assess in deep the components performances and to extend spatially their results, without the necessity of recoding any part of the component but just making use of connective scripts.

This is, obviously not the first effort in such direction, and some other good softwares were produced as 
SolarFlux (in ArcInfo GIS) (Dubayah and Paul, 1995; Hetrick et al., 1993), Solar Analyst (Fu and Rich, 2000), SRAD (by Moore, 1992, and documented in Wilson and Gallant, 2000- but you can see this), Solei (Miklanek, 1993 - see this) or r.sun (Hofierka and Suri, 2002), and often integrate the models in GIS. 
Our modelling, making treasure of these previous efforts, is also in line with those tools that try to respond to the increase demand of modularity and interchangeability in hydrological and biophysical models and have been developed in the last decades.

Executable, Data, and Documentation of the system can be found following this link (not yet operational - give me a little time for preparing it, with the objective that everyone can reproduce the results of the paper). Source code is partially available through the jgrasstool page. 

From Yesterday Just Open Data from the Italian Administrations ?

Yesterday Mr. Monti government Agenda Digitale decree-Law was transformed in state law. Hopefully this would make open all the environmental data produced by public administrations.
Let's see, and hope.

The text of the law can be found here.

Solar Radiation Physics and Geometry for hydrologists

I started to realize that a hydrologist has to learn about radiation, since the first moment I put my sight a little beyond rainfall-runoff moedelling, at least fifteen years ago. This was in building the first bricks of GEOtop: the first problem we faced was to avoid give radiation to point in shadows, i.e. calculating shadows. At the time this was not very much documented, even if the papers by Ralph Dubayah, Jeff Dozier, and others were already out since a few years. I started with giving a look to IPW and I tried to understand what and how it works, but eventually I finish to try my own modules.  To summarise some of the topic,  I renovated my slides on slideshare, which now better cover the notation, and can be considered as a preliminary reading before going to the Formetta et al. 2012 paper (see it on GMDD).

Radiation  treated in the slides mostly regards: geometry (where we finally followed the work of Javier Corripio), shortwave radiation treatment, and longwave radiation.

If, from many points of view working  with a real model of atmospheric absorption would be more exciting, hydrologists use mostly parameterizations, and that is what I mostly summarised in my slides together with a little of astronomical geometry.

Anyway, I found really useful the papers you see listed below.

References

(Mostly about short-wave radiation)

Bird, R. and Hulstrom, R.: Simplified clear sky model for direct and diffuse insolation on horizontal surfaces, Tech. rep., Solar Energy Research Inst., Golden, CO (USA), 1981.

Bird, R. E.,  Riordan, C, 1986: Simple Solar Spectral Model for Direct and Diffuse Irradiance on Horizontal and Tilted Planes at the Earth's Surface for Cloudless Atmospheres. J. Climate Appl. Meteor., 25, 87–97. doi: http://dx.doi.org/10.1175/1520-0450(1986)025<0087:SSSMFD>2.0.CO;2

Boland, J., Scott, L., and Luther, M.: Modelling the diffuse fraction of global solar radiation on a hori- zontal surface, Environmetrics, 12, 103–116, 2001.

Corripio, J.: Modelling the energy balance of high altitude glacierised basins in the Central Andes., PhD dissertation, University of Edinburgh, 2002.

Corripio, J.: Vectorial algebra algorithms for calculating terrain parameters from DEMs and solar radi- ation modelling in mountainous terrain, International Journal of Geographical Information Science, 17, 1–24, 2003.

Dozier, J. and Frew, J.: Rapid calculation of terrain parameters for radiation modeling from digital elevation data, Geoscience and Remote Sensing, IEEE Transactions on, 28, 963–969, 1990.

Dubayah, R.: Modeling a solar radiation topoclimatology for the Rio Grande River Basin., Journal of vegetation science : official organ of the International Association for Vegetation Science, 5, 627–640, http://ukpmc.ac.uk/abstract/AGR/IND20453099, 1994.

Dubayah, R. and Paul, M.: Topographic solar radiation models for GIS, International Journal of Geo- graphical Information Systems, 9, 405–419, 1995.

Duguay, C.: Radiation modeling in mountainous terrain review and status, Mountain Research and Development, pp. 339–357, 1993.

Erbs, D., Klein, S., and Duffie, J.: Estimation of the diffuse radiation fraction for hourly, daily and monthly-average global radiation, Solar Energy, 28, 293–302, 1982.

Gubler, S., Gruber, S., and Purves, R.: Uncertainties of parameterized surface downward clear-sky shortwave and all-sky longwave radiation., Atmos. Chem. Phys, 12, 5077–5098, 2012.

Helbig, N., Lowe, H., Mayer, B., and Lehning, M.: Explicit validation of a surface shortwave radiation
balance model over snow-covered complex terrain, Journal of Geophysical ResearchAtmospheres, 115, D18 113, 2010.

Long, C. and Ackerman, T.: Surface measurements of solar irradiance: A study of the spatial correlation between simultaneous measurements at separated sites, Journal of Applied Meteorology, 34, 1995. Mikla ́nek, P.: The estimation of energy income in grid points over the basin using simple digital elevation model, in: Annales Geophysicae, vol. 11, 11 European Geophysical Society, Springer, 1993.

Orgill, J. and Hollands, K.: Correlation equation for hourly diffuse radiation on a horizontal surface,
Solar energy, 19, 357–359, 1977.

Ranzi, R. and Rosso, R.: Distributed estimation of incoming direct solar radiation over a drainage basin, Journal of Hydrology, 166, 461–478, 1995.

Spencer, J.: Fourier series representation of the position of the sun, Search, 2, 172, 1971.

Tovar, J., Olmo, F., and Alados-Arboledas, L.: Local-Scale Variability of Solar Radiation in a Mountainous Region., Journal of Applied Meteorology, 34, 2316–2328, 1995.