Google Earth Engine

If I would say it all, I would prefer to have a non profit organisation financed by states doing it. However I cannot deny that the project is fascinating and offers (but I have to explore it) interesting possibilities. I am talking of the Google Earth Engine of which I came to know few hours ago.

Here you can watch to a YouTube video talking about the project:

The page of the system is: https://earthengine.google.com/
Having your opinions, thoughts, impressions of use, would be an interesting feedback.

Grids: Notes for an implementation

This contains some hints and discussions about how to implement Grids  (that I learned to call CW-Complexes) in a Object Oriented language. Specifically the discussion is made with Java in mind, but obviously, not limited to it.  These slides do not contain very much bibliography and are, by far, not a complete treatment of the subject. They hope to be, however, some useful "food for thought" to start with.

Clicking on the above figure you'll be redirected to the presentation that contains the seeding for a deployment, at least in our overall system.

Evaluation of Dr. *** work and research. For getting tenure

It often happens that an Academic is asked to assess his/her peer work. This below is an example of one letter that I made recently. However, in boldface I put comments in a less traditional language that I think would give more insight with respect to the letter I actually wrote both to people who judge the researcher. Here you find it below:

The short answer to your Institution request is: there is no doubt that Dr. *** deserves to get his/her tenured position, s/he is one of the best in the world in what s/he does.

C: The short answer is: this man/woman is damned good. S/he is far above the average and you would be stupid not to keep him/her.

The long assessment is easy too. My own research work intersects very much with Dr. ***'s one. So I came to know and follow her/his production since ten years ago. Her/His paper are among those I cited more frequently in my recent research and s/he is one of the two or three researchers younger than me that I believe it is necessary to follow when working on hydrological modelling.

C: I do not need to read her/his papers. I already read and frequently cited her/him. So: s/he is good. S/he regularly publish just on the best journals. S/he has a rate of citation slightly than mine. So, maybe s/he is better than me. Stop. 

S/he also publish regularly with some of the best researchers in the field and this does not usually happen to everyone.

C: S/he is well connected. The group s/he frequents is extremely good but tends to be self-referential. This exposes her/him to the threat to follow the main stream (which helps in publishing) and not be a paradigms breaker (which could be hard to sustain). However, if s/he is smart, as I believe, s/he will avoid the pitfalls of situation.

Finally her/his DUPER environment is a milestone in recent literature.

C: Her/his framework is damned good. It is one of the main contenders of mine. If I would be a super rampant guy, I would try to obstacle it. But I am a professor (sigh).

It happens, obviously, that I disagree on some details of her/his research, but this is matter of normal dialectics between peers.

C: S/he made some choices different from mine and sustains them with some statements that I judge debatable. But respect to the shit I see around there is by far no competition. 

From the recent papers s/he chose for evaluation, I could see that s/he also enlarged her/his view on using Bayesian techniques for estimating model’s parameters.

C: S/he is able to understand where the fashionable stuff is. This attitude does not produce science by itself but for sure is one of the characteristics that good Academics must have at least a bit (sometimes to avoid to follow fashionable stuff).

His/Her approach is sophisticate and requires certainly some deepening from myself, but I could appreciate its novelty.

C: The last paper is really technical and if s/he insists too much in this direction s/he can fall into hydrological mathematistry. 

Whilst her/his initial production reflects also %%%% view of the matter, it is quite clear now what is Dr. *** own contribution and evolution.

C: Her/his former and influential boss quite determined her/his initial research, but s/he seems having coming out from her/his boss old fashionable approach, which was not up-to-date and could have brought her/him to produce crappy stuff. Which was not. 

To summarise, when s/he writes: "My research areas in catchment modelling can be broadly classified as: (1) the development of flexible models,which provide the building blocks for the construction of models, (2) the formulation of guidelines for model development, and (3) the use of models to interpret catchment behaviour and to produce reliable predictions. Through these research areas I have contributed to important developments in hydrological sciences in the past years, such as the use of models for hypothesis testing, the incorporation of experimental knowledge inthe modelling process, and the understanding of large scale hydrological processes and their controls.”, s/he give an image of her/himself that I fully share and think is appropriate.

C: S/he knows what s/he is and does. S/he is ready for his role. 

S/he individuates three areas of progress for his research: a) Flexible models development; b) Theory of (hydrological) models developments; c) Understanding and prediction of catchment behavior.

C: This intersects with my activities, that’s why I reformulated the name of her/his second area of interest. My, I think, is more appropriate. 

Recently there was, among many colleagues, the idea that the topics of surface hydrology were mature and that research has to move to hybrid fields like socio-hydrology, eco-hydrology (with an accent to ecology more than hydrology), ecosystem services and so on.

C: Most of colleagues, even gifted ones, tends to give for granted was is actually not (at all).

With respect to these new, certainly exciting directions, the focus of Dr. *** seems quite traditional. However, I fully approve it. The kind of research s/he is pursuing is fundamental and necessary after years of blown dilettantism that has relevant consequences in research and practice.

C: With respect to these topics most colleagues are either wrong or superficial. (Someone has to use these occasion to push out his frustration). 

Conclusions and extension obtained from ill-conceived concepts, improperly used models, and lack of hypothesis testing brings to wrong interpretations of hydrological facts and can have negative consequences on engineering applications and cause the choice of wrong policies.

C: They use wrong findings based on misconceptions and move to new subfields with wrong information. GIGO. 

Dr. *** work is important and its importance is going to become more and more clear in the next years. Hopefully just a few model infrastructures will emerge from the present fragmentation, and I believe the one from the evolving work of Dr. *** will be in the group.

C: Just a few framework will survive when people finally will understand the limits of hydrological dilettantism. Possibly the work of this researcher will survive and for sure, it will do for the next decade.

Probably, If I would be her/him, I would try to broaden a little the perspective including, besides the hydrologic response, the whole set of hydrological processes that concern catchments’ budget more seriously, and, in particular, evapotranspiration that, in some environments covers even sixty per cent of the water budget.

C: Engineers, even some ecohydrologists, are obsessed with discharges. These are just a part of the game. Future frameworks have to play the full game.

I would also devote some attention to the approach with travel time distributions which could open new perspectives in modeling of nutrients and pollutants, a field which Dr. *** already came across.

C: S/he forget some fundamental aspects concerning her/his own area of interest. S/he should not.

Finally, I have no doubt that Dr. *** will continue to improve and bring great contributions to your Institution.

C: I criticize her/him as I do with those I really like. S/he is a good person. Everybody can work with her/him and have benefits. They know that they want to keep her/him and I appreciate it. This will made her/him more free and more brave.

P.S. - They keep him/her

A smooth introduction to some Algebraic Topology topics

Since the previous post on Grids touched some topics on algebraic topology, I  selected a few sites where I found some interesting information from our point of view that can help the reading of the previous slides and, in general, of the references already presented.

• This science4all site is a very visual and simple introduction to most of the concepts formalized in slides.
• Isabel Darcy's video lectures  on Applied Algebraic Topology are also enjoyable (You can stop anywhere but I put them all)
• On Euler's characteristics (We did not mention it in slides, but is worth to know)
• Simplicial, Addition, Orientation
• Modular Mathematics
• Addition and free vector spaces
• Triangulation and simplicial complexes
• Creating a simplicial complex from data 
• Persistent Homology  - I
• Persistent Homology - II
• Persistent Homology - III 
• A quick example calculating the column space and the nullspace of a matrix
•  Clustering via persistent Homology

 

• Enzo Tonti's lectures about his book

• The reasons of analogies in Physics - Part I - Part II - Part III - Part IV - Part V - Part VI

I do not share many of Tonti's opinions, but some of his talks and books are, indeed, enlightening.

Meshes, Grids, CW-Complexes

Representation of space (and time) is a necessary step to implement any Physics. However, the topic is seldom faced with the appropriate generality, and this reflects into implementations in softwares that do not have a general structure. This is the rational for talking here about meshes or grids.
From a quick view of the material found in literature, it appears that a lot of work has been done, by few people (group). There are at least two pathways to follow. The first is the Ugrid mesh specification. We can describe it as the classification work of mesh by power users, i.e. people who use meshes for describing (especially environmental) numerical problems. Their work is concentrated on semantics and explaining what the mesh are,  with the scope to insert them in NetCDF, a self explaining file format conceived to contain environmental data.
The second approach, in Berti (2000) starts from more fundamental mathematical work which is also used in Heinzl (2007, but referring to the paper Heinzl, 2011, could be convenient).
In general, the first two chapters of Berti’s dissertation are a must-to-read for those who deals with scientific computing.
A subsequent number of papers cover two topics: how to store mesh in databases and how to give to these structures the right flexibility to be parallelized. Interestingly some of the mathematical work actually flowed into the creation of C++ libraries, in particular the GRAL libraries, developed by Berti himself.

Browsing around, mesh are rigorously defined in Algebraic Topology (e.g. Hatcher, 2001) and this was  recognized in various papers, since the sixties (e.g. Branin, 1966, and references therein). A general discussion, which involve the nature of Physical laws, was produced by Tonti (2013) and somewhat pushed also by Ferretti (2015).  These treatments could bring in the matter some new insight and the general understanding. What we did with the slides was to try to synthetize some of the above work, especially in view of an implementation of some Java libraries. The idea suggested by the readings was to use generic programming, design patters (Gamma et al, 1994, Freeman et al., 2005) and programming to interfaces (which BTW we already have in mund: we found what we were looking for). 
But the detail of the implementation will be the topic of a future post (but you can have a glance to literature browsing the bibliography below). Now get (a little of) theories by clicking on the figure above.

References

• Berti, G. (2000, May 25). Generic Software Components for Scientific Computing.
• Branin, F. H. (1966). The algebraic-topological basis for network analogies and the vector calculus (pp. 1–40). Presented at the Symposium on generalized Networks, New York.
• Ferretti, E. (2015). The algebraic formulation: why and how to use it. Curved and Layer. Struct., 2, 106–149. http://doi.org/10.1515/cls-2015-0007
• Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1994). Design Patterns Elements of Reusable Object-Oriented Software (pp. 1–417). Addison-Wesley.
• Hatcher, A. (2001), Algebraic Topology. 
• Freeman, Eric, Freeman, E., Sierra, K., & Bates, B. (2004). Head First Design Patterns (pp. 1–672). O'Reilly.
• Heinzl, R., & Schwaha, P. (2011). A generic topology library. Science of Computer Programming, 76(4), 324–346. http://doi.org/10.1016/j.scico.2009.09.007
• Naftalin, M., & Wader, P. (2009). Java Generics and Collections (pp. 1–286). O'Reilly.
• Nikishkov, G. (2006). Object oriented design of a finite element code in Java. Cmes, 11, 81–90. http://doi.org/10.3970/cmes.2006.011.081
• Nikishkov, G. (2010). Programming Finite Elements in Java (pp. 1–394). Springer-Verlag 2010.
• Simonelli, C. (2015, November 22). Programmazione parallela strutturata in Java8: Muskel2. (M. Danelutto, Ed.).
• Tonti, E. (2013). The Mathematical Structure of Classical and Relativistic Physics (pp. 1–537).  

Notes

Some of my  students asked for somewhat a milder introduction to algebraic  topology.  I dedicated a new short post to it.

Weather Generation (according to Korbinian Breinl)

How one can reasonably cope with simulating future hydrometeorological forcing for hydrological purposes ? Clearly, since meteorology is dominated by unpredictable phenomena (in the sense of chaotic ones) and we cannot pretend to simply use forecasts, when we are looking just a little far away. An option would be to use climatic models and doing dynamic downscaling of their outcomes. The previous lecture given by Jeremy Pal (GS), followed this research path. However, we can produce statistical weather scenarios using stochastic weather generators (SWG) too, once  we have an idea of what will be the mean characteristics of such system.
Literature is full of SWG that covers mainly temperature and rainfall, but it seems there exists systems also that covers other meteorological variables, like wind and radiation.
Today we had a talk on the subject given at our Department of Civil, Environmental and Mechanical Engineering given by Korbinian Breinl. He is at present a post-doc at Uppsala University with Giuliano Di Baldassarre (GS) and we are collaborating in the SteepStreams project.

As usual you can find his presentation by clicking on the figure above. But you can see the talk on YouTube

Besides flooding (and solid flooding) which is one of the scopes of the projects, I hope we succeed in modeling all the main components of the hydrological cycle by a combine use of Korbinian’s Generator and JGrass NewAGE. I have also the video record of his presentation, but not yet the approval to share it publicly on YouTube. However you can ask it to me writing to abouthydrology @ gmail.com.

Korbinian's generator is written in Matlab, and it is available through Github.

Please find below a reference list which include, besides Korbinian’s one, some other references that I could gather through time.

References

• Apipattanavis, S., G. Podesta´, B. Rajagopalan, and R. Katz, 2007: A semiparametric multivariate and multisite weather generator. Water Resour. Res., 43, 1–19. 
• Baigorria, G. A., & Jones, J. W. (2010). GiST: A stochastic model for generating spatially and temporallycorrelated daily rainfall data. Journal of Climate, 23(22), 5990-6008.
• Baxevani, A., and J. Lennartsson (2015), A spatiotemporal precipitationgenerator based on a censored latentGaussian field, Water Resour. Res., 51,4338–4358, doi:10.1002/2014WR016455.
• Beersma, J. J., and T. A. Buishand, 2003: Multi-site simulation of daily precipitation and temperature conditional on the atmospheric circulation. Climate Res., 25, 121–133.
• Brissette, F. P., M. Khalili, and R. Leconte, 2007: Efficient stochastic generation of multi-site synthetic precipitation data. J. Hydrol., 345, 121–133.
• Breinl, K., Turkington, T. and Stowasser, M. (2015), Simulating daily precipitation and temperature: a weather generation framework for assessing hydrometeorological hazards. Met. Apps, 22: 334–347. doi:10.1002/met.1459
• Breinl, K., Di Baldassarre, G., Lopez, M.G., Hagenlocher, M., Vico, G., Rutgersson, A.,2017. Can weather generation capture precipitation patterns across different climates,spatial scales and under data scarcity? NATURE Scientific Reports 7.
• Burton, A., C. G. Kilsby, H. J. Fowler, P. S. P. Cowpertwait, and P. E. O’Connell, 2008: RainSim: A spatial–temporal stochastic rainfall modelling system. Environ. Modell. Software, 23, 1356–1369.
• Cannon, A., 2008: Probabilistic multisite precipitation downscaling by an expanded Bernoulli–Gamma density network. J. Hydrometeor., 9, 1284–1300 
• Cordano E. and Eccel E. (2016), Tools for stochastic weather series generation in R environment, Italian Journal of Agrometeorology <doi:10.19199/2016.3.2038-5625.031> (Codes and manual can be retrieved here).
• Donatelli, M., Bellocchi, G., Habyarimana, E., Bregaglio, S., Confalonieri, R., Baruth, B., CLIMA: a weather generator framework, 18th World IMACS / MODSIM Congress, Cairns, Australia 13-17 July 2009 http://mssanz.org.au/modsim09
• Fatichi, S., Ivanov, V.Y., E. Caporali (2011). Simulation of future climate scenarios with a weather generator, Advances in Water Resources, 34, 448–467, doi:10.1016/j.advwatres.2010.12.013 (code here)
• Fowler, H. J., C. G. Kilsby, P. E. O’Connell, and A. Burton, 2005: A weather-type conditioned multi-site stochastic rainfall model for the generation of scenarios of climatic variability and change. J. Hydrol., 308, 50–66.
• Geng, S., Penning de Vries, F.W.T., Supit, I., 1986. A simple method for generating daily rainfall data. Agric. ForestMeteorol. 36, 363–376.
• Hodges, T., French, V., LeDuc, S.K., 1985. Estimating solar radiation for plant simulation, models. AgRISTARS Tech. Rep. JSC-20239; YM-15e00403, Columbia, MO, USA.
• N. Z. Jovanovic , J. G. Annandale , N. Benadè & G. S. Campbell (2003) CLIMGEN-UP: A user-friendly weather data generator, South African Journal of Plant and Soil, 20:4, 203-205, DOI: 10.1080/02571862.2003.10634936
• Ivanov, V. Y., R. L. Bras, and D. C. Curtis (2007), A weather generator for hydrological, ecological, and agricultural applications, Water Resour. Res., 43, W10406, doi:10.1029/2006WR005364. (code here)
• Keller, D.E., A weather generator for current and future climate conditions, Ph.D. Dissertation, ETH Zurich, 2015
• Khalili, M., R. Leconte, and F. Brissette, 2007: Stochastic multisite generation of daily precipitation data using spatial autocorrelation. J. Hydrometeor., 8, 396–412.
• Koutsoyiannis, D. and Onof, C. Rainfall disaggregation using adjusting procedures on a Poisson clustermodel. J. Hydrol. 246 (2001).
• Leander, R., and T. A. Buishand, 2009: A daily weather generator based on a two-stage resampling algorithm. J. Hydrol., 374, 185–195.
• Müller, H. & Haberlandt, U. Temporal Rainfall Disaggregation with a Cascade Model: From Single-StationDisaggregation to Spatial Rainfall. J Hydrol Eng 20 (2015).
• Palutikof, J. P., C. M. Goodess, S. J. Watkins, and T. Holt, 2002: Generating rainfall and temperature scenarios at multiple sites: Examples from the Mediterranean. J. Climate, 15, 3529– 3548.
• Qian, B., J. Corte-Real, and H. Xu, 2002: Multisite stochastic weather models for impact studies. Int. J. Climatol., 22, 1377– 1397
• Richardson CW. 1981. Stochastic simulation of daily precipitation,temperature, and solar-radiation. Water Resour. Res. 17: 182–190
• Richardson, C.W., Wright, D.A., 1984. WGEN: A Model for Generating Daily Weather Variables. U.S. Department of Agriculture, Agricul- tural Research Service, ARS-8.
• Mikhail A. Semenov, LARS-WG A Stochastic Weather Generator for Use in Climate Impact Studies. 2002
• Mikhail A. Semenov, Roger J. Brooks, Spatial interpolation of the LARS-WG stochastic weather generator in Great Britain, CLIMATE RESEARCH,Vol. 11: 137–148, 1999
• Stöckle, C.O., Nelson, R.L., Donatelli, M., Castellvı`, F., 2001. ClimGen: a flexible weather generation program. In: Bindi, M., Donatelli, M., Porter, J.R., Van Ittersum, M.K. (Eds.), Proceedings of the Second International Symposium on Modelling Cropping Systems, Florence, Italy, pp. 229e230.
• Srikanthan, R., and G. G. S. Pegram, 2009: A nested multisite daily rainfall stochastic generation model. J. Hydrol., 371, 142–153.
• Wilby, R. L., O. J. Tomlinson, and C. W. Dawson, 2003: Multi-site simulation of precipitation by conditional resampling. Climate Res., 23, 183–194.
• Wilks, D. S., 1998: Multisite generalization of a daily stochastic precipitation generation model. J. Hydrol., 210, 178–191
• Wilks, D.S., Wilby, R.L., 1999. The weather generation game: a review of stochastic weather models. Progress in Physical Geography 23, 329e357.

Other available codes

• GMLWGEN (R)
• WeatherGen (C++)