Uncertainty and Information Theory

We all are persuaded that uncertainty is a big topic, in life but also, in hydrology. So important that many hydrologists dedicate their life to its estimation, in connection to hydrological processes. Uncertainty since it is uncertain also generate confusion, and some of tis literature is  confuse and confusing (I don't want to cite negatively anyone, but I could).
Whatever the case, one of the best talk I attended to at last Fall American Geophysical Union Meeting, was the invited lecture by Hoshin Gupta. Hoshin has an outstanding (really outstanding, I mean) carrier in finding calibration methods, indentifiability of parameters and understanding uncertainty in models. Recently (see for instance Gong et al., 2013) he started to apply concepts derived from information theory to hydrology.  BTW, you can find the pdfs of his AGU’s presentations here: on the necessity to apply information theory concept to evaluate models structural hypotheses, and another one about Information theory and Bayesian inference in hydrology (both with a lot of citations).

I never really understood why hydrologists do not use information theory  concepts. I-Theory is a well developed mathematica theory with a lot of tools, and could help to get out from the fuzziness around  the determination of uncertainty in models. Besides, using the concept of I-Theory information/uncertainty one can gain knowledge about the complexity of processes outputs and, possibly, infer something about the "complexity" of models required to mathematically account for it in a proper way (remind: "Everything should be made as simple as possible but not simpler").

Hoshin is not the only one that was attracted by information theory. In my occasional browsing of the topic, I also found some other interesting papers: the first one, by  Majda and Gershgorin, is concerned by climate models. This is encouraging, because climate models are certainly at least as involved as hydrological models are, and, if not, even more. A second is Weijs et al. (2013): this is concerned with time series: we compare time series, therefore knowing how much information is hidden in a time serie (at least with reference according to some encoding key) is certainly useful. For Wejis and van de Giesen, this paper is just a coming back to the topic (see also Weijs et al., 2010, and Weijs CV)

Another paper came from  Rudell (GS) on EOS remarkably highlighting that the I-Theory applications to hydrology attracted last year  many more people than use to be.
For making me feeling among the smarter, I  bought a book, by Mezard (see also, and GS) and Montanari (Andrea, not our colleague Alberto who also has quite a production on uncertainty: please see his website) which can be a further source of ideas and thoughts.

So far, I never actually read carefully any one of the papers (or the book), but excited at the idea to have time to do it in deep.

References

Gong, W., H. V. Gupta, D. Yang, K. Sricharan, and A. O. Hero III (2013), Estimating epistemic and aleatory uncertainties during hydrologic modeling: An information theoretic approach, Water Resour. Res., 49, 2253–2273, doi:10.1002/wrcr.20161.

Mézard, M. and Montanari, A. , Information, Physics, and Computation, Oxford University press, 2009

Majda, A. J.,  and Gershgorin, B., Quantifying uncertainty in climate change science through empirical information theory, PNAS, August 24, 2010, vol. 107,no. 34, 14958–14963

Ruddel, B.L, N. A. Brunsell and P. C. Stoy, Applying Information Theory in the Geosciences to Quantify Process Uncertainty, Feedback, Scale, Eos, Vol. 94, No. 5, 29 January 2013

 Weijs, S. V.;  Schoups, G.  and van de Giesen, N., Why hydrological predictions should be evaluated using information theory, Hydrol. Earth Syst. Sci., 14, 2545-2558, 2010, www.hydrol-earth-syst-sci.net/14/2545/2010/, doi:10.5194/hess-14-2545-2010

Weijs, S. V., van de Giesen, N. and Parlange, M. B., Data compression to define information content of hydrological time series, Hydrol. Earth Syst. Sci., 17, 3171–3187, 2013 www.hydrol-earth-syst-sci.net/17/3171/2013/ doi:10.5194/hess-17-3171-2013

An introduction to Forecast, Errors, Uncertainty in Models, and Forecast Correction

Following the link below, you can find  the presentation I gave at the Pensa Trasversale initiative. I talked about modelling and its uncertainty, and I had, obviously, in mind my experience as a hydro-geomorpho-logical modeler.
The last topic in the title, forecast correction, is not as deep investigated as I would have. But in writing I realised that I had to speak to High School Students.
The presentation is in Italian, but I can translate it in English if someone is interested.  In the case, I should do a further investment in making some parts more technical and add some examples.

The presentation can be found, as usual, on slideshare. A companion of the presentation, also in Italian, of this presentation can be this paper that I wrote more than twenty years ago about modelling, entitled: Introduction to the Mathematical Models of Environment.  I dedicated this small contribution to Sandro Marani with whom I discussed a lot the topic.

The scientific "validation" of hydrological models

The title is controversial. Obviously both in epistemology and hydrology. A Reviewer of one of my recent papers asked for us taking away any recurrence on the word validation. The same asked a coauthor of another paper.  But, since I am not afraid of the word,  let's agree that with this word I do not mean anything ontological but instead  I try to indicate a grid of rules that identify a process of scientific knowledge accumulation which  could be accepted by the community of scientists as a "good practice" to do science.

The entire presentation of the talk can be found, as usual on SlideShare, here. In this post, I summarize instead its main points.
I did not face the problem directly but analyze two models, the GIUH model (in its width function version) and the Topmodel (Beven and Kirkby, 1979).

What I observed is that they gained the status of good scientific model from:

- having been tested by different researchers group independently  from the theoretical point of view: they tested the consistence of the assumptions on which the models were built, and their formal mathematical structure (the simplicity of the algorithms involved allowed in fact multiple implementation of the theories);
- field campaigns that tested the correspondence of model's results with measures (with respect to hydrograph reproduction);

In the case of GIUH,

-  Furthermore   the assumptions  was  derived from some general physical principle (e.g. minimum energy dissipation theories).

I also showed that both the models are wrong with respect to some set of measures, i.e. tracer measures for the GIUH, and measures of soil moisture distributions in hillslopes for the Topmodel. Nevertheless, I argue that they remain good models, when applied respecting their assumptions, and for limited scopes.  Their generalizations, on the other hands, are indicated by their own failure.

So models that can be falsified (in Popperian style) are good scientific examples, and what is validated is a shared procedure that make our knowledge to increase about phenomena. These models, in particular, even falsified remains good tools, (which, however, should not be abused) because built with simplicity in mind (remember the Occam's razor), and therefore maintain a reasonable success in describing a well defined set of users' cases.

In fact we deliberately do simplifications and errors in hydrology, with the scope to obtain simple and fast models, on the opposite of complex and slow ones, with the hope that their errors cancel each other (“ You cannot deny that our universe is not a chaos; we discern in it beings, things, stuff that we name with words. These beings or  things are forms, structures endowed with a certain stability; they fill a certain portion of space and perdure for a certain time", R. Thom, 1975) because  reality has built-in scales, and different laws at different scales.
When searching  for reconstructing a unique hydrograph at the closure of a basin, it could be reasonable to think that this canceling of errors can happen for the nature itself of the process which collects information (water) all around a basin and concentrates (sums) it at the outlet (as discharge).  In other cases, that simplifications work and that errors cancel could not be so heuristically defendable.

In general, when building our models, we should have a clear and disenchanted vision of their limits,  a theory for their errors, and the idea of the measures (if we do not have controlled experiments) to falsified them. The best would be to have a theory correlating the information (of the signal) we need to reproduce  with the complexity of the model needed to get it, so we do not exaggerate with detailed descriptions of the (micro-)physics, at finer scales, which are not required at the larger ones.

However, if models need to be simple they should not be simpler (as A. Einstein said). Therefore we should give attention to those phenomena which are not well described and add complexity when needed. In turn, when adding complexity,

- any model addition should be tested independently and not just on the basis of the benchmark quantities (like discharge) that were already investigated by testing the simpler model (i.e., if we add discharge with a snow-melt model, we should test directly and independently the snow-model before, and have the certainty -!- that snow was really present in the catchments, and that it, in fact, did melt).

A further note regards the code.  Meanwhile more complexity is required, models becomes more complicated in their equations,  and more and more models' code becomes the "real thing" that is used to do prognoses (the real model).  Therefore models' code should be open, and open to third parties inspections. I talked other times on this issue in this blog (e.g. http://abouthydrology.blogspot.it/2012/05/paper-in-nature-on-scientific-software.html), and I will not repeat myself again.

I finished my talk with some rhetorical polemics with those (great indeed) hydrologists who discuss restless about validation and uncertainty, instead of trying to do better models. But this was actually just for fun.

Quantifying uncertainty

I saw on EOS an announcement regarding a new website, QUEST, or quantifying uncertainty in ecosystem studies. Hydrology in fact needs an effort to quantify uncertainties, but this is usually ignored.

It is quite a few years that I fly around of this issue, and probably next year I would try to dig a more little deep in literature.

The sources of uncertainty in hydrological modeling are at least three:

- the input data (which can derive from chaotic dynamics)
- the approximation contained in equations
- the parameterizations of constants which can be heterogeneous (highly variable, if not random in space)

When thinking to inputs, the paradigm is rainfall. It is usually estimated just a few points in the domain with large errors. Then, the local estimates needs to be interpolated and extrapolated in space, introducing further errors. Rainfall itself is very irregular in time and space at all of the scales, which means that you can capture just the statistics of its behavior (and this left you out in the cold with other errors).

When looking at flows, i.e. at their mathematical description in equations, one has to think that they are eminently a thermodynamical product where some fluctuations need to be neglected and described with suitably averaged properties, which could be no possible (significant).
Besides, usually the system described is made up of many non-linearly connected subsystems, and in practical implementations the nonlinearities and the feedbacks are simplified or even neglected. Moreover, equations need to be discretized on grid, which introduce itself approximations.

Finally, saying that some processes are governed by heterogeneities, we also state that the information they contain is algorithmically incompressible (e.g. Chaitin), and there is no way to represent it in short strings. The latter syndrome is the one well described by Borges in "The Exactitude of Sciences", but also in Noam Chomsky's book, Rules and Representation, where at page 8 he cites Stephen Weinberg, and goes so deep in asking if we can really know reality, and what it means.

In any case, the hydrological community, started to take care of it from a long time (here it is a recent abstract with hopefully a good literary review, and here, the work by Beven, Gupta and Wagener), but they started to work especially on the assessment of parameter uncertainty (even if GLUE pretends to be of more general validity). A recent assessment is also in this work by Goetzinger and Bardossy which can provide access to further concepts and bibliography.

However, many hydrological models produce just time series, and therefore the uncertainty reduce to understand (and sometimes to compare) a couple of time series: the measured time serie and the modeled time serie. Good hydrological models are those that reproduce the time series with a good agreement. This is quantified, often but not always, with the use of indexes. The mean square error or its root, the Nash-Sutcliffe, the minimax objective function, average absolute percentage error, the index of agreement, the coefficient of determination, are a few of them.

This is certainly a narrow perspective to look at the topic. Both the measured and the simulated series are, in fact, affected by errors, and therefore one should not compare the two series directly , but the time series including their errors. I believe that this would coincide to adopting a Bayesian perspective of the problem (e.g D'Agostini 2003 - Bayesian Reasoning in Data Analysis, A critical Introduction) and will turn into data assimilation (e.g. Kalnay, Atmospheric Modeling, Data Assimilation and Predictability, 2003) with the defect that, at this point, data and models are so entangled that it would be difficult to extricate them (but not impossible, I guess).

We can also observe that a model usually produces more than a single time series. So "a prediction" becomes the "predictions" and the uncertainty spreads in all of them.

Besides, we did not mention, spatial patterns: before we claims for their uncertainty, we have to recognize that we should quantify them. How can we do ? And for extension are we able to identify spatio-temporal patterns ? And therefore when we can decide if two of these patterns are the same (neglecting noises). Indicators of statistical equality would probably give miserable scores if applied to two or three dimensional fields.

Someone has ideas ?