Graphs, Correlation and Causality

A week ago I started to read “The book of why: the science of cause and effects" by Judea Pearl. (see also his website). This is part of my search for new mathematics for describing entangled hydrological and ecological processes (see also this post and links therein). It is a dissemination book and not intended to grow too technical. However, it arrives the moment when understanding technicalities becomes part of the full understanding, if you do not want to remain a tourist of the new knowledge and become instead an active user of it. Pearl says he dedicated most of his research life to these problems and, therefore, pretending to fill the gap in few weeks is a nonsense.
I require more time to go deeper but presently I have no time. Therefore let me take some annotations here for making easier future efforts. 

 
To go to the details, one can go to the more technical book by Pearl himself, Causality. However, it happened I went to browse some chapters of the very good Shalizi's book on statistics. Chapters from the 20th are a reasonable starting point. Shalizi book itself is not fully explicative but a compromise where some theorems are not deminstrated but assumed and make explicit. Nice enough but requiring in any case the appropriate dedication. Shalizi seems to be a voracious reader, and in the bibliography of his chapter 21, he cites some fundamentals work to put in line for a full understanding the topic. His subsequent chapters also enlarge the vision to information theory, and is connections between the science of causal statistics. Cool. While postponing the study and trying to grasp concepts, I fully report the bibliography I came across here below (mostly from Shalizi's).
All of these are also a good reading for those who believe that data science is a practice which springs out from nowhere.


References

• Chalak, K., & White, H. (2012). Causality, Conditional Independence, and Graphical Separation in Settable Systems. Neural Computation, 1–60.
• Cover, Thomas M. and Joy A. Thomas (2006). Elements of Information Theory. New York: John Wiley, 2nd edn.
• Dinno, A. (2017). An Introduction to the Loop Analysis of Qualitatively Specified Complex Causal Systems (pp. 1–23).
• Guttorp, Peter (1995). Stochastic Modeling of Scientific Data. London: Chapman and Hall.
• Jordan, Michael I. (ed.) (1998). Learning in Graphical Models, Dordrecht. Kluwer Academic.
• Kindermann, Ross and J. Laurie Snell (1980). Markov Random Fields and their Ap- plications. Providence, Rhode Island: American Mathematical Society. URL http://www.ams.org/online_bks/conm1/. 
• Lauritzen, S.L., Dawid, A.P., Larsen, B.N., Leimer, H.G. (1990), Independence properties of directed Markov fields, Networks, 20, 491-505
• Lauritzen, S.L. (1996) Graphical Models. New York: Oxford University Press.
• Loehlin, John C. (1992). Latent Variable Models: An Introduction to Factor, Path, and Structural Analysis. Hillsdale, New Jersey: Lawrence Erlbaum Associates, 2nd edn.
• Moran, P. A. P. (1961). “Path Coefficients Reconsidered.” Australian Journal of Statis- tics, 3: 87–93. doi:10.1111/j.1467-842X.1961.tb00314.x.
• Pearl, J. (2000). Causality- Models, Reasoning, and Inference (pp. 1–386). Cambridge University Press.
• Wright,S., The Method of Path Coefficients. Annals of Mathematical Statistics 5:161-215.
• Wysocki, W. (1992). “Mathematical Foundations of Multivariate Path Analysis.” Inventiones Mathematicae, 21: 387–397. URL https://eudml.org/doc/263277. 

Uncertainty and Information Theory

We all are persuaded that uncertainty is a big topic, in life but also, in hydrology. So important that many hydrologists dedicate their life to its estimation, in connection to hydrological processes. Uncertainty since it is uncertain also generate confusion, and some of tis literature is  confuse and confusing (I don't want to cite negatively anyone, but I could).
Whatever the case, one of the best talk I attended to at last Fall American Geophysical Union Meeting, was the invited lecture by Hoshin Gupta. Hoshin has an outstanding (really outstanding, I mean) carrier in finding calibration methods, indentifiability of parameters and understanding uncertainty in models. Recently (see for instance Gong et al., 2013) he started to apply concepts derived from information theory to hydrology.  BTW, you can find the pdfs of his AGU’s presentations here: on the necessity to apply information theory concept to evaluate models structural hypotheses, and another one about Information theory and Bayesian inference in hydrology (both with a lot of citations).

I never really understood why hydrologists do not use information theory  concepts. I-Theory is a well developed mathematica theory with a lot of tools, and could help to get out from the fuzziness around  the determination of uncertainty in models. Besides, using the concept of I-Theory information/uncertainty one can gain knowledge about the complexity of processes outputs and, possibly, infer something about the "complexity" of models required to mathematically account for it in a proper way (remind: "Everything should be made as simple as possible but not simpler").

Hoshin is not the only one that was attracted by information theory. In my occasional browsing of the topic, I also found some other interesting papers: the first one, by  Majda and Gershgorin, is concerned by climate models. This is encouraging, because climate models are certainly at least as involved as hydrological models are, and, if not, even more. A second is Weijs et al. (2013): this is concerned with time series: we compare time series, therefore knowing how much information is hidden in a time serie (at least with reference according to some encoding key) is certainly useful. For Wejis and van de Giesen, this paper is just a coming back to the topic (see also Weijs et al., 2010, and Weijs CV)

Another paper came from  Rudell (GS) on EOS remarkably highlighting that the I-Theory applications to hydrology attracted last year  many more people than use to be.
For making me feeling among the smarter, I  bought a book, by Mezard (see also, and GS) and Montanari (Andrea, not our colleague Alberto who also has quite a production on uncertainty: please see his website) which can be a further source of ideas and thoughts.

So far, I never actually read carefully any one of the papers (or the book), but excited at the idea to have time to do it in deep.

References

Gong, W., H. V. Gupta, D. Yang, K. Sricharan, and A. O. Hero III (2013), Estimating epistemic and aleatory uncertainties during hydrologic modeling: An information theoretic approach, Water Resour. Res., 49, 2253–2273, doi:10.1002/wrcr.20161.

Mézard, M. and Montanari, A. , Information, Physics, and Computation, Oxford University press, 2009

Majda, A. J.,  and Gershgorin, B., Quantifying uncertainty in climate change science through empirical information theory, PNAS, August 24, 2010, vol. 107,no. 34, 14958–14963

Ruddel, B.L, N. A. Brunsell and P. C. Stoy, Applying Information Theory in the Geosciences to Quantify Process Uncertainty, Feedback, Scale, Eos, Vol. 94, No. 5, 29 January 2013

 Weijs, S. V.;  Schoups, G.  and van de Giesen, N., Why hydrological predictions should be evaluated using information theory, Hydrol. Earth Syst. Sci., 14, 2545-2558, 2010, www.hydrol-earth-syst-sci.net/14/2545/2010/, doi:10.5194/hess-14-2545-2010

Weijs, S. V., van de Giesen, N. and Parlange, M. B., Data compression to define information content of hydrological time series, Hydrol. Earth Syst. Sci., 17, 3171–3187, 2013 www.hydrol-earth-syst-sci.net/17/3171/2013/ doi:10.5194/hess-17-3171-2013