Thursday, October 10, 2019

Virtual Reality, Augmented Reality and Mixed Reality in Hydrology and Earth System Science

I did some searches on the web to see if there are application of virtual reality (VR), augmented reality (AR) or mixed reality (MR) (see also here for a tech-magazine type of introduction) around that could be used in hydrology.
On the general issues there are many information on the web, but if you think that reading a book can save a lot of time, this book is Augmented Reality by Dieter Schmalstieg and Tobias Höllerer.
These technologies are largely advertised by the big IT companies (e.g. Google). Incredible user experiences and interactions with data can be seen in the scifi movies we watch daily. Application in Earth Sciences are scarce though. 

To be short for what regards hydrology: I found application in Iowa, in Germany and a group of people that moves around the Virtual Geosciences Conferences (from which I robbed the image) and in general, geologists found simple and direct applications of it. In hydrology just relatively few papers: by Su et al (2008), by Billen et al. , (2018), and those cited below of the Julich Forschungszentrum (JF).
In Iowa there is an active group of hydroinformatics. They have a VR/AR/MR page here and because their involvement in real time flood forecasting, their work is mainly in that direction. In the US there is NOOA that has done some activity, but especially looking to the global planet and the space. NOOA, not surprisingly (?), is more interested in dissemination of results through virtual reality than using AR, MR in supporting Earth Science Research.
In Germany I found a few experiences. One is the group working at "Data Assimilation for Improved Characterization of Fluxes across Compartmental Interfaces” that includes ESA, a few good Universities and research centers. Their site is, however cryptic about results. Their focus is to support data assimilation. The JF plays in it a central role with its models, and they, in fact, have something to show. In fact, browsing Lars Bike website, one can also find some publications (e.g., Yan et al, 2019Rink et al., 2018Rink et al, 2017Helbig et al., 2015, Bilke et al., 2014) that can be useful to read for understanding some directions to go.
To have the very best experiences of the VGS, the best thing is to browse their 2018 proceedings

There is not very much around beyond that, and therefore, I think there is a lot of room to use them in Earth sciences.
From my model builder and user I see exciting the possibility to interact in a more immersive way with input and output of models.
Notwithstanding we are usually looking for the Holy Grail of simplified models, we have to get used to manage and have a supervision of more large data sets, maybe also as a necessary step to get simpler models.
Input data are many and complicate to grasp if the model has to describe complex reality and patterns are not that visible at the first sight since we have to educate our eyes, as any guy using a microscope, or a telescope knows, since the Galileo times.
As a modeler, I would like the data must be visualized and modified promptly for changing simulations behavior. I also dream that the models could be driven by human interaction in real time, changing parameters on the fly copying with the changes the flow of measures impose. Like we were driving a starship.
Obviously this model/data navigation needs to be recorded and re-analyzed afterwards to learn better what has happened (an this further requires tools)
In our field, visual data are usually spatially distributed datasets or graphs and distributed datasets could cover static quantities like the terrain topography, the landscape, or dynamical quantities as soil moisture, temperature, water velocity and, possibly, also some other less trivial quantities like entropy fluxes, water celerity, nutrients concentration.
Brought to the fields those data can be useful to setup measures, tp drive field inspection, “learn by seeing" data and the situation together in an immersive environment, where discrepancy between what expected and what seen can become evident than in traditional situations.
There are non secondary application to education which seems trivial to VR/MR/AR experts, because virtual reality classes seem actually be already available. However if we look carefully, these cover usually very elementary topics and seldom support high education (with exceptions, see Aubert et al, 2015). What I could find is here, and here for example, with incursions ob the psychology of learning here.
Clearly VR/AR could interact also with crowd science initiatives, as has been already envisioned, but could be certainly enhanced.

For the interested there is also a Journal, Virtual Reality, where something interesting about water can be actually found (but I confess I am not able to evaluate how good the journal is).


Aubert, A. H., Schnepel, O., Kraft, P., Houska, T., Plesca, I., Orlowski, N., and Breuer, L.: Studienlandschaft Schwingbachtal: an out-door full-scale learning tool newly equipped with augmented reality, Hydrol. Earth Syst. Sci. Discuss., 12, 11591–11611,, 2015.
Bilke, L., Fischer, T., Helbig, C. et al. Environ Earth Sci (2014) 72: 3881.
Billen, M. I., Kreylos, O., Hamann, B., Jadamec, M. A., Kellogg, L. H., Staadt, O., & Sumner, D. Y. (2008). A geoscience perspective on immersive 3D gridded data visualization. Computers & Geosciences, 34(9), 1056–1072.
Kromer, R. (2018). VGC2018, 1–101.
Helbig C, Bilke L, Bauer H-S, Böttinger M, Kolditz O (2015) MEVA - An Interactive Visualization Application for Validation of Multifaceted Meteorological Data with Multiple 3D Devices. PLoS ONE 10(4): e0123811.
Rink, K., Bilke, L., Kolditz, O., (2017) Setting up Virtual Geographic Environments in Unity, in Workshop on Visualisation in Environmental Sciences (EnvirVis), Rink K., Middel A. , Zeckzer D. and Bujack R. Eds., 978-3-03868-040-6
Karsten Rink, Cui Chen, Lars Bilke, Zhenliang Liao, Karsten Rinke, Marieke Frassl, Tianxiang Yue & Olaf Kolditz (2018) Virtual geographic environments for water pollution control, International Journal of Digital Earth, 11:4, 397-407,
Su, S., Cruz-Neira, C., Habib, E., & Gerndt, A. (2009). Virtual hydrology observatory: an immersive visualization of hydrology modeling. In I. E. McDowall & M. Dolinsky (Eds.), (Vol. 7238, pp. 72380H–9). Presented at the IS&T/SPIE Electronic Imaging, SPIE.
Yan C., Rink K., Bilke L., Nixdorf E., Yue T., Kolditz O. (2019) Virtual Geographical Environment-Based Environmental Information System for Poyang Lake Basin. In: Yue T. et al. (eds) Chinese Water Systems. Terrestrial Environmental Sciences. Springer, Cham

1 comment: