Subsurface-Surface Water Flow and the Nested Newton Algorithm by Vincenzo Casulli

 On March 19th we had in Trento Martyn Clark (GS), Louise Arnal (GS) and Wouter Knoben (GS) to discuss about common research pathways. It was a dense day of discussions and exchange of ideas that started from the Mesiano terrace, continue inside, at the pizzeria Korallo, and later with an aperitif in Piazza Fiera and could have continued for days. You realize that the afternoon is missing. The afternoon was even better because Vincenzo Casulli (GS) gave a couple of talks about how to integrate subsurface and surface flow, and about his Nested Newton algorithm for solving a special class of non linear problems. Vincenzo agreed to record his talks and now you can appreciate them here below.  

Commanding OMS3 simulations from the Command Line (or from a Jupyter Notebook)

 The normal way to start OMS3 simulation is the use of its console. However, it could be convenient to command project simulations directly from the command line of a terminal (in Windows, in Mac OS, Linux users  know what a terminal is). 

The presentation above (click on the Figure) explain how to do it. Jupyter Notebooks can also be used as terminal. The presentation connects to a Notebook where it is done. 

Preparing for a CAMELS-Po dataset

CAMELS stand for Catchment Attributes and MEteorology for Large-sample studies. The clearest introduction is in Krazterk er al. It tells “Detailed datasets combining hydroclimatic time series, landscape attributes, and/or hydrological response variables like streamflow exist for many experimental catchments, in many cases spanning decades. ... In parallel, there also exist tens of thousands of gauges monitoring rivers across the world. .... Gupta et al. argued that large sample sizes allow for assessment of the generality of hydrological models and research findings. Large sample sizes also allow for large-scale research like detecting and attributing systematic shifts in terrestrial water availability at regional to global scales. ... Recognizing this has led to the development of a sub-discipline in the hydrological sciences called large-sample hydrology (LSH), which relies on data from hundreds to thousands of catchments”. (Please read the references for more information).

The initiative has also the scope to enable multiple approaches on catchments studies that were in the past often too much limited to single case studies with poor ability to get a satisfying generality. In a sense, the CAMELS initiative pursues some of the same objectives that the global hydrology trend, where a cohort of researchers aims to cover the whole Earth catchments with simulations often with high resolution models (Wood et al., 2011 ). The latter trials, whilst providing ubiquitous hydrological forecast are scanty in providing accurate local estimates for lack of information and characterization (as well as some Earth observation/data driven global solutions). It is easy then to foresee that there will be soon a new effort in doing large simulation on hyper-resolved catchments with these new ground-based datasets that will open and reanalysis era in catchment hydrology encompassing scales, regions and climates.

The initiative of the Po river project, supported by the River Po basin Authority goes in this direction, in the limited but extremely complicated are of the largest basin in Italy. As a preliminary effort we we inaugurate the CAMEL-Po actions. What the CAMELS-Po action aim is to serve several objectives:

_____________________________________________________________________________________

TASK 1 - SHAREABLE CATCHMENTS DELINEATION AND GEOMORPHOLOGY

1.1 - to give a common and shareable watershed delineation of the Po river at a scale of 10 square kilometers, eventually  to be refined at 5 kilometer an to the hillslope extents in future passages (obtained with open source softwares Horton Machine)

1.2- Include major lakes and reservoirs in this delineation

1.3- to estimate some common geomorphic quantities to marks watershed characteristics and their spatial variations (as already standard in the CAMELS datasets)

TASK 2 - OPEN HYDROMETEOROLOGICAL REANALYSIS BASED ON INTERPOLATION OF AVAILABLE GROUND DATA AND SUBSEQUENT IMPROVEMENTS

2.1 - to build a solid restitution of the last 30 years of hydro-meteorologic data over the catchments, initially with one point  per catchment, eventually over a standard regular grid, i.e. https://land.copernicus.eu/pan-european/high-resolution-layers, and eventually make it public (including all the quantities available as ground data)

2.3 -Providing a re-analysis of such data at very high resolution blending various source of information, like weather reanalysis, radar and EARTH observation datasets

TASK 3 - HUG WITH EARTH OBSERVATIONS

3.1 Providing facilitate access to EO and other data catchment by catchments in the logic of the Digital Earth Twins of Hydrology (DARTHs)

TASK 4 - OPEN SNOW

3.2 Providing a reanalysis of the snow cover over the whole Po during the last 30 years by using ground observations, GEOtop and GEOframe at 1 km scale

_____________________________________________________________________________________

These objective are a minimal part of the Po project, which is more ambitious and involve several modelling tasks of the various quantities and several scientific investigations.

For who interested in these follow ups, stay tuned. For who interested in the logic of CAMEL sets, see the references below.

Cited and non cited References

• Addor, Nans, Andrew J. Newman, Naoki Mizukami, and Martyn P. Clark. 2017. “The CAMELS Data Set: Catchment Attributes and Meteorology for Large-Sample Studies.” Hydrology and Earth System Sciences 21 (10): 5293–5313. https://doi.org/10.5194/hess-21-5293-2017.
• Alvarez-Garreton, Camila, Juan Pablo Boisier, René Garreaud, Jan Seibert, and Marc Vis. 2021. “Progressive Water Deficits during Multiyear Droughts in Basins with Long Hydrological Memory in Chile.” Hydrology and Earth System Sciences 25 (1): 429–46. https://doi.org/10.5194/hess-25-429-2021.
• Alvarez-Garreton, and Mendoza. 2018. “The CAMELS-CL Dataset: Catchment Attributes and Meteorology for Large Sample studies–Chile Dataset.” Hydrology and Earth System Sciences. https://hess.copernicus.org/articles/22/5817/2018/.
• Chagas, Vinícius B. P., Pedro L. B. Chaffe, Nans Addor, Fernando M. Fan, Ayan S. Fleischmann, Rodrigo C. D. Paiva, and Vinícius A. Siqueira. 2020. “CAMELS-BR: Hydrometeorological Time Series and Landscape Attributes for 897 Catchments in Brazil.” Earth System Science Data 12 (3): 2075–96. https://doi.org/10.5194/essd-12-2075-2020.
• Coxon, Gemma, Nans Addor, John P. Bloomfield, Jim Freer, Matt Fry, Jamie Hannaford, Nicholas J. K. Howden, et al. 2020. “CAMELS-GB: Hydrometeorological Time Series and Landscape Attributes for 671 Catchments in Great Britain.” Earth System Science Data 12 (4): 2459–83. https://doi.org/10.5194/essd-12-2459-2020.
• Delaigue, and Brigode. 2022. “CAMELS-FR: A Large Sample Hydroclimatic Dataset for France to Explore Hydrological Diversity and Support Model Benchmarking.” IAHS-2022. https://meetingorganizer.copernicus.org/IAHS2022/IAHS2022-521.html?pdf.
• Fowler, Keirnan J. A., Suwash Chandra Acharya, Nans Addor, Chihchung Chou, and Murray C. Peel. 2021. “CAMELS-AUS: Hydrometeorological Time Series and Landscape Attributes for 222 Catchments in Australia.” Earth System Science Data 13 (8): 3847–67. https://doi.org/10.5194/essd-13-3847-2021.
• Hao, Jin, Xia, Tian, and Yang. 2021. “Catchment Attributes and Meteorology for Large Sample Study in Contiguous China.” Earth System Dynamics. https://scholar.archive.org/work/fn4n5jgzlreyjfvjna2fklyo5a/access/wayback/https://essd.copernicus.org/preprints/essd-2021-71/essd-2021-71.pdf.
• Jehn, Florian Ulrich, Konrad Bestian, Lutz Breuer, Philipp Kraft, and Tobias Houska. 2019. “Clustering CAMELS Using Hydrological Signatures with High Spatial Predictability.” Hydrology and Earth System Sciences Discussions, April, 1–21. https://doi.org/10.5194/hess-2019-129.
• Klingler, Christoph, Karsten Schulz, and Mathew Herrnegger. 2021. “LamaH-CE: LArge-SaMple DAta for Hydrology and Environmental Sciences for Central Europe.” Earth System Science Data 13 (9): 4529–65. https://doi.org/10.5194/essd-13-4529-2021.
• Knoben, Wouter, and Martyn Clark. 2022. “CAMELS-Spat: Catchment Data for Spatially Distributed Large-Sample Hydrology.” Copernicus Meetings. https://doi.org/10.5194/egusphere-egu22-6609.
• Kratzert, Frederik, Daniel Klotz, Claire Brenner, Karsten Schulz, and Mathew Herrnegger. 2018. “Rainfall–runoff Modelling Using Long Short-Term Memory (LSTM) Networks.” Hydrology and Earth System Sciences 22 (11): 6005–22. https://doi.org/10.5194/hess-22-6005-2018.
• Kratzert, Frederik, Daniel Klotz, Guy Shalev, Günter Klambauer, Sepp Hochreiter, and Grey Nearing. 2019. “Towards Learning Universal, Regional, and Local Hydrological Behaviors via Machine Learning Applied to Large-Sample Datasets.” Hydrology and Earth System Sciences 23 (12): 5089–5110. https://doi.org/10.5194/hess-23-5089-2019.
• Ma, Kai, Dapeng Feng, Kathryn Lawson, Wen-Ping Tsai, Chuan Liang, Xiaorong Huang, Ashutosh Sharma, and Chaopeng Shen. 2021. “Transferring Hydrologic Data across Continents – Leveraging Data‐rich Regions to Improve Hydrologic Prediction in Data‐sparse Regions.” Water Resources Research 57 (5). https://doi.org/10.1029/2020wr028600.
• Siber, Rosi, Marvin Höge, Martina Kauzlaric, Ursula Schönenberger, Pascal Horton, Jan Schwanbeck, Daniel Viviroli, et al. 2022. “CAMELS-CH - Building a Common Open Database for Catchments in Switzerland.” https://doi.org/10.5194/egusphere-egu22-9859.
• Sterle, Perdrial, Li, and Adler. 2022. “CAMELS-Chem: Augmenting CAMELS (Catchment Attributes and Meteorology for Large-Sample Studies) with Atmospheric and Stream Water Chemistry Data.” Hydrology and Earth System Sciences. https://hess.copernicus.org/preprints/hess-2022-81/.
• Vasquez, Cepeda, and Gomez. 2019. “Exploring the Relation between Meteorological, Physiographic and Hydrological Similarities through Catchment Classification.” Geophysical and Astrophysical Fluid Dynamics. https://search.ebscohost.com/login.aspx?direct=true&profile=ehost&scope=site&authtype=crawler&jrnl=10297006&AN=140493270&h=NKYXshyG%2Bbu%2FK%2FLUfFm%2FEMY20NhOmBgJvpEtFInXc7Jkq9BHvL5fOilpNzcqVPISF9tBXQDvKfqoPbel4FIfNQ%3D%3D&crl=c.
• Wood, Eric F., Joshua K. Roundy, Tara J. Troy, L. P. H. van Beek, Marc F. P. Bierkens, Eleanor Blyth, Ad de Roo, et al. 2011. “Hyperresolution Global Land Surface Modeling: Meeting a Grand Challenge for Monitoring Earth’s Terrestrial Water: OPINION.” Water Resources Research 47 (5): 54. https://doi.org/10.1029/2010WR010090.

Just questions on celerity

 “Celerity, broadly speaking, is the rate additional potential energy supplied by water inputs propagates through the landscape and ultimately does the work of mobilizing water into the stream and out of the watershed. Thus, celerity determines how quickly the stream responds to the storm and how long it takes for discharge to decline after the storm. Velocity, on the other hand, determines how long it takes water molecules to traverse the watershed and so controls the age composition of discharge.” (Harman, 2019).

However, in Richardson-Richards equation, celerity and velocity are tightly connected by the soil water retention curves. So much water, so much pressure. Therefore:

Are SWRc to be modified to account for pressure transmission through the medium ? No pressure transmission is allowed in vadose water phase discontinuous medium ? Pressure transmission is possible only in saturated soil ?

Reference

Harman, C. J. 2019. “Age ranked Storage discharge Relations: A Unified Description of Spatially Lumped Flow and Water Age in Hydrologic Systems.” Water Resources Research 55 (8): 7143–65. https://doi.org/10.1029/2017wr022304.

WHETGEO-2D an open source tool for solving the 2D Richards equation

For new readers, WHETGEO stands for Water HEat and Transport in GEOframe. It is a set of models that deals with the processes in soils. The paper describes the 2D version of it, at least its initial deployment, since the system was built on the idea of expandability. The paper and its abundant supporting material a a step towards a series of further analyses of  the water behaviour in the vadose and saturated zones.

DeAnn Presley painting (https://www.agronomy.k-state.edu/about/people/faculty/presley-deann/)

The paper was submitted to environmental Modelling and Software and can be found  here.  Extensive material and video about WHETGEO 2D can be found among the material of the GEOframe Summer School. 

Nobody applied to this call ! Reactions by Solomon Vimal ( How to Mobilize Top Tech Talent Towards Hydrology)

In a blog post, followed by a list-serv email, Prof. Rigon exclaimed,  “Digital Hydrologists from around the world unite! Nobody applied for this call”! His project focuses on building digital earth twins for the hydrologic cycle using advanced cyber infrastructure and software architecture solutions that are unusual in the field of hydrology in most hydrology groups around the world. He added, “Klemeš, V. (1986) reflected on dilettantism in Hydrology [... we cannot] manage complex interactions and feedback with poorly based informatics”. Prof. Rigon’s outcry indeed makes one think: “Ehi, out there, is anybody interested? Anybody interested in a PhD in this topic?”

It prompted me (Salomon Vimal, see below note by rr) to think -- what can be done to attract top informatics talent into hydrology?

Asterisks marks rr notes that you can find at the bottom.

Hydrology is a tiny field. We are a small sub-field of civil engineering. Among say 100 students who leave high school, conservatively, perhaps 1 in every 3-5 (maybe more!) years of graduation will reach the field of hydrology if lucky, and often by chance. This is an estimate on the basis that nobody in my circles from high school even knows what hydrology is. A subset of them may sign up to list-servs. A small subset (say 1/10) of professors would forward it to interested students in relevant MS programs. The incentives of a PhD salary in Italy, especially $ *, is relatively small compared to what most people with above average skills in tech industry get, so this filters a good portion of them out. Taken together with the fact that 95% of good software tech is developed by only a tiny fraction of the top percentile of the coders (more on this below), it is important to look for the tech talent with a different strategy to attract the cream-dela-creme. These call for a closer look at the tech talent pool and incentives to have them move to the small field **.

Let’s look at Stack Overflow’s (SO) skilled pool of talent. Let's assume SO is a good representative slice of the world's tech talent (which it indeed is). Some observations based on data on distribution of tech talent that can be drawn from Stack Overflow data queries at data.stackexchange.com. It may be helpful to look at how many people, with what skill level, contribution, and reputation are out there. There are 18M total SO users (see the query here). 95% of all activity is by 1M people with a 80 reputation. 80% of all activity is by 0.3M people with 850 reputation. 70% of all activity is by 100K people with 2K reputation. 50% of all activity is by 37K users with 7K reputation. The query example here can be modified to examine percentiles. But the point made here is simple. Most of the work comes from a very small fraction of programmers. For reference, I asked as well as answered 17 questions (a balanced karma) and only a few of my questions were actually appreciated and this puts my reputation score at ~880 (Aug, 2022) which accrued over 5 years or so, and I fall into the bucket of the top 0.3M users, i.e. top 1.6 percentile (by score) reputed users who make 80% of all contributions. My score is nothing compared to the top 50-100K users. While I generally feel comfortable to say that I am good with programming (with over 10 years of experience in Python), I certainly would not apply for the informatics job above, as I do not think I have sufficient skills to start contributing to the informatics/architectural aspect of the project without proper training in CS/IT. I would perhaps defer this position to someone with a score of >2-5K in SO or a degree in CS/IT. Using the query engine and the SQL language offered in Stack Data, one may try to find good metrics like rising stars on SO using one of the many threads on such metrics and pick the good ones to hire from such lists and write to them individually rather than wait for them to find the PhD job post***.

We may consider joint-specializations in PhD: In my previous company (RMS.com), flood models were built using GPU computing. Most of the hydrologists were unable to program with the GPU, but it was indeed developed in academia in a PhD work by someone at Penn State who had a PhD major in hydrology and minor in informatics — computational science, if I remember correctly. Most PhD programs do not offer such minor course work based specialization. Maybe universities can team up with the CS departments and offer a joint PhD in CS - this could be a solution to be attractive to a broader audience. I have no idea about the ground realities of bureaucracy that may impede such a solution, but I recall a double PhD in river science was offered at U. of Trento some years ago. ****

Let’s not forget the blackhole which attracts and retains top talent: Google alone has ~27K engineers (here, estimated for 2022) a good majority of whom are on SO, like the top user. The bulk of the good ones who do 50% of all activities are perhaps already taken by such Big Tech companies so PhD programs are not left with any good ones who have high reputation in SO. I know that Big Tech companies do have a similar approach to identifying top tech talent. If one prefers to do hiring in the more traditional way, one may write to CS department heads in some select universities in global cities that are in countries that would typically consider a move to Italy for PhD.

Consider the places where the bulk of tech talent comes from: The bulk of the top CS talent is arguably in US/China - clearly leaders in tech companies, products and talent - most of them perhaps won't move to Italy for PhD. This basically rules out many top university graduates. We are then left with some of the good institutes in other countries well-known for their contribution to global tech talent. India, for example, has a very strong IT workforce. But, it is better to focus on cities rather than countries in the case of CS expertise. In my experience cities that are Silicon Valley Like (SVL) in various countries might be good places to look. If I were to hire tech talent, I would simply look in the developing SVLs such as Tel Aviv, Bangalore and the NCR Region of India -- which certainly have a very strong talent pool in CS, perhaps not comparable to Bay Area or Shenzhen. I do not know the equivalent SVs of other countries but numerous articles like this one exist which enlist them. 

Finally, salaries: Horton wrote about a related issue in hydrologic research (Horton, 1937): "Men of highest degree of knowledge and experience in hydrologic matters are not likely to be employed by government salaries". The cream-dela-creme of the SVLs worldover will get salaries that will be well beyond the incentives of the PhD. In such a scenario, it may be sensible to adopt a different hiring approach and reach out to CS departments of universities strategically and identify candidates who have a demonstrable great level of skill in informatics which we are interested in and fully incentivize them to move to hydrology. For example CS graduates with say 3-5 years of experience from such SVLs may be interested in moving to Italy for the international experience and the good (alternate) life experience, even if the salary cannot per se incentivize a move. *****

It is indeed important for the field of hydrology to thrive with good software personnel who are also sought in all fields of science and all other domains of human pursuit. I hope we all agree this is a core problem with hiring good IT talent. Hydrologists need to hire top IT talent comparable with Big Tech companies to truly create good software as most of the good stuff comes from a small portion of skilled coders. It may be a good idea to give such hiring strategies a careful consideration. I hope my rambling is useful to hydrologists to reflect and find a good solution or hiring strategy.

Solomon Vimal

Solomon Vimal is an avid reader of the About Hydrology blog since 2011. He completed a PhD in hydrology in 2022 from University of California, Los Angeles on the topic, “Climate Change Impacts on Millions of Lakes”. He is now the founder and CEO of Geothara, a tech company currently incubated as a startup company at UCLA’s summer accelerator program and part of the 2022 cohort of the Runway Postdoc program at Cornell Tech in New York City. 

About Geothara: Geothara offers geographic change detection and visualization software to help its users explore the changing world using satellite images and a novel, up to 30% more accurate trend detection technique via a patent-pending technology that enables smarter location-specific decisions.

Comments by rr

* However, I would make not just a point of salary. The salary of a PhD student is not high but enough to live in Italy, if you are a young guy. Obviously you expect that the sacrifice bring some benefits. I cannot judge myself, but I believe that you can learn something from studying in Trento that you cannot learn elsewhere.

** small field (if we restrict our focus on digital hydrology) but important for life on Earth and for any community if we think that all is about water availability. After all everything which is alive requires water.

*** when we have our son, I felt somewhat inadequate to the role of father. Then I simply did (and I am doing) it. Never the fear should to succeed to stop you. Try ! PhD is a learning process.

**** That degree would be great to be set up but the start it is subject to the uncertainties and foresight of academic policies

***** unfortunately in Italy the salary is established once forever by the state (ok, maybe the single Universities could give some incentives). I will investigate what is possible.

Nobody Applied to this call ! Follow up by Keith Beven

 Keith Beven (GS), answered to my previous post and I think it is relevant to bring to the knowledge of everyone his point of view: 

"Dear Riccardo,

 

This is clearly something I have been interested in for a long time – in fact I was writing about models of everywhere long before digital twins became the parlance to use.

 

I attach some papers that you have not cited but would seem to be relevant. "

He also added in a second mail:

"I should perhaps have added that if you look at those papers you will see that I think there is one really important innovation associated with models of everywhere that carries over to the digital twins (where the latter is applied at appropriate scales) and that is the use of visualisations in testing models as hypotheses – including the use of information from local stakeholders.    You gain little from applying models at hyperresolution when hyperresolution is 1km for example (see the hyperresolution ignorance paper).   "

From Keith's "Still Dynamics" book (click on the image for the book)

Here the paper he cited:

Beven, Keith J., and Ruth E. Alcock. 2012. “Modelling Everything Everywhere: A New Approach to Decision-Making for Water Management under Uncertainty.” Freshwater Biology. https://doi.org/10.1111/j.1365-2427.2011.02592.x.

Beven, Keith, Hannah Cloke, Florian Pappenberger, Rob Lamb, and Neil Hunter. 2015. “Hyperresolution Information and Hyperresolution Ignorance in Modelling the Hydrology of the Land Surface.” SCIENCE CHINA Earth Sciences 58 (1): 25–35. https://doi.org/10.1007/s11430-014-5003-4.

Beven, Keith. 2019. “How to Make Advances in Hydrological Modelling.” Hydrology Research 50 (6): 1481–94. https://doi.org/10.2166/nh.2019.134.

Blair, Gordon S., Keith Beven, Rob Lamb, Richard Bassett, Kris Cauwenberghs, Barry Hankin, Graham Dean, et al. 2019. “Models of Everywhere Revisited: A Technological Perspective.” Environmental Modelling & Software 122 (December): 104521. https://doi.org/10.1016/j.envsoft.2019.104521.

Panta rhei is new picture book by Keith, while Still Dynamics is only available on pdf now. 

P.S. - On the same topic we recently submitted a paper to HESSD:

Rigon, Riccardo, Giuseppe Formetta, Marialaura Bancheri, Niccolò Tubini, Concetta D’Amato, Olaf David, and Christian Massari. 2022. “HESS Opinions: Participatory Digital Earth Twin Hydrology Systems (DARTHs) for Everyone: A Blueprint for Hydrologists.” Hydrology and Earth System Sciences Discussions, 1–38.  Updated manuscript here