Hydrology 2020 lab

This is the material of the Laboratory Class of the Hydrology Class 2020. You can find the class material here. 

1: Programs Installation

• Installations (Zoom2020). For installations you can also see this Installation of the geoframe-vicenza environment with Anaconda (Zoom2020)
• Material to download: ex01.zip, geoframe_vicenza.yaml, TestNotebook.zip

 2: Installations problem solving

• FAQ  (Zoom2020)

• L - Reading an Excel File with PANDAS (Notebook, Data I, Data II, YouTube Live Video)

This is a placeholder for the material which is going to be uploaded

Hydrological Modelling 2020 Lab

Welcome to the Hydrological modelling lab class. The theoretical/conceptual lectures are here.

• Software Installations (Zoom2020)
• Where are my data (Zoom2020)
• A little of GIS 
• Cartografia tradizionale 
• An brief introduction to GIS (Zoom2020)
• A little about QGIS (Zoom2020 Part I, Zoom2020 Part II)
• Some maps to download
• Time Series
• Some time series of discharges treated can be seen here.
• Time Series in GEOframe
• Some material on the time series exercise 

About Hydrological Models

Starting from some philosophy I try to give an idea of what a scientific model is, on the example and limitations of the Galileian approach. I claim that hydrology is mostly a science where you have to move away from experiments and you have the deduct the models from observations more than experiments. This introduces some difficulties and arrive to the arrive to the conclusion that we do models of the data and models of the models of the data. Because of the presence of multiple phenomena interacting, hydrology gain a space in the science of complexity. But yet we do not know how to characterize it, if we do not use the classical mechanics and/or the usual thermodynamics. As a matter of facts, we have to deal with an entire zoo of models born with different scopes and evolved sometimes erratically.But at this point is where we do stand.

Please find the slides here, and the YouTube video where I comment them here.  I also makes a little summary of the talk here.
These slides can ideally be followed by those I used during the 2020 GEOframe Winter School:

• The GEOframe type of models (YouTube2020)
• Representing the Hydrological Dynamical Systems (YouTube2020)
• The Embedded Reservoir Model (YouTube2020)

Entrance Of The Rivers

Beloved of the rivers, beset

By azure water and transparent drops, 

Like a tree of veins your spectre 

Of dark goddess biting apples: 

And then awakening naked 

To be tattoed by the rivers, 

And in the wet heights your head 

Filled the world with new dew. 

Water rose to your waist,

You are made of wellsprings 

And lakes shone on your forehead. 

From your sources of density you drew 

Water like vital tears 

And hauled the riverbeds to the sand 

Across the planetary night, 

Crossing rough, dilated stone, 

Breaking down on the way 

All the salt of geology, 

Cutting through forests of compact walls 

Dislodging the muscles of quartz.

Poem by Pablo Neruda

I've see it in Muscles of Quartz blog by Sean Turner

About papers Authorship in Hydrology

Because many often collaborate to a part of a research, it is worthwhile to try to establish some good rule for authorships. A standard text is the one provided by The International Committee of  Medical Journals Editors (ICMJE). I think it gives some reasonable indication which can be adapted where necessary for Hydrological Sciences. Here below you will find verbatim their rules with comments in Italics for Hydrology.

1. Why Authorship Matters

Authorship confers credit and has important academic, social, and financial implications. Authorship also implies responsibility and accountability for published work. The following recommendations are intended to ensure that contributors who have made substantive intellectual contributions to a paper are given credit as authors, but also that contributors credited as authors understand their role in taking responsibility and being accountable for what is published.
Because authorship does not communicate what contributions qualified an individual to be an author, some journals now request and publish information about the contributions of each person named as having participated in a submitted study, at least for original research. Editors are strongly encouraged to develop and implement a contributorship policy. Such policies remove much of the ambiguity surrounding contributions, but leave unresolved the question of the quantity and quality of contribution that qualify an individual for authorship. The ICMJE has thus developed criteria for authorship that can be used by all journals, including those that distinguish authors from other contributors.

2. Who Is an Author?

The ICMJE recommends that authorship be based on the following 4 criteria:


• Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND
• Drafting the work or revising it critically for important intellectual content; AND
• Final approval of the version to be published; AND
• Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved

Comment: In case of Hydrology, besides the conception, the design and the data, there can be Software writing or Working out the mathematics (when they are present) which should be added to point (1).  However, please be careful to observe that when I say new “software writing”, I mean new software and when I say “Working out the mathematics”, I mean for the first time. We can call Software and Mathematics writers, “Developers”. Developers may reserve the right for the first scientific application of their schemes but not claim to be inserted in any paper using their method or software later on. The citation of the first paper should be  enough in most of the cases. Please also notice that Project Administration and Funding acquisition is usually not considered as a sufficient reason to be co-author.

Therefore for hydrologists, the points for claiming authorship could be:

1. Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; or software and mathematics development of the work; AND
2. Drafting the work or revising it critically for important intellectual content; AND
3. Final approval of the version to be published; AND
4. Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
5. Corollary: Developers may reserve the right for the first application of their scheme, as well as data providers can claim their first publication.

In addition to being accountable for the parts of the work he or she has done, an author should be able to identify which co-authors are responsible for specific other parts of the work. In addition, authors should have confidence in the integrity of the contributions of their co-authors.
All those designated as authors should meet all four criteria for authorship, and all who meet the four criteria should be identified as authors. Those who do not meet all four criteria should be acknowledged—see Section II.A.3 below. These authorship criteria are intended to reserve the status of authorship for those who deserve credit and can take responsibility for the work. The criteria are not intended for use as a means to disqualify colleagues from authorship who otherwise meet authorship criteria by denying them the opportunity to meet criterion #s 2 or 3. Therefore, all individuals who meet the first criterion should have the opportunity to participate in the review, drafting, and final approval of the manuscript.
The individuals who conduct the work are responsible for identifying who meets these criteria and ideally should do so when planning the work, making modifications as appropriate as the work progresses. We encourage collaboration and co-authorship with colleagues in the locations where the research is conducted. It is the collective responsibility of the authors, not the journal to which the work is submitted, to determine that all people named as authors meet all four criteria; it is not the role of journal editors to determine who qualifies or does not qualify for authorship or to arbitrate authorship conflicts. If agreement cannot be reached about who qualifies for authorship, the institution(s) where the work was performed, not the journal editor, should be asked to investigate. The criteria used to determine the order in which authors are listed on the byline may vary, and are to be decided collectively by the author group and not by editors. If authors request removal or addition of an author after manuscript submission or publication, journal editors should seek an explanation and signed statement of agreement for the requested change from all listed authors and from the author to be removed or added

Comment: The first Author should be the major contributor to the paper. It is a fair policy that Ph.D. student can claim first Authorship for the work coming out from their dissertation. In Hydrology (and in EU funding schemes) is also growing the habit to consider the last Author as the senior Author who possibly provided funding, the starting ideas and the supervision of the work (besides the other necessary requirements).

The corresponding author is the one individual who takes primary responsibility for communication with the journal during the manuscript submission, peer review, and publication process. The corresponding author typically ensures that all the journal’s administrative requirements, such as providing details of authorship, ethics committee approval, clinical trial registration documentation, and disclosures of relationships and activities are properly completed and reported, although these duties may be delegated to one or more coauthors. The corresponding author should be available throughout the submission and peer review process to respond to editorial queries in a timely way, and should be available after publication to respond to critiques of the work and cooperate with any requests from the journal for data or additional information should questions about the paper arise after publication. Although the corresponding author has primary responsibility for correspondence with the journal, the ICMJE recommends that editors send copies of all correspondence to all listed authors.
When a large multi-author group has conducted the work, the group ideally should decide who will be an author before the work is started and confirm who is an author before submitting the manuscript for publication. All members of the group named as authors should meet all four criteria for authorship, including approval of the final manuscript, and they should be able to take public responsibility for the work and should have full confidence in the accuracy and integrity of the work of other group authors. They will also be expected as individuals to complete disclosure forms.

Some large multi-author groups designate authorship by a group name, with or without the names of individuals. When submitting a manuscript authored by a group, the corresponding author should specify the group name if one exists, and clearly identify the group members who can take credit and responsibility for the work as authors. (....)

3. Non-Author Contributors

Contributors who meet fewer than all 4 of the above criteria for authorship should not be listed as authors, but they should be acknowledged. Examples of activities that alone (without other contributions) do not qualify a contributor for authorship are acquisition of funding; general supervision of a research group or general administrative support; and writing assistance, technical editing, language editing, and proofreading. Those whose contributions do not justify authorship may be acknowledged individually or together as a group under a single heading (e.g. "Clinical Investigators" or "Participating Investigators"), and their contributions should be specified (e.g., "served as scientific advisors," "critically reviewed the study proposal," "collected data," "provided and cared for study patients", "participated in writing or technical editing of the manuscript").
Because acknowledgment may imply endorsement by acknowledged individuals of a study’s data and conclusions, editors are advised to require that the corresponding author obtain written permission to be acknowledged from all acknowledged individuals.

Examples of Authorship recognition required by some Journals

1 - Conceptualization, M.B. and S.M.; Methodology, M.B., S.M. and R.R.; Software, M.B., S.M. and R.R.; Validation, M.B. and S.M.; Resources, S.M.; Data Curation, M.B.; Writing—Original Draft Preparation, M.B.; Writing—Review & Editing, M.B., S.M. and R.R.; Supervision, S.M.; Project Administration, S.M.; Funding Acquisition, S.M.

2 - M.B., G.F. and F.S. developed the model code integrated in the GEOframe-SIK 10 package. M.B., F.S., M.B. W.A. designed the experiments and performed the simulations. M.B. and R.R. prepared the manuscript with contributions from all coauthors. R.R. provided funding, supervision and methodology.

3 - S.F. and T.M. designed the study. J.P., R.R., B.S. and M.B. contributed with data. P.H. contributed to the optimization of the parallel simulations. T.M. performed the simulations and the analyses. T.M. designed the figures with contributions from G.M and S.F. The results were synthesized by T.M., S.F., C.P. and P.M. T.M. and S. F. wrote the manuscript with contributions from all other authors.

Is the scientific paper outdated ?

There is a provoking paper by Ryan Abernathey (GS) that appeared on  The Chronicle of Higher Education which is entitled: The Scientific paper is outdated. The Author explains why, in his opinion.

I read it and I fully agree with the arguments. I dedicated part of my last 25 year of research in producing good designed software and I know that I sacrificed more than a paper to this task. Fortunately journals like GMD or EM&S built a bridge between paper writing and software building. The Author realizes also that just publishing software should be enough in most of the cases, but he is probably unaware that now this is actually possible with Joos, a new type of "journal" which is also commented here.

4 Researchers Who Want to Develop Hydrological Models Without Reinventing the Wheel

Maybe you can save efforts and you do not need to reinvent the wheel. Therefore I wrote about our project that lasts since almost fifteen years. We use an open infrastructure, the Object Modelling System v. 3 and developed an open system, called GEOframe that has unmatched capabilities to be expanded while not wasting the old work done and maintaining your independence as developer.

The best way to understand OMS and see how it works is probably to give a look to the material of our last winter School on our system GEOframe. Specifically, this is the page of introduction to OMS. But also starting from installations of the material can give insights.
There you can find links to the OMS v3 pages at CSU and original material. Our choice of OMS and Java was long meditated. This was my assessment almost ten years ago. This is instead the assessment made by NIWA six years ago.

Here  you can find some thinking about the choice of programming  languages. We use, in practice, a mix of Python and Java. Python for treating and visualizing data. Java for writing our models.
When properly used Java is not so slow (twice slower of well written C++) and was favored by having an incredible suite of tools helping proper programming, project building and code maintenance that make the experience to develop in C++ a poor one. Python (specifically some of its libraries) and the Jupyter lab are instead a very nice experience for treating data and, besides, is supported by a vibrant community of developers, even among hydrologists, who really make the experience better and better everyday. Python is also becoming a “lingua franca” among scientists and this makes less difficult and steep to share results. I have a couple of pages to introduce Python to hydrologists that you can access starting from here. For what regards of Java one can start from here.
In any case Java is not mandatory for the use of OMS. Also FORTRAN and C++ codes can be used: NIWA did it with FORTRAN, but we never tried it.

OMS code is distributed with the MIT Open Source license. It is not available on a public repository though, but Olaf David, its chief architect, gave us access to it and does not have problem to give it to others. I recognize that this is a limitation but with more users, it would not be so difficult to get a branch of the source code in a public repository, I guess.
Our own source code is at the GEOframe components Github repository. The software is under GPL3 and the repository is open to anyone willing to collaborate.

The structure of the OMS allows to easily add components with any type of licence, the one more suited to you, and does not oblige you to release the software under the GPLv3 license.
Worth to mention is that OMS has a server side companion, CSIP of which you can find any information here.

Once Olaf David mentioned to me that the overall cost of OMS was closely 10 million of dollars. From my side I guess i invested around 1 million and more on developing GEOframe and its ancestors, Just to give a measure of the investment needed for such an enterprise. But building on top of them is now much less expensive and much more immediate.
Our current task are finishing the 1D-2D-3D solving of Richards equation (extended to treat groundwater and surface waters) and its coupling with the energy budget, evaporation and vegetation dynamics. This would replace our older GEOtop in two years, except for snow modelling for which we do not have started anything so far. We have also ongoing work for hillslope stability and a well established set of tools for doing lumped modelling, including the estimation of travel times and tracers/pollutants concentrations. Some other developments, more on the side of informatics were done by a former students of mine and are reported in his Ph.D. Dissertation. It contains also some work on neural networks.
For the future, an effort should be made to bring OMS to Java 9, which, I think, could make the code more understandable. Classes are a good thing but understanding the connection between them is what matters. Java 9 Modules should help to get the code more clean, and Gradle should do the rest for external libraries.  A few years go I started to dig into the code. I could not complete the task for my limited time but I believe one Java programmer can master OMS3 code in six intense months or so.  I also envision that, having a good project, OMS could be completely embedded in Jupyter lab making it usable through it making its use more familiar to a multitude of users. 

Obviously, we are willing to give any help, if you join us. I hope this information can help you to make your decisions.