Why Excel is Evil

Why Excel is Evil is not a campaign against a commercial product. Instead, it is an analysis of the requirement for having a solid scientific product.  Thanks to Colin Caprani for this valuable contribution.

EGU Wien 2019: Doing open science in practical hydrological modeling

Yes, we are doing open science. At least we try to do it. I gave some talks about Replicable Research, I put some posts, and I repeat something here. However there are some few practical things that needs to be further presented, and that excites us. Here they are:

The presentation at least make a list of them, and give reference to more information. Click on the Figure for the presentation. Enjoy.

Open Science

Nothing really original in this post. I just recollect what already said in the FosterOpenScience web pages. Their definition is:

"Open Science represents a new approach to the scientific process based on cooperative work and new ways of diffusing knowledge by using digital technologies and new collaborative tools (European Commission, 2016b:33). The OECD defines Open Science as: “to make the primary outputs of publicly funded research results – publications and the research data – publicly accessible in digital format with no or minimal restriction” (OECD, 2015:7), but it is more than that. Open Science is about extending the principles of openness to the whole research cycle (see figure 1), fostering sharing and collaboration as early as possible thus entailing a systemic change to the way science and research is done." The wikipedia page is also useful.

This approach get along with the one of doing reproducible research which I already talked about several times. I do not have very much to add to what they wrote, but I also want to make you note that "there are in fact multiple approaches to the term and definition of Open Science, that Fecher and Friesike (2014) have synthesized and structured by proposing five Open Science schools of thought" .

In our work a basic assumption is that openness require also the appropriate tools, and we are working hard to produce them and use those other that make a scientific workflow open.

The tale of open source codes

Prologue

Why did I choose to produce with the people directly working with me (ph.D students, master students, postdocs) open source software ?

- because is good for science
- because I am paid by a public institution
- because it is a neutral conditions that can serve the rights of all the participants (in particular mine of freely use and modify the software at my will an defend myself from who, people or institution, would like to close the software, even against me). On the other side,  my intention is clearly that my projects serve as a seed for developments of my students (or others) who can freely use the products of my research and maintaining it alive beyond me and despite me. *

I use GPL (for its interpretation, see here) but many others licences could work.

A declaration

In this way, I think, I have the right to claim to be able to use or peruse the software outcomes from my group. I declare that I want to use “fair play” rules, but, it should be clear that these rules cannot extend to limit my research freedom. People who claim the participation to papers where they give no contribution, except having producing the code that we produced together, have wrong arguments. People who claim to be involved in projects or researchers, without any other reason that I want to use the software they contribute (under GPL), have wrong arguments.
Neither they can claim that I have to warn and tell them personally what I am going to do in my research with the common code, for having their consent.
They would be right to protest, only if I would not enlighten their contribution on previous work properly.
My research for my own belief is actually very public and its evolution too. It can be found at the abouthydrology blog. My core research is shared with my teamwork. This includes just the people of whom I have direct responsibility for age and rule (Master students, Ph.D. students and postdocs) and whom I sustain with funding, my own time and ideas.

With all others, including my masters, and my former students, colleagues, friends, women and men that like my research topics and achievements, and me, I can have collaborations. This means that we can share part of our views, beliefs, discussions, fightings, friendship, papers, parts of code. However our own agendas, in this imperfect world, do not coincide, and if they do, this happens for an incredibly short time. It seems it is a declaration of distance, but it is just consciousness of how life works, and the first step to start an effective and respectful collaboration.

Q&A

Can my students refuse to develop OS software ?
No, as soon as it is the product of common intellectual efforts in which they, maybe, write the code, but I will say what to write.

Do I start collaborations in which not OS code can be developed ?
Never say never. However there should be very strong reasons because, from my side, I to support this. Certainly in projects there could be partners that develop non open source software, but this falls in the responsibilities of who gives the financial support.

Is the requirement of open sourceness enough ?
No, it isn't. Open Sourceness is useless if not followed by good practices of using open repositories and collaborative modalities of action.

Can my students refuse to learn these practices ?
For the common work no. I am not responsible for the rest. I tend to fully book their time, though.

Do I start collaborations where these practices are not followed?
I would prefer not, but I do. Certainly collaborations can be at different levels and rarely they are about co-producing software. I would not participate to joint projects where I put ideas and expertise and others write closed codes, unless they pay me or my group a lot. Really a lot. I can participate to projects where other subjects put their ideas, or ideas from literature, in their own closed code, and I put mine in OS codes. However, the situation I prefer would be a common production, as a community, of open source codes.

My own use cases

Here below I summarised (with quite large simplifications) my software history in order to further justify what I wrote above.

Professor means who puts science, time, and money (funds derived by projects). Student means who puts time and science. Companies means they put time and money and business related efforts. Agents, Subjects are generic actors of the play (they can be either students, professors or someone else). Community is the informal group that happened to gather around the projects and, eventually,  evolve them.

Case 0

Professor Z writes the initial library. On top of that A builds radiation budget. Student B writes surface water flows. Student C implements soil-atmosfere interactions. Students D writes vadose zone components. Student E writes snow treatment. Student F rewrites snow components, then rewrites most of the codes interacting with student G and student H. Student I writes codes for landslides triggering treatment. Student G writes a small but important portion of the a little but successful part of the freezing soil hydrology. Professor L hires F. F continues to rewrite parts. G start a huge operation of cleaning the code, moving it to C++, uploading it to an open repository. C comes back and starts to use the code in his research and occasionally hires H to do some ancillary work for treating data. In meantime G  has founded a company where the common code is the basis of the business. M company, initially hired by L, works on the code to refactor and enhance it. M works collaboratively with G and F. M to setup continuos integration. Student N starts to produce executables for the main operating systems and eventually on Cooker (fictional name). M embraces immediately this philosophy.


Case 1

Professor Z writes the initial library. Z writes more than fifty tools for terrain analysis. Student A (not the same as above) ports them to a major Open Source GIS. Z and A start the construction of a new GIS, say JG. Initially JG contains just the the terrain analysis tools and some simple hydrological model. They start to do schools for financing their project. This works for some years. Student B, in the meanwhile, has joined the crew and A & B funded the company AB. They live with schools, supports from a main project of Z and other resources (a main research projects). A cleans the tools' suite and inaugurates the name JGT for them. Z uses JGT in his classes.
Z, A, and B decided to join the development of UGIS. Some research projects supports them together with resources raised by the company AB on its own. Students C and students D write some further modules. UGIS  funding disappears, and UGIS slowly becomes an almost inactive project. AB brings JGT to an intermediate product ST. Z continues to USE JGT in his classes. AB finally joins the development of a new GIS, say GS.
(In the middle,  A adds new tools, AB wrote an Android app, Aapp, and expands its business. Aapp is not  related to JGT, but worth to mention). During the years A and B get a Ph.D. whose topics are related to the GIS work. Student E with a small effort brings back JGT also to a platform, OI that Z uses with his students.

Case 2

Thanks to an unexpected financial support from project 00, Professor Z hires a five students to build from the scratch a new modelling platform. For this new software enterprise, he and company AB (funded by his former students A and B) chooses the open source framework OI. He hires former student C, to help software developments and former student D and E for the general management of the project and data gathering, respectively. C works more on improving and enriching JGT (see case 1 above) which serves as a basis for the terrain analysis functional to modeling. A and B develop a full suite of model components (the new paradigm) for: temperature and rainfall interpolation, rainfall-runoff, evapotranspiration and various tools to visualise components' inputs and outputs. AB also designs and populates an SQL database that contains all the data of the projects. The projects 00 ends. The Institution that supported the project close it in a drawer.

With other financial support, former 00 project's tools are maintained in life. Open source framework OI is changed for open source framework OM with a notable reduction of code lines (but it is a huge code effort, indeed, almost entirely on AB shoulders). With embracing OM, also starts a research collaboration with professor U and W.

Student E comes into play. He realises that rainfall-runoff does not work well. AB company has to survive on its own and cannot give very much support (https://vimeo.com/144089061). E implements a new rainfall-runoff model. AB, however, hires F for a small project where he works on radiation. Eventually, E refactors F's work and highly expands it. E adds a new snow modelling component and does/refactors evapotranspiration. In doing this (pouring sweat and blood) he, however, has the guidelines of the open sources codes already written. E spends some periods at U and W. E also refactors and enhances the Kriging code. Eventually E graduates and starts his career as post-doc elsewhere. In the meanwhile he finalises his research in a series of papers.
Student G comes. He does not have programming skills, but quietly learns to use the components of E and produces some interesting papers where E is co-author.
A new student, H, comes into the game. She works first on radiation on top of E code, then she starts to implement tools for travel time analysis and another rainfall-runoff component.
Student L comes. He  has a strong attitude for informatics. He brings-in new ways to manage projects. H and L implement the OpenOpenSoftware repository, and the site BeatifulGEO (names are fictional, but tools real). H refactors the old code,  and together with L (who, sort of, leads the learning process), introduces design patterns for increasing code reusability. L provides the trickery to have continuous integration on OpenOpenSoftware using GETIT and connects software deployment to ISTOREIT to store official versions of the components. Students M and N come in to stage and start to use the code. Professor Z (with the help of H) starts to use the components with his students for his classes. Student L evolves the original OM capabilities to allow for more flexibility and to increase the computational power of the models. H brings-in her models into the new infrastructure.

Discussion and Conclusions

The above is a summary (where, I say again, I simplified many passages) of my main software enterprises. Could have they been evolved all differently (and better) if I would not have applied an open source strategy ? Probably yes, but I should have constrained the students to a contract about the property of the software. In this way I would have deprived my students of parts of their own work.
At the same time, I could not have left the software simply to them. The histories themselves show that I built my own work and research on the software we develop, and being free to use it and modifying it was a necessity. If I have needed to ask permission to use it, to sign a contract or so with someone (for instance who gave financial support), all the development would have been much more difficult to pursue. The same apply for other Actors who invested time and resources in the software development just because it was open. They are usually singles or low budget companies that could not have afforded expenses related to other type of licenses and be subjected to limitation of the software use.
Other researchers used the model. Being it free and open source was a clearly an added value for them.

Keeping the software close and commercial, besides not having scientific reasons (which require the contrary), would have obliged me to change myself in a businessman and turned away from my science. There are several cases of scientists that turned to captains of companies. But, for instance Stephan Wolfram, a gifted scientist, did not give very much contributions to science after he devoted his energies to Mathematica. Mathematica (probably the best computing environment ever) itself is his main achievement (which is not depreciable), despite his own claims on "New kind of Science"s.
The overburden required for managing a commercial software is not for all and has its own dynamics, that personally I  could not bear.

The fact that my code is free and open source has allowed (not without difficulties) self-instruction of new incomers. Various Agents had the possibility to start experiments and investigate new directions of development. Nobody needed to ask for starting them. Asking is a process that would have decreased dramatically people or groups pro-activity.

The Community had benefits from this policy. In some cases, single Actors could have thought that their contribution was not recognised enough and did not give to them an advantage. Their argument is  flawed. All of them had advantages from the collaborative environment and nobody (me included) could have produced what s/he has achieved without building on the shoulder of others and other open source projects.

Forgetting the above, some feel that their work is not enough protected, and being all open source, newcomers can more easily jump in and take advantage of their work.
Uncertainty on future, a competitive society, the pure necessity to find something that pays you for a decent life incline even to bests to a moderate selfishness or a moderate parasitic behavior. They do not want to give back to the community, after having got a lot from it, and act defensively.

Well, this behavior is absolutely possible if their developments do not use the original code that was produced as GPL. In particular, the components strategy used in project 2 above allows for building on top of the open source material new, undisclosed material, that anyone can use for his/her own profit, with a non open license.
I have to warn, however, that if the moderate parasitism grows too much, enthusiasm that is always necessary decreases,  the projects die, the source of benefits disappears and the community falls.

I would say that a mild parasitism is functional to the community if it is necessary to sustain the collaborative Subjects, and if eventually the Subjects give something back to the community. Parasitic Subjects themselves act in favour of the community by spreading and advertising the products, and sooner or later this will be bring benefits back (so do not blame them, they are, in any case, part of the stream).

Some Subjects actually wants an opaque management of the GPL philosophy in which people maintain an informal (but they pretend recognized) property of the software that goes beyond the copyleft and the intellectual recognition of their contribution. This would imply, in their mind: preferential redirection of funds towards them; involvement in papers or conferences contributions that use their code; veto power towards actions of thirds.
These desiderata are based on misunderstandings. It is clear that they will be involved in papers, conference, and decision. Any (wo)man and community of good-will will apply this policy in their favour, if they do not grow too greedy. But these actions are not mandatory and not even necessary. GPL does not implies them.

To be more clear, especially in hydrology, the market out there treats our model and softwares as a fungible commodity, that is, the market tends to treat all the codes as equivalent or nearly so with no regard to who produced them. (I think this is wrong, highly wrong, when brought to an excess).
But also the internal market, inside the community, treats them as commodities, meaning that, it would be dysfunctional, it would cause a waste of precious time, but any contribution is perceived as a thing that can be replaced (this is part of the not said history of 0,1,and 2 projects). Everybody is important nobody is necessary.


The A. paradox

One common argument of reluctant open sourceres  is: “I did not have still tapped the results of my own work and I should share it (statement 1)”, or "if I share it, others will use it without me and I will have no personal gain(statement 2)".
The first danger can be overcome, by an appropriate delay of the disclosure of documentation and explanatory material (I would not argue that keeping industrial secrets is useless, in general, however). That is: it is matter of having strategies that prevent the negative cases. In our field, however, being everything perceived as a commodity (see above) nobody will care to use our model or achievement instead than another one that gives what is (wrongly perceived) as similar, especially if our code is not known. Being open source with proper support actions helps model spreading.
Besides, looking at my histories (see also here), software changes fast and is, by no means, immutable. Histories 0,1, and 2 are signed by change. So the advantage one has with a new code in hands is ephemeral. In my own estimates you have just a a year of advantage for small codes, and a few years of advantage with a large and complex code. This small advantage, if you are smart, can be appropriately managed and used to produce new and more innovative code and so on. (Open sourceness is against stagnation).
Often, however, it is not the the fear of far away threats that makes problems, but the fear of close by Agents. Guy A fear that B in the group who came in after her/him, will get positions or funding with his/her work. I would say that this could happen but it is difficult. In a fair (not fear) competition A always wins over B, if the quality of B can just be attributed to codes that A developed. The real problem is when B is much better that A. But in that case, having A work for B is not important. B will get rewards instead than A almost always. For A, the best thing, in the medium range, is to collaborate with B.
What, finally I really call the A. paradox is in statement (2). If it is so easy to grab your work, then it would be equally easy to anyone to replicate it. Therefore your work is not giving to you any competing advantage, even if you keep it secret for a while. If it is not easy to grab, then, who wants to use it proficiently needs you. So you are the winner, not because you keep your code top secret, but because all the issues it solves require a complex expertise that only you, the author can have. So ….

Epilogue

Professor eventually Z disappears. Not because he dies (please do exorcisms), but because his role, in the growing group of people around projects has become more and more marginal. Subjects also acquired maturity and as well as the will to maintain the advantages that the work has produced with respect to competitors.
This passage requires that the initially informal community establish as a formal Community (they wrote here for Academics) with its rules, etiquette, and wise management. This, in turn, requires Subjects coordinate and share alike their views, plan together new developments, plan events to make the common work to grow. Balkanisation of the code (which GPL could allow) and internal conflicts (never avoidable, having the Subjects different agendas) should be managed appropriately, and this requires clear agreements, smart actions, good will, and wise arguments.
If the community grows, everybody would be safer, because cooperating is better than competing (see also coopetition).
A partial adoption of the Open Source strategy is instead very useless. Open source codes that are practically not available (as those that are open source but not freely downloadable) cannot grow a healthy community and, sooner or later, die.

* A final note

Actually even if in my intention is a project also for my students, not a few of my students do not deeply endorse it. Reasons for this can be, maybe found in their personal history, the chemistry of their bodies and minds, or something else, which is hidden to me. So far, I  overreacts feeling myself betrayed, when they dismiss in what I believe it is right. So, probably my attitude is not is not correct. Sons do whatever they want, and probably they are right to try to find their way. So I have to conclude that the above is MY dream, and I will not be upset anymore, if my academic sons search their own in a different way.

Topics and concepts Titus Brown is excited about

This is kind of easy for me. Taking Titus Brown's (GS) post from my blogroll, and addressing my readers to it. He is writing about open science and research reproducibility. His guidelines for his lab are also an effort worth to read for my students.

You better go directly to his blogpost to read it, but let me summarise the topics he covered for a minimum of information.

1. The wonderful ongoing discussion around significance and reproducibility.

2. Blogging as a way to explore issues without prior approval from Top People.

3. Open source as a model for open science.

4. Computational narratives as the engine of collaborative data science.

5. mybinder: deploy running Jupyter Notebooks from GitHub repos in Docker containers

6. Overlay journals.

7. Bjorn Brembs.

8. Idea futures or prediction markets.

9. Open peer review by a selected papers network.

10. A call to arms: make outbreak research open access.

You have just to click on the link to get it.

Zenodo and Joss

Here I am robbing from Living in an Ivory Basement blog. Titus Brown (GS) introduces two interesting tools. Maybe the word “tools” is an inappropriateas description.

Zenodo is a way to archive software products, under the idea that GitHUB is not an archival:

“A common point of concern, however, is that GitHub repositories are not archival. That is, you can modify, rewrite, delete, or otherwise irreversibly mess with the contents of a git repository. And, of course, GitHub could go the way of Sourceforge and Google Code at any point. … But! Never fear! The folk at Zenodo and Mozilla Science Lab (in collaboration with Figshare) have solutions for you! … Zenodo is an open-access archive that is recommended by Peter Suber (as is Figshare). … Items will be retained for the lifetime of the repository. This is currently the lifetime of the host laboratory CERN, which currently has an experimental programme defined for the next 20 years at least.“

For more information, nothing better than reading the original blog post.

JOSS stands for Journal of Open Source Software and it promises to be a vehicle to publish software, not what software is about, or a description of the software and its peculiarities, but of the software, which seems to go through a process of peer review, extremely useful and new. You can find the Joss post here.

About graphs, DSL and replicable research in Francesco Serafin's work

This is the summary of what Francesco Serafin (his blog) did in its first year of doctoral studies, defending for his admission to his second year Ph.D. Undoubtedly he did a lot of work and he programs to do even more. Three are the lines of his research:

• implementing a new flexible structure based on graphs for commanding simulations of interacting systems; 
• implementing a domain specific language for doing environmental models (and particularly to solve ordinary and partial differential equations); 
• deploying a system that makes easier to do replicable science. 

Clicking on the Figure above you will find his presentation. Obviously the committee admitted him to the second year of his Ph.D. studies.

Scientific Software's Myths

In preparing the work of the new Ph.D. student that will arrive this fall, I think a reading of there two posts "The myths of bioinformatics software" by Dr. Lior Pachter (GS) and the response by Dr. Titus Brown (GS) can help a lot. While I do not marry the negativist attitude of Patcher, I cannot deny that many of his statements are, in my experience, correct. I endorse more Brown's position, however.

I suggest that Ph.D. applicants look carefully both the positions (and the intricate series of links). For me, I certainly remain with GPL. At the same time,  I would warn the applicants that I am oscillating towards a thesis topic which would sound like "Design Patterns for Hydrology" (not forgetting, certainly what I wrote on GEOtop 3.0).

P.S. -I also totally endorse this Brown's statement: "From my few involvements with working on non-free software, I would also add that selling software is a tough business, and not one that automatically leads to any profits; there's a long tail, just as with everything else, and I long ago decided that my time is worth more to me than the expected income from selling software would be. (I would be thrilled if a student wanted to try to make money off of our work, but my academic work would remain open source.)".  I saw so many colleagues that believe they can sell their software or keep it close, failing miserably and missing the leadership they could have. I know also some remarkable exceptions though (Hydrus, Flo2D especially).

Theory and Practice of Reproducible Research

In the occasion of the GII Ph.D. days, I am giving a talk about Research Reproducibility and Replicability. You can find my presentation on Slideshare, as usual. The slides below the figure are actually those of the talk I gave in Perugia October 12, 2016

The old presentation (which correspond to the youtube video, is here).

Please find the reference to tools and resources inside the slides. My talk is available trough my youtube channel:

Enjoy it!

Best practices for Scientific Computing

I was actually looking for other stuff, but I across this paper on Plosone entitled:

Best practices for Scientific Computing, by Wilson et al., 2014

Enjoy it ! One of the Authors, Titus Brown has a blog that we follow (see the Related blog)

Five steps into Reproducible Research

I am worried about Wuletawu being worried about the steep path in doing reproducible research. However,  taking inspiration from Tim-Berners Lee  five star/steps into open data I tried to implement a similar five steps sequence for RR:

@ Do not wait! Make your stuff available on the Web (whatever format) under an open license^1.

@@  Make it available with documentation (e.g. a README file for any data set and for any model)

@@@ Provide examples of runs, and give some reference. Structure your documentation. Include figures and their making. 

@@@@ Use URLs and providers like Github to store code and data, so people can point at your stuff, and browse it freely^2

@@@@@ Maintain a user group (and answer to questions when asked). Provide any run you do on the web with the appropriate metadata^3,4. 

Then you start to be a professional  of RR and you can face more complex task, and using structured tools like those presented in the COURSERA classes brought to my attention by Wuletawu.

1- Same as Tim Berners-Lee - Waiting to have it in better shape will delays the publication forever, and your contribution will be lost (like tears in rain).  

2 - Almost the same as in Tim Berners-Lee

3 - See: https://earthsystemcog.org/projects/es-doc-models/

4 - See: http://abouthydrology.blogspot.it/2014/10/naming-things-in-hydrological-models.html

Lost and (not yet) found

Recently I posted the history of JGrass-NewAGE, and I was wondering where the old material of JGrass-NewAGE version 0 was. All was more than prototypical and quite operative (with maybe some flaws in the science) and  worth to remain  alive. Wasn't the use of OMS etc. looking to some reproducibility and maintainability ?
Finally I recovered in my HD the part of the documentation of that version which can be downloaded from here (unfortunately is in Italian).

The database developed in PostgreSQL/PostGIS referred in the "manual" was found stored in a computer accessible trough the password "idrologia", and we are going to see if it is still usable and extendable. Actually, as apparent from my previous post, most of the components of the project can be considered obsolete, enhanced by those available with JGrass-NewAGE version 1. However, at least one component is still missing, and I hope to be able to get it back on-line: the one integrating the de Saint-Venant Equation in 1-D. It was actually saved in a public repository (so, it should be), but the repository was changed, and not maintained anymore. So people have to look at old backups to get it back.

Lessons learned: 1 - there should be an explicit will to maintain old stuff (quite an obvious conclusion) and not to make it vanish as vapour in summer. 2 - Even if the project was produced as Open Source, if it is not maintained, or at least left in a public site which is maintained unaltered, the effort is meaningless.   Useful stuff can be easily lost. 3 - Data can be lost even easier than software. A repository - Github-like for data should be mandatory. We'll see what we can do.

Research Reproducibility on Nature

Luca Brocca brings to my attention two papers on Research Reproducibility and the policy followed by Nature journal.

They are the editorial "Code Share" and the paper "Open Code for Open Science" by Steve M. Easterbrook

Again on reproducible research

I've got this from a post on linked-in by Mariza Costa-Cabral, author of a pioneering paper on digital elevation models, and now senior scientists at Northwest Hydraulic Consultants, Inc.:

"I hope this is the start of something good for the sake of reproducibility of research: Mozilla analysis of scientific software. And if such automated or semi-automated debugging can be done, then I hope we can get to use it much BEFORE submitting the paper! :-) Imagine, how great it would be to get help with debugging! Also, please read the particularly sensible 2010 article by Nick Barnes: "

The paper had several comments around the web, including those by Titus Brown. Here further posts of mine on the same topic. 

The JGrass-NewAGE informatics

As I tried to convey in previous posts, since more than five years, I am working to the idea to built a hydrological model by components. Well, this should have been the first paper in row, but as often happens, is the fourth to have been submitted.

The paper contains, by Formetta et al., the main ideas behind this type of modelling and shows that the system actually works, it is not just a matter of  speculations. We did it. At the moment it is at a late stage of review on Environmental Modelling & Software, and we eventually ask for it being open access.

The rational of the paper is expressed in its introduction:

"Scientists demand more and more the availability of simulation model’s source code since it has become a key factor for the understanding, validation, and advancement of science (e.g. Ince et al. [2012]). However, this is not enough, even if the source code would be available, the growing complexity of modelling code makes model development progress challenging to understand and manage. In fact, if model code distribution is matter of policies (e.g. Annan et al., [2013]), external inspection and analysis of models, improvement and contribution are difficult or even impossible when the software is inadequately engineered. The implementation of many environmental processes intimately interlinked (as snow, runoff production, evapotranspiration, in the hydrological case), is usually difficult to understand per se, but models writing in traditional monolithic forms, as defined in Rizzoli et al. [2005], makes their implementations overwhelming hard to follow, and models themselves practically impossible to be verified (Quesnel et al. [2009]). As a matter of facts, the traditional modelling practices preclude easy understanding, rapid reuse and improvement of the source code, and eventually obstacle seamless advancement in science.

Part of models’ obscurity has its foundation in bad documenting practices, and many researchers’ community that rely on computational methods and techniques as part of their day-to-day activities proposed shared protocols, like the so called Overview, Data concepts, Details method (ODD, e.g. Grimm et al., [2006]), to improve documentation effectiveness. However, “reproducible-research systems” (RRSs), making easier to document any step during research from model implementation and data preparation to output analyses, would greatly help correct policies to be adopted. Actually from a RRS system we would expect, besides model codes sharing, tools to allow the researchers, on one hand, to repeat the simulations in the same conditions and, on the other hand, to spend more time on their own science.

Many of software infrastructures or modeling frameworks (MF) were actually designed and built to streamline the process of a sound scientific production (e.g. Wesselung et al. [1996], Argent [2005], Rizzoli et al. [2005]). Among those that specifically target the support of hydrological modelling are the Spatial Modelling Environment (SME, Maxwell and Costanza [1997]), The Invisible Modelling Environment (TIME, http://www.toolkit.net.au/Tools/TIME) and hydrological derivative tools like, E2 (Argent et al. [1999]), OpenMI (http://www.openmi.org/), Moore and Tindall [2005]), and the Object Modelling System (OMS, David et al. [2002, 2013]), Common Component Architecture (CCA, Bramley et al. [2000]) and Earth System Modeling Framework (ESMF, Hill et al. [2004]).

However, most of the above MF require a quite significant learning curve that not all scientists, even proficient modellers, are willing to make.
Therefore, in order to ease the transition into modern programming environments, some modelling efforts and projects recently focused on providing code generation support and reducing the invasiveness of frameworks (Lloyd, 2010) into the model. Especially the third version of OMS and the BioMA project (BioMA, 2012) revealing promising perspectives.

A RSS would not be complete without including data visualization. Gardner and Manduchi (2007), among others, emphasize that in order to optimize scientific productivity, a RRS infrastructure should include not only the computational cores but also visualization and data-processing tools necessary to synthesize knowledge from high volumes of inputs and outputs.

Indeed, tools of choice for the visualization of hydrological processes have been for a long time Geographic Information Systems (GIS) (Maidment [1993]; Grayson et al. [1992]). However, traditional GIS are usually designed for managing static, non- temporal information layers. They are not designed to interact with the dynamic modelling (e.g. Burrough et al. [1998]; Wesselung et al. [1996]). In fact, the interaction between models and GIS can be described as “off-line” and it is performed with integration strategies that affect either the functionality of GIS tools or the usability of models.

Instead, the MF listed above offer a proper abstraction to streamline the interaction with a GIS. They promote the separation of the model into well defined modules or components, each with a well-defined way to interact with others through specified interfaces. Through their interfaces the modules can communicate and exchange data.

Therefore it is also timely for GIS and hydrological model components to constitute a pool of interoperable tools that can be blended together to create software that is accurately tailored to geosciences."

The revised version of the paper prior to publication, in pdf format, can be retrieved from here (or clicking below the picture). The final version is here instead.

The publication of geoscientific model developments v1.0

Geoscientific Model Development (a.k.a. GMD) has become for many, including us, a Journal of reference. Its open access policy with the fact that it covers a gap in scientific literature, has made of it a journal that quickly gained an impact factor of 5 (which is the higher among the journals in which I publish). Its success made to increase  the papers submitted exponentially, and, at this point, the Editors decided to restrict the policy for publication of the papers. The new editorial for publication is here (or clicking under the image).

I suggest that any modeller reads it. 

No code No paper

This is entirely from: Simply Statistics » R, and I completely agree with it. It applies the very same way to hydrological literature.

"I think it has been beat to death that the incentives in academia lean heavily toward producing papers and less toward producing/maintaining software. There are people that are way, way more knowledgeable than me about building and maintaining software. For example, Titus Brown hit a lot of the key issues in his interview. The open source community is also filled with advocates and researchers who know way more about this than I do.


This post is more about my views on changing the perspective of code/software in the data analysis community. I have been frustrated often with statisticians and computer scientists who write papers where they develop new methods and seem to demonstrate that those methods blow away all their competitors. But then no software is available to actually test and see if that is true. Even worse, sometimes I just want to use their method to solve a problem in our pipeline, but I have to code it from scratch!

I have also had several cases where I emailed the authors for their software and they said it “wasn’t fit for distribution” or they “don’t have code” or the “code can only be run on our machines”. I totally understand the first and last, my code isn’t always pretty (I have zero formal training in computer science so messy code is actually the most likely scenario) but I always say, “I’ll take whatever you got and I’m willing to hack it out to make it work”. I often still am turned down.

So I have a new policy when evaluating CV’s of candidates for jobs, or when I’m reading a paper as a referee. If the paper is about a new statistical method or machine learning algorithm and there is no software available for that method – I simply mentally cross it off the CV. If I’m reading a data analysis and there isn’t code that reproduces their analysis – I mentally cross it off. In my mind, new methods/analyses without software are just vapor ware. Now, you’d definitely have to cross a few papers off my CV, based on this principle. I do that. But I’m trying really hard going forward to make sure nothing gets crossed off.

In a future post I’ll talk about the new issue I’m struggling with – maintaing all that software I’m creating."

A paper in Nature on Scientific Software

The news was brought to me by Martin Davis who had from Stefan Steiner.  The paper is:  The case for open computer programs and was published in Nature.

Here it is what Martin says:

"The paper raises the argument for open source software to a higher plane, that of being a necessary component of scientific proof. It points out that the increasing use of computational science as a basis for scientific discovery implies that open source must become a standard requirement for documentation. Apparently some journals such as Science already require source code to be supplied along with submissions of articles. Amongst other advantages, access to source code is an essential element of peer review.
An interesting example they mention is the infamous HadCRUT and CRUTEM3 meteorological datasets. One of the (few) salient criticisms levelled at this information during Climategate was the inability to reproduce the results by re-running the software. (Mind you, the software was probably a pile of crufty old Fortran programs mashed up by Perl scripts, so maybe it's just as well)"

This clearly reflect what I already wrote in some of my posts:

http://abouthydrology.blogspot.it/2011/03/going-beyond-present-stato-of-art-in.html
http://abouthydrology.blogspot.it/2012/02/reproducible-research-and-papers.html

Reproducible Research (and papers)

Many papers I read in hydrology present research that is very difficult to reproduce. Because, as a scientist, I would like to reproduce the results of what I read (this is science, indeed!) some rules should be followed. I found this group of scientists that initiated a Reproducible Research web site, especially directed to image processing colleagues, but easily extendible to Hydrology, and related fields. They offer a how to guide, which is verbatim reported here below:

"Of course, it all starts with a good description of the theory, algorithm, or experiments in the paper. A block diagram or a pseudo-code description can do miracles! Once this is done, make a web page containing the following information:

• Title 
• Authors (with links to the authors' websites) 
• Abstract 
• Full reference of your paper, with current publication status, and a PDF of your paper 
• All the code to reproduce all the results, images and tables. Make sure all the code is well documented, and that there is a readme file explaining how to execute it 
• All the data (images, measurements, etc) to reproduce all the results, images and tables. Add a readme file explaining what the data represent 
• A list of configurations on which you tested your code (software version, platform) 
• An e-mail address that people can use for comments and remarks (and to report bugs) 

Depending on the field in which you work, it can also be interesting to add the following (optional) information to the web page:

• Images (add their captions, so that people know what Figure xx is about) 
• References (with abstracts)

For every link to a file, add its size between brackets. This allows people to skip large downloads if they are on a slow connection.
For examples, see this list of reproducible papers at LCAV, EPFL. "

Obviously they also link a blog and links to various RR resources. RR, incidentally, the same as my initials.

Now there is also a book about reproducible research made with R by Christopher Gandrud. The chapter can be found here.