planet.donarmstrong.com

The "Critical Assessment of Open Science" meeting, or CAOS, was convened by Sage Bionetworks in New Orleans in early February. About 30 open science practitioners and advocates were invited by Sage to a day long meeting in New Orleans to consider the last 10 years of progress and failures in open science. The meeting was attended by scientists, policy experts, funders, and others. While the emphasis was on the biosciences, many themes were discussed in a broader context of all of science.

You can read more about the motivation for the meeting, and see a series of summary blog posts, here.

This post is my attempt to summarize the entire meeting, based on notes I took during the meeting.

The meeting was organized in a series of "call and response" engagements, in which two participants "called" for 5 minutes to one of five broad themes, and then a responder summarized, contextualized, and responded to their call. There were multiple such calls & responses in each session, for about 5 sessions. Audience participation was lively!

The meeting was held under Chatham House rules, so below I am reporting my takeaways without reference to specific individual comments or revealing details. There should be some form of publication output in the future so you can see who attended and get a more global view of the meeting; I'll link to that below when it is out.

Thank you to Sage Bionetworks for coordinating this meeting & inviting me!

Main themes that emerged (for me)

We hoped that open science would lead to new and better practices; what we too often got was practices that fed into the same broken system.

As the value of analytics and data becomes ever more apparent, there is ever more interest by commercial interests in capturing that value in closed systems. Often, the data creators and/or owners seem to be unaware of this capture, especially when the data is secondary to their primary mission (e.g. in universities). This lack of awareness Has Consequences.

Governance and sustainability of open institutions (especially open source projects) is on a lot of people's minds. Sage has a large team focused on this! (John Wilbanks says "call me!")

We talked a fair bit about the challenge of convincing individuals and groups that increased opportunity for unpredictable serendipity was worth giving up predictable (but smaller) gains in fame/power/money.

The invisibility of successful "open" came up repeatedly - the modern data science ecosystem is built on R and Python, preprints in the life sciences, open & FAIR data, and open source especially. That successful open practices achieve near instant adoption is wonderful; that they are not highlighted as successes of open in the open science community is unfortunate; and their invisibility means that their sustainability is often not strongly considered. (You can see a longer blog post by me on this topic, here.)

It was great to see multiple statements about how the idea of one consortium/community building THE platform for analysis in an area was a non-starter. Functional interoperability, collaboration, and ecosystem thinking within and across platforms is seen as critical, even by the most senior researchers.

In concert with that, I see that every functional system is a compromise between various requirements and design considerations. Therefore building multiple differently functioning systems is a good ecosystem bet.

Several different people referred to the increased attack surface that open practices offer: e.g. by making your methods and data open, you increase the ability of others to attack your conclusions. While this is an important aspect of open science, it is also something that discourages everyone, with disproportionate negative impact on already marginalized populations. Sharing within "club" structures, or gated communities, was seen as one possible solution.

We noted the need for & challenge of placing "do no harm" restrictions on use and reuse of data; community codes of conduct were discussed as one example of a governance structure that (combined with not-entirely-open communities) could enforce such restrictions.

Diversity and inclusion was a frequently mentioned topic. Lack of diversity in communities can be seen as empirical evidence of missing structure in communities that is not clearly visible from within; I think this is important when it comes to formal governance discussions that can externalize internal culture (hopefully accurately).

Another interesting theme was the extent to which some saw that grassroots communities of practice could be an antidote to the "monkey's paw" or "shitty genie" of requirements generation. Often, engineers building infrastructure want detailed use cases and requirements specification, which then leads to the wrong thing being built (and the associated blame), while if the engineers are brought into the community of practice they are more likely to build the right thing due to shared understanding and iterative/continuous participation.

The challenge of analyzing all the interesting data sets was frequently mentioned. While not discussed at the meeting, in my view, training is a way to bring prepared minds and hands to tackle the analysis of interesting data sets. This training needs to be built in rather than bolted on to projects, however.

My own POV: the critical role of communities of practice

Again and again, I saw that communities of practice presented a key ingredient to solutions for problems in governance, training, infrastructure, methods, etc. Communities of practice bring the people to the problems! Fundamentally, I think open systems do not work without a community of practice underpinning them.

Creating, growing, and sustaining these communities is, I think, one of the most important tasks to be tackled. More on that as I have time to write.

Concluding thoughts

One of the organizers closed out the meeting by asking everyone to highlight one theme that surprised and/or dismayed them. This was a productive if depressing way to extract essential takeaways!

"The cavalry isn't coming." One of the more sobering conclusions from this part of meeting was that, given the seniority of the people in the room, we had no one but ourselves to blame for failing at open in the next decade. If we couldn't figure out how to coordinate and incentivize open, then it was unlikely that someone else would step in to help us out. We are the cavalry. (And existing, closed, institutions are more resilient than we realized.)

Consumers are often very happy to trade data for convenience. This is a challenge for open!

Open science can be weaponized by opponents of science, e.g. reproducibility challenges can lead to the conclusion that all science is wrong; there are many politicians eager to attack science. The dangers of further deligitimizing science in the eyes of the world are real!

While scientists always start in and often revert to competitive mode, they can also switch to cooperative mode with ease, given the proper incentives and structure. (I personally recommend reading Kathleen Fitz's book Generous Thinking, which focuses on this issue!)

A generational (?) concern was that DIY biology will eat all of biology, and that this meeting could be viewed as a bunch of PDP-11 engineers discussing the intricacies and importance of time sharing system design. I personally think millenials are more sophisticated about data ownership, more invested in sharing (and more sophisticated about its tradeoffs), and are likely to seriously upset current apple carts, but I'm an optimist :).

There was a repeated concern that open biomedical science has to translate into better outcomes, and a shared concern that open science is an ideology built on practices that don't really work 80% of the time.

My own (depressing) conclusion was that it is not possible for open to be truly open, and that completely open institutions are extremely vulnerable to attack (for my previous thoughts on this in open source projects, see "How open is too open?"). There are gates that must be kept (hodor)! I'll expand on this theme in another blog post when I have time!

In general, I'm happy to expand on themes as time permits, if people have questions!

Immediately after writing this, I happened to revisit Denisse Alejandra's article, "Reimagining Open Science Through a Feminist Lens", and I was encouraged by the overlap and relevance of a lot of what was discussed at the CAOS meeting to this reimagination!

--titus