Consequences of the Scientific Reform Movement · Journal of Trial & Error

This could be fun. 200 word proposals are due May 18th, 4000 word contributions are then due on September 30th.

In this call, we aim to inspire scholars to take heed and critically reflect on the direction in which the scientific reform movement is heading. Is scientific thought and practice being steered in a good direction? What are the known unknowns and unknown unknowns that challenge the field? Are there lessons that can be learned from other fields that have struggled with similar issues?

With the goal of stimulating a critical discussion in the field of scientific reform, we invite submissions that focus on “second-order” effects or “second-generation” challenges: issues that arise during or as a result of the resolution of a primary issue. We encourage a diversity of contributions, but all contributions should discuss a second-generation challenge for the scientific-reform movement. Examples of potential contributions include:

  • Historical and philosophical articles
  • Opinion pieces
  • Articles communicating insights from other fields
  • New data
  • Empirical syntheses
  • Formal theoretical models

Potential topics include:

  • Unintended consequences
  • Tipping points in complex systems
  • Second-order responses to changing incentive structures
  • Insights from innovation sciences and cultural evolution about cumulative knowledge accumulation
  • Excessive use of scientific resources (on e.g., open access in high-profile journals)
  • Gaming open science
  • Improved methodological rigor being circumvented novel Questionable Research Practices
  • Mindless check-box scientific practices
  • Issues with measuring scientific progress
  • The effects of reforms on scientific creativity
1 Like

Two topics that I’d be interested in writing an article on are: 1) the focus on technology/tool development over culture change, and 2) the issue of relevance during research question/problem selection.

Would anybody be interested in collaborating on a proposal for one of these?

1 Like

Hi @Gavin , I’d be interested in collaborating on a short contribution. I hasten to clarify that my tight schedule would not allow me to spend a lot of time on this project, but (i) I absolutely adore the Journal of Trial and Error and I want to support them as much as possible; (ii) I would be happy to collaborate with you (I greatly appreciate your work at IGDORE).

Should you be interested in starting a collaboration despite the caveat above, I find the first of your proposed topics closer to what I see in my everyday job at Erasmus University Rotterdam, so perhaps I would be a bit more useful. Another topic that I am very passionate about is the almost ubiquitous lack of accountability in academia. One way this manifests itself in the context of open science is (i) the abundance of policies (e.g., on open access or open data) whose compliance is not checked, and (ii) no sanctions when policy is not followed. This is particularly evident in grant allocation, where funding agencies may have a clear set of policies and guidelines but compliance is not verified and sanctions – e.g., grant freeze – are not applied (with very rare exceptions). When such an important piece of the complex puzzle of academia does not stand its ground, it contributes to a delayed implementation of open research practices at the institutional and individual level.

Curious to hear your thoughts!

2 Likes

Thanks for your interest in collaboration, Antonio! And for your appreciation of my work :smiley:

I feel accountability to, and enforcement of, mandates is important, but I am not so familiar with this issue, and it would be hard for me to either lead or contribute to this. So I think culture change could be a better topic for us to collaborate on. I wonder if @brucecaron might also be interested in collaborating with us on this? (anybody else is also welcome to express interest)

The call suggests the following contributions:

  • Historical and philosophical articles
  • Opinion pieces
  • Articles communicating insights from other fields
  • New data
  • Empirical syntheses
  • Formal theoretical models

I feel they are likely to receive many opinion pieces, so I wonder if there would be a good way to approach this from either a research or review perspective. One idea for a research project would be to look at how culture change efforts (maybe the presence of a ReproducibiliTea journal club) change the rate at which Open Science practices are adopted compared to reference universities that aren’t making such efforts. This would probably be feasible if the relevant data (preprint/preregistration/OA rates) could be easily pulled from public databases - I’d need to look into how easy that is. I’m also happy to hear other suggestions (and if something similar to my suggested analysis has already been done!).

I stumbled on this opinion article in Nature from 2016, which reminded me of the topic of this call.

Research integrity: Don’t let transparency damage science

Transparency has hit the headlines. In the wake of evidence that many research findings are not reproducible1, the scientific community has launched initiatives to increase data sharing, transparency and open critique. As with any new development, there are unintended consequences. Many measures that can improve science2 — shared data, post-publication peer review and public engagement on social media — can be turned against scientists.