This month’s article is from the authors of the preprint Metascience as a scientific social movement which criticaly reflects on the structure of the current scientific reform movement (we previously discussed it here). Peterson and Panofsky now put forward Arguments against efficiency in science (not OA, but preprinted), which is a short response to Hallonsten’s Stop evaluating science: A historical-sociological argument.
Peterson and Panofsky note:
The arguments of the proponents of evaluations and metascientific reform are firmly rooted in the values of liberal society: transparency, accountability, and productivity. Counterarguments are easily cast as defensiveness or obscurantism. This is not just an academic problem. Scientists we interviewed told us that they felt constrained expressing their skepticism of reforms because, while reformers can draw on popular rhetoric of how science should operate, critics must wade into the murky waters of real scientific practice. … Ultimately, our goal is not to suggest that the concept of efficiency has no place in science but, rather, that efficiency is only one value in a cluster of values that includes utility, significance, elegance and, even, sustainability and justice. That efficiency is the easiest to articulate because it accords with other dominant bureaucratic and economic values should not allow it to win policy discussions by default. The fact that the argument against efficiency is challenging makes it all the more pressing to make it.
I am inclined to agree, I just skimmed Stop Evaluating Science - it is interesting, but draws a rather nebulous argument that incorporates discussions on the economization, distrust, democratization and comidification of science to ultimately end in rhetorical argument against scientific evaluation: Questions like ‘has science been productive enough?’ and ‘how can it be proven that science has been productive enough?’ shall first and foremost be answered with a rhetorical question, namely, ‘how else do you suppose that we have achieved this level of wealth and technical standard in Europe and North America?’. But as Hallonsten then notes:
In spite of the overwhelming logic of this rhetorical counter-question, and the historical evidence that supports it, champions of the view that science is insufficiently productive and must be made productive and held accountable through limitations to its self-governance and the use of quantitative performance appraisals will demand evidence that they can comprehend and, preferably, compare with their own simple and straightforward numbers. A list of counter-examples will therefore probably not suffice, since it can be discarded as mere ‘anecdotal evidence’ against which also the shallowest and most oversimplified statistics usually win.
Peterson and Panofsky present a brief argument against efficiency based on two key points. Firstly, efficiency shouldn’t be equated to scientific progress because we don’t agree on what progress is:
Our inability to chart basic scientific progress undermines the ability to measure efficiency. The notion of efficiency only makes sense in the context of established means/ends relationships. The goal is to organize the means in the optimal way to achieve the desired end. The problem is that, in the area of basic science, the end is unknown. … There is little agreement among the scientists themselves about what constitutes a significant contribution. There is reason to believe this dissensus is not a mere technical deficiency, but is a constitutive feature of the cutting edge of science (Cole, 1992: 18). Rather than clarity, these accounts underscore the complexity of conceptualizing progress in science.
Secondly, the incentivizing efficiency may have counterproductive outcomes compared to which existing inefficient practices are preferable. Incentives may be particularly difficult to apply in academic environments as:
scientific cultures are not Lego sets that can be broken down and rebuilt anew. They have organically evolved their own systems of communication and evaluation. They interpret broadly accepted, but abstract, values like skepticism, verification, and transparency in ways sensible to their particular contexts. Applying blanket rules to maximize efficiency in such systems can lead to unintended and, even, counterproductive outcomes.
The mistaken assumption of trying to make science more efficient stems from misinterpreting scientists as nothing more than value-maximizing, incentive-driven agents. Reformers in science have adopted economic language and, in so doing, have treated scientists as actors primarily motivated by material rewards (e.g., Harris, 2017; Nosek et al., 2012). This can be compared to a Mertonian account which would view them motivated by the interlocking system of scientific norms. Under an economic account, the best way to change behavior in science is to alter the incentive structure to reward or punish specific behaviors. Rational scientists will then react to those incentives and outcomes can be ensured.
The problem with incentive-based legislation has been detailed in a recent book by economist Samuel Bowles (2016). He argues that trying to engineer social systems by treating actors as thoroughly self-interested and incentive-driven ignores the useful role that preexisting cultural values play. In the reformer’s mind, newly introduced incentives and existing preferences are ‘additively separable’ from existing values. That is, if actors already value a behavior, then adding an incentive can only have a positive, cumulative effect. Yet, this need not be the case. Bowles details laboratory and field studies that show how the introduction of incentives can reduce or even reverse existing values.
I’m quite partial to the second point as I feel that grassroots cultural change in academia is more likely to lead to beneficial scientific reform than the use of top-down incentives and rules. Still, data on the effectiveness of institutional policies at promoting Open Science practices should be starting to become available, so this point may prove easier to resolve than the first.
I’m in favour of promoting Open Science, but I do think this paper was a thought-provoking critique that provided:
the beginnings of a counterargument, so that any reform dressed in the language of efficiency must address what it means by efficiency and how it might impinge on other values. Science reform should be a slow, reversable process with input from funders, institutions, those who study science, and, most importantly, the scientists themselves. And although defensiveness and obfuscation are enemies of science, resistance to reforms may have reasonable roots.