Worldview as a hidden methodological variable — is this an under-discussed source of irreproducibility?

Open and replicable science
, , , ,
1

A question I’d like to put to this forum, since it sits adjacent to most of what we discuss here but rarely gets named directly:

Replicability conversations focus heavily on the visible parts of methodology — preregistration, sample sizes, statistical practice, data sharing. All necessary. But there’s a layer underneath that I think shapes results just as much and is almost never audited: the researcher’s implicit worldview.

Concretely:

  • Whether matter is fundamental and mind/information are derivative — or the other way around — determines which questions you treat as scientifically tractable in physics-of-life, cognition, foundations of QM.

  • Whether you implicitly treat agency as real or as a useful fiction determines what counts as a “result” in animal cognition, social science, AI alignment work

  • Whether you implicitly treat first-person experience as data or as noise determines what gets studied in consciousness research, mental health, well-being interventions

  • Whether causation is something “real” in the world, a counterfactual relation, an information-theoretic flow, a chaotic-system artifact, or just a model of something that doesn’t truly exist out there — determines what your equations are claiming to be about in complex systems, epidemiology, economics

  • Whether you implicitly believe science uncovers truth or constructs frameworks determines how you write your discussion section

These aren’t idle philosophy questions — they’re operating in your modeling choices before any preregistration document gets written. Two researchers preregistering the “same” study can be running different experiments because they’re studying different objects. And the worldview that shaped the choice is invisible to the reader, to peer review, and often to the researcher themselves.

I’ve been thinking about this as a kind of epistemic debugging — a structured practice of surfacing where one’s own framework has quietly constrained the question space, where Goodhart effects (publications, citations, fundability) have substituted for actual scientific judgment, and where the implicit answer to “what counts as real” is doing more work than any explicit method.

I’d be curious whether others here see this as:

  1. A real and underexplored source of methodological variance worth taking seriously

  2. Already adequately handled by existing open-science / preregistration practices

  3. A philosophy-of-science problem rather than a methodology problem

  4. Something else entirely

We’re actually running a workshop on exactly this in September — Honest Science Workshop, a small research gathering for people working at the intersection of complexity science and contemplative studies. Full details and a companion retreat are in the Events post here.

But independent of the workshop — I’m genuinely curious how this community sees the question. Where does worldview-debugging fit (or not fit) in the broader open-science project?

-Pavel

1 Like
2

Conceptual and theoretical standpoints of researchers are definitely a critical aspect of the replication crisis. My experience leads me to believe that insistence on the “visible” parts of methodology is consequential to the same epsitemic issues you describe. Moreover, technical solutions may also be better suitable for profitable outcomes (see for example, the notion that AI can be used as a predictive tool for research validity and reliability). I would also point out that there is nothing revolutionary in stating that epistemological and ontological positions need to be rendered explicit (regardless of discipline or methodological approach), in fact, it was customary in classical scientific work to integrate these reflections in methodological description. I think that the hyper marketization of research and the accelerated speed demanded on the production of scientific outputs has led to a general amnesia relative to this part of science. I am now a middle aged woman (not that old at the end of the day!) but still remember studying these aspects thoroughly as an undergraduate at an Italian university. I have been involved in making sense of the replication crisis for some time now, and most of this time, I feel like we are just trying to reinvent the wheel. Hope these reflections are useful.

1 Like
3

I am happy to see this simple yet essential thread which explicitly recognises the very new and yet strongly embedded (as Elisabeth pointed out) dangers of “constructed universalism of zero-point epistemology”. What you mention Pavel, critiques the notion of universal knowledge as objective and neutral, exposing it as a construct rooted in specific cultural and historical contexts, as there is no single foundation or one starting point of knowledge. These false ‘universal claims’ highlight limited/false binaries* (*abyssal thinking). The effects of this specific mindset on people’s experiences lead to epistemic violence (as you noted already) and further entrenches power relationships by rewarding epistemic injustice to ‘get ahead’ in systems for our career, etc..

Instead, we need to unlearn this ourselves first and foremost, by re-opening ways to express realities more fully and openly, and beyond binaries and as ‘universal’. We can do this via e.g.: reflexivity (shown to us briefly by Elisabeth) and reintegrate our positionality into methods as mentioned and indeed every aspect of the scientific work we do.

There is a lot of work on “the researchers implicit worldview” - positionality to reiterate what Elisabeth says here which has been forgotten. It is only a very recent development (and far too widespread) that science seems to have forgotten that it is not a fixed institution rendering it dogmatic as our practices seem to perpetuate.

So happy to be having this discussion here in IGDORE :slight_smile: