Replication: for falsify theory
Issues with replication
the outcome of replication should be only used to testing experiment
it should not be used to test replication
acceptable replication is within generalization
Variable Range of Replication
replicate successfully with similar variable strengthen narrow theory
replicate successfully with different variable strengthen broad generalization
replicate failure with similar variable undermines narrow theory
replicate failure with different variable undermines broad generalization
think this problem like adding new data to dataset, would it change the original connection of the neurons in a neural network?
Integrative Replication: confirming a theory by playing with it
Replication in social science: given same data with same tool, would you produce similar findings? (data/code needed to provide)
p-hacking: choosing significant findings often decrease the robustness of model (over-fitting by select models that score the highest on validation set)
scientists' value bias the research: allow publishing research once data collected
replication for interpretive work:
"replication" is forcing interpretive work to be non-interpretive
only fact checking on small detail possible
Why not perform replication? Why not accept replication?
performing "successful" replication is hard in social science
replication is not a pure replication, but instead, adding new "data" to our knowledge. Therefore, similar research (from what we want to replicate) is not more valuable than different research. We don't have infinite research resource.
replication studies are not "interesting enough" for journals if successful
More complicated when replication "fails"?
I have the same argument as the author
we should not treat replication as pure replication because there is not distinct difference between "original" and a "replication". That is, there exists no such relationship as "tested" and "tester". When result don't agree, they don't agree. It is impossible to tell which one is correct without extra information.
Pre-registration help "replication crisis"?
It help replication crisis, but create another crisis
Since researcher has guaranteed publication, researcher's own non-epistemic bias can be reduced to produce more robust, therefore replicate research.
however, that is to say "all research are equal" once you have data. Then research will be careless when analyzing data. It leads to produce more "useless" research because journals can no longer pick "valuable" research.
Interpretative social science in replication? Interpretive social science should not be subject to the replication requirement. Interpretive work often uses data that needs interpretive methods to analyze, and therefore the outcome of such analysis is strongly dependent on the analyzer's value and assumptions. Since it is very difficult to communicate all the assumptions the analyzer uses to analyze the data, a pure replication is unlikely to be successful. If replication is made a requirement, then research resources have to be devoted to confirming or denying a study. Since this replication is highly unlikely to be successful, replication of interpretive work is a waste of research resources because it generates little information. Although it should not be a requirement, replication should be made recommended so that it encourages researchers to unearth their underlying assumptions and state them clearly in their research articles.
Table of Content