Lecture 025

Reading

Replication: for falsify theory

Issues with replication

1. replications are not exact
2. replication requires imagination (requires epistemic value)
3. therefore whether replication success or fail is ambiguous, value needed

experimenter's regress:

• the outcome of replication should be only used to testing experiment

• it should not be used to test replication

acceptable replication is within generalization

Variable Range of Replication

• replicate successfully with similar variable strengthen narrow theory

• replicate successfully with different variable strengthen broad generalization

• replicate failure with similar variable undermines narrow theory

• replicate failure with different variable undermines broad generalization

think this problem like adding new data to dataset, would it change the original connection of the neurons in a neural network?

Integrative Replication: confirming a theory by playing with it

Replication in social science: given same data with same tool, would you produce similar findings? (data/code needed to provide)

p-hacking: choosing significant findings often decrease the robustness of model (over-fitting by select models that score the highest on validation set)

• can be half-prevented by pre-registration

scientists' value bias the research: allow publishing research once data collected

replication for interpretive work:

• "replication" is forcing interpretive work to be non-interpretive

• only fact checking on small detail possible

Questions

1. Why not perform replication? Why not accept replication?

2. performing "successful" replication is hard in social science

3. replication is not a pure replication, but instead, adding new "data" to our knowledge. Therefore, similar research (from what we want to replicate) is not more valuable than different research. We don't have infinite research resource.

4. replication studies are not "interesting enough" for journals if successful

5. More complicated when replication "fails"?

6. I have the same argument as the author

7. we should not treat replication as pure replication because there is not distinct difference between "original" and a "replication". That is, there exists no such relationship as "tested" and "tester". When result don't agree, they don't agree. It is impossible to tell which one is correct without extra information.

8. Pre-registration help "replication crisis"?

9. It help replication crisis, but create another crisis

10. Since researcher has guaranteed publication, researcher's own non-epistemic bias can be reduced to produce more robust, therefore replicate research.

11. however, that is to say "all research are equal" once you have data. Then research will be careless when analyzing data. It leads to produce more "useless" research because journals can no longer pick "valuable" research.

12. Interpretative social science in replication? Interpretive social science should not be subject to the replication requirement. Interpretive work often uses data that needs interpretive methods to analyze, and therefore the outcome of such analysis is strongly dependent on the analyzer's value and assumptions. Since it is very difficult to communicate all the assumptions the analyzer uses to analyze the data, a pure replication is unlikely to be successful. If replication is made a requirement, then research resources have to be devoted to confirming or denying a study. Since this replication is highly unlikely to be successful, replication of interpretive work is a waste of research resources because it generates little information. Although it should not be a requirement, replication should be made recommended so that it encourages researchers to unearth their underlying assumptions and state them clearly in their research articles.