skip to content

The Cambridge-INET Institute - continuing as the Janeway Institute

WP Cover

Menkveld, A., Dreber, A., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Neusüss, S., Razen, M., Weitzel, U., Linton, O.

Non-Standard Errors

JIWP Number: 2112

Abstract: In statistics, samples are drawn from a population in a data generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Online appendix available at

Please note a full list of authors is available in the working paper.

Keywords: Market Efficiency, P-hacking, Publication bias

JEL Codes: A14 C10 C12 C59 C90 G14 G40

Author links: Oliver Linton  

PDF: jiwp2112.pdf

Open Access Link: 10.17863/CAM.79374

Theme: empirical