Reproducibility in the absence of selective reporting: An illustration from large‐scale brain asymmetry research

Xiang‐Zhen Kong, Clyde Francks, Samuel R. Mathias, Tulio Guadalupe, Christoph Abé, Ingrid Agartz, Theophilus N. Akudjedu, Andre Aleman, Saud Alhusaini, Nicholas B. Allen, David Ames, Ole A. Andreassen, Alejandro Arias Vasquez, Nicola J. Armstrong, Phil Asherson, Felipe Bergo, Mark E. Bastin, Albert Batalla, Jochen Bauer, Bernhard T Baune, Ramona Baur‐Streubel, Joseph Biederman, Sara K. Blaine, Premika Boedhoe, Erlend Bøen, Anushree Bose, Janita Bralten, Daniel Brandeis, Silvia Brem, Henry Brodaty, Dilara Yüksel, Samantha J. Brooks, Jan Buitelaar, Christian Bürger, Robin Bülow, Vince Calhoun, Anna Calvo, Erick Jorge Canales‐Rodríguez, Dara M. Cannon, Elisabeth C. Caparelli, Francisco X. Castellanos, Fernando Cendes, Tiffany Moukbel Chaim‐Avancini, Kaylita Chantiluke, Qun‐lin Chen, Xiayu Chen, Yuqi Cheng, Anastasia Christakou, Vincent P. Clark, David Coghill, Colm G. Connolly, Annette Conzelmann, Aldo Córdova‐Palomera, Janna Cousijn, Tim Crow, Ana Cubillo, Udo Dannlowski, Sara Ambrosino Bruttopilo, Patrick Zeeuw, Ian J. Deary, Damion V. Demeter, Adriana Di Martino, Erin W Dickie, Bruno Dietsche, Nhat Trung Doan, Colin P. Doherty, Alysa Doyle, Sarah Durston, Eric Earl, Stefan Ehrlich, Carl Johan Ekman, Torbjørn Elvsåshagen, Jeffery N. Epstein, Damien A. Fair, Stephen V. Faraone, Guillén Fernández, Claas Flint, Geraldo Busatto Filho, Katharina Förster, Jean‐Paul Fouche, John J. Foxe, Thomas Frodl, Paola Fuentes‐Claramonte, Janice M. Fullerton, Hugh Garavan, Danielle Santos Garcia, Ian H. Gotlib, Anna E. Goudriaan, Hans Jörgen Grabe, Nynke A. Groenewold, Dominik Grotegerd, Oliver Gruber, Tiril Gurholt, Jan Haavik, Tim Hahn, Narelle K. Hansell, Mathew A. Harris, Catharina A. Hartman, Maria Carmen Valdés Hernández, Dirk Heslenfeld, Robert Hester, Derrek Paul Hibar, Beng‐Choon Ho, Tiffany C. Ho, Pieter J. Hoekstra, Ruth J. Holst, Martine Hoogman, Marie F. Høvik, Fleur M. Howells, Kenneth Hugdahl, Chaim Huyser, Martin Ingvar, Akari Ishikawa, Anthony James, Neda Jahanshad, Terry L. Jernigan, Erik G Jönsson, Vasily Kaleda, Clare Kelly, Michael Kerich, Matcheri S. Keshavan, Sabin Khadka, Tilo Kircher, Gregor Kohls, Kerstin Konrad, Ozlem Korucuoglu, Bernd Krämer, Axel Krug, Jonna Kuntsi, Jun Soo Kwon, Nanda Lambregts‐Rommelse, Mikael Landén, Luisa Lázaro, Irina Lebedeva, Rhoshel Lenroot, Klaus‐Peter Lesch, Qinqin Li, Kelvin O. Lim, Jia Liu, Christine Lochner, Edythe D. London, Valentina Lorenzetti, Michelle Luciano, Maartje Luijten, Astri J. Lundervold, Scott Mackey, Frank P. MacMaster, Sophie Maingault, Charles B. Malpas, Ulrik F. Malt, David Mataix‐Cols, Rocio Martin‐Santos, Andrew R. Mayer, Hazel McCarthy, Sarah Medland, Mitul Metha, Philip B. Mitchell, Bryon A. Mueller, Susana Muñoz Maniega, Bernard Mazoyer, Colm McDonald, Quinn McLellan, Katie L. McMahon, Genevieve McPhilemy, Reza Momenan, Angelica M. Morales, Janardhanan C. Narayanaswamy, José Carlos Vasques Moreira, Stener Nerland, Liam Nestor, Erik Newman, Joel T. Nigg, Jan Egil Nordvik, Stephanie Novotny, Eileen Oberwelland Weiss, Ruth L. O'Gorman, Jaap Oosterlaan, Bob Oranje, Catherine Orr, Bronwyn Overs, Yannis Paloyelis, Paul Pauli, Martin Paulus, Kerstin Jessica Plessen, Georg G. Polier, Edith Pomarol‐Clotet, Maria J. Portella, Jiang Qiu, Joaquim Radua, Josep Antoni Ramos‐Quiroga, Y.C. Janardhan Reddy, Andreas Reif, Gloria Roberts, Pedro Rosa, Katya Rubia, Matthew D. Sacchet, Perminder S. Sachdev, Raymond Salvador, Lianne Schmaal, Martin Schulte‐Rüther, Lizanne Schweren, Jochen Seitz, Mauricio Henriques Serpa, Philip Shaw, Elena Shumskaya, Timothy J. Silk, Alan N. Simmons, Egle Simulionyte, Rajita Sinha, Zsuzsika Sjoerds, Runar Elle Smelror, Joan Carlos Soliva, Nadia Solowij, Fabio Luisde Souza‐Duran, Scott R. Sponheim, Dan J. Stein, Elliot A. Stein, Michael Stevens, Lachlan T. Strike, Gustavo Sudre, Jing Sui, Leanne Tamm, Hendrik S. Temmingh, Robert J. Thoma, Alexander Tomyshev, Giulia Tronchin, Jessica Turner, Anne Uhlmann, Theo G.M. Erp, Odile A. Heuvel, Dennis Meer, Liza Eijk, Alasdair Vance, Ilya M. Veer, Dick J. Veltman, Ganesan Venkatasubramanian, Oscar Vilarroya, Yolanda Vives‐Gilabert, Aristotle N Voineskos, Henry Völzke, Daniella Vuletic, Susanne Walitza, Henrik Walter, Esther Walton, Joanna M. Wardlaw, Wei Wen, Lars T. Westlye, Christopher D. Whelan, Tonya White, Reinout W. Wiers, Margaret J. Wright, Katharina Wittfeld, Tony T. Yang, Clarissa L. Yasuda, Yuliya Yoncheva, Murat Yücel, Je‐Yeon Yun, Marcus Vinicius Zanetti, Zonglei Zhen, Xing‐xing Zhu, Georg C. Ziegler, Greig I. Zubicaray, Marcel Zwiers, Karolinska Schizophrenia Project, David C. Glahn, Fabrice Crivello, Simon E. Fisher, Paul M. Thompson, Lars Farde, Lena Flyckt, Göran Engberg, Sophie Erhardt, Helena Fatouros‐Bergman, Simon Cervenka, Lilly Schwieler, Fredrik Piehl, Karin Collste, Pauliina Victorsson, Anna Malmqvist, Mikael Hedberg, Funda Orhan, Carl Sellgren,
Human Brain Mapping. 2020-08-25; 43(1): 244-254
DOI: 10.1002/hbm.25154


Read on PubMed

---

Kong XZ(1)(2); ENIGMA Laterality Working Group; Francks C(1)(3).

Collaborators: Kong XZ, Mathias SR, Guadalupe T, Abé C, Agartz I, Akudjedu TN,
Aleman A, Alhusaini S, Allen NB, Ames D, Andreassen OA, Vasquez AA, Armstrong
NJ, Asherson P, Bergo F, Bastin ME, Batalla A, Bauer J, Baune BT, Baur-Streubel
R, Biederman J, Blaine SK, Boedhoe P, Bøen E, Bose A, Bralten J, Brandeis D,
Brem S, Brodaty H, Yüksel D, Brooks SJ, Buitelaar J, Bürger C, Bülow R, Calhoun
V, Calvo A, Canales-Rodríguez EJ, Cannon DM, Caparelli EC, Castellanos FX,
Cendes F, Chaim-Avancini TM, Chantiluke K, Chen QL, Chen X, Cheng Y, Christakou
A, Clark VP, Coghill D, Connolly CG, Conzelmann A, Córdova-Palomera A, Cousijn
J, Crow T, Cubillo A, Dannlowski U, de Bruttopilo SA, de Zeeuw P, Deary IJ,
Demeter DV, Di Martino A, Dickie EW, Dietsche B, Doan NT, Doherty CP, Doyle A,
Durston S, Earl E, Ehrlich S, Ekman CJ, Elvsåshagen T, Epstein JN, Fair DA,
Faraone SV, Fernández G, Flint C, Filho GB, Förster K, Fouche JP, Foxe JJ, Frodl
T, Fuentes-Claramonte P, Fullerton JM, Garavan H, do Santos Garcia D, Gotlib IH,
Goudriaan AE, Grabe HJ, Groenewold NA, Grotegerd D, Gruber O, Gurholt T, Haavik
J, Hahn T, Hansell NK, Harris MA, Hartman CA, Del Carmen Valdés Hernández M,
Heslenfeld D, Hester R, Hibar DP, Ho BC, Ho TC, Hoekstra PJ, van Holst RJ,
Hoogman M, Høvik MF, Howells FM, Hugdahl K, Huyser C, Ingvar M, Ishikawa A,
James A, Jahanshad N, Jernigan TL, Jönsson EG, Kaleda V, Kelly C, Kerich M,
Keshavan MS, Khadka S, Kircher T, Kohls G, Konrad K, Korucuoglu O, Krämer B,
Krug A, Kuntsi J, Kwon JS, Lambregts-Rommelse N, Landén M, Lázaro L, Lebedeva I,
Lenroot R, Lesch KP, Li Q, Lim KO, Liu J, Lochner C, London ED, Lorenzetti V,
Luciano M, Luijten M, Lundervold AJ, Mackey S, MacMaster FP, Maingault S, Malpas
CB, Malt UF, Mataix-Cols D, Martin-Santos R, Mayer AR, McCarthy H, Medland S,
Metha M, Mitchell PB, Mueller BA, Maniega SM, Mazoyer B, McDonald C, McLellan Q,
McMahon KL, McPhilemy G, Momenan R, Morales AM, Narayanaswamy JC, Moreira JCV,
Nerland S, Nestor L, Newman E, Nigg JT, Nordvik JE, Novotny S, Weiss EO,
O’Gorman RL, Oosterlaan J, Oranje B, Orr C, Overs B, Paloyelis Y, Pauli P,
Paulus M, Plessen KJ, von Polier GG, Pomarol-Clotet E, Portella MJ, Qiu J, Radua
J, Ramos-Quiroga JA, Reddy YCJ, Reif A, Roberts G, Rosa P, Rubia K, Sacchet MD,
Sachdev PS, Salvador R, Schmaal L, Schulte-Rüther M, Schweren L, Seitz J, Serpa
MH, Shaw P, Shumskaya E, Silk TJ, Simmons AN, Simulionyte E, Sinha R, Sjoerds Z,
Smelror RE, Soliva JC, Solowij N, Souza-Duran FL, Sponheim SR, Stein DJ, Stein
EA, Stevens M, Strike LT, Sudre G, Sui J, Tamm L, Temmingh HS, Thoma RJ,
Tomyshev A, Tronchin G, Turner J, Uhlmann A, van Erp TGM, van den Heuvel OA, van
der Meer D, van Eijk L, Vance A, Veer IM, Veltman DJ, Venkatasubramanian G,
Vilarroya O, Vives-Gilabert Y, Voineskos AN, Völzke H, Vuletic D, Walitza S,
Walter H, Walton E, Wardlaw JM, Wen W, Westlye LT, Whelan CD, White T, Wiers RW,
Wright MJ, Wittfeld K, Yang TT, Yasuda CL, Yoncheva Y, Yücel M, Yun JY, Zanetti
MV, Zhen Z, Zhu XX, Ziegler GC, de Zubicaray GI, Zwiers M, Project KS, Glahn DC,
Crivello F, Fisher SE, Thompson PM, Francks C, Farde L, Flyckt L, Engberg G,
Erhardt S, Fatouros-Bergman H, Cervenka S, Schwieler L, Piehl F, Agartz I,
Collste K, Victorsson P, Malmqvist A, Hedberg M, Orhan F, Sellgren C.

Author information:
(1)Language and Genetics Department, Max Planck Institute for Psycholinguistics,
Nijmegen, The Netherlands.
(2)Department of Psychology and Behavioral Sciences, Zhejiang University,
Hangzhou, China.
(3)Donders Institute for Brain, Cognition and Behaviour, Radboud University,
Nijmegen, The Netherlands.

The problem of poor reproducibility of scientific findings has received much
attention over recent years, in a variety of fields including psychology and
neuroscience. The problem has been partly attributed to publication bias and
unwanted practices such as p-hacking. Low statistical power in individual
studies is also understood to be an important factor. In a recent multisite
collaborative study, we mapped brain anatomical left-right asymmetries for
regional measures of surface area and cortical thickness, in 99 MRI datasets
from around the world, for a total of over 17,000 participants. In the present
study, we revisited these hemispheric effects from the perspective of
reproducibility. Within each dataset, we considered that an effect had been
reproduced when it matched the meta-analytic effect from the 98 other datasets,
in terms of effect direction and significance threshold. In this sense, the
results within each dataset were viewed as coming from separate studies in an
“ideal publishing environment,” that is, free from selective reporting and p
hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%,
min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger
effects and in larger datasets. Reproducibility was not obviously related to the
age of participants, scanner field strength, FreeSurfer software version,
cortical regional measurement reliability, or regional size. These findings
constitute an empirical illustration of reproducibility in the absence of
publication bias or p hacking, when assessing realistic biological effects in
heterogeneous neuroscience data, and given typically-used sample sizes.

© 2020 The Authors. Human Brain Mapping published by Wiley Periodicals LLC.

Know more about