{"id":56,"date":"2023-01-19T17:09:47","date_gmt":"2023-01-19T17:09:47","guid":{"rendered":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/chapter\/2-6-3-learn-it-the-replication-crisis\/"},"modified":"2025-11-04T13:16:28","modified_gmt":"2025-11-04T13:16:28","slug":"2-6-3-learn-it-the-replication-crisis","status":"publish","type":"chapter","link":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/chapter\/2-6-3-learn-it-the-replication-crisis\/","title":{"raw":"The Replication Crisis: Learn It 3\u2014Replicating Experiments","rendered":"The Replication Crisis: Learn It 3\u2014Replicating Experiments"},"content":{"raw":"<h2>Does it Replicate? The Messy World of Real Science<\/h2>\r\n<p>A few years after the McCabe and Castel study was published, some psychologists[footnote]Robert B. Michael, Eryn J. Newman, Matti Vuorre, Geoff Cumming, and Maryanne Garry (2013). On the (non)persuasive power of a brain image. Psychonomic Bulletin &amp; Review, 20(4), 720-725.[\/footnote] at the University of Victoria in New Zealand, led by Robert Michael, were intrigued by the results and they were impressed by how frequently the paper had been cited by other researchers (about 40 citations per year between 2008 and 2012\u2014a reasonably strong citation record). They wanted to explore the brain image effect, so they started by simply replicating the original study.[footnote]They actually tried to replicate Experiment 3 in the McCabe and Castel study. You read Experiment 1. These two experiments were similar and supported the same conclusions, but Dr. Michael and his colleagues preferred Experiment 3 for some technical reasons.[\/footnote]<\/p>\r\n<p>In their first attempt at replication, the researchers recruited and tested people using an online site called Mechanical Turk. With 197 participants, they found no hint of an effect of the brain image on people\u2019s judgments about the validity of the conclusions of the article they read.<\/p>\r\n<p>In a second replication study, they tested students from their university and again found no statistically significant effect. In this second attempt, the results were in the predicted direction (the presence of a brain image was associated with higher ratings), but the differences were not strong enough to be persuasive.<\/p>\r\n<p>They tried slight variations on instructions and people recruited, but across 10 different replication studies, only one produced a statistically significant effect.<\/p>\r\n<section class=\"textbox tryIt\">[ohm2_question height=\"400\"]3435[\/ohm2_question]<\/section>\r\n<p>So, did Dr. Michael and his colleagues accuse McCabe and Castel of doing something wrong? Did they tear apart the experiments we described earlier and show that they were poorly planned, incorrectly analyzed, or interpreted in a deceptive way?<\/p>\r\n<p>Not at all.<\/p>\r\n<p>It is valuable to see how professional scientists approached the problem of failing to replicate a study. Here is a quick review of the approach taken by the researchers who did not replicate the McCabe and Castel study:<\/p>\r\n<ul>\r\n\t<li>First, they did not question the integrity of the original research. David McCabe[footnote]David McCabe, the first author of the original paper, tragically passed away in 2011 at the age of 41. At the time of his death, he was an assistant professor of Psychology at Colorado State University, and he had started to build a solid body of published research, and he was also married with two young children. The problems with replicating his experiments were only published after his death, so it is impossible to know what his thoughts might have been about the issues these challenges raised.[\/footnote] and Alan Castel are respected researchers who carefully reported on a series of well-conducted experiments. They even noted that the original paper was carefully reported, even if journalists and other psychologists had occasionally exaggerated the findings: \u201cAlthough McCabe and Castel (2008) did not overstate their findings, many others have. Sometimes these overstatements were linguistic exaggerations\u2026Other overstatements made claims beyond what McCabe and Castel themselves reported.\u201d [p. 720]<\/li>\r\n\t<li style=\"font-weight: 400;\">Replication is an essential part of the scientific process. Michael and his colleagues did not back off from the importance of their difficulty reproducing the McCabe and Castel results. Clearly, McCabe and Castel\u2019s conclusions\u2014that \u201cthere is something special about the brain images with respect to influencing judgments of scientific credibility\u201d\u2014need to be taken as possibly incorrect.<\/li>\r\n\t<li style=\"font-weight: 400;\">Michael and his colleagues looked closely at the McCabe and Castel results and their own, and they looked for interesting reasons that the results of the two sets of studies might be different.\r\n\r\n<ul>\r\n\t<li style=\"list-style-type: none;\">\r\n<ul>\r\n\t<li style=\"font-weight: 400;\"><strong>Subtle effects<\/strong>: Perhaps the brain pictures really do influence their judgments, but only for some people or under very specific circumstances.<\/li>\r\n\t<li style=\"font-weight: 400;\"><strong>Alternative explanations<\/strong>: Perhaps people assume that irrelevant information is not typically presented in scientific reports. People may have believed that the brain images provided additional evidence for the claims.<\/li>\r\n\t<li style=\"font-weight: 400;\"><strong>Things have changed<\/strong>: The McCabe and Castel study was conducted in 2008 and the failed replication was in 2013. Neuroscience was very new to the general public in 2008, but a mere 5 years later, in 2013, it may have seemed less impressive.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n<p>Do images really directly affect people\u2019s judgments of the quality of scientific thinking? Maybe yes. Maybe no. That\u2019s still an open question.<\/p>\r\n<h2>One Final Note<\/h2>\r\n<p>When we wrote to Dr. Alan Castel for permission to use his stimuli in this article, he not only consented, but he also sent us his data and copies of all of his stimuli. He sent copies of research by a variety of people, some research that supported his work with David McCabe and some that did not.\u00a0The goal is to find the truth, not to insist that everything you publish is the last word on the topic.<\/p>\r\n<p>Scientists disagree with one another all the time. But the disagreements are (usually) not personal. The evidence is not always neat and tidy, and the best interpretation of complex results is seldom obvious.<\/p>","rendered":"<h2>Does it Replicate? The Messy World of Real Science<\/h2>\n<p>A few years after the McCabe and Castel study was published, some psychologists<a class=\"footnote\" title=\"Robert B. Michael, Eryn J. Newman, Matti Vuorre, Geoff Cumming, and Maryanne Garry (2013). On the (non)persuasive power of a brain image. Psychonomic Bulletin &amp; Review, 20(4), 720-725.\" id=\"return-footnote-56-1\" href=\"#footnote-56-1\" aria-label=\"Footnote 1\"><sup class=\"footnote\">[1]<\/sup><\/a> at the University of Victoria in New Zealand, led by Robert Michael, were intrigued by the results and they were impressed by how frequently the paper had been cited by other researchers (about 40 citations per year between 2008 and 2012\u2014a reasonably strong citation record). They wanted to explore the brain image effect, so they started by simply replicating the original study.<a class=\"footnote\" title=\"They actually tried to replicate Experiment 3 in the McCabe and Castel study. You read Experiment 1. These two experiments were similar and supported the same conclusions, but Dr. Michael and his colleagues preferred Experiment 3 for some technical reasons.\" id=\"return-footnote-56-2\" href=\"#footnote-56-2\" aria-label=\"Footnote 2\"><sup class=\"footnote\">[2]<\/sup><\/a><\/p>\n<p>In their first attempt at replication, the researchers recruited and tested people using an online site called Mechanical Turk. With 197 participants, they found no hint of an effect of the brain image on people\u2019s judgments about the validity of the conclusions of the article they read.<\/p>\n<p>In a second replication study, they tested students from their university and again found no statistically significant effect. In this second attempt, the results were in the predicted direction (the presence of a brain image was associated with higher ratings), but the differences were not strong enough to be persuasive.<\/p>\n<p>They tried slight variations on instructions and people recruited, but across 10 different replication studies, only one produced a statistically significant effect.<\/p>\n<section class=\"textbox tryIt\"><iframe loading=\"lazy\" id=\"ohm3435\" class=\"resizable\" src=\"https:\/\/ohm.one.lumenlearning.com\/multiembedq.php?id=3435&theme=lumen&iframe_resize_id=ohm3435&source=tnh&show_question_numbers\" width=\"100%\" height=\"400\"><\/iframe><\/section>\n<p>So, did Dr. Michael and his colleagues accuse McCabe and Castel of doing something wrong? Did they tear apart the experiments we described earlier and show that they were poorly planned, incorrectly analyzed, or interpreted in a deceptive way?<\/p>\n<p>Not at all.<\/p>\n<p>It is valuable to see how professional scientists approached the problem of failing to replicate a study. Here is a quick review of the approach taken by the researchers who did not replicate the McCabe and Castel study:<\/p>\n<ul>\n<li>First, they did not question the integrity of the original research. David McCabe<a class=\"footnote\" title=\"David McCabe, the first author of the original paper, tragically passed away in 2011 at the age of 41. At the time of his death, he was an assistant professor of Psychology at Colorado State University, and he had started to build a solid body of published research, and he was also married with two young children. The problems with replicating his experiments were only published after his death, so it is impossible to know what his thoughts might have been about the issues these challenges raised.\" id=\"return-footnote-56-3\" href=\"#footnote-56-3\" aria-label=\"Footnote 3\"><sup class=\"footnote\">[3]<\/sup><\/a> and Alan Castel are respected researchers who carefully reported on a series of well-conducted experiments. They even noted that the original paper was carefully reported, even if journalists and other psychologists had occasionally exaggerated the findings: \u201cAlthough McCabe and Castel (2008) did not overstate their findings, many others have. Sometimes these overstatements were linguistic exaggerations\u2026Other overstatements made claims beyond what McCabe and Castel themselves reported.\u201d [p. 720]<\/li>\n<li style=\"font-weight: 400;\">Replication is an essential part of the scientific process. Michael and his colleagues did not back off from the importance of their difficulty reproducing the McCabe and Castel results. Clearly, McCabe and Castel\u2019s conclusions\u2014that \u201cthere is something special about the brain images with respect to influencing judgments of scientific credibility\u201d\u2014need to be taken as possibly incorrect.<\/li>\n<li style=\"font-weight: 400;\">Michael and his colleagues looked closely at the McCabe and Castel results and their own, and they looked for interesting reasons that the results of the two sets of studies might be different.\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\"><strong>Subtle effects<\/strong>: Perhaps the brain pictures really do influence their judgments, but only for some people or under very specific circumstances.<\/li>\n<li style=\"font-weight: 400;\"><strong>Alternative explanations<\/strong>: Perhaps people assume that irrelevant information is not typically presented in scientific reports. People may have believed that the brain images provided additional evidence for the claims.<\/li>\n<li style=\"font-weight: 400;\"><strong>Things have changed<\/strong>: The McCabe and Castel study was conducted in 2008 and the failed replication was in 2013. Neuroscience was very new to the general public in 2008, but a mere 5 years later, in 2013, it may have seemed less impressive.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Do images really directly affect people\u2019s judgments of the quality of scientific thinking? Maybe yes. Maybe no. That\u2019s still an open question.<\/p>\n<h2>One Final Note<\/h2>\n<p>When we wrote to Dr. Alan Castel for permission to use his stimuli in this article, he not only consented, but he also sent us his data and copies of all of his stimuli. He sent copies of research by a variety of people, some research that supported his work with David McCabe and some that did not.\u00a0The goal is to find the truth, not to insist that everything you publish is the last word on the topic.<\/p>\n<p>Scientists disagree with one another all the time. But the disagreements are (usually) not personal. The evidence is not always neat and tidy, and the best interpretation of complex results is seldom obvious.<\/p>\n<hr class=\"before-footnotes clear\" \/><div class=\"footnotes\"><ol><li id=\"footnote-56-1\">Robert B. Michael, Eryn J. Newman, Matti Vuorre, Geoff Cumming, and Maryanne Garry (2013). On the (non)persuasive power of a brain image. Psychonomic Bulletin &amp; Review, 20(4), 720-725. <a href=\"#return-footnote-56-1\" class=\"return-footnote\" aria-label=\"Return to footnote 1\">&crarr;<\/a><\/li><li id=\"footnote-56-2\">They actually tried to replicate Experiment 3 in the McCabe and Castel study. You read Experiment 1. These two experiments were similar and supported the same conclusions, but Dr. Michael and his colleagues preferred Experiment 3 for some technical reasons. <a href=\"#return-footnote-56-2\" class=\"return-footnote\" aria-label=\"Return to footnote 2\">&crarr;<\/a><\/li><li id=\"footnote-56-3\">David McCabe, the first author of the original paper, tragically passed away in 2011 at the age of 41. At the time of his death, he was an assistant professor of Psychology at Colorado State University, and he had started to build a solid body of published research, and he was also married with two young children. The problems with replicating his experiments were only published after his death, so it is impossible to know what his thoughts might have been about the issues these challenges raised. <a href=\"#return-footnote-56-3\" class=\"return-footnote\" aria-label=\"Return to footnote 3\">&crarr;<\/a><\/li><\/ol><\/div>","protected":false},"author":20,"menu_order":32,"template":"","meta":{"_candela_citation":"[{\"type\":\"original\",\"description\":\"Brain Imaging and Messy Science\",\"author\":\"Patrick Carroll for Lumen Learning\",\"organization\":\"Lumen Learning \",\"url\":\"\",\"project\":\"\",\"license\":\"cc-by\",\"license_terms\":\"\"}]","pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"part":22,"module-header":"learn_it","content_attributions":[{"type":"original","description":"Brain Imaging and Messy Science","author":"Patrick Carroll for Lumen Learning","organization":"Lumen Learning ","url":"","project":"","license":"cc-by","license_terms":""}],"internal_book_links":[],"video_content":null,"cc_video_embed_content":{"cc_scripts":"","media_targets":[]},"try_it_collection":null,"_links":{"self":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapters\/56"}],"collection":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/wp\/v2\/users\/20"}],"version-history":[{"count":9,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapters\/56\/revisions"}],"predecessor-version":[{"id":7065,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapters\/56\/revisions\/7065"}],"part":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/parts\/22"}],"metadata":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapters\/56\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/wp\/v2\/media?parent=56"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/pressbooks\/v2\/chapter-type?post=56"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/wp\/v2\/contributor?post=56"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/content.one.lumenlearning.com\/introductiontopsychology\/wp-json\/wp\/v2\/license?post=56"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}