The Replication Crisis: Learn It 2—The McCabe and Castel Experiment

Method

To test their hypothesis, McCabe and Castel asked 156 college students to read the three articles and rate them for how much they made sense scientifically. Everybody read the same articles, but the picture that accompanied the article differed to create three experimental conditions. Each participant read all three articles: one with a brain image, one with a bar graph, and one without any illustration (the control condition). Across all the participants, each article was presented approximately the same number of times in each condition, and the order in which the articles were presented was randomized.

The Brain Image Condition: For the article in the brain image condition, subjects saw one of the following brain images to the side of the article:

3 different images. The first is the brain activation fMRI showing activity in the brain, the other shows an overhead fMRI of activation and a statement that says "meditation enhances creative thought." The last shows another brain scan saying "playing video games enhances attention."
Figure 1. Subjects in the experimental condition were shown ONE of the applicable brain images with each article they read.

The Bar Graph Condition: The figures below show the graphs that accompanied the three articles for the bar graph condition. The results shown in the graphs were made up by the experimenters, but what they show is consistent with the information in the article. Graphs are a common and effective way to display results in science and other areas, but most people are so used to seeing graphs that (according to McCabe and Castel) people should be less impressed by them than by brain images.

3 bar graphs. Participants in the bar graph condition were shown one of the following with each article they read: The first bar graph called “Watching TV is Related to Math Ability" shows two bars of about the same height measuring parietal lobe activation. The second bar graph called “Meditation Enhances Creative Thought” shows similar levels of activation in the occipital, temporal, parietal, and frontal lobes. The third bar graph called “Playing Video Games Enhances Attention” shows shows more activation in the parietal lobe in gaming experts than in gaming novices.
Figure 2. Participants in the bar graph condition were shown ONE of the bar graphs with each article they read.

The Control Condition: In the control condition, the article was presented without any accompanying figure or picture. The control condition tells us how the subjects rate the articles without any extraneous, but potentially biasing, illustrations.

Ratings

Immediately after reading each article, the participants rated their agreement with three statements: (a) The article was well written, (b) The title was a good description of the results, and (c) The scientific reasoning in the article made sense. Each rating was on a 4-point scale: (score=1) strongly disagree, (score=2) disagree, (score=3) agree, and (score=4) strongly agree. Remember that the written part of the articles was exactly the same in all three conditions, so the ratings should have been the same if people were not using the illustrations to influence their conclusions.

Before going on, let’s make sure you know the basic design of this experiment. In other words, can you identify the critical variables used in the study according to their function?

Results

How do you think participants responded to each of these questions, based on the type of article they read?

Results for (a) Accuracy of the Title

For the question about the title, the experimenters predicted that there would be no difference in the ratings between the 3 articles, and their prediction was correct. Subjects gave about the same rating to the titles in all three conditions, agreeing that it was accurate.

Results for (b) Quality of the Writing

For question (b) about the quality of the writing, the experimenters found that the two conditions with illustrations (the brain images and the bar graphs) were rated higher than the control condition. Apparently, just the presence of an illustration made the writing seem better. This result was not predicted.

Results for (c) Scientific Reasoning Assessment

The main hypothesis behind this study was that subjects would rate the quality of the scientific reasoning in the article higher when it was accompanied by a brain image than when there was a bar graph or there was no illustration at all. If the ratings differed among conditions, then the illustrations—which added nothing substantial that was not in the writing—had to be the cause.

Use the graph below to show your predicted results for the scientific reasoning assessment. Move the bars to the point where you think people generally agreed or disagreed with the statement that “the scientific reasoning in the article made sense.” Higher bars mean that the person believes the reasoning in the article is better, and a lower bar means that they judge the reasoning as worse. Click on “Show Results” when you are done to compare your prediction with the actual results.

Discussion

McCabe and Castel conducted two more experiments, changing the stories, the images, and the wording of the questions in each. Across the three experiments, they tested almost 400 college students and their results were consistent: participants rated the quality of scientific reasoning higher when the writing was accompanied by a brain image than in other conditions.

The implications of this study went beyond brain images. The deeper idea is that any information that symbolizes something we believe is important can influence our thinking, sometimes making us less thoughtful than we might otherwise be. This other information could be a brain image or some statistical jargon that sounds impressive or a mathematical formula that we don’t understand or a statement that the author teaches at Harvard University rather than Littletown State College.

But, there’s another catch…this study hasn’t replicated well. What does that mean, exactly? Read on to find out.