The Reproducibility Crisis of Psychology and What It Is Trying to Tell Us

By Doug Marman

Over the last few years, a raging crisis has hit the field of psychology: Most published studies can’t be replicated by others. For example, 100 experiments published by highly respected psychology journals were recently tested and only 36% produced results in agreement with the original reports.[1] This is called the “reproducibility crisis.”

It’s a complicated problem. It isn’t caused by fraud, except in rare cases. Many factors are involved, as explained by this article. For example, designing psychology experiments is more difficult than it sounds, and drawing conclusions often involves complex statistical analysis. Even the experiments aimed at reproducing experiments have been found wanting.[2]

This has created a rift among psychologists, with half saying that the problem is more about the way reproducibility tests are run, with the other half feeling “the academic ground give up beneath their feet.” This led one reporter to ask:

“Crisis or not, if we end up with a more rigorous approach to science, and more confidence in what it tells us, surely that is a good thing?”[3]

No, I don’t think that is the answer. In fact, I believe it will make the reproducibility problem worse. The rigorous approach of traditional science is part of the problem. It is time to put a spotlight on how objectivity can interfere with psychology experiments. Otherwise, we are going to continue casting doubt on valid scientific experiments.

Take, for example, an experiment that is literally a textbook case:[4] In the 1980s, Fritz Strack and his co-workers showed that when a person smiles, it improves their mood. Many well-known psychologists, such as William James, and scientists, such as Charles Darwin, have said that expressions create emotions. It makes sense. The challenge was how to design an experiment that scientifically verifies this.

You can’t just ask people to smile, because that automatically makes them conscious of what they’re doing. That will invalidate the results. Strack and his co-workers needed to find a way to get people to move their mouths into a smile, or a pout, without them knowing what they were doing. They found an ingenious solution.

When they asked people to hold a pen in their mouths, with their mouths closed, they automatically moved their faces into a sort of pout. When they asked another group to hold a pen between their teeth without closing their lips, they naturally formed a smile. The subjects had no idea what the test was really about. They were told that the experiment was studying people trying to do two things at the same time. They needed to hold the pen in their mouths while evaluating a series of Far Side cartoons.

Images from an experiment that tested the influence of smiling versus pouting.

The results showed that the group with smiles found the cartoons funnier than the group who was pouting. In other words, just putting your face into a smile naturally brightens your day.

The experiment has been verified countless times over the last twenty-five years, by many researchers. Some have expanded and tested the idea in new ways, besides smiles and pouts, and found similar results. For example, if you take a confident stance, in front of a group, you feel more confident.

So, Strack volunteered to have his classic study be tested by a team of researchers who wanted to reproduce psychology experiments. He wasn’t concerned. It had already been validated before.

Unfortunately, results from the replication experiment contradict Strack’s conclusion. The new test was run by seventeen scientists, across eight countries, using 2,000 subjects. They found no evidence that an unintentional smile or pout made any difference in the funniness of cartoons.[5]

How can this be?

Strack questions the conclusions and the set-up of the experiments. He voiced his concerns even before the testing began, after looking over their approach. At first, as I read Strack’s complaints, it felt like he was trying to defend his original work. But a number of things made me question my first impression.

First, Strack himself offered his experiment to be tested for replication and willingly supplied his original notes and evidence. Second, it had been confirmed successfully many times by other researchers. Third, he questioned the impact of the replication experimenters excluding the results of 600 subjects because they felt those subjects were holding the pens incorrectly or their answers were too wildly divergent. Did their selection to exclude certain results introduce a bias? Fourth, Strack pointed out that many of the subjects were psychology students. Since this was a textbook case, they could have recognized the experiment and its true purpose. That would have prevented them from acting naturally. They should never have been involved.

But it was the fifth point he made that jolted my attention. Strack said that he didn’t like the addition of cameras in the room watching the subjects because it could make the participants self-conscious. That jogged my memory. I had seen this scenario before.

It was one of the most famous early studies in psychology. In 1897, George Stratton strapped on a pair of lenses over his eyes that inverted and reversed his field of view.[6] He knew that our eyes have built-in lenses that produce the same effect: All of the images hitting our retinas are flipped upside-down and reversed. Stratton wanted to see if his mind would naturally find a way to invert and correct his vision.

Sure enough, after five days of looking through inverting lenses, he saw everything as right-side-up. After a week, his new vision felt completely normal.

The results were so startling that hundreds of follow-on experiments were run to reproduce the results. Many did, but some could not. For example, David Linden, a hundred years later, called Stratton’s theory of achieving upright vision a myth.[7] This has created an ongoing controversy.

I studied dozens of experiments with inverting lenses to find an explanation for what was going on. Why were the results so different? I finally found an answer in the longest study ever performed (40 days).[8] Ivan Kohler discovered, unexpectedly, that when he tried to examine the subjects every day with a battery of clinical tests, it interfered with their ability to adapt. They actually regressed.[9]

At first, Kohler thought lab tests would help show the progress his subjects were displaying. Just as Linden did, Kohler brought them in for examination on a daily basis. However, the tests made things worse. The subjects reverted back, losing the gains they had made. What’s going on, he wondered? Kohler had to alter his tests before figuring out the problem. As soon as the experiments were designed to resemble the everyday world, the problem disappeared:

“When the subject was asked to ‘aim’ at something, or to put up his hands in protection when danger threatened…he made correct responses. But when he was asked, ‘Please point this marker in the direction the light is coming from,’ errors occurred.”[10]

That’s when Kohler realized that the subjects were adapting instinctively to the real world. The moment they tried to think critically and objectively about what they were seeing, it broke their “perceptual set.” They reverted back to pre-experimental ways of seeing the world. Asking them to analyze what they were doing prevented them from adapting.

This was hard to understand, Kohler wrote. It took weeks to solve the mystery. For example, after fourteen days of fencing practice, subjects with inverting lenses were able to respond to their opponent’s blade without errors. When it came to fencing, the correct reaction was all that mattered. But if he asked them the question, “Where do you see the rapier point?” it forced them to think critically about what they were experiencing, breaking their lens of perception. They immediately reverted back to old ways of seeing. His question interfered with their instinctive responses.

Getting the subjects to think objectively about what they were doing prevented them from adapting to upright vision. This was the mistake Linden had made. Even though Linden ran his experiment thirty years after Kohler, he didn’t realize the negative impact of objectivity. No wonder all his subjects failed to achieve upright vision.

This is the same affect that cameras can have on subjects. Strack was right: It would make them conscious of being recorded and seeing what they were doing objectively. It makes the experience less natural. On top of this chilling effect of cameras, all of the instructions telling the subjects what to do were presented by a recorded video, in a closed room with no other people, making the experience even more sterile and impersonal.

Can this explain why the subjects showed no positive effects from their unintentional smiles? I think it does. Remember, Strack was trying to study an unconscious effect. He designed his experiments specifically to avoid any interference of conscious thought on the part of the subjects. If moving their mouths into the shape of a smile influences their mood, it is going to happen unconsciously. This means they need to feel at ease and natural, or it isn’t going to work. Thinking critically and objectively about what they were doing is going to interfere.

Think of the irony: Subjecting the subjects of psychology experiments to rigorous, clinical objectivity prevented the very thing they were trying to study—natural responses. They intentionally used cameras and pre-recorded instructions to eliminate outside biases, and without knowing it they introduced a new bias that was just as powerful—objectivity.

Imagine what would happen to a loving relationship if you started analyzing your life partner or lover objectively. Do you think your relationship is going to get better or worse? Is it going to warm up or cool down your natural and playful back-and-forth exchanges?

Psychology research projects have noted the detrimental impact of objectivity on natural relationships. For example, in the last few decades, psychologists have looked closer at the way people learn new skills. John Flach, Professor of Psychology at Wright State University, offers an interesting illustration for how skill-based learning works: Look at the process a child goes through when first learning how to walk, then how to skate on ice, next how to do a handstand, and finally how to walk on stilts.

Each skill needs a “different type of coordination pattern,” a different way of acting to achieve control.[11] In other words, they each require a different lens of perception, a different way of seeing, to master these skills. They learn this unconsciously through trial and error.

Skill-based learning starts with actions. Trying something gives the child feedback, such as falling on their faces or flipping onto their backs. Then they try a new approach. With each loop of trial and error they gradually figure out how to balance and how to move. Learning at this stage is non-verbal and not mediated by thought: The child can’t explain how to balance on stilts. They don’t know how they learned to walk on their hands or skate on ice. They just did it.

This natural learning process is the best way to acquire new skills. No one teaches babies how to talk. They learn it themselves by making sounds and hearing the sounds they make. They learn how to use their bodies the same way: They form working relationships with their muscles and cells. They figure it out without thinking about it.

This is different from academic study, where we consciously think to understand new ideas and what they mean. Our natural process for learning new skills, on the other hand, is largely unconscious and critical thinking can interfere with this natural process.

Psychology experiments are not easy to design. The more rigorous and objective you make them, the more artificial they become, preventing the natural responses you are looking for. You end up learning less about how people act in the real world and more how they behave in a clinical lab.

This is why, as I said above, I believe more objectivity will make the reproducibility crisis worse, not better. What is needed is a better understanding of our lenses of perception, and where to use them. For example, objectivity, as a way of seeing, shouldn’t be the goal of science, but as a tool for double-checking and verifying our experiments. If we want our relationships with others and with our bodies to be natural and spontaneous, we need a relational lens instead, not objectivity.

Over the last century, psychologists have tried to become more rigorous and objective—to become more like physicists. At the same time physicists have come to realize that objectivity can’t explain the behavior of subatomic particles. This is the lesson they learned from quantum mechanics: How you set up an experiment alters the results, and there is nothing you can do to avoid this. In other words, there is no such thing as a fully objective perspective because all measurements influence the outcome.

This same principle applies to the study of natural human responses. It can’t be avoided. Objectivity and critical analysis can and will interfere. If we understand this better, I believe psychology experiments will become easier to reproduce.

I think Katie Palmer got it right when she said that the reproducibility crisis comes down to this:

“The field [of psychology] may have to think differently about how it thinks about itself.”


[1] Open Science Collaboration (over 260 co-authors), “Estimating the Reproducibility of Psychological Science,” Science, August 28, 2015: Vol. 349, Issue 6251.

[2] Daniel T. Gilbert, Gary King, Stephen Pettigrew, Timothy D. Wilson, Comment on ‘Estimating the Reproducibility of Psychological Science,’” Science, March 4, 2016: Vol. 351, Issue 6277.

[3] Ed Young, “Psychology’s Replication Crisis Can’t Be Wished Away,” The Atlantic, March 4, 2016.

[4] Fritz Strack, Leonard L. Martin, Sabine Stepper, “Inhibiting and Facilitating Conditions of the Human Smile: A Nonobtrusive Test of the Facial Feedback Hypothesis,” Journal of Personality and Social Psychology, Vol 54(5), May 1988, 768-777.

[5] Daniel Engber, “Sad Face,” Slate magazine,  August 28, 2016.

[6] George M. Stratton, “Vision without Inversion of the Retinal Image,” Psychological Review 4, no. 4 (1897), p. 341-360.

[7] David E. J. Linden, Ulrich Kallenbach, Armin Heinecke, Wolf Singer, Rainer Goebel, “The Myth of Upright Vision,” Perception 28, no. 4 (1999), p. 469-481. Also posted at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.294.9093&rep=rep1&type=pdf.

[8] Ivo Kohler, The Formation and Transformation of the Perceptual World, tr. Harry Fiss (New York: International Universities Press, 1964).

[9] Doug Marman, “Lenses of Perception: A Surprising New Look at the Origin of Life, the Laws of Nature, and Our Universe,” (Ridgefield, Washington, Lenses of Perception Press, 2016.), p. 88-90.

[10] Ivo Kohler, The Formation and Transformation of the Perceptual World, p. 153-155.

[11] John M. Flach and Fred Voorhorst, “What Matters?: Putting Common Sense to Work,” (Dayton, Ohio, Wright State University Libraries, 2016), p. 104-105.

The Experience of Consciousness vs Knowing Our Own Mind

By Doug Marman

The New York Times just ran an opinion piece that is a good example about how articles on neuroscience often get the big issues wrong.

Photo by Miranda Knox.

Picture by Miranda Knox.

The author, Alex Rosenberg, isn’t ignorant of the topic. He’s a co-director of the Center for the Social and Philosophical Implications of Neuroscience. In other words, he is fully informed of the science of the brain. So, he clearly has every right to state his opinions. Unfortunately, he misses the point badly.

Right from the opening paragraph, Rosenberg misdirects and misrepresents the issue. I don’t mean to say that he is doing this intentionally. I believe he is stating the problem honestly as he sees it. He’s just using the wrong lens.

Here is how he begins: Ever since Plato, philosophers have made it sound like a truism that we know the reality of our own thoughts:

“They have argued that we can secure certainty about at least some very important conclusions, not through empirical inquiry, but by introspection: the existence, immateriality (and maybe immortality) of the soul, the awareness of our own free will, meaning and moral value.”[1]

Rosenberg then goes on to berate two recent authors for continuing with this tradition, as if something that seems so fundamentally true can “trump science.” Not so, he tells us. We might think that we know what’s going on in our own minds, but numerous studies show that this simply isn’t true. We don’t know.

Here’s the first problem with this article: Plato wasn’t talking about knowing our mind. He was talking about knowing our self. He never said that we can ever truly know our own mind our even the true nature of our thoughts. The fundamental truth that Plato and many other philosophers have pointed to is the experience of being conscious.

Using “introspection” to study our thoughts isn’t even in the same ballpark as the experience of consciousness. Experiences are far more fundamental than thoughts.

A lot of neuroscientists mix these up. They do so for a good reason: They are using third-person lenses. In other words, they are taking the traditional scientific approach of viewing the matter as if they are outside observers—as if they are completely outside of the mind or the experience of consciousness and looking in. This is the objective approach, and it has long been used in science for a good reason, because it is excellent at understanding cause-and-effect relationships like we see in mechanisms and chemical reactions.

However, this is the wrong lens to use for understanding the experience of consciousness. If we insist on using a third-person approach, then we have assured our failure to see it at all. The only way to understand the nature of experience is through experience, not by mental analysis.

Trying to understand the mind by thinking about it with the mind is like trying to find reality in a hall of mirrors. Photo by Bjoern Lotz.

Trying to understand the mind by thinking about it with the mind is like trying to find reality in a hall of mirrors. Photo by Bjoern Lotz.

We might as well use a telescope to look for microbes in a drop of water. We will see nothing. Even worse, we can fool ourselves into thinking that microbes don’t even exist, because we can’t see them.

We need to use the right lens, the right tool. In this case, the only perspective that works is a “first-person” lens. This is how we experience everything, whether it be a new car, eating lunch with a friend, or our own consciousness. Every experience is a first-person perception.

What does an experience mean? That’s a different story. That’s a question we ask with our minds, as if we could interpret an experience or reduce it down to a thought. As soon as we start thinking about our experiences we’ve left the first-person world behind.

Therefore, the point that Rosenberg is making does not prove that science trumps experience. Quite the opposite. It shows us that science doesn’t understand consciousness. This is exactly why philosopher David Chalmers calls consciousness the hard problem. He writes:

“Consciousness poses the most baffling problems in the science of the mind. There is nothing that we know more intimately than conscious experience, but there is nothing that is harder to explain.”[2]

Third-person lenses don’t work because they move us outside the world of experience. Outsiders can’t see consciousness. This is why we need to use a first-person lens. Chalmers says the same thing:

“If one takes the third-person perspective on oneself—viewing oneself from the outside, so to speak—these reactions and abilities are no doubt the main focus of what one sees. But the hard problem is about explaining the view from the first-person perspective.”[3]

Unfortunately, this isn’t the only problem with Rosenberg’s article. In his zeal to show how much scientific evidence there is that we don’t know our mind, he makes some rather serious blunders. He writes:

“In fact, controlled experiments in cognitive science, neuroimaging and social psychology have repeatedly shown how wrong we can be about our real motivations, the justification of firmly held beliefs and the accuracy of our sensory equipment. This trend began even before the work of psychologists such as Benjamin Libet, who showed that the conscious feeling of willing an act actually occurs after the brain process that brings about the act—a result replicated and refined hundreds of times since his original discovery in the 1980s.”

The first sentence in the above paragraph is right. Subconscious influences affect our choices and decisions all the time. We often try to “explain” our behavior as if it is rational, when, in fact, our subconscious colors everything we do. So, the point Rosenberg is making—that we don’t fully know our own minds—is right.

It’s the second sentence that is the problem. Benjamin Libet did not show “that the conscious feeling of willing an act actually occurs after the brain process that brings about the act…” And no other experiment has proven this either. It is easy to show why Rosenberg is just plain wrong about this. Here is how I explained it in my book,

“None of the experiments show the brain making a decision before the person did. Scientists can’t prove such a claim, since they have no way of determining when a choice is made. Decision-making is a subjective process. They can’t observe it scientifically. No instrument can measure the act of choosing. They can only detect outer activity in the brain, not the inner content of consciousness.”[4]

In fact, not only is Rosenberg wrong about what Libet’s experiment shows us, there are quite a few experiments that contradict his conclusion and one shows clearly that he is wrong. In that case, the “readiness potential” brain signals that Libet detected show up whether a person decides to do something or not, so they can’t be an indicator of a decision being made:

“Judy Trevena and Jeff Miller, psychologists from New Zealand, asked a group of subjects to press a key every time they heard a tone. A second group was told to do the same thing—press a key on a computer after a tone sounds—but only half of the time. It was their choice when to push the button and when not to.

“It didn’t matter whether the subjects in the second group pressed the key or not, the same readiness potential signals were detected. This is proof that this brain activity is not the same as a conscious decision. In fact, it suggests that the term ‘readiness potential’ was right all along. The brain is simply getting ready to act.”[5]

Rosenberg makes the matter worse. He goes on to say: “there is compelling evidence” that our own self-awareness is simply our brain trying to guess at what we ourselves might be thinking. This is a misrepresentation. I’m giving Rosenberg the benefit of the doubt when I say this.

If you interpret “self-awareness” the way I do, as the experience of our own consciousness, then Rosenberg is flat out wrong. But I think what Rosenberg is getting at here is that we often guess about our own behavior and our intentions, the same way we guess at the intentions of others. He is absolutely right about that, but this is not the basis of our self-awareness.

If Rosenberg limited his conclusion to the ideas that we form about ourselves and the picture we might have of who we are, then I would agree with him. But that isn’t self-awareness. That’s our ego he is talking about—the image we have about who we are and how we fit in the world.

Self-awareness is something we gain through the direct experience of our consciousness. No thought involved. No guesswork. It is purely an experience—not an interpretation. That’s what makes this an issue that “trumps science.” Science can’t crack that nut, but we can prove to ourselves the reality of it through our own awareness.

Then Rosenberg really does it. He makes an absolutely ridiculous statement that has no scientific foundation at all, while acting as if it is shored up by empirical evidence. He writes:

“The upshot of all these discoveries is deeply significant, not just for philosophy, but for us as human beings: There is no first-person point of view.”

Photo by Gabor Kalman

Photo by Gabor Kalman

Here is the logic that Rosenberg just used to arrive at this conclusion: If you first decide to use a third-person lens, and only a third-person lens, to study the problem, then you will discover that first-person perception doesn’t exit.

Well of course it doesn’t exist if you use a lens that requires you to be an outsider looking in. How could you ever experience consciousness that way? How could you ever experience anything?

What’s the real upshot of all this? Science can’t see, detect, measure, or photograph the experience of consciousness. So, what do some scientists do? Well, they make up a story as if they understood the mind well enough to know that it is just making up the experience of consciousness. In other words, they are doing exactly what Rosenberg was telling us the mind does: guessing at the things it doesn’t understand.

If Rosenberg is right that we can’t know our mind through introspection, and I agree with him on this, then how could anyone ever come to the conclusion that the mind is fabricating the experience of consciousness? That makes no sense.

If a person is smart enough to make such a statement, why wouldn’t they be smart enough to realize that the only way it could be true is if they really did understand their mind?

It baffles me. I don’t have the answer to this question, but if you do, please explain it to me. I really would like to know.


[1] Alex Rosenberg, “Why You Don’t Know Your Own Mind,” The New York Times, July, 18, 2016.

[2] David J. Chalmers, “Facing up to the Problem of Consciousness,” Journal of Consciousness Studies 2, no. 3 (1995), p. 200. Also posted on http://consc.net/papers/facing.pdf.

[3] David J. Chalmers, “Moving Forward on the Problem of Consciousness,” Journal of Consciousness Studies 4, no. 1 (1997), p. 3–46, Section 2.2. Also posted on http://consc.net/papers/ moving.html.

[4] Doug Marman, Lenses of Perception: A Surprising New Look at the Origin of Life, the Laws of Nature, and Our Universe (Washington: Lenses of Perception Press, 2016), p. 277.

[5] Ibid., p. 278-281.