Skeptics & mavericks

Editorial note: This series was originally published in the Parents PACK newsletter, a free monthly e-newsletter for the public that addresses vaccines and related topics. To learn more about the program, visit vaccine.chop.edu/parents.

Throughout the course of the COVID-19 pandemic, a few scientific papers have gained attention because they were considered to be proof of popular points of view. Typically, a single paper does not prove anything; instead, it supports a theory or adds evidence to a body of knowledge. Indeed, often when a study is enthusiastically supported for a popular social idea, a deeper look finds that the conclusions are not as clear-cut as they are being deemed. Often, this is because the findings were “cherry-picked.” Unfortunately, cherry-picking, or selectively choosing parts of the data to accept while ignoring other parts, is one of many tactics used by people who want to fit science to their beliefs, but that is not how science works.

In this three-part series, we address common misconceptions about the practice of science (and scientists) and use some specific examples of studies to help you sort out the headlines and wade through your social media feeds. Part two looks at some studies that ultimately did change our understanding of previously accepted science, and part three addresses some studies that were misinterpreted with a look at what aspects were at the heart of the confusion and why.

So, let’s get started.

There is no such thing as “my science and your science”

If your car won’t start, you’re likely to go through a variety of possible issues that could be preventing the car from starting — battery, gas, engine, etc. Your goal is to figure out why the car won’t start. This is similar to how science is done. Scientists start with a problem or a question and they go through a variety of experiments to get more information about the matter.

Now back to your car. You may have an idea that the problem is the battery. This “hunch” probably comes from existing information — the age of your battery, a hesitation you noticed when you started the car last week, etc. Your hunch about the battery is your educated guess. In science, it is called your hypothesis. If you find that the battery is not the issue, you will keep looking for the problem. It does not matter that your hypothesis was not correct; it just means something else is going on. The same is true of science. Each study adds a bit more information, and a scientist needs to let go of or adjust a hypothesis that does not agree with their findings.

If we think about that in terms of your car problem, it would not help you to keep focusing on the battery when you have evidence that it’s not the problem because your car still won’t start.  The same is true for science. When done properly, science aims to find an answer — not to support a position. Scientists can’t start out with the end in mind. They can make a hypothesis about what they think they will find, but they can’t work as though the answer is already known. They need to remain open minded and be OK with their hypothesis being wrong. They also can’t assume they are right and that the experiment is wrong just because what they wanted to be true is not. Science is a way of knowing about the world. It is about both finding and ruling out information about a topic.

Maybe you are hoping the problem is your battery because you have a spare one in your garage. In this scenario, you have a pre-existing bias toward the battery being the problem. Because you are biased toward this outcome, you may not be as careful in your evaluation of other possibilities. Rather you may just go get the battery and put it in your car — only to find that your car still won’t start. Your bias got in the way of your assessment. While everyone has biases, scientists are trained to study in a manner that reduces their personal biases in the outcome. Now, just like any profession, some scientists are better at overlooking their biases than others, but it is important to realize that even though scientists can be biased, science cannot. Said another way, even if you really want the issue to be your car battery, your car won’t start simply by replacing the battery if something else is going on — no matter how much you want it to.

Scientists are skeptics

Generally speaking, people who practice science are skeptics. They are not just going to believe what they are told; they will seek out weaknesses and evaluate the validity of the conclusions. This is where the peer-review process comes in. Anytime a study is published, other scientists critically evaluate it, meaning they take a long and careful look under the hood. What method was used? How many samples were taken? Were the appropriate statistics used? Do the conclusions follow from the data? From the outside, this process can look like taking sides based on one’s own beliefs, but it’s not that simple. Peer review requires more than just saying a study isn’t good. It also requires substantiating that feedback with evidence.

Because most journals only publish studies that have been peer reviewed, the process often keeps poorly constructed studies from being published, but that is not always the case. For example, some journals allow people to publish simply by paying a fee. Now, publishing in one of these journals does not necessarily mean the study is poorly done, but it is important to know whether a study was peer reviewed or not because it helps us understand if other scientists got to evaluate it before it was published, and even after publication, studies will continue to be evaluated as more and more scientists review them. In some cases, as more scientists review a study, concerns arise about its quality. On very rare occasions, these concerns can lead to a study being retracted, or removed from publication, but more often, the study will just fade into the background of scientific literature. You can think of studies like pieces of a giant puzzle with scientists from all over the world working to add pieces to their own little corner of that puzzle. Some studies will add a very important piece to the puzzle, filling in a hole and tying together several other findings, but most studies add one little piece to the edge — important, but still a baby step toward seeing the big picture.

Scientists who repeatedly set up studies to get a pre-determined outcome will not contribute much to the puzzle, nor will they gain the respect of their colleagues. Other scientists working on the puzzle will come to realize that those individuals or groups are pushing an agenda rather than practicing science.

Mavericks are rare

Sometimes a scientist keeps publishing studies that go against the generally accepted body of knowledge about a topic. As their work continues to be dismissed or criticized by colleagues, they often take their ideas directly to the media and the public, assuming the role of victim. Sometimes they attempt to position themselves as mavericks whose findings are so earth-shattering that they are upsetting established understanding about a topic. They suggest that they are being targeted for their “breakthrough,” but vow to keep fighting. Often this rhetoric is accompanied by suggestions of a cover-up — by the government, by companies interested only in profits, by scientists, or by some combination of these. To this, we ask: Have you ever tried to get a group of people to keep a secret? How’d that go? When someone is suggesting a massive cover-up, start to think about how many people would need to be involved, and then think about the last surprise party that you planned.

While sometimes scientists do uncover novel findings that dispute established understanding, those who respect the scientific enterprise welcome the criticism of their colleagues and go back into the lab to generate more data that will support their findings or refute their critics. Practicing science means having a thick skin; scientists have to get used to accepting criticism and using it to make their work even better. If they have uncovered something real that will substantially change how we think about a particular topic, they’ll continue generating data that will make their case stronger. Likewise, others will try to repeat the findings. Similar findings by other groups of scientists add credibility to the findings. This is another reason mavericks are rare because as more people find similar results, they too start supporting the ideas and what was originally an idea of one becomes that of many. The saying popularized by Carl Sagan comes to mind, “extraordinary claims require extraordinary evidence.” So, if a scientist is in front of the cameras presenting themselves as a victim, consider that they should be in their lab collecting some extraordinary evidence.

The takeaways

1. It takes more than one study to support a new idea or change current thinking about a topic. Keep watching for a theme to emerge from many papers.
2. Individual scientists may be biased, but science isn’t. As such, don’t rely on a single expert’s point of view. Look to what is being said about a topic by multiple scientists. Do this by not only looking at multiple news outlets, but also by actively seeking out information from science-based organizations, like National Geographic, Smithsonian, American Association for the Advancement of Science (AAAS) or National Academy of Science (NAS).
3. Be wary of scientists who portray themselves as victims.

Resources for evaluating information

Find resources that can help when evaluating information.

Download a PDF version of this article.