Skip to content

Podcast

Why Is There So Much Fraud in Academia?

Freakonomics Radio

Source ↗ ← All highlights
  • Cheating in Academia
    • Cheating is a fundamental economic act, getting more for less.
    • Academic researchers are not immune to cheating. Transcript: Stephen J. Dubner I rarely do this, but today I’m going to start by reading a couple sentences from Freakonomics, which Steve Levitt and I published in 2005. Cheating, we wrote, may or may not be human nature, but it is certainly a prominent feature in just about every human endeavor. Cheating is a primordial economic act, getting more for less. So when you think about it, why shouldn’t we expect cheating even among scientific researchers? Consider this. Today, it is thought that Ptolemy, the second century Greek astronomer, faked his observations to fit his theories. And a new study in the journal Nature found that last year, more than 10,000 research articles were retracted, easily breaking the old record. (Time 0:07:43)
  • Research Retractions
    • Over 10,000 research articles were retracted in 2023, a new record.
    • While many retractions come from U.S. universities, Saudi Arabia, Pakistan, Russia, and China lead. Transcript: Stephen J. Dubner While a lot of the recent headlines are about scholars at big-name American universities, the countries with the most retractions were Saudi Arabia, Pakistan, Russia, and China. Brian Nosek Fraud has existed since science has existed. And that’s primarily because humans are doing the science. And people come with ideas, beliefs, motivations, reasons that they’re doing the research that they do. And in some cases, people are so motivated to advance an idea or themselves that they are willing to change the evidence fraudulently to advance that idea or themselves. That (Time 0:08:30)
  • Human Element in Science
    • Fraud in science exists because humans, with their motivations and biases, conduct it.
    • Some researchers manipulate evidence to advance their ideas or careers. Transcript: Stephen J. Dubner Is Brian Nosek, a psychology professor at the University of Virginia. In 2013, he founded the Center for Open Science, a nonprofit that tries to improve the integrity of scientific research. Just to get it out of the way, I asked Nosek where his funding comes from. Brian Nosek Our funders include NIH, NSF, NASA, and DARPA as federal sources, and then a variety of private sources such as the John Templeton Foundation, Arnold Foundation, and many others. And that diverse group of funders, and it’s quite diverse, I think share the recognition that the substantive things that they are trying to solve won’t be solved very effectively If the work itself is not done credibly. Stephen J. Dubner In other words, the stakes here are high, higher than just individual academic researchers trying to advance their careers. If your goal is to improve medicine or transportation or immigration policy, any area where decisions are based on academic research, you don’t want that research to be compromised. Brian Nosek There are specific cases where a finding gets translated into public policy or into some type of activity that then ends up actually damaging people, lives, treatments, solutions. One of the most prominent examples is the Wakefield scandal relating to development of autism and the notion that vaccines might contribute. And that has had an incredibly corrosive impact on public health, on people’s beliefs about the sources of autism, the impacts of vaccines, et cetera. And that is very costly for the world. There’s also a local cost in academic research, which is just a ton of waste. So even if it doesn’t have public downstream consequences, if a false idea is in the literature and other people are trying to build on it, it’s just waste, waste, waste, waste. (Time 0:09:10)
  • Ideals of Academic Research
    • Academic research, ideally, operates under strict rules of data accuracy and peer review.
    • It aims to be impartial and free from personal or financial interests. Transcript: Stephen J. Dubner But the fact that Brian Nosek has been kept very busy with his Center for Open Science suggests that my faith in academic research has been misplaced. I asked Nosek to walk me through how he went from being a researcher himself to being a new sort of referee. Brian Nosek Yeah, I have always had an interest in how to do good science as a principled matter. And in doing that, we in the lab would work on developing tools and resources to be more transparent with our work, to try to be more rigorous with our work, to try to do higher powered, More sensitive research designs. And so I wrote grant applications to say, can we make a repository where people can share their data? This is like 2007. And they would get polarized reviews where some reviewers would say, this would change everything. It’d be so useful to be more transparent with our work. And others saying, but researchers don’t like sharing their data. Why would we do that? (Time 0:12:35)
  • Academic Reward System
    • The academic reward system prioritizes publication, creating a disincentive for data sharing.
    • Researchers fear losing control of their work will harm their careers. Transcript: Brian Nosek And why would researchers not want to share their data? Yeah, it’s based on the academic reward system. Publication is the currency of advancement. I need publications to have a career, to advance my career, to get promoted. And so the work that I do that leads to publication, I have a very strong sense of, oh my gosh, if others now have control of this, my ideas, my data, my designs, my solutions, then I will Disadvantage my (Time 0:13:39)
  • Reproducibility Project
    • Brian Nosek’s Reproducibility Project found that less than half of replicated psychology studies had successful replications.
    • This highlights potential issues with research reliability. Transcript: Stephen J. Dubner Can see how these incentives create a problem. If a system has a built-in bias against transparency, not only will there be less transparency, but also more opportunity to cheat. Nosek and his colleagues set out to address this by trying to replicate the result of papers that had already been published in academic journals. They called their idea the Reproducibility Project. Brian Nosek And in the end of that, 2015, we published The Findings, which was a 270 co-author paper of 100 replications of findings from three different journals in psychology. We got a little less than half of the findings successfully replicated. You did not mishear Brian Nosek. Stephen J. Dubner That’s what he said. Brian Nosek A little less than half of the findings successfully replicated. (Time 0:15:18)
  • Data Colada’s Motivation
    • The Data Colada team investigated research methodologies because they often didn’t believe published findings.
    • They questioned findings that didn’t align with their intuition. Transcript: Uri Simonsohn They focus on examining the methodology used in this kind of research. What motivated our whole journey into methodology was that we would go to conferences or read papers and not believe them. And we would find that whenever a finding didn’t align with our intuition, we would trust our intuition over the finding. And that was, it sort of defeated the whole purpose. Like if you’re only believing things you already believe, then why bother? This guy, Daryl Bem, published a nine study paper with eight studies worth of statistically significant evidence that people have ESP. And most people were like, what is going on? Like this, this cannot be a true finding. So the idea was like, how do we show people that you can really very easily produce evidence of anything? So we thought, let’s start with something that’s obviously false. We said, okay, something that’s quite hard to do is to make people younger. We’ve been trying forever. We never succeeded. So let’s show that we can do that in a silly way. (Time 0:24:10)
  • False-Positive Psychology
    • Data Colada manipulated data to show how easily false positives can be generated.
    • They published a paper demonstrating how listening to “When I’m 64” could make people younger. Transcript: Stephen J. Dubner Ran real lab experiments with real research subjects who had real birthdates and played them real songs. When I’m 64 and two others. A control snobble, it’s called Abelie Calumba by Mr. Uri Simonsohn Scropp. And then we had another song that was meant to go in the other direction, and it didn’t work, so we just didn’t report it, which was Hot Potato. Leif Nelson Hot potato, hot potato, hot potato. Stephen J. Dubner They essentially manipulated and cherry-picked their data to produce the absurd finding they wanted, that listening to When I’m 64 does lower your age. By a full year and a half, it turns out. They published their article in Psychological Science, one of the top journals in the field. Their piece was called False Positive Psychology. Undisclosed flexibility in data collection and analysis allows presenting anything as significant. They wrote, these studies were conducted with real participants, employed legitimate statistical analyses, and are reported truthfully. Nevertheless, they seem to support hypotheses that are unlikely or necessarily false. So we were surprised when the paper got accepted. And then the immediate aftermath was shocking. (Time 0:25:26)
  • Signing at the Top
    • The “signing at the top” paper, co-authored by Dan Ariely and Francesca Gino, claimed that signing at the top of forms promotes honesty.
    • The study became highly influential. Transcript: Stephen J. Dubner Number three, two of the five co-authors on the paper were among the best-known people in this field, Dan Ariely and Francesca Gino. And four, there was already evidence that something was up with the original paper because its authors had published a second paper saying that their original findings didn’t replicate. The failure to replicate, as we heard earlier, does not necessarily mean fraud. But now the Data Collada investigators claimed to have proof that, yes, there was fraud in the original paper. That original paper was written by Dan Ariely, Francesca Gino, along with Nina Mazar, Lisa Hsu, and this man. I’m Max Bazerman, and I’m a professor at the Harvard Business School. (Time 0:36:05)
  • Bazerman’s Role
    • Max Bazerman co-authored a paper with Francesca Gino and Lisa Hsu on the “signing at the top” effect, but he didn’t personally interact with the raw data.
    • He trusted his junior colleagues to oversee the data collection process. Transcript: Stephen J. Dubner Max Bazerman, a professor of business administration at the Harvard Business School, is considered an elder statesman in the field of behavioral science. For decades, he has been publishing well-regarded research papers and books, and he’s known as a wise and caring mentor to younger scholars. That last bit, it would seem, is the best explanation for how Bazerman wound up being a co-author on what would turn out to be a very, very, very problematic research paper. This is the signing at the top paper we heard about before the break, the paper published in PNAS in 2012, which claimed that you are more likely to be honest if you sign a form at the top Before you fill in the information than if you sign at the end. This paper actually started out as two separate research projects. Max Bazerman Here’s Bazerman. So 2011, Lisa Hsu and Francesca Gino and I had a working paper that got rejected from a couple of journals that basically claimed to show if you sign a document before you fill it out, you’re More likely to tell the truth. Our studies were done in the laboratory. Stephen J. Dubner Lisa Hsu was a doctoral student at the time. Bazerman was her advisor, one of the chairs of her dissertation committee. As for Francesca Gino? Max Bazerman Francesca started visiting the Harvard Business School as a doctoral student from Italy. By 2004, she was attending my doctoral seminar, and we started to interact pretty regularly. And eventually, I was on her dissertation committee and played a pretty active role in advising her. Bazerman liked (Time 0:39:21)
  • Trusting Junior Colleagues
    • Bazerman compares his role as a senior researcher to a restaurant owner who trusts head chefs to manage ingredients.
    • This delegation allows him to focus on other tasks. Transcript: Stephen J. Dubner You think that most people, let’s say the median American who maybe holds a decent opinion of university life and academic research, which maybe that’s not the median person, maybe The median person doesn’t hold such a decent opinion. But for someone who might read an article that’s based on an academic study and say, oh, that’s interesting, I’m going to file that away as a useful, probably true piece of information. How surprised do you think that person would be to learn that a senior colleague like you, who’s co-author on a lot of papers with junior colleagues, that you personally don’t interact At all with the original data? How surprising do you think most people would find that? Max Bazerman So I wouldn’t say don’t interact with all. I certainly would read the results section pretty carefully, but I would read it with the intent of seeing if there was any error along the way. Stephen J. Dubner But how can you tell if there’s error if you’re not in the, you know, it’s like this whole issue reminds me a little bit of being, let’s say I’m a chef in a restaurant and I’m given the ingredients To cook, but I’m not allowed to examine them. Max Bazerman So I don’t know if they’re rancid or fresh or even fake. So I like that example. So instead of being a chef of a restaurant, let’s imagine that you’re the owner of 12 different restaurants. And you have a head chef in each. And that head chef is going to be what I think of as the most senior of my more junior colleagues on the project. And over time, I’ve come to trust that they’re going to be doing a really good job of overseeing the ingredients that go into the research process. And by doing so, there’s other things that I can do. I can work on, you know, making sure that we have the funds available. I could, you know, work on whatever particular problems come up administratively. I could work with more young scholars because my time is more available. So there’s lots of good by this efficiency of trusting the assistant professor on the project or the head chef at a particular restaurant so that I’m not examining the specific ways In which the sausage is made. (Time 0:42:32)
  • Widespread Adoption
    • The “signing at the top” paper gained popularity and was implemented by organizations like Lemonade Insurance and government agencies.
    • This widespread adoption highlighted the paper’s practical implications. Transcript: Stephen J. Dubner Journals. Bazerman says they got feedback suggesting their argument about signing at the top would be more believable if, in addition to the lab results that Francesca Gino provided, they had Some results from the real world as well. That’s what researchers would call a field experiment versus a lab experiment. As luck would have it, another researcher, a friend of Gino’s, no less, apparently had some good field results. Max Bazerman We collectively heard that Dan Ariely was presenting a very similar result based on a field experiment having to do with an insurance company. And Francesca reached out to Dan and we basically combined efforts to pull the three studies into one paper. Stephen J. Dubner Ariely’s data included the number of miles the customers of this insurance company reported having driven in a year. If you think about how insurance works, a customer might have an incentive to under-report their mileage in the hopes of lowering their insurance bill. Arielli’s data showed that customers who were asked to sign this mileage statement at the top reported having driven more miles than customers who were asked to sign at bottom. Again, suggesting that signing at the top makes people more honest. Max Bazerman now received the first draft of a new paper that combined the Ariely and Gino studies. Max Bazerman I’m reading the insurance field experiment for the first time at this point. And as I’m reading it, I have some questions. I wasn’t remotely thinking about fraud as an issue. I just thought there was something wrong. And what I saw as wrong was that we were reporting that the average driver in the database had driven between 24 and 27,000 miles per year. And I just looked at that and I said, that seems off. Stephen J. Dubner It seemed off to Bazerman because the average American only drives around 13,000 miles a year. So I asked some questions about it. Max Bazerman And Ariely, who was the point person for that data, the origins of which are not completely clear, sent back a very quick email saying the mileage is correct. And I continued to say, well, we need to clarify what’s going on here. It seems off that people have driven so many miles, particularly when you’re talking about tens of thousands of drivers. And eventually, Ariely comes back with the drivers are senior citizens in Florida. Sounds like they should drive even less than 24,000 miles then. Exactly my thought. So my questions continue, and I don’t get very good answers. And literally, this goes on for months, and I’m seriously considering taking my name off the paper. At the time, Lisa Hsu is a doctoral student on the job market, and she’s presenting this work. And how concerned were you about damaging her prospects? I was very concerned that if I dropped off the paper, that there’s something suspicious about Lisa’s presentation. So I keep on asking questions, but I don’t withdraw. And by early 2012, I’m attending a conference and I arrive at the conference and in the big hallway, I run into Lisa, who’s my advisee, my friend, my co-author, somebody who I like a lot. And she’s with Nina Mazar, who I had never met before. So Lisa introduces us and I believe I was expressing my unhappiness with the lack of clarity on this mileage issue. Stephen J. Dubner Nina Mazar is Ariely’s collaborator on the insurance study, correct? So that’s a little bit unclear. Max Bazerman Okay, so when we asked Ariely to join forces, he said, fine, but Nina would be part of the project as well. So I always thought she was part of the insurance study. Later on in life, Nina claimed she had no more connection to the insurance study than I did, that she first connected to it when this five-author paper comes together. Stephen J. Dubner But in any case, at this conference, she assuaged you to some degree. Yeah, exactly. Max Bazerman So she basically pleasantly and openly pulled up the database on her computer. And I said, so what’s going on? And she said, I think what’s going on is that we don’t know that the period between time one and time two for assessing the number of miles driven was one year. We know when time two is collected, but it may have been more than one year for when time one was collected. And in my mind, what becomes clear is that that makes our study noisier. But as long as a real experiment was run, this is actually pretty good news. All we need to do is correct the presentation in the paper, which we did. The paper’s submitted. It’s published. I develop a belief that this effect is true. And people love this result. And from a theoretical standpoint, it’s a shockingly simple idea. From a practical standpoint, it’s just perfect. It’s so simple that organizations can easily implement it. And who did implement it? A lot of people implemented it. You know, I think Lemonade Insurance, under Ariely’s advice, implemented it. Lemonade Insurance, by the way, didn’t just implement Ariely’s advice. They hired him as their chief behavioral officer. And many government agencies implemented, including the U.S. Government. (Time 0:44:50)
  • Replication Failures
    • Bazerman’s attempts to replicate the signing-first effect online failed repeatedly, leading to a re-evaluation of the original findings.
    • He questioned the validity of the initial research. Transcript: Stephen J. Dubner No. Did you leave because the finding was fraudulent? No. No, I have a terrific relationship with Slice. Okay, so going back to 2016, Max Bazerman wanted to help his newfound cousin learn whether signing at the top would be as effective in an online setting, the way it seemed to work with Paper documents. So Bazerman and some junior colleagues set out to test that question. And how was Bazerman feeling at the time about the original sign at the top finding? We know it works. We know the effects are big. We know the world is intrigued by it. Seems perfect. Media descriptions of this signing-first phenomenon, that if they encounter a form where they’re asked to sign-first, they will now know that they are under some kind of scrutiny, Perhaps, and therefore they’re more likely to be honest because they know about it? Max Bazerman I think as of the time we were doing these online studies, and there are many of them, I don’t think that there was widespread public awareness of the signing of first effect. You know, Ariely and Gino and I talked to lots of executive classes, but I wouldn’t say it was a well-known social phenomena. But I could be wrong. So your methodological critique could be viable. But anyhow, signing first with or without a placebo doesn’t work online. We got nothing. How surprised were you? Very. And I kind of said, well, let’s take a look at what we did. Let’s see sort of how we might have messed up the design. Let’s try again. So we make some changes. We do it a second time. We make some changes. We do it the third time. Still, no effect, no effect, no effect. And recall, the 2012 paper not only has effects, it has effects across three different studies that are all statistically significant, and the effects are large. So the project clearly transforms somewhere between replication failure three to five. It’s transforming from how do we get people to tell the truth online to a massive replication failure of a pretty visible academic effect. (Time 0:52:36)
  • Evidence of Fraud
    • Data Colada presented Bazerman with evidence of fraud in both the insurance and lab studies related to the “signing at the top” paper.
    • This revelation shocked Bazerman and led to further investigation. Transcript: Max Bazerman So there’s a Zoom, and the first part of the Zoom is the Data Collada team showing me the evidence for fraud in the insurance paper. And it’s kind of overwhelming. These guys are careful and they’re thorough. And like, they convinced me that there was fraud in the study. Stephen J. Dubner This insurance study was one of three studies in the sign at the top paper that Bazerman had co-authored some years back. The data collada researchers had scrutinized the data that Dan Ariely had used and found several things suspicious. The most obvious one was a data chart called a histogram showing the number of miles driven each year by the people in his study. For data like this, a histogram will typically look like a bell curve with a lot of people clustered around the center and then some outliers sloping off toward the upper end and lower End. But this histogram showed a nearly uniform distribution of drivers from zero miles to 50,000 miles. This is not what real data look like, the data Collada team wrote on their blog post, and we can’t think of a plausible, benign explanation for it. And what’s Max Bazerman thinking now? I’m just kind of overwhelmed with the fact that I’m the author of a fraudulent paper. And later, there was new information from the insurance company that had given Dan Ariely the data. They told the Planet Money podcast that the data Ariely published was significantly different from what they had given him. And in their original data, there was no difference between those who signed the forms at the top and those who signed at the end. Although Ariely declined to do an interview for this episode, he did send a written statement. As someone who spent many years studying dishonesty, he wrote, I appreciate the irony of being accused of dishonesty. There’s no question that the data underlying the 2012 study I co-authored with four other researchers about dishonesty was, well, dishonest. I have looked diligently to figure out what went wrong, but given that this took place more than 15 years ago, I can’t tell for sure what happened. He added, all five co-authors of the study in question participated in review sessions with people from the insurance company and asked questions about the data. Ultimately, we all felt satisfied with the answers we got and collectively decided to move forward with the paper. (Time 1:02:30)
  • Investigation of Gino
    • Data Colada began investigating Francesca Gino’s work after receiving tips from a graduate student and an anonymous researcher.
    • They suspected fraud in multiple papers authored by Gino. Transcript: Stephen J. Dubner Data Collada had begun investigating Francesca Gino after they received a tip from a graduate student named Zoe Ziani and another anonymous researcher. In addition to the sign at the top paper, the Data Collada team wrote, quote, we believe that many more Gino authored papers contain fake data, perhaps dozens. Leif Nelson Professor Gino has indicated that she has done nothing wrong. And we have said that the data in those four papers contain evidence that strongly suggests that there is fraud. (Time 1:08:00)
  • Incentives and Cheating
    • Simine Vazire argues that academic incentives encourage cheating.
    • A rational, self-interested researcher would likely cheat given the current system. Transcript: Joseph Simmons Yeah. So if you were just a rational agent acting in the most self-interested way possible as a researcher in academia, I think you would cheat. I think that is absolutely the way the incentives are set up. I don’t think most people do, but not because of the incentives. (Time 1:15:03)