Podcast
Balaji Srinivasan- The Network State
The Knowledge Project
- Replication of the Digital Academic urban legends can be traced back to their source. Theories are often based on rumors that have never been proven or disproved. Research is done by building a chain of citations and checking them against each other. If you disagree with it, or question something, you drill down an look at its citations. So either it’s data is correct or incorrect; maybe the data itself is fraudulent. Transcript: Balaji S. Srinivasan Like, is spinach a good source of iron, like the myth? Shane Parrish Well, you’d be able to trace everything back, right? So if something was unproven or didn’t replicate, then you would also know all the implications of it through all the different, how it’s weaved its way through science, the term used Loosely there, not reproducible science, but how we’ve used that information to make decisions or how it’s influenced future studies or implications. Balaji S. Srinivasan Yeah, exactly. So here’s like this, there’s an interesting study, which is basically about trying to backtrace and find out whether, this story is complicated and I don’t want to paraphrase it. Academic urban legends, okay, old Bjorn Rektal. Many of the messages presented in respectable scientific publications are in fact based on various forms of rumors. Some of these rumors appear so frequently that we can think of them as academic urban legends. The explanation for this phenomenon is usually that authors have lazily, sloppily, or fraudulently employed sources and peer reviews and editors have not discovered these weaknesses In the manuscripts during evaluation. To illustrate this phenomenon, I draw upon a remarkable case in which a decimal error appears to have misled millions into believing that spinach is a good source of nutrition. Through this example, I demonstrate how an academic urban legend can be conceived and born and continue to grow and reproduce within academia and beyond. And so he just goes into this story and the sub-myths and myths and myths and so on. Point is, something you might think was a solved problem, like is tracking back citations, is not a solid problem. In fact, it’s actually something where the evidentiary base is actually not as firm as you might think, especially in biomedicine. And so what you actually want to do is click, click, click, and see the original table in 1963 and the specific rows and columns that are giving this parameter that is being used on millions Of people, right? It is basically just like in any complicated software engineering project today, you have the set of dependencies, and dependency management is a whole thing. If you’re coding Node or something like that, you have like 30 libraries that your program depends on, and there’s specific version numbers, and this is a whole thing. So now you bring that kind of rigor into actually the management of dependencies for scientific paper. And those are essentially the input premises that are then expanded upon by the paper that leads to the conclusion of that paper and its incremental contribution to literature, right? So now, you know, what I’m describing here, by the way, is of the complexity of Wikipedia, but not greater than that. You’re basically talking about taking the scientific literature, putting it into truly reproducible research format and putting it on chain such that it’s permalinked and then Building a citation map back to each other. Could you do this? Yes. Is it a large project? Yes. Is it a finite project? Yes. Because the relevant scientific papers are only a subset of the literature, right? So you can focus on that area first. Once you do this, you have at a minimum, the digital aspect is auditable. You can look at a paper if you disagree with it, or you question something, you drill down, look at citations. So either it’s data is correct or incorrect. Maybe the data itself is fraudulent. There’s a Benford’s law kind of thing you can run against it. So you can see whether it’s internally consistent, like the stuff that accountants do. There’s very statistical tests that you can run to see if it looks like it was faked or what have you, right? So you can internally check it. You can plot it and graph it yourself and look at it. Sometimes you’ve rotated data and you’re like, oh, that’s an artifact I can see on this axis that I couldn’t see looking at an XY plane, but it’s obvious in another plane. So you can analyze the data, you can look at the premises, you can analyze it a few different ways. And (Time 0:33:36)