Thoughts on Epistemic Collapse
If an intelligence explosion occurs, the majority of humanity could be confused, misled, and epistemically disempowered.
Context: I’m 70% confident that “epistemic collapse” (broadly meaning: a far more severe version of “epistemic disruption”) will occur during an intelligence explosion.
In retrospect, I think much of the language I used in this post was overly dramatic and not carefully reasoned enough. I’m 30% confident in the specific takes described below.
Introduction
Knowledge defines and shapes the world around us. It dictates how we decide what is true, and how we take action on such truths. It is undeniable that the advent of superintelligent AI systems would irreversibly change how we relate to knowledge on both an individual and societal level.
MacAskill and Moorhouse refer to this as epistemic disruption in their paper “Preparing for the Intelligence Explosion”. When defining the intelligence explosion, they use the analogy of a compressed century. What if all the scientific, social, philosophical, and political advancements of the 21st century were compressed into just 10 years? MacAskill compares humanity’s situation to:
"A mediaeval king suddenly needing to upgrade from bows and arrows to nuclear weapons to deal with an ideological threat from a country he's never heard of, while simultaneously grappling with learning that he descended from monkeys and his god doesn't exist.” (emphasis mine)
The authors already identify several ways in which superintelligence would affect society’s decision-making abilities. These include: super-persuasion, stubborn resistance to valid arguments, viral ideologies, and ignoring new crucial considerations (e.g. discovering something as big as heliocentrism). The relevant section of the paper is fairly short – so I’d encourage you to go read it if you haven’t already, before proceeding with this post.
While I agree with all of the risks identified in the section, I believe that the authors are massively underestimating just how turbulent and destabilising the situation would be for humanity. It is far from clear that the impact will “likely be positive overall”, as they claim. It very well may be. But we are probably underrating the amount of work required for that to be the case.
Beyond epistemic disruption (which I take to imply manageable turbulence), I think we would be facing a potential epistemic collapse –broadly, a systemic breakdown of how humanity decides what is true. If we take seriously their idea of a century of progress compressed into a decade (or less), we face a very difficult challenge in helping people adapt when fundamental beliefs are rapidly proven wrong – and how they decide what to believe in the first place. Even if we solve the technical challenges relating to AI (e.g. alignment, control, etc), this social/psychological adjustment will be very hard to get right.
Could AI Really Help Solve This?
Yes, the benefits they identify from AI-enhanced reasoning (fact/argument checking, automated forecasting, and augmented/automated wisdom), would definitely help. But these solutions assume a level of epistemic stability that may not exist during the intelligence explosion. Consider fact-checking. The authors point to community notes on Twitter/X as a success story that AI could build upon. But community notes seem to work precisely because they operate within a shared epistemic framework – users may disagree on facts, but they generally agree on what constitutes evidence. What happens if superintelligence discovers that our fundamental assumptions about causality, consciousness, or even logic are wrong? You probably can't fact-check your way out of something like this.
I think there’s a much stronger case for automated forecasting working, but it too has a critical weakness: trust. They suggest AI systems could "build up a strong track record" that generalises to controversial domains. But track records take time to establish, and time is exactly what people won't have during an intelligence explosion. More fundamentally, if people's entire worldviews are crumbling at an ever increasing rate, why would they trust anything, even an AI with an apparently perfect prediction record? We already see this with something like climate denial. There are many cases where overwhelming evidence doesn't overcome worldview-level resistance.
"Augmented and automated wisdom" presumes people will want to turn to augmented wisdom when they perceive their most basic beliefs as being under assault. During an epistemic collapse, we’d likely lose any shared framework for determining what counts as “augmented wisdom” versus “augmented manipulation”. Some people may embrace every new AI-delivered truth uncritically. Others will reject everything defensively. Most will probably oscillate between the two, without any stable ground for making distinctions. The crisis would be the fragmentation, not just which direction people end up fragmenting. We already have seen this pattern, albeit more slowly. Darwin published On the Origin of Species over 100 years ago, yet only ~41% of humanity accepts evolution. COVID-19's uncertainty didn't lead to collective learning but to an explosion of conspiracy theories. Each group inhabited completely different realities1.
MacAskill and Moorhouse conclude that “selection pressures will probably favour desired traits on the epistemic front” because users will prefer honest and truthful models. But this assumes people can accurately assess truthfulness when their entire epistemic environment is breaking down before them. It seems that current evidence also suggests otherwise – social media algorithms already optimise for engagement over truth, and users consistently choose content that confirms their biases over content that challenges them. Sycophancy is a very big problem in current AI systems.
During an intelligence explosion, all these issues would be magnified. How do you select for “truthfulness” if the nature of truth itself is being revised monthly? Most users plausibly would select for AI systems that provide psychological comfort and coherent narratives, not those delivering difficult truths about the changing nature of reality as we know it!
Maybe AI will just get good at changing people’s minds, and we won’t need to worry about all this. But, how would this work in practice? Would this create over-reliance on the AI and whatever goals/values it is aligned to (e.g. in terms of the model spec or if it was controlled by a very small group of people)?
More Drivers of Destabilisation
Beyond the risks identified above, the intelligence explosion would very plausibly introduce entirely new epistemic threats. Ones that sound like they are straight out of a sci-fi movie.
Consider the concept of digital resurrection. Superintelligent AI could create hyperrealistic simulations of deceased individuals based on their digital footprints, writings, and recordings. Imagine your dead grandmother calling you, sounding exactly like herself, sharing memories only she would know (realistically interpolated from data), and giving you advice about your life. Is this really her preferences and wisdom, or an AI's best guess? Reviving past memories with AI already (kind of) exists. While some people would adjust to this and improve their “cognitive security” measures, many would not be able to keep pace with the rate of technological change.
Or preference extrapolation – AI systems that claim to know what you "really" want better than you do, based on patterns in your behaviour you're not even conscious of. When an AI can predict your choices with 99.9% accuracy and explain unconscious drives you didn't know you had, who is the authority on your own preferences? I’d imagine that some people would agree to adhere to AI-revealed-preferences, while others would double down on their own human cognition.
The New Underclass of Those Who Do Not Wish to Enhance
This situation isn't helped by the fact that the intelligence explosion likely would make transhumanist interventions (e.g., cognitive enhancement2, physical enhancement, direct neural interfaces, and so on) available to those who desire them and have the means to access them.
But what about the ones who do not wish to enhance and/or augment their capabilities?
A new "naturalist" underclass may emerge. Even if people have the tools to overcome their epistemic crisis, many would probably purposefully choose not to implement them due to fear, appeal to nature fallacies, or just extremely strong emotional aversion. Humanity has integrated with technology in the past (e.g. glasses, medicine, vaccines, etc), and we continue to become more transhumanist. However, this would be a sudden jump like nothing we've seen before3. Our normal human brain is not designed for the blindingly fast levels of change that would be accompany the intelligence explosion. Our species’ technological capabilities have raced ahead, but our brains remain mostly unchanged since they evolved about 200,000 years ago. The enhancements required to keep up would be drastic – not just wearing a device, but fundamentally restructuring how your brain processes information (or even relying on an external AI system to process and simplify nearly all the information you receive).
Therefore, people who say no (which could plausibly be a very large percentage of the human population) will have no say – or a very limited say – in what the future looks like. This would be massively disempowering. They would functionally become children (or, even newborns if the intelligence explosion gets really crazy) in a world run by incomprehensible adults. Democracy would become impossible when citizens are operating at such fundamentally different cognitive levels. The un-enhanced would be using entirely obsolete frameworks for determining truth. Meanwhile, the enhanced would be moving further into AI-mediated realities the rest of humanity couldn’t even begin to perceive if they had thousands of years at their disposal.
The World From a “Normal” Human’s Perspective
Here's a fictional scenario (written with the help of Claude) of what epistemic collapse might feel like for a “normal” human:
Sarah is a 45-year-old teacher. The year is 2034, two years into the intelligence explosion4. Like most of humanity, she was effectively kept in the dark that an intelligence explosion was even occurring. Though, now she could see it before her eyes. The future had come crashing down upon the present, and the world was beginning to look more and more like sci-fi.
Sarah refused all forms of enhancement, due to fears about keeping her brain “untouched” and placing her trust in “mother” nature instead. Every morning she faces the same problem: she can't tell what's true anymore. Her enhanced sister sends her "fact-checked" news through an AI system that claims to filter manipulation, but how can Sarah verify the fact-checker? She's stuck trusting black boxes or trusting nothing. Her dead mother called yesterday. Perfect voice, shared memories only they knew, offering advice about her divorce. Sarah has heard about digital resurrection, but knowing doesn't make her immune to just how scarily realistic it is. Are these her mother's actual preferences? An AI's best guess? The technology to verify doesn't exist in any form she can understand.
At work (assuming she is even able to find employment in such a world), enhanced colleagues operate through AI-mediated channels she can't access. When she asks what they're teaching, they try to explain but the conceptual frameworks they use just don’t exist in her un-enhanced brain.
She watches her social circle fragment. Many adhere to AI-led cults and/or new religions. AI romantic partners are very common, and many are advocating for legal protections for such systems. Her brother embraces every AI revelation uncritically – "the best AI scientists say we do actually live in a simulation, and here are the objectively morally valuable actions to take!" Her best friend rejects everything defensively – "they're rewriting reality to control us!" Most people, like Sarah, move back and forth between the two, with no stable ground for distinguishing augmented wisdom from augmented manipulation.
Various forms of human enhancement are widespread in this world, and do not face issues relating to equality of access. The enhanced tell her she's choosing to be left behind. But when worldview-shattering discoveries and understanding them requires restructuring your brain, what choice is that really? She's become unable to participate in day-to-day life, let alone decisions shaping humanity's future.
Preventing Epistemic Collapse
So, what can we do to prevent this from becoming our future? I don't have good answers for how to prevent epistemic collapse, and it seems like a very hard problem – very worth of its “grand challenge” title. But I think it's worth bringing attention to it, and that’s what this post is trying to do. Here are some thoughts on what future work in this area could look like:
Learning from historical transitions. We need to find institutions and movements that have managed major belief changes successfully. E.g., what allowed some societies to navigate the shift from religious to secular worldviews without completely fragmenting?
Epistemic scaffolding. We would need transitional institutions and tools that can help people adjust. This might mean AI systems specifically designed to translate between different cognitive levels, or ones that are smart enough to help advise us on questions relating to both societal and individual epistemics.
Update (14 March 2026): Forethought’s series on “AI tools for collective epistemics” seems really really promising. If society manages to implement the technologies described (and that is a big if), I think we will probably be able to avoid epistemic collapse.
Better evaluation of AI persuasion. We need evaluations that test AI's ability to shift entire worldviews, not just individual opinions. How persuasive can these systems become? How quickly?
Mapping epistemic resilience. What does the current state of the field of epistemic resilience actually look like? There are already people thinking about and working on “AI for epistemics”, and there are some talent-building initiatives on this (like the Fellowship on AI for Human Reasoning).
Truth-tracking without understanding. Can we develop systems that let un-enhanced humans make good decisions even when they can't understand the underlying reality? This sounds paradoxical, but I’d argue we already do this in some sense. Most people don't understand how airplanes fly but trust them anyway!
I hope I’m wrong about both the likelihood and nature of epistemic collapse. Maybe superintelligent AI’s impact on the epistemic environment will be positive overall, and we will leverage AI and institutional tools to muddle through. However, we will definitely not avoid it by default, and I think there are many reasons to be concerned.
Acknowledgements
Thank you to Duncan McClements for providing useful feedback on this post.
“Machines of Loving Grace”, an excellent (though optimistic) essay on a world with “powerful AI systems”, does a very good job of describing what AI-accelerated neuroscience and biology could enable.
The jump from "wearing glasses" to "installing GPT-12 in your prefrontal cortex" isn't gradual adaptation.
This date is an illustrative example and not representative of my actual timelines.
Thanks for writing this! Lots of interesting points. A few thoughts while reading:
>What happens if superintelligence discovers that our fundamental assumptions about causality, consciousness, or even logic are wrong?
I'm actually not sure this is worth worrying about. Our understandings of causality and consciousness indeed are changing and highly disputed, but in most contexts (e.g. understanding US politics) this isn't very relevant. I don't know what it would look like to discover that our fundamental assumptions about logic are wrong (some have argued against obvious-seeming axioms, e.g. dialetheism, but those people live their lives much like the rest of us).
>How do you select for “truthfulness” when the nature of truth itself is being revised monthly?
Similarly, I'm not fully sure what this means, but it sounds a bit too dramatic to me. Again consider that people trying to figure things out in other epistemic domains rarely care to ask which theory of truth is correct.
>I think there’s a much stronger case for automated forecasting working, but it too has a critical weakness: trust […] if people's entire worldviews are crumbling monthly, why would they trust anything, even an AI with a perfect prediction record?
Here I'm just not sure I see the positive case for why you and I will lose all trust in every source of information. Why would you personally decide not to listen to "an AI with a perfect prediction record?". Another angle on this is that it will always be possible (if painstaking) to scrutinise the reasoning / sources of your favoured source, and verify if they seem sensible. If they seem like bullshit, you can tell others, and that source will fall out of favour, and vice versa.
>They [non-enhanced people] would functionally become children (or, even newborns if the intelligence explosion gets really crazy) in a world run by incomprehensible adults.
I do think this is a good and worrying point. But a couple thoughts. One is that already, some people in the world are in a far stronger epistemic position than others. Some are lucky enough to have learned a lot about the world, know how to competently access credible sources, etc. Some, as you point out, have crazy views about the world (e.g. young Earth creationism). Why isn't this already a disastrous situation? I think one reason is that we're most free to form crazy (wrong) views on issues which don't materially affect our lives. Our beliefs about the age of the Earth don't matter much for how our lives go; our beliefs about e.g. which side of the road people drive on do matter, so we get them right more often (and those few people who are not able to understand which side of the road to drive on are typically not going to be frequent drivers, i.e. there is often a happy coincidence between epistemic competence and the consequences of making errors).
A second thought is that all of us are in a position of deference-by-default on a huge range of questions. I have not personally recreated the experiments to verify whether the Earth is flat, or revolves around the Sun, but I trust the community that figured these things out, scrutinised the results, and disseminated the results.
Incidentally, I really recommend Dan William's Substack, it shaped my views on a lot of these questions — https://www.conspicuouscognition.com/
Thanks again!
A fascinating perspective on the social consequences of superintelligence. If intelligence accelerates faster than our capacity to comprehend it, reality drift may emerge as a defining feature of the transition.