I’ve always found the infectious nature of misinformation to be an interesting thing to consider, but I’m a little uncomfortable with how often new developments in AI are making me consider it.
I saw this article from 404Media covering a research paper about recurring names appearing in text generated by large language models the other day, and it immediately reminded me of one of my favorite bits of sci-fi horror writing; SCP-922.
For those unfamiliar, SCP-922 is one of the many SCP files that make up the online collaborative sci-fi writing project known as the SCP Wiki. In the shared "SCP universe" of the wiki there exist paranormal objects, entities, and phenomena which defy normal science, and a secret "SCP Foundation" seeks to study and contain these anomalies* so they don't disrupt normal civilization. The anomalies—ocassionally referred to within the Foundation as simply "SCPs"—each have a numbered foundation file which outlines the Special Containment Procedures** in place to contain them, gives a description of their characteristics, and often includes a log of experiments or incidents which divulge how those characteristics have been observed. It's usually within this description and appended log information that the ramifications of each anomaly are explored, and through calculated academic language, their horror slowly and coldly revealed.
Specific discussion of SCP-922 begins here. You should probably just go read it, it's a pretty short read with very interesting details, but I have summarized most of the relevant sections to provide context for my thoughts below in case you won't.
SCP-922s file explains that it was first discovered a simple discrepancy in a university's records, noticed when several students attempted to drop a course only to find it didn’t exist in the university course catalog. A search through additional records of the campus registry revealed more odd discrepancies; records of university custodial staff falsified with information from existing staff entries, students with schedules and IDs but no record in the university system, and duplicate student registrations that students would inexplicably identify with despite their details not matching up. The situation escalated further when records showing 1,399 students assigned to a campus residence hall with a capacity of only 450 were investigated by Foundation agents; while assigned personnel identified the residence hall as a seven-story building with with signage listing a capacity of 1,500, four agents who were not recorded as having been dispatched to investigate reported that the residence hall did not exist.
It’s at this point that the SCP-922 file reveals that these discrepancies were not limited to the university’s records, but appeared in Foundation records as well.
*Philosophy of science tangent: The use of the term "anomaly" to refer to SCPs appears to be in reference to—or at the very least consistent with—Thomas Kuhn's use of the term in his influential 1962 book, The Structure of Scientific Revolutions. In the book, Kuhn argues that science develops through periodic "paradigm shifts" rather than through the refinement of one model's lineage. For example, science didn't arrive at the heliocentric model through the continued refinement of the geocentric model. Instead, the dominant scientific understanding was forced to change when the model placing the sun and planets in orbit around Earth could no longer explain what was being observed of the universe as well as a model where these bodies orbit around the sun. A paradigm shift occured when observed phenomena that defied explanation by the existing model—or as Kuhn termed them, anomalies in the current paradigm—became prominent enough that defying the base assumptions of the existing model became necessary to explain what was being observed.
**The acronym "SCP" can stand for either "Special Containment Procedures", refering to the practices put in place to attempt containment of the anomalies, or the Foundation's motto: "Secure, Contain, Protect".
Contamination is generally bad enough on its own, but I worry that we may soon be forced to appreciate that a contaminant is usually imagined as a small portion of intruding material, as opposed to the greater majority that it is contaminating. The last addendum of SCP-922 and the possibility introduced by generative AI in our reality are both paranoia inducing because they suggest it might not be our knowledge that’s being contaminated by misinformation, but rather our misinformation that’s being contaminated with knowledge.
Given how hard it is to measure the extent of misinformation, how can we know whether it's fraying the edge of the truth or drowning it out entirely?