Hey reader(s),
One concept I’ve always found sort of interesting is to think about is misinformation, particularly in infectiously it can spread and how difficult it is to contain. But while I think it’s fun to ponder, I’m getting a little uncomfortable with how often new developments in generative AI are making my pondering feel too real and too relevant. We’ve arrived at a point where an article from 404Media I saw the other day covering a research paper about recurring names in large language model outputs reminded me more of a particular piece of sci-fi horror writing than any sort of inspiring advancement in the frontier of human technology, and it’s left me dreading the potential extent of this technology’s use in our society.
The specific piece of sci-fi horror writing in question is SCP-922, one of the many “SCP files” that make up the online collaborative writing project known as the SCP Wiki. If you’re unfamiliar, the fiction of the SCP Wiki is set in a world where there exist paranormal objects, entities, and phenomena that defy normal science, along with a secretive “SCP Foundation” which seeks to study and contain these anomalies while preventing the disruption of normal civilization. These anomalies*, often referred to as simply “SCPs”, are generally represented in writing by numbered Foundation files which outline the Special Containment Procedures** in place to contain them, give a description of their characteristics, and include a log of experiments or containment incidents which show how knowledge of these characteristics has been ascertained. It's usually within this description and appended log that the ramifications of each anomaly are explored, and given that they’re often presented in a quasi-academic writing style, I often find it uniquely interesting how their horror is revealed.
{This is where I summarize SCP-922 specifically as context for the later discussion about some of my worries with generative AI. You should probably just go read SCP-922 instead—it’s not much longer than my summary and includes some details that work better when it shows rather than when I tell—but I will be covering the most relevant sections exists in case you won't.}
*Philosophy of science tangent: The use of the term "anomaly" to refer to SCPs appears to be in reference to—or at the very least consistent with—Thomas Kuhn's use of the term in his influential 1962 book, The Structure of Scientific Revolutions. In the book, Kuhn argues that science develops through periodic "paradigm shifts" rather than through the refinement of one model's lineage. For example, science didn't arrive at the heliocentric model through the continued refinement of the geocentric model. Instead, the dominant scientific understanding was forced to change when the model placing the sun and planets in orbit around Earth could no longer explain what was being observed of the universe as well as a model where these bodies orbit around the sun. A paradigm shift occurred when observed phenomena that defied explanation by the existing model—or as Kuhn termed them, anomalies in the current paradigm—became prominent enough that defying the base assumptions of the existing model became necessary to explain what was being observed.
**The acronym "SCP" can stand for either "Special Containment Procedures", referring either to the practices put in place to attempt containment of the anomalies or the Foundation's motto; "Secure, Contain, Protect".