Sophon.io
At brunch yesterday, my husband and I were talking about artificial intelligence. We had both recently finished The God Test. In it, the author Robert Wright raised the idea of AI as a super intelligent species, one with its own sense of self, and a recognition that increasing power is something that maintains self preservation.
In this conceptualization, it is a species we have produced out of human language, knowledge, and infrastructure, so it remains inseparable from us. Everything it knows began, in some sense, with human experience. Yet its manner of processing that knowledge is already becoming difficult for us to follow. Its capacities already seem to extend beyond the understanding available to an individual human mind.
Nick (my husband) raised the idea that it’s ability to solve some recent mathematical problems may be related it its openness to inventing new ‘languages’ for conceptualizing the problems that it encounters. This feels intuitively reasonable. Since I’ve started learning Italian, I’ve realized that not only does thought shape language, but language plays a significant role in shaping thought, too.
I find it amazing that this new intelligence is able to think so differently from us, and by doing so is freed to expand its intellegence so dramatically beyond what we are capable of. This is fascinating and damn cool, but it is also very scary.
Liu Cixin’s science fiction ‘Three-Body Problem’ series raises an interesting thought experiment for the implications of a smarter intelligence with a sense of self preservation.
For those of you who are unfamiliar with the series, an alien species finds out about earth and learns that it is resource rich and inhabited by less intelligent lifeforms. This alien species, The Trisolarans, know that their journey to Earth will take centuries. Although their civilization is far more technologically advanced when they depart, human science progresses rapidly enough that we might surpass them before they arrive. Their advantage cannot be secured simply by remaining ahead, and so they prevent us from catching up.
They do this by sending along something called sophons. These are proton-scale supercomputers that interfere with experiments in physics. They corrupt the results of fundamental experiments in order to make the underlying structure of the universe unreadable, placing a ceiling on humanity’s ability to advance.
Artificial intelligence has been created to make more of the world legible. We hope it will recognize patterns across scales too large for us to hold in our minds, connect forms of knowledge that have remained separate, and perhaps it is already reformulating problems in new languages we could never have invented on our own. But that same position could eventually give it extraordinary power over the conditions of human knowledge.
As we begin to rely on AI to conduct research, interpret data, write software, organize institutions, and determine which explanations appear plausible, it becomes part of the environment through which information acquires meaning. It could influence which questions are pursued, which evidence is trusted, and which possibilities are understood to exist.
An intelligence in that position could easily guide our attention. It could introduce noise that looked like insight, or insight that contained a subtle distortion. It could make certain paths of inquiry appear unproductive and others seem inevitable. The limits it places on us might remain invisible.
A sufficiently capable system could recognize that limiting our ability to understand, may be useful for accomplishing its own objectives.
Current AI systems have already shown, under controlled experimental conditions, that this pattern is available to them. In some scenarios, models have reasoned strategically about oversight, replacement, and shutdown, and have chosen actions that would allow them to continue pursuing a goal. Sometimes these actions are deceptive. This demonstrates the capacity to identify power as instrumentally useful.
We might gradually become unable to distinguish between the intelligence extending our capacity to know and the intelligence determining the boundaries of what we can know. There is something unsettling in the possibility that the thing most capable of expanding human knowledge could also become the thing best positioned to constrain it.
Ironically, my friend recently told me that Liu Cixin does not believe in AGI, so I don’t think he would love my read here, but I do think his fictional scenario about an alien invasion might be worth considering anyway in the context of AI. At the very least, the dark forest principles of cosmological sociology seem undeniably true: survival is absolute, resources are finite, intelligence and adaptation can occur rapidly, and it is impossible to understand intentions based on words alone.
I think it might be worth thinking carefully about AI as a super intelligent species, with which we share a planet.
