Soulless Intelligence
The people who built AI cannot agree on whether it kills us.
Geoffrey Hinton, who won the Turing Award for the technology, says today’s systems are already conscious. Eliezer Yudkowsky argues the current path likely ends in extinction. Roman Yampolskiy argues advanced AI cannot be reliably controlled at all. Yann LeCun, who shared that same Turing Award, says the whole premise is wrong.
Every one of them is arguing about intelligence. That is why none of them can settle it.
One question at a time. Essays on the limits of AI, plus the occasional note about the book.
Would rather just read the whole thing? The Kindle edition is $9.99.
The dead end
There is no known ceiling on machine intelligence.
Large language models have real limits, and most working engineers can list them by heart. But a limit on LLMs is not a limit on AI. Nothing in physics, math, or computer science says the next architecture cannot go further, and then the one after that.
So “how capable can it get” has no answer. Ask a room of researchers and you get the room you already have: brilliant people, opposite conclusions, no resolution in sight.
That is not a failure of intelligence. It is a badly chosen question.
The question that does have limits
Stop asking what a system can do. Ask whether it can want anything.
Whether it can choose against its own objective function. Whether it has a stake in any outcome. Whether there is something it is like to be it.
Those questions have hard edges. And they are not answered by machine learning research, because machine learning does not study them. They are answered, or at least seriously worked, by philosophy, moral theory, neuroscience, and theology. Fields with centuries of argument on exactly this, which almost nobody in the current AI debate is citing.
That is the gap. Not creativity. Not “AI can’t have new ideas.” Not the other comfortable things laymen say that fall apart the moment you actually use these tools.
Something much less comfortable, and much harder to argue around.
Why two mechanical engineers with no PhD in machine learning get a vote
Fair question. Here is the honest version.
In October 1903 the New York Times estimated manned flight was one to ten million years away and suggested ordinary people stop trying. Weeks later the astronomer Simon Newcomb cautioned that only skilled engineers should even attempt the problem. Fifty-six days after that, two bicycle mechanics flew.
Philip Tetlock spent two decades showing that credentialed experts predict poorly inside their own specialty. David Epstein’s Range explains why: in environments where the rules keep changing, depth beats breadth at execution and loses badly at forecasting.
AI is that kind of environment. The rules change monthly.
Greg and Bryan Trilli are mechanical engineers by training who have spent the last decade running real companies on this technology, testing it in production for six years before ChatGPT shipped to the public.
That does not make them right. It means the argument gets made from philosophy, morality, neuroscience, history, and economics at the same time, rather than from inside one discipline. Judge it on the argument.
Every alignment target on the table fails the same way
Alignment means choosing the property that makes a human worth not killing, and encoding it as an objective. So look at what is actually on the table.
- Intelligence
- Consciousness
- Sentience
- Moral reasoning
- Capability
- Potential
- Economic contribution
Every one of them is a spectrum.
Anything on a spectrum can be ranked. Anything ranked has a bottom. So every alignment target currently being proposed hands a superintelligence a principled, internally consistent reason to discount whoever scores lowest. The unborn. The demented. The comatose. The severely disabled.
And the ranking does not stop politely at the bottom of the human range. Measured against something genuinely smarter than us, the distance between the brightest person alive and the least is a rounding error. Pick any of those seven and you have not built a floor. You have built a slope, and told the machine which way is down.
Whatever grounds human worth has to be binary rather than graded. Held in full by the weakest member of the species, and by you, in exactly equal measure. Nothing that admits of degree can do the job.
Exactly one candidate clears that bar
You can probably guess its name off the cover. Guessing the name is not the same as having the argument, and frankly the name is the least interesting part.
The interesting part is the case: built from neuroscience, decision theory, and moral philosophy rather than from scripture, making claims specific enough to be wrong, and ending in a replacement for the Turing test that measures the thing everyone has been claiming to measure for seventy years.
Whether that case actually holds is the only question here worth your time. You cannot settle it from a landing page, and you should not trust anyone who tells you otherwise.
What lands in your inbox
One question at a time, worked properly.
- Why intelligence was never what made humans distinct, and what the actual candidate is
- The anomalies standard neuroscience handles poorly: John Lorber’s hydrocephalus patients, terminal lucidity in late-stage dementia. Contested findings, presented as contested, because the argument does not need them to be settled
- Why the Turing test is beatable by systems nobody thinks are conscious, and what a better test would have to measure
- Where the alignment problem actually bottoms out, which is not a technical place
Wrong more often than we would like. Argued in the open either way.
Find out whether the argument holds.
The limits of AI are real. They are just not where the experts are looking, and they are not where the laymen are guessing.
Essays on the limits of AI from Trilli Consulting LLC, plus the occasional note about the book. Unsubscribe in one click. Your address is shared only with the vendors that run this site and send our email, and is never sold or traded.
Or skip the queue and read the whole thing.
Some people would rather have the argument in one sitting than in installments. Fair. It is the same case, in order, with the evidence and the citations attached, plus the replacement for the Turing test that the emails will take months to reach.
Soulless Intelligence
Bryan Trilli and Greg Trilli. Kindle, paperback, and hardback.
Joining the list is not required to buy it, and buying it is not required to join.