Blog

In this post, we explore the path to intelligent truth by meandering through an abstract logical village. We show that A.I. is incapable of true intelligence as identifying the rules and base truths of an argument are beyond computation. This leads to a clarification of why A.I. hallucinates, why agents need human intervention, and ultimately why modern computers may never be able to reach AGI.

The abilities of a computer are constrained by constructive logic. This type of logic acts to derive conclusions through step-by-step derivation of proof. The Curry-Howard correspondence shows there is a direct equivalence between constructive logic and computation. However, constructive logic does not cover all possible provable objects. Axioms come under classical logic, and are self-evident truths that cannot be proved by construction.

The Analogue

Let’s explore this, by first visualising it. Imagine that propositions are represented by houses in a village. Each house is painted blue if it is a true proposition, and red if it is not. In classical logic, this is all you have; truth and falsity are simply asserted. To expand into constructive logic, we connect the houses by a series of one way roads, that allow you to travel from one to the other. These roads are the constructed proof; you start at one house (proposition), and move your argument on using a set of rules (logic) until you reach your conclusion. The computer is essentially only able to move from house to house via the roads; realising the colour of the house when it arrives by a road.

In the analogy, a topos is akin to a village; a series of houses connected by roads. It is a self-contained geometric landscape following it’s own internal logic. However, in some way the road network and colour and placement of houses is abstract. The combination can be applicable to many ideas, not just the starting proposition. Ideas that are similar structural, or connect with similarly structured rules and can be interposed with each other. This is the nature of a topos, and abstraction that defines structure between propositions on a meta level.

The Computer

Now, there are some problems here. How does a computer know at which house to start ? We know that a computer proceeds through constructive logic. It can only know where to go, by starting at one house, travelling on a road, and arriving at another.

The houses that have no roads leading to them as self-evident truths, or axioms. These are truths, or falsehoods that are asserted by classical logic. A computer cannot determine these, as it can only live in the realm of constructed logic. It can only follow roads, not jump to a house.

In constructive logic, there is hope that axioms are not truly houses with no roads leading to them; but that the roads are yet undiscovered. This is an open question, and there is no such proof of existence; and we know from Godel’s Incompleteness theorem that there are pockets of truth that no road can ever reach.

The Human

This leads to point where humans enter in. The starting houses, and the roads define the logical universe in which the village exists. We know the computer can use the roads, and discover them through training. It does this through taking small steps through the village, mimicking maps already laid out by humans. But we know that the A.I. must be prompted to act, a human must define the starting house, and more essentially, select the village to start in. This is an essential part of intelligence.

This is selecting the topos; picking the logical universe in which the proof exists, and what it can be constructed from. We know this is outside of constructed logic, and must be non-computable by definition.

The process of training the machine is discovering a set of pre-computed roads by which the computer can travel. Again, the topos is interchangeable, the houses may represent analogous propositions connected by the same road network. So, a computer can adapt, it can use what it has learnt from one set, and apply that to a different village; but this leads to problems.

The Hallucination

If we now swap out the propositions of the houses, and the road map is not identical, then the computer will arrive at houses that don’t exist, or produce contradictions. The road map still connects them, but the output is nonsensical; this is hallucination.

It is an artefact of knowing the connections between propositions, but not the topos itself. The two are intimately connected; the houses, and the roads. It can be improved by further training, crawling through more and more villages, but will lead to failure, as it is just tunnel vision in a intricate landscape.

The Agent

We can now see why A.I. finds it difficult to be truly agentic; why it relies on constant feedback from humans. The human must constantly reset the point that the agent is starting at, in different villages. The human must help the agent backtrack from dead ends. They can be constantly frame constraints, don’t follow that road. This is because the computer cannot see further than the point it is at, and the road it is on. It gets lost, it doesn’t know where to start. We have made some progress in automating this through frameworks, but it is still a perennial problem.

The AGI

This means ultimately a human must always be in the loop. The act of identifying and selecting the topos is inherently non-computational. It is based on identifying axioms that define it, which cannot be discovered constructively. A.I. can approximate this process, but ultimately will find situations where it will fail. We know that non-constructive logic is non-computational, and so this is a problem that cannot be physically overcome by computers. This pours cold water on the potential of computers to achieve AGI.

A fairly recent discussion on X occurred between Demis Hassabis and Yann LeCun [link]. It was regarding human intelligence and whether or not it is general. This sparked a few thoughts in my mind, and I wanted to note them down here. These thoughts centred around that general intelligence is often misframed at the level of the individual mind. While human cognition is clearly specialised and resource-limited, this specialisation is not a weakness but a necessary filter over an overwhelmingly noisy reality. What we call “general intelligence” may instead be an emergent property of human society; one arising from abstraction, cumulative culture, and the collective exploration of ideas over time.

In summary, Yann argues that humans are very specialized in their intelligence. That there is clear evidence that many animals have better capacity in many areas; and that we are generally bad at chess (in comparison to machines). Our intelligence is a consequence, and an adaption of our interaction with the physical world, and we do not have general intelligence as such.

Demis then ripostes by marvelling at the complexity of the human mind, and how from simple hunter gatherer minds we have achieved so much. That even though we are limited by physical capacities (memory, time) regarding our ability around chess; we are its originators, a leap of creative thinking. He then proceeds to celebrate the achievements of modern civilisation such as science, and engineering. His overall claim is that Yann is confusing Universal Intelligence with General Intelligence. That the adaptability of the human mind is what makes it’s intelligent; not the fact that we can imagine all things possible in the universe.

The response by Yann is a more theoretical one. He reminds us that a human mind, given enough pens and paper, is Turing complete; but this does not make it efficient. Therefore, the human mind must be selective in what it processes. Using something entropy calculations, he shows that the human mind only has the capacity to represent an infinitesimal slice of possible signals just from the optic nerve, and so must be specialised.

In my own thoughts, this framing around specialisation of thought as a entropic analysis does not ring true though. If we sample the space of RGB images, where each is composed of three channels of 256, we know that most images are nothing but noise. The fact that the human brain cannot understand patterns in the noise, is not a failing of the human mind, but a understanding of reality. Noise serves no purpose, it is without form, and is therefore not an artefact of reality, but a consequence of process. It may be that the noise signals something that is useful for us. For example, we know that EM radiation exists beyond visual range, but conveys information. However, we know that signals exist that give us information about stellar bodies, or enable us to see heat signatures of animals on earth. We have overcome these limitations, by creating devices that enable us to extract this useful information.

However, we do see that humans struggle with concepts outside our daily experience. It is notoriously difficult to visualise Quantum Mechanics, or the effects of General Relativity. These things are different from our perception of continuity and linear time and so become difficult. This is where mathematics allows us to bring structure to concepts, and work with them, by “sticking to the rules”. In that way, our brains are limited, and specialised. This abstraction has allowed science, engineering and mathematics to flourish. Chess, could be seen as an abstract battle between two forces, something that does exist in reality.

This abstract ability could be seen as our general intelligence; a way to move beyond our daily experience. We must, however, remember that this does not come easily to humans. For millenia, we slowly developed mathematics, and the scientific method. We spent much of our time in the realms of magic and superstition. These abstractions have come late in our development. Why is this ?

One possible explanation, is that general intelligence is not an individual trait, but a societal one. The genius of humanity comes from the brute force application of the variability of genetics, and personal experience. That real abstract discovery, like chess, is the culminations of small advances based on the “shoulders of giants”, or the stacking of regular sized people that are made to look like giants. We forget so much of what humanity has tried, and failed to do, through survivorship bias. If we were to collect everything together, I suspect human advancement would look much more like random exploration, that guided abstract advancement.

So, my thoughts can be summarised as this. Yes, human intelligence does not understand every random pattern that can be measured by the optic nerve, but this is correct, most is noise; not signal. This is the specialisation of the human brain. However, this specialisation, predicated on nature and nurture of individuals has allowed humanity to explore abstraction together. In my mind, this points to not the general intelligence of the individual, but the general intelligence as an emergent behaviour of society.

The replacement of humans, who do white collar work, is a constant topic of conversation in the present times. The world is awash with talk of the intelligence of LLMs, and the oncoming threat of AGI. But what are the limitations of LLMs, and what, if anything, do humans have as an advantage to LLMs.

I’ll list what I think below. Some of these may seem l

The Personal Touch

This article is written by a human*, but the image above is generated by an AI (ChatGPT 4o). The power and productivity leveraging by AI is clear-as-day; the image above would have taken me many hours of work to produce such a clean result, and AI has provided it in seconds. However, it is not my image, it is not what I pictured in my mind when I thought of the image first, but it does the same job. It is something that fits the prompt, however vague that is.

Now for someone, who is an expert artist (not me), to produce their minds-eye image, it may be easier to create the image, than to describe exactly what they want; and in fact, for any written work (code/writing etc.) it is in the very act of writing that you are describing what you want. If what you describe in a prompt, is enough to produce the correct result, then by definition the prompt is enough. The rest is just the act of translation, the presentation of the information in an easily digestible form. However, the actual information content will be the same.

Obviously, this changes if you do not know what to write, or you do not know enough to create the content in the first place. Then the AI acts as a search engine, gathering and collating information, as you would do normally on a day-to-day.

So, what does this mean. The human value is therefore, the personal touch, the uniqueness that you bring to the table, that is a result of your own experience. This manifests as your ability to describe your own inner thoughts, be it through writing, images, or any creative outlet. It is this unique perspective that has always been difficult to describe to others, and is equally difficult to describe to an AI.

Symbolic Reasoning/Abstraction

This is a complex one, and requires an understanding of what symbolic reasoning is, and how it leads to abstraction. Of the current LLMs, including “reasoning” ones, this is the major downfall, and is something (at time of writing) that is still being worked on.

According to ChatGPT 4o, symbolic reasoning is:

The process of manipulating abstract symbols according to logical rules to draw conclusions or solve problems.

This definition not only applies to mathematics, but also to the symbolism of literature. The ability to use metaphor to substitute one symbol for another in a film or book, is a way to make abstract topics relatable. For example, in horror films, primal fears, become manifest in monstrous form, and the situation and context is built around them with is analogous to reality, or how that fear comes about in reality. The ability to swap the symbol of a monster in place of fear, is a kind of symbolic manipulation, and the understanding of the context is the reasoning.

So, symbolic reasoning is key to abstract and higher level thinking. It is in some ways the embodiment of intelligence, but is missing from LLMs. To understand why, you need to understand the transformer architecture. I won’t go into a large amount of detail here (please refer to the link article), but LLMs excel at pattern matching.

Pattern matching is LLMs can be demonstrated by a counter-example, something that is difficult for them to solve (without external assistance). Let’s take a string of words, and ask LLMs to sort them:

banana, apple, cucumber, apricot

We then provide the LLM with a prompt:

Please sort the following: banana, apple, cucumber, apricot

Now the transformer uses something called self-attention, that is it looks at each word in the prompt, and asks which words are most relevant for the other words. Crucially, this attention mechanism is built into the system, and is fixed, but does change based on the words in the prompt. You can view the first four words as a kind of instruction to prep this matrix; but the matrix has been trained, not on the four words following (or more specifically on their ordering). To produce the next word in the response, the attention matrix itself would have to shift attention to the word apple; but the statistical nature of the training means this won’t be the most common case. In fact, in general, the attention should be even amongst all the words. This leads on to symbolic reasoning.

From the brief explanation above, it is clear to see that the LLM does not see symbols. In a human mind, each position in the list is a different symbol, and so we can manipulate them as such; not simply something to shift attention to. An LLM does not see it that way, and never will. This is clearly demonstrated by the ARC-AGI challenge which humans can do almost instantly. How human brains do it is a mystery, but I try to explore it here.

If anyone ever reads this, I know they will try the above prompt, and see that it does in fact work. This is because behind-the-scenes, the AI models have access to tools. They can convert the request to code, and run the code locally. It may seem like this solves the problem, but the point is that the symbolic reasoning should be embedded in the intelligence to perform true abstract reasoning.

* All but the headline, and otherwise stated, that is.

Automation has been a thread that has run through all of human history. Previously, it has largely replaced heavy and some skilled manual labour. Now it is replacing repetitive intellectual work. This means that the realm of human work is being compressed into a smaller and smaller subset of viable economic work. Without repetition the work becomes exploratory, work that is traditionally risky, and difficult to value.

Pattern based intelligence has appeared as if from nowhere in the last few years. Its ability to generate code, writing or art automatically has sent a shock through industry; devaluing the perceive value of human work. Partial automation has brought apparent devaluation before; photo editing, code autocomplete, spreadsheets; but ultimately led to increased productivity. However, this is different, it is seemingly more dynamic; adapted to any well specified task.

The question remains, what intellectual work is left for humans to perform ? The answer is frontier work; work not defined by well understood and repeated patterns. However, this work is inherently exploratory, with long feedback loops, and high failure rates. It naturally has the economics of research, and innovation.

This means there is a mismatch between standard labour markets and the new economic model. The vast majority of jobs are repeatable, with predictable output. Traditionally, research-like activity is a small part of most companies, who shy away from gambling with their bottom-line. Universities, public funded laboratories, and a handful of large firms get around this with subsidies, monopolies, or regulation; but most do or cannot.

The new form of intelligent automation is intensifying this. The replacement of repetitive pattern based work at the heart of the labour market, will lead to employment being compressed into the outer edge; the frontier. The result is not mass unemployment, but growing instability: fewer roles, higher variance, delayed validation, and increased pressure on individuals to absorb uncertainty personally.

All that remains if for employees to absorb uncertainty on behalf of others. The difficulty is that most labour markets are not designed to pay for uncertainty, only for output. Until that changes, disruption is not an anomaly; it is the natural state of the system.