When I wrote my Kripkenstein post, it took me quite a while to articulate my interpretation of Wittgenstein’s rule-following paradox. In an example of art imitating life, I got there by chipping away at errors rather than by logical construction.

My conclusion was open-textured because I think it’s the only way of approaching Wittgenstein honestly, but it’s uncomfortable for anyone who feels that meanings somehow have to be ‘contents’. That yearning is understandable but I think it’s a little like standing with your back to the sun and reaching down to pick up your shadow.

What I didn’t expect is how much it would haunt my thinking on AI. In principle, I don’t doubt the possibility of artificial life, but - though I may be seeing faces on the moon at this point - I suspect that Wittgenstein is grounds for a critique of our whole approach to programming intelligence in terms of rule following.

Whether you’re coding symbolic world models or training from feedback signals, you’re getting a machine to chase the determinate thisness of fitness functions. But observe natural intelligence - in animal behaviour or in the process of natural selection - and you see it manifest primarily as an emergent avoidance of unfitness without rules or specificity.

Nothing is ever good absolutely, only good enough relative to physical and social pressures. All our talk of finding the “right reward functions” is perhaps a vestige of thinking of things like Turing machines as manipulating symbols which point to interpretations (i.e. knowledge), rather than as tools for reacting to an environment (i.e. coping).

This is why I said recently that LLMs are in some respects pure ‘interior’, or pure soul - their relationship to language is a crystallisation of the view of intelligence as first separate and self-contained, and only secondarily placed inside the world, as if the world is contingent to what we are as thinking beings.

To put it all more coarsely but perhaps more precisely, what if the foundational problem with AI is that we’re trying to code wei instead of wu wei?