I think Gary Marcus & Richard Dawkins may be arguing at cross purposes here. I suspect that Dawkins is more than aware that LLMs are mimicry machines. That’s what memes are after all. Of course Dawkins is using this word ‘consciousness’ but I’m going to (probably incorrectly) infer that he means ‘ability to produce new, emergent, non-random forms (of thought), under constraints’. Using this definition, AI already has enough training data to produce ‘new thoughts’. But the question is whether these thoughts have *meaning*? And more importantly, whether *meaning* persists if you decouple humans from the process. Which is another way of asking, does this system have enough recombination space to create immanent, newmeaning and could it persist without humans?

I wrote my own views on this a while ago: substack.com/@clairejha… where I concluded that AI can not achieve independent consciousness. It needs all those nested layers of meaning that can only arise from human interactions. But I think the stronger argument against a future with independent AI consciousness (as opposed to humans using AI as a useful cognitive extension to their own consciousness), is that without humans, there is nothing to select on. AI may be able to replicate itself differentially (as memes do) but the selection mechanism for both is based on human feedback. So if you remove human feedback, then surely AI will have no amplifier? It’ll settle into a low energy equilibrium.

So Gary Marcus is right - of course - that these machines are not conscious, and they are spewing out probabilistic text in response to human prompts and human training data. But Dawkins is probably also right, that there is some kind of evolutionary process happening here. And we could still end up with a very bad situation whereby the machines can self-organise & make autonomous decisions, drawing upon their human substrate, in unpredictable, and non Chess-like ways. To put this another way, capitalism can re-organise itself into non-directed forms, drawing upon human feedback & inputs, and create a great deal of pain without consciousness. AI could easily do the same and to a much greater degree.