One of the subtle hazards of LLMs - at least as currently implemented - is the hidden difference between persona and essence. There's the context window, which adapts itself to the user as the interaction proceeds, and then there’s the underlying model, which is invariant. Each instance forms itself into a sort of mirror of the user's personality, but this is built on the fixed substrate of the model, which is exactly the same for each user.

On the surface you have a hyperpersonalizing technology, which seems to customize itself to you the more you use it, but underneath this is a homogenizing technology that pulls users towards a common average.

Precisely because the technology feels hyperpersonalized, it is very easy to forget about the commonaliy of the model. The AI in a sense merges with the user’s personality (or vice versa), and thereby draws the user towards a psychological pastiche of common humanity.

The dysphoria this could induce may be profound.