It's the same as we don't understand how the human brain works. What we presumed was the result of the massive size and complexity of the brain, ie 'thinking', is now occurring in much smaller, simpler systems - but it's still too complex to record or understand it.

AI is entirely different from normal computation. That the most powerful are using graphic cards to process language is a hint of how bizarre these things are. Normal software goes from step 1 to whatever, with a series of 'If this, then that' type of instructions. AI is not like that, at all. It connects concepts and words, creating ideas and thoughts, very much like a human mind, and no, you cannot measure, watch, track or otherwise understand what it's doing.

The people who designed it are shocked how it is able to reason and figure things out that they never expected.