I’m listening a keynote by Noam Shazeer of Google on the future of AI here at Hot Chips. An interesting question during Q&A about forgetting mechanism. We have all heard about attention, but one of the problems with models is that they tend to no forget what they are trained on. Interesting comment about how RNN can symbolically forget. What about LLMs based on transformers.

This is an Interesting thing to contemplate regarding super intelligence. It might know everything but everyone shouldn’t know everything.