Meta just introduced “Large Concept Models”, an AI architecture that, instead of predicting words, predicts higher-level abstractions called Concepts.

This is in line with human cognition, which operates at multiple levels of abstraction (rather than words) when analyzing information and generating creative content.

In their paper, they assume a “Concept” to correspond to a sentence and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities.

The Large Concept Model is then trained to perform autoregressive sentence prediction in this embedding space.

Their results are quite impressive.

LCMs show amazing zero-shot generalization performance to many language tasks, outperforming existing LLMs of the same size.

Alongside, they also perform well on many language tasks. (Although they do not necessarily outperform LLMs on all these tasks.)

The general model architecture and the SONAR embedding space are shown below.

Check out their ArXiv research pre-print here:

arxiv.org/pdf/2412.08821

Curious about AI? Follow along:

intoai.pub