Introducing TikTokenViewer app 🫣🫣

Yet another weekend project update 🤓

After watching Andrej Karpathy’s Lets build the GPT Tokenizer video (finally!), where he also introduced the TikTokenizer vercel app to visualize, how a tokenizer works based on BPE but also how it varies on the underlying encoding scheme / model, I couldn't prevent myself from re-implementing the Token visualizer app myself.

It was really a fun and rewarding journey to implement this app in 🐍 . Hands down he is one of the best educator.

Implemented using OpenAI's TikToken, Streamlit and other streamlit components, I'm able to decently mimic the original app (just couldn't mimic the hover-over effect yet 👨🏻‍💻).

I have hosted TikTokenViewer app in streamlit cloud and source available in Github.

Y’all can access the app here : tiktokenviewer.streamli…

Go ahead and try it. Have fun!