LLMs before RLHF are like a genius with no social skills:

They've read every book on the internet.

They have no idea what you actually want to know.

They'll give you a 10,000-word answer when you asked "what's for dinner."

RLHF is basically finishing school for AI. It teaches manners.