It seems to me that this is most directly addressed the first amendment's guarantee that the people have the "right of the people peaceably to assemble. "

Or, presumably, the right not to assemble.

Why would refusing to do business with a person whose beliefs you consider beyond the pale be unprotected?

I know that would completely trash most of civil rights law and jurisprudence, but ?