Groups in favor of free speech tend to be on the outside looking in with regards to political power. Full stop.
Ed Meese comes to mind.