Generally speaking the man's colleagues would have no idea what happened, or assumed that the man was guilty, rather than being OK with false accusations or spurious accusations. There are some actual sexual harassers, and there are some people offering or accepting grades and promotions in exchange for sex. If you hear that Prof H was fired for sexual harassment you might think it was his own doing. Listening to people defending themselves against false accusations and trying to work out who's telling the truth is a huge amount of tedious work, and taking sides with the accused can be risky if they turn out to actually have been exploiting their position.