It's not even necessary for them to be angry. Merely ambitious. Want to clear out space above you? Simply accuse him of touching your arm for a second too long. Or, hell, making a slightly less than sensitive joke.