It increases the size of people "oppressed by white people" (non-white) but excludes them from the group of the most oppressed people (black). Not really about color exactly since some people who are considered to be black are lighter skinned than some actually brown skinned people.

There is probably not a generalized affinity between so called black and brown people and it is more about an anti-white political purpose.