There is no rule that applies to all gouramies because the group is so varied with diverse sizes and temperaments as well as behaviour and prefferences.
You would probably have difficulty in mixing three-spots, pearls and dwarfs. Three-spots can be notoriously aggressive and, unless you have a large, planted tank, even a pair can often not get along once fully mature. Males can be very aggressive to both females and to other gouramies. Personaly, I'd keep a single male in a smaller tanks (20 or so gallons) and a trio in anything large.
Also, different color morphs of three-spots often show less aggression towards each other whilst males of the same color would often fight to the death.
I don't like keeping most gouramies in pairs in fact as the male often chases the female and constantly stresses her. A trio is usualy better as long as you have the space (1 male, 2 females).
In contrast to three-spots, pearls are usualy very peaceful fish and a pair will work fine. I still preffer to keep them in threes but this is only my own opinion. Surprisingly, pearls can often handle the aggression of three-spots when mature and keeping them together is not impossible. I would advise against it myself though as pearls can become shy if kept with aggressive adult three-spots.
Young three-spots are not very aggressive.
Dwarfs will fight for territory amongst each other but usualy wont show any interest in other species. Females can be rare and people often keep 2 males together without trouble though a trio would still be my preffered way of keeping them. A pair, with these fish, will also work but I wouldn't try it in a very small tank where the female wont be able to escape the male's consistent attentions.
Pearls and dwarfs get along but I'd steer away from combining dwarfs and three-spots even though they get along to an extent in larger, planted tanks.
As for whether it is sex-specific, well I'm not sure what you mean but generaly the males are the territorial and aggressive ones while the females are relatively peaceful and rarely territorial. IME, you can keep several females of any of the more common species and have no trouble. You can usualy keep 2 male dwarfs, and often 2 male pearls, without any females, but I'd never purposefuly try this with 2 male three-spots (especialy if they are both of the same color morph) as they will usualy kill each other as they mature.
