It depends entirely on the species. Asking if fish prefer to be in groups of their own species is like asking if mammals eat grass. Some do, some don't.
Of the fish that are commonly sold as community fish, some (tetras, glass catfish, corydoras, most barbs, danios) are schooling fish, so they need to be in groups of their own kind to feel safe. How badly they need it or how large the groups need to be depends on the species. Some can actually die if they are kept singly, others tend to get aggressive, yet others are simply less fun to watch and seem not to be enjoying themselves without the school.
Others (like gouramis, bettas, red-tail black sharks etc) are territorial which means you can only keep one of the same species or (depending on species) only one male. Cichlids are also strongly territorial, but you need to ask about individual species for these.
Then several common community fish are what you might call sociable, they thrive with company of their own kind, but it does not actually have to be in shape of a large group. I'd say most common livebearers belong here (but beware of putting more than 1 male swordtail in the tank). IME they can derive a similar benefit from being with a related fish with similar habits, i.e. a female guppy kept with female platies; I currently have a merry widow who seems happy with the portholes.
