Questions Swordtails And Hi-Fin Lyretail Swordtails Fighting

[warsword]

 have the following fish in a 60 gallon planted tank:

• a male swordtails is isnt full red but close
• a male hi-fin lyretail swordtails red
• two female hi-fin lyretail swordtails red
• four female hi-fin lyretail swordtails pineapple
• two female hi-fin lyretail swordtails tox

 
my male swordtail keep attacking the two female hi-fin lyretail swordtails red and the male hi-fin lyretail swordtails red none stop...
the other ones he does no attack...
 
is this normal? 
should i just take him out of the tank?