I would think that it does matter. The size of the fish refelcts its age, and if one is much smaller than the other, it might be very young or might not be fit to breed (resulting in small batches of eggs or fry).
If they are big fish, then an inch or so would not be that big of a deal, IMO, especially if one gender is naturally smaller than the other. If it is a much bigger difference, you should wait for the smaller one to get bigger or replace the smaller fish with a bigger one, or the bigger one with a smaller one.
I would go with the latter, since younger fish means the fish live longer with you, though you would have to wait for sexual maturity.
Do you have a fish in mind to breed, by the way?
