How big is the goldfish in comparison to the perch?
Scales are a fishe's armour and unless the fish is unhealthy, ie. has dropsy and is bloated to distort its scales the only other explaination is that it is being bullied when you are not watching. There is another explaination-parasites. This could be making the fish flick itself against hard surfaces and thereby damage the scales. You will need to watch very carefully, because where the fish has exposed parts, it is guaranteed to be a site for infection. Various pathogens will eventually break the slime defense all fish have under their scales.