This is a very complicated subject, and way beyond my level of understanding. But I have picked up some bits over the years.
Fish like all animals are born with algorithms implanted into their brain. These involve sight certainly, but likely other senses too. And they can be surprisingly acute. Studies have proven that a guppy will avoid eating its own offspring, while eagerly gobbling up the offspring of another guppy in the same tank. Scientists believe that some fish can distinguish their own siblings, which avoids interbreeding in the wild. Other studies have shown that fish prefer to hunt with specific other fish of their species when it has been to their advantage in the past. Male guppies have been shown to ignore females they have previously mated with and select others, probably for the purpose of spreading their distinctive genes further afield, something that we now know drives the evolution of new species.
Eyesight is a major factor, as scientific studies have shown. Images from the eyes are transmitted to the brain, and the optic lobe is the largest part of a fish's brain. Studies have shown that fish can be "tricked" into recognizing another member of the species or a predator in the shape and colour but made of wood or plastic placed outside the tank.
Smell and chemical reception also plays into this. Fish release pheromones and allomones that are sufficiently distinctive to be read by other fish for a number of different purposes. Fish of the same species pick up the pheromones, and these can signal food, danger, spawning, stress, etc. They affect behavioural and physiological responses in the receiver. Other species pick up the allomones, which benefit the originator but not the receiver; these are primarily used as a form of defense. The sense of smell in fish is quite strong; corys for example can smell food in the sand when they cannot see it. I always feed my fish from the side of the tank (length-wise) closest to the filter return so the current carries the scent down-tank, and this makes a difference so obviously the fish are picking it up.
Byron.