It isn't anything they would typically find in the wild, so it isn't anything that their body would naturally know what to do with. Their body would have to break it down to its basic components, and probably generate a lot of waste because of the larger number of things that their bodies don't know what to do with. Organs like the liver and kidneys would have to work significantly harder than they would otherwise. It is the same reason that you shouldn't feed your fish chicken and beef and pork -- it just isn't something that the fish are going to consume very often in the wild, and evolutionarily, they just haven't developed the biological processes to handle those kinds of material.
I think that it is probably a bad idea unless you have no food at all for them. I think that really finely crushed flake is probably better than powdered milk, because at least the flake food is based on things that they would normally consume in the wild, and you can always crush up flake food in between your fingers.