Think of a stream, river etc. Water is constantly changing due to the flow of said body of water. If you can imagine your fish tank inserted into the body of water you can see how quickly the water around a particular group of fish would change. Fish in tanks are essentially swimming in the same water constantly. Even a 100 gallon tank is no match for a small meandering stream when it comes to water changes.
Fish excrete waste, hormones and other stuff. Your tank may have decaying live plants (dead leaves), uneaten food and other potential hazards.
I assume that when you say that your water is "healthy" you have checked ammonia, nitrites and nitrates, as well as possibly pH, GH and KH. I have yet to see a "hobby" water test that checks for hormones. Hormones may or may not be detrimental to the health of your fish, so why take the chance?
For the health of your fish, you should try to replicate nature as much as possible, the easiest way is to change your water frequently.
I once left a LFS because the owner said "this tank is so big I never have to change the water, it takes care of itself." If that's what he thought, then I was extremely cautious about the health of his stock.