He would be stunted by the small tank as many have said. He won't grow much beyond this point regardless of the size aquarium he is put into, so he doesn't need a very large tank.
With a 4 year old tank - ammonia poisoning is not likely. Nitrate poisoning is a bigger threat. Also, the fish is small and likely not fed much, so he probably doesn't need multiple 75% changes per week. For many, this is impractical anyway.
Stunting in goldfish does NOT necessarily affect their health or lifespan. Many, many small or stunted fish live perfectly normal lifespans and are perfectly healthy, they're just small. The oldest recorded goldfish were all stunted little runts who lived in tanks smaller than 10 gallons their entire lives.
This is NOT to say that I advocate for small tanks for goldfish, as they kill a LOT fish by ammonia poisoning as the tanks cannot handle the waste. That said - fish that aren't killed by such poisoning and are tough enough to survive it or owners who keep the tank fairly clean despite the size often have small healthy fish for many years.
With that said - a bigger tank would probably be more enjoyable for him. Get the largest tank you can and move the current filter over to that tank, and keep doing what you've been doing.
