I thought since this tank was bigger, water changes with my tap water wouldn't effect it because, it (the ammonia my tap water contains) would be diluted alot once in the tank from the larger water volume, unlike my old 29 galllon!
It doesn't matter if your tank is the size of a sugar cube, or if it is the size of an art museum. If you have 2ppm of ammonia in your tap water, then the following will always be true:
100% water change = 2ppm ammonia in tank
50% water change = 1ppm ammonia in tank
25% water change = 0.5ppm ammonia in tank
20% water change = 0.4ppm ammonia in tank
10% water change = 0.2ppm ammonia in tank.
So, what you said made perfect sense .. you did a 25% water change and your tank ended up with 0.5ppm ammonia in it! Exactly!
The Seachem Prime will bind that ammonia to ammonium for about 24 hours. (ammonium still registers as ammonia on the API test kit). So, as long as your bio filtration can process that ammonia (and resulting nitrite) in 24 hours (in addition to the existing fish waste it is already handling) .. you are doing pretty good.
By the way, building up nitrates and doing regular water changes to get rid of them is perfectly normal. Your (uncle was it?) who never changes his tank water ... is a very unique situation. I have never heard of such a thing! ... unless he has a heavily, heavily planted tank. In some cases, a VERY heavily planted tank might be able to process all the nitrates.
Looks like your biofilter is processing all that ammonium rather quickly, well before the bond is broken and it turns back into free ammonia ... so you should be fine ... just keep your water changes small .. personally, I would not change more than 20% at a time.
You might consider, doing frequent, even smaller water changes (like 10%). That way, you will only end up with 0.2ppm ammonia in your tank after each water change. The down side, is that you would have to do it pretty frequently, and it would be hard to keep track of what small part of your gravel bed your were able to clean each time. But, it looks like your filter will handle the 0.5ppm ammonia in time.