Yes, you can get fert runoff that can up the nitrates and phosphates and it can vary. Its true that some species do fine with 400ppm nitrate and there are reports of river catfish withstanding 1000ppm. There's not much of any real data on most of our species though.
The recommendation we often make here in the beginners section is that 15 to 20ppm *above* whatever your tap water nitrate level is is a good rough max to shoot for. The important thing to learn though is that nitrate is really just a "symbol" that we use to keep a loose watch on how our tank maintenance is going. There are hundreds of organic and inorganic things in the tank that we don't want to concentrate too much but we don't have the time, ability or money to test for them. Nitrate serves as a good "canary in the coal mine" chemical to test for (even though the darn stuff is unfortunately tough to have a nice reliable simple test for!) The symbolic numbers we're watching however are the -tank- ones, not the -tap- ones, so if it goes up in your tap then that's not your fault and isn't saying saying anything bad about your maintenance of course. (The maintenance we're talking about here is both your gravel-clean-water-changing and your filter maintenance.)
So, to be able to judge how you're doing, you would really need some data in your aquarium notebook. Perhaps a once a month set of readings of tap and tank nitrate would be reasonable. That's just my guess on a compromise between the irritation of doing it versus the usefulness of having the data down the road.
I doubt you've got a problem. Possibly your own maintenance hasn't slacked at all!
~~waterdrop~~