Yes you shouldn't be touching the sponges every week you only need to touch them every so often, you go by your water stats.
You only touch one when there really mucked up with fish waste.
The only thing you rinse is the filter floss which get rid of fine particles in the water.
Leave the sponges then.
Ask in tropical discussion about how to use a gravel vac with sand as there a method to doing it.
Don't touch the filter you need to hoover your sand if it hasn't been done in a while.
Also remove all ornaments so you can do a good hoover.
You can get bad bacteria pockets in tanks when there not being hoovered.
What does your tank water smell like.