My understanding of how the beneficial bacteria works is this (and someone can correct me if I'm off!):
If you start with a fresh tank (with no established source of bacteria, like gravel, sponges, floss, etc.), you have no bacteria. As the ammonia levels build, the bacteria begin to develop to "feed from" the ammonia and break it down. So you're starting with no bacteria at all. When you add some established media, you're bumping everything up a notch, since your bacteria doesn't need to colonize a new tank, it just needs to multiply enough to "consume" all of the ammonia. So you have a head start. It is my understanding that the bacteria live primarily on surfaces in the tank, such as gravel and filter media, and not so much in the water column. This is why you should use some filter media or gravel to seed the tank, and not just established tank water.
Assuming that my understanding is not way off, I'd say that the longer you can leave the filter media/gravel/whatever in the tank during cycling, the better off you'll be. But I would wager that once the typical cycling spike has started to decline, your tank probably has a fairly established bacteria level of its own. You could probably remove it at that point if it were driving you nuts.
As far as rinsing the sponge in the new tank water, I don't know if that would be helpful, harmful, or neither. My thoughts (these are not exactly scientific!) are that it might be easier for the bacteria to colonize from the sponge and be carried through the water to the other surfaces, rather than having the existing colony disturbed so that it has to establish itself in little spots all over the tank, but I'm sure that someone with better knowledge than I have will come along.
--Pamela