just keep the whole tank running the bacteria will decrease as there is no ammonia or nitrate for it to feeb upon.
when you come to restock build up very slowly.
every fish you add will send tank through a small cycle.
test for amonnia and nitrite after every addition,when the levels go back down to zero the cycle has finished you can move on.
whilst empty continue with water changes as you are doing
the key to it is to remember that every thing you add or do to your tank will have an impcat.
the bacteria in your filters will adjust to what is going on in your tank and your tank is constantly cycling, so they naturally increase and decrease.
the polyfilter has taken about a week to get the stuff out of my tank, my last loss was 3 days ago,but am not going to add anything for a couple of weeks.
Hope this helps
Sam