"good" bacteria will grow on pretty much anything inside your filter maisy, be it the white sponge or the black carbon sponge. Ideally when you setup a filter, you shouldn't have any carbon sponges/pads/bags inside it but when you buy them brand new, they always come with one. Most experienced fish keepers will swap them out for a white sponge or some ceramic media which provive a good surface for bacteria to grow on. Since you've not done this, it would be wise just to carry on as you are, removing it now and replacing it with a new white sponge would reduce the bacteria in the filter, meaning it cant clean your water of ammonia and nitrite as quickly thus it would put you into a mini cycle until the bacteria grow back. So my advice is to chuck it all back into the filter and set it running again then next time, when your filter is really really dirty swap over the carbon sponge for a white bio sponge and do nothing more for a few weeks. That way the dirty mulm in the filter will help seen the new white bio sponge and in theory, you shouldnt have a mini cycle. Worst case would be 3 or 4 water changes over the period of one week to get it back on track.