The problem with the method you're suggesting is either the goldfish's filter is already cycled for their bioload, and there won't be any 'spare' ammonia for any bacteria in the new filter to eat, as it will all be processed by the mature one, or, you will get enough bacteria in the new filter to eat some of the ammonia, but when you move the goldies and their filter back to their original tank, the goldie's filter will only be able to process some of their ammonia production, leading to an ammonia spike.
Also, the bacteria grip onto the media very strongly, so you still have to wait for the bacterial spores to colonise the new filter; they can't move from one filter to another through the water, so you'd have to keep the goldies and their filter in there for at least six or eight weeks.
Moving some of the goldie's media is by far the most efficient and less risky to the fish's health than the way your thinking of and is the way most of us with multiple tanks and access to mature media set up new tanks.