It depends on what exactly the 'gravel' is. Some filters might have a ceramic medium like little balls, others have zeolite in them, a white gravelly substance. Some have zeolite and carbon mixed together. Does your filter manual say what the 'gravel' is?
If the gravel is ceramic, put it in a mesh bag inside the new filter.
Zeolite is not needed in a filter. In fact it isn't a good thing to have at all. Depending on exactly where it is, before or after the sponge, it can absorb the ammonia before it gets to the bacteria so they don't grow properly. But if zeolite has been in there a while, it will be full and bacteria will be growing on it.
If it was me, and the gravel is zeolite, I would move the sponge over, filling any gaps with the new media that comes with the new filter, and throw the zeolite away. But you would need to keep a close eye on your ammonia and nitrite as you would be throwing some bacteria away and may well have a mini spike. If you do start to see ammonia and/or nitrite, do water changes to lower them until the bacteria catch up.