It can depend...A LOT. If the gravel is coarse, uneaten food and fish/plant waste can get down under where it decomposes to pollute the water and can become a 'nitrate factory'. This is one of the short comings of the undergravel filter when water flows are too great and detritis gets pulled deeply into the gravel. Without routine gravel vacuuming, water quality may suffer.
The above condition coupled with over feeding and perhaps infrequent periodic water changes can result in very high nitrates that may go unchecked.
On the other hand, if you don't over feed, the gravel size is modest, AND you do routine partial water changes of sufficient volume/frequency, gravel vacuuming becomes less important.
AND there are different schools of thought. My friend Greg Sage of Select Aquatics of Erie Co. is convinced that mulm should be removed as it results in a decrease in water quality that inhibits fry growth. While Charles Clapsaddle of Goliad Farms feels that mulm is beneficial and there are several inches in most of his otherwise bare bottom vats of fish...but he has massive plant filtration.
One other factor. Once organic material decomposes the resulting 'sludge' is relatively inert so one could make the case that with sufficient plant filtration and/or sufficient frequency/volume of fresh water partial water changes, mulm or mulm in the substrate is insignificant...which can make grave vacuuming quite optional.