Depends on the size of the tank, and the amount of bio filtration. Rotting food creates ammonia, which won't be a problem if you are lightly stocked or overfiltered in regards to bio filtration. You should siphon them out if they are there more than around 12 hours.