I agree with Andy. It depends on how rushed I feel. If at all possible I like to have given myself enough time for a good clean. It can be a very good thing to take the ornaments out (they can be scrubbed in the sink if needed) and even some of the plants (the ones that don't mind their roots being disturbed.) I do a very deep gravel clean all over the tank and usually remove 50 to 60% of the tank water doing it. I try for those smoky clouds of stuff that can sometimes be found deep in the gravel (this is assuming you've got gravel and not sand!) After the refill, I reshape the gravel, re-scape the plants that have been moved, re-set the ornaments and generally make everything perfect. The next morning I'm rewarded with a crystal tank that looks great.
Alternatively, there are times when I feel rushed and I leave the ornaments and plants in place for these gravel cleans. I usually still to a full sponge-down of the inside tank walls though, to remove any tiny beginnings of algae that I can't even see. I always do that prior to the gravel clean, so any loose algae will have a greater chance of being sucked out.
Its my feeling that your focus about the fish during tank cleaning shouldn't be so much about them being disturbed but instead should focus on not harming them with the gravel cleaner. These gravel cleaner things are quite clunky and more than one fishkeeper and hurt or even killed their fish with these things. The fish are going to be disturbed like it or not but its really very rare that tank cleaning is harmful to them.
~~waterdrop~~