First thing to do it test your water to check if anything's out of balance (do you have filter, and is it cycled?), then do a water change.
Don't feed the fish for two or three days (that won't do him any harm at all, I can assure you), then try him with some pea (lightly cooked, taken out of its 'shell' and chopped up) or some daphnia (live or frozen are both fine).
If he's not improved, or you're unsure about anything, do please post again x