In my opinion, all fish are sensitive to ammonia and changes in a tank.
Usually, it's best not to make too much changes after the fish have settled in.
But there are times when you have to make changes where you have no choice.
For example, if the fish started to breed in a community tank, some people will move the fish out to a separate tank.
Or there are time where you want to move them to a bigger tank.
Though I may not be an expert, but I believe that as long as you can maintain the ammonia & nitrite to zero, have the same water chemistry (pH, GH, KH) and with plenty of plants, driftwoods, caves, hiding places, etc, they will be fine.
By the way, I think the biggest stress for the fish is when they are being transported from the fish farm to the fish stores and then from the fish store to our aquariums.
Recently, I saw a LFS at my place where they kept the Apistogramma in its original packing/bag even after it had arrived at the store for a few days.
I looked at the label on the packing and it was shipped 5 days ago and yet the fish is still sitting in the plastic bag that hasn't been opened. It was horrible.
Imagine a fish sitting in the same plastic bag with the same water and oxygen after 5 days. I think the fish would die soon.
And guess what kind of plastic bag they used?
It's a small bag and without much air inside. Most fish farms will pump oxygen into the bag to make it inflated but not for this case.
Anyway, it's ok to start a new tank when you really need it.
Just make sure to reuse some of your old filter medias and have frequent water change if you need to.
But for 3 small fish in a 20 gallons tank, I don't think there will be any problem at all.