There are a few factors involved, but we can find natural watercourses that contain fish and have a pH around 4. The Rio Negro, the largest blackwater river on the planet, and home to thousands of fish such as discus, greeen neons, pencilfishes, etc. has a pH range of 2.9 to 4.2 [ https://www.feow.org/ecoregions/details/314 ]. Simply Discus mentions Rio Negro pH being tested at 3.9 and 4.5 with the latter apparently the more frequent. Biotope Aquarium also has 2.9 to 4.2 and I assume is citing the Freshwater Ecosystems of the World link I gave.
I have no idea how low the pH is in some of my tanks, as the lowest test I have had went down to 5 and the yellow was very much stronger so I assume it is/was down closer to 4. I only keep very soft water species, most are wild caught.
I would water change the minerals back and dilute the leaf compounds, but when I want a higher KH (alkalinity), I keep a DIY pantyhose bag of crushed coral in the filter. I don't have to use very much to keep pH at 7.4 (KH 4), only 1/2 tablespoon per 30 gallons. I don't like to let the pH go lower than 7.2 because of the reasons someone mentioned, the nitrogen cycle is inhibited at that pH level. Any ammonia in the tank is not processed, but kept in a safe state as ammonium. So when you do a water change with higher pH water, the pH shoots up and ammonium converts to toxic ammonia. Plus the fluctuating pH is stressful for the fish. So if you want to maintain a tank at pH 4, you would have to pre-condition your new water at every water change or use RO water. If your not planning to keep a soft water tank longterm, remove some of the leaves and slowly bring the KH up with water changes.
Have you tested your tap? Usually a couple water changes is enough to bring levels back up. But are you intending to keep it that low? If you add less leaves, it won't alter so much. Plus I can't think of any options for a 5 gallon with low pH.