oxygen and CO2 are not linked as is commonly thought. Airating the water (and increasing the O2) will not reduce the CO2 directly, it just increases the oxygen, although the increased surface agitation means the CO2 will diffuse out faster.
You could try reducing the airation, just keep an eye on the fish that they arent at the top gasping, as this may help the CO2 remain in the water longer (thus lowering pH)
IMO your best bet is to mix your tap water with RO water to lower pH and hardness