It really depends on what you mean by "builder's gravel". If it's in a bag and says it's been washed, then it my be fine (depending on the vinegar test and your fish). If it's just someting you would be getting off the ground or from a construction site, I wouldn't use it as you don't know what type contaminants it may contain.