I have in the past(many years ago) and had no problem but I did clean them. Many I used were from the beach that was on an island where my grandmother lived in Maine. Deanasue makes a good point it really depends on where and the quality of the water.
As others have mentioned, it is never "safe" to use rock or wood collected outside unless you know about the material and composition and the site.
Calcareous rock will dissolve calcium and magnesium into the water, raising GH, KH and pH. This may be OK if you have hard water fish species, but not if you have soft water species. And this is only the obvious GH issue, there is no way to determine what if any toxic substances the rock (or wood which is even more risky) may have come into contact with and "soaked up" to leech out eventually.
Having said that, I use river rock acquired from a landscape outlet in some of my tanks. This, as the name implies, are smooth pieces of rock that have been worn down by water action and collected from rivers and streams. This is generally safe.