Can it be scratched, dented, broken? Sure. So can steel.
If you're going to be putting your weapons into situations where a polymer gun would be destroyed and a steel one would survive, then by all means buy a steel gun.
Just keep in mind that a 20,000 round torture test was performed on the XD:
http://www.hs2000talk.com/viewtopic.php?t=20502 , part of which (if I'm not mistaken) involved a truck running over and being parked on top of the weapon, as well as being tossed off a cliff. No breakage, no failures to fire, no nothing. Scratched? A little (last page in that thread has post test pics)...but that's just character.

I don't think a 1911 wouldn't have come out looking so rosy.
It's tough as nails. What are you afraid of?