I'm in the same boat. I'm not very hairy to begin with. If I shaved, no one would probably even notice, and I doubt it would improve my riding.
In fact, I've done it once, and all it did was give me uncomfortable chaffing for about a week. I didn't like it and didn't try it again.