My mic has low volume for external callers

When I call external customers and when external customers call me, they frequently ask me to speak up because they can barely hear me. This happens whether I use my headset (Plantronics CS500 XD) or the handset (Polycom VVX 411). Calls to people in the company work just fine. For example, there is a notable difference between me calling a coworkersring central phone vs calling their personal cell phone. I rebooted the phone, but that doesn't seem to have improved my audio. And this isn't an issue with audio quality. My voice isn't choppy/broken/whatever. It is just very quiet.