When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
You can try cleaning the switch contacts first to see if that is the issue. If you clean the switch and the symptoms remain you may need to rebuild or replace the mirror assembly because the motors inside are failing.
I just went through this on my 2005. My driver mirror would function perfectly except for the up and down. I cleaned the switch and nothing changed. I then ordered a new mirror and once it was installed everything worked perfectly again.