I would try cleaning the switch first. No need to remove the door panel to remove the switch. Here is someone else who fixed a similiar problem with cleaning.
http://us.lexusownersclub.com/forums...hl=door+switch

 

Thank-you for the suggestions. Can anyone instruct me on how to pry off the metal/aluminum covering (I have a Silversport) around the window switches? I just don't want to damage or bend any of the metal.

Thanks in advance.

 

I also have a SS but haven't had the need yet to remove the switches, but I believe you don't actually pry off the aluminum trim. I think the switch housing is held into the trim with clips and will pry out from the trim. You may want to post over at the link I provided to confirm this or try PM-ing the original poster. Of course he didn't have a SS so he will be talking wood trim instead of metal but the switch should still be secured in the same way.