Before I got your message, I managed to find a useful writeup (actually about installing an Ipod adapter), but gave good pointers on how to remove the glovebox and surround
a. undo the 2 screws along the top edge of glovebox. Let the glovebox unit stay, held in place by the 3 top clips (good Lexus engineering design!)
b. remove the horizontal trimpiece under the glovebox (3 pinch clips at front then it slides forward)
c. remove the door trim piece (with Lexus nameplate), just enough to reach the 10mm bolt holding the lower corner of the glovebox unit.
d. undo the 10mm on the opposite lower corner (near transmission tunnel)
e. Unclip the wiring harness for the tailgate switch, then you can pull out the whole glovebox unit!