To my knowledge there is no way to turn off nav guidance voice commands through the voice guidance the feature. You can also delete your destination which will cancel the voice guidance feature but that take a few mouse clicks and may take your eyes off the road unless you are pulled over and stopped. Deleting your destination will not remove if it was on your pre set list in the nav option
If you refer to the "Navigation System Owners Manual" on pages 62 and 63 it lists the screens for the voice settings. I could not find any reference to a voice command that will turn off the destination voice guidance without getting into the voice screen and making changes to either voice settings or auto voice guidance. Also look at pages 316 and 317 where it lists the expanded voice commands and the option to turn off voice guidance through voice command is not listed. In my opinion the Lexus voice command system is marginal at best when using the standard voice commands.
You may want to contact your dealer to get final closure on this issue....hope this helps