It ain't blown. I've got that same system in my ES. Most likely the person who put it in didn't bypass the factory amp. The sound your hearing is two amps competing with one another - the factory one, and the amp inside the d3.
The harness you need to bypass the amp is Metra 70-1761 (http://www.clublexus.com/forums/es30...tallation.html
) Second to last post on the first page.
The stock stereo harness will not be used. It just sits back there. Also, as a result of not using the factory amp, you supposedly (so I've read) lose your sub output, which I believe is that speaker you're talking about.
I just installed the D3 in my ES about a month ago. Pain to bypass the amp and get all the wires situated, but once done, it works nicely! Good luck!