You just need to carefully pry out the plastic piece for the backseat air vents and the screws for the leather armrest piece should be accessible. No need to remove the whole center console.

 

Other option...build a custom center console. I just did a center console delete on my 01' and i gotta say, as high quality as our cars are, a lot of the interior componenrs are plasticy and ugly...this car deserves something hight quality...perhaps a nice hardwood, stainless steel, etc...My buddy actually replaced the center console on his Tundra years ago with custom baltic birch subbox sporting an 8 inch high output subwoofer.