The aux adapter assy in our cars is used on many Toyota vehicles. It was something I had to replace. The part number is 86190-53010 and it takes about 20 seconds to replace. I'd go buy it from Toyota at half the price of Lexus. Go out to your car, pug it in, if it doesn't work go right back into Toyota and tell them you changed your mind
See if they will take it back.