Yep, the "finish" has worn off. The majority of leathers used in the automotive industry are NOT vat-dyed. They have a sprayed-on "finish" (top coat) that provides the color & in some cases, the texture. Hey, it's cheap and easier to clean!
Anyway, unless you can find some sort of touch-up stuff (I hate to call it paint, but that's about what it is), you will need to have the armrest recovered. Doubtful that you will get a 100% color match, so consider recovering the passenger side at the same time.
Your Lexus dealer may be able to get new covers. Won't hurt to ask.