Rear View Mirror Bracket Question
 

So I am putting in a new mirror in my 98 gs400. I had the spring loaded bracket and ball bearings, and the ball bearing/plastic part of them broke. My question is, is there a reason I can't just glue the two brackets together and put the screw in the mirror to hold it in? I can't for the life of me figure out the function of those stupid ball bearings.....