Well he has said he was unable to compete in 2010 because of the intense allergic reaction, hence the TUE in 2011 to help him. He didn't have a TUE in 2009 when he finished third while riding for Garmin. Or a TUE for any of his successes before and after 2012. Without TUEs he was arguably the finest pursuiter the track has ever seen, multiple Ilympic golds and Word Championships including endurance events. He won stage races like Paris-Nice and the Daupine on the road without a TUE, he won the ITT World Champs without one, and the Hour record. His ability as a bike rider beyond the microcosm of 2012 (which we'll agree he didn't win because of individual brilliance, and wasn't even the best rider on the team) has always been world class, he is clearly an extraordinary athlete when you take his palmeres into account, omitting the TdF win. Even that year he was winning stage races before the TUE, so you can argue that the TUEs were an insurance he was riding at his best, without the allergies.
Tl:dr his ability on a bike is proven time after time, the TUE was insurance he was able to perform to that ability