He isn't brimming with personality or charisma, which I think does matter as he comes across a bit emotionless and unenthusiastic.
He came to prominence for a lot of people by challenging someone (Wiggins) who was generally quite well-liked.
His riding style is not pretty.
His tactics - power meter, tempo riding - have come across as boring (obviously not even close to valid after this year's Giro)
He rides for a very unpopular team.
He lives in Monaco.
He is a white Kenyan.
I think some or all of those have contributed to people finding him hard to warm to, and these are just the legitimate or semi-legitimate reasons.
The more I see of him, the more likeable I find him though.