What's all the aggression about?

The thing you did wrong was not apologise in the first place. It's the same as walking into someone and not apologising - why would you do it? Yes, no harm done, but IMO you lack manners as well as the car driver. The fact you (seemingly) ignored your misdemeanour in the first is probably what wound him up.

the car passed him and hit his arm because it was too close?