To me, the minor statistical advantage isn't worth the effect I believe my wearing hivis will have socially:
People may think that its the responsibility of a cyclist to avoid being run over by a driver
People may think that cycling is so dangerous you must take every available precaution to avoid being run over by a driver
People may think that if you don't wear a special jacket then you are responsible for being run over by a driver
People may all start wearing hi-vis, which may decrease the already minor efficacies of hi-vis, meaning the same amount of cyclists get run over by drivers but now we all look stupid as we die
People may normalise wearing hi-vis, expect cyclists to wear hi-vis, and become less able to notice cyclists not wearing hi-vis. Wearing hi-vis may well be a zero sum game. The more visible I am, the less visible others are, until we all wear hi-vis and look stupid as we die.
People may think that cycling is uncool
I don't want to forward any of the above, hence no hi-vis.
The reason that people don't want to have a "reasonable discussion" about hi-vis is that to many even asking the question, or having the same, repeatable, pointless discussion, is in itself a form of deflecting blame, taking up the time of the vulnerable (in this case non car road users) to help assuage the guilt of others, and in the end, doesn't really ever change anyone's opinion.