Sugar

I'm still scared of it though.Why are there so many people avoiding it?

According to the wiki page, scare mongering in the 70's and again in the 90's. According to a source on the wiki page

"The safety testing of aspartame has gone well beyond that required to evaluate the safety of a food additive. When all the research on aspartame, including evaluations in both the premarketing and postmarketing periods, is examined as a whole, it is clear that aspartame is safe, and there are no unresolved questions regarding its safety under conditions of intended use."
That's just from the abstract.