There isn’t really a straightforward answer as to why speed wobbles (also called tankslappers in the motorcycling world) occur. Essentially something knocks the front wheel off line and the self-correcting nature of the front wheel in the fork moves the wheel back towards the midline but overcorrects and the wheel goes the other way where the reverse happens, thus setting off a to and fro motion.
There are many factors that can lead to this including speed, road surface, frame material and geometry and weight distribution. How the various factors interact is not well studied but certainly situations where the front wheel is unloaded and has less traction seem to make wobbles more likely. In your case, having loaded rear panniers plus one or more of the other factors could cause it.
Personally, I’ve never experienced it on any of my bikes in 50 years of cycling but there is always a first time.
If you fancy seeing a really good speed wobble: