Weirdly I have this problem on both my bikes; a Raleigh 531Club usually with a single rear pannier on for commuting, and a Holdsworth path racer with a front rack. Happens cycling no-handed (look mum!) at v. low speeds, and gets quickly worse with speed.
Seems to be pivoting around the seat tube. I've gone through various bar, stem, wheel and rack set ups and the problem is always there.