So ... I just swapped the default Cannondale stem for a shorter FSA SLK stem on my new Synapse Carbon. The conical spacer plus one 5mm spacer below, the rest on top (as it was before).
Somehow I've introduced some horizontal movement between the fork and frame. Checked everything is tightened using a torque wrench, and everything is pushed down as tight as possible. This is the limit of my powers, what am I missing?