You’re spot on. The flats I occasionally swap on and off the bike have a very different stack height than the SPDs/shoes combo I use. If for more than a ride to the park, I need to adjust saddle height. The fore-aft has never been an issue for me, but height for sure. Cleat position, combined height of pedal/cleat/shoe, type of flat pedals, thickness of shoe sole, etc., all contribute.
I tend to position myself based on what ‘feels right’ for pedaling. Seems to work OK for me.