Indeed. One person amongst 5000 (on something PBP) isn't going to change anything.
10 people on a 300 person event, as you say, may start to have a noticeable effect for some.
The danger is that if it gets known that it's ok to tag along without entering the insignificant number that did it before may become significant to the detriment of the event itself.
There has been the odd Audax that the organiser decided not to put on again because of non-entrants riding, even though they used commercial controls. The ride had a limit on the number of entrants due to what one cafe could reasonably cope with (even with extra staff) and when many more people than expected turned up (many riding without entering because they missed out on an official place) they were swamped and refused to be a control for the next year. No controls = no Audax.
I'm sure things like Ride London 100 plan on there being a certain number of people who just jump on the route and ride it. If it becomes a significant problem I expect they'll start to do something about it.