Who organised the scaffolders and the tower? If you then your problem, if the builders then that's on them and lol at paying before the job is complete (ie: works finished, site cleared). How can you be sure that the works won't be damaged as the scaffolding is being dismantled?
Your builder's invoice will have payment terms that will be, or should be, longer than the end of the week. Just say that everything is good and that you'll be making payment this week and see how it plays out. Nearer the time mention the scaffolding and say you'll make transfer when it's being driven away from your property.