If all things are equal no I don't see how it could much, maybe a a millisecond to get from say 7psi to 15psi but once the boost is on it comes on hard and you wouldn't notice it.
But like said above low boost will have lighter wights, run less octane, and prolly not need an intercooler which is also gonna help, with a smoother/less restrictive air flow and no pressure drop.