See this comment by @slh regarding USB devices passed from host to VMs. In short, you should expect issues, unless you do PCIe pass-through on the USB controller, in which case you may still get issues (or not). You will just have to try and see.
P.s. I am not speaking from personal experience, just bringing to your attention the opinion expressed by someone else.