This is a combination issue with the VFAT driver being slow (or, arguably, thorough) and the software used for copying. Windows Explorer, when copying a file, first asks the destination to pre-allocate the total size. To do that Samba calls ftruncate() ... which is really slow with FAT filesystems (I believe I remember it writes the whole file with zeroes first, which is not done with filesystems like ext4 that support sparse files).
It depends on the program that does the copying, though. Unlike Windows Explorer, some other file explorers do not call for this pre-allocation. And it's not inherently a bad thing, Windows Explorer is probably just trying to avoid copying data that will not fit in the end.
Google "samba vfat ftruncate" to find folks having the same problem for decades.
(Edited for clarity)