A bit more digging about has shed some light on this problem, and also a fix.
All the files which were incorrectly shown as directories had the xattr
user.DOSATTRIB="", according to
getfattr -d on the router. All other files which were fine had a base64(?) value of this attribute, e.g.
Removing the invalid
"" value of
setfattr to either change it to something else or remove it altogether, meant the problem files were no longer displayed as directories. They were correctly shown as files and these mp4 files could be opened and played with VLC. Setting the value back to
"" caused them to be displayed as directories again. So the problem can be reliably reproduced and fixed simply by changing this xattr, without having to disable the
macos=1 option in the config file.
Another way to restore a valid
user.DOSATTRRIB xattr is, with a Windows client, to right click on the problem file (displayed as a directory), then toggle the "Archive" property off and then on again, clicking Apply each time. This sets the xattr to a base64 string rather than the invalid
"" value, so is probably preferable to just deleting this xattr.
So, it seems that somehow these files were either created with an invalid value for
user.DOSATTRIB, or had it set at a later time. After getting rid of this invalid value using one of the methods above the files are fine. I'm not sure what actually caused this in the first place, but at least there is a simple fix.
I've opened a Github issue about this, which contains roughly the same information as in this post. As of 8th May 2020 the issue is still open, as we're waiting to see whether the problem returns. But if it was just a one-time issue with a bunch of files in a single file copy, it's unlikely to be a bug in the samba4-server package.
I'll mark this thread as resolved, so if in future anybody else has the same problem they can see the solution here: just change
user.DOSATTRIB to something other than
"" by e.g. using
setfattr -x user.DOSATTRIB <file> on the openwrt router or by toggling the Archive property of the file on a Windows client.