Yes agreed. It shouldn't be necessary. I found the same behaviour as you and could not track down the cause either. Which is why I resorted to using the force unload option. So I'm definitely interested in the answer if you can track it down.
So technically speaking, one needs to rmmod -f to force unload a module even with this kernel option configured. What I found was that if I did not have it enabled, I could not unload modules, similarly to you.
With it enabled I could unload modules with a simple rmmod without the -f flag.
root@OpenWrt:~# lsmod | grep usdm_drv
usdm_drv 90112 0
root@OpenWrt:~# rmmod usdm_drv
root@OpenWrt:~# lsmod | grep usdm_drv
root@OpenWrt:~# dmesg | grep usdm
[92866.360317] usdm_drv: Unloading USDM Module Version 0.7.1...
So, this shows the driver unloading and its exit routine being called. However, without the FORCE_UNLOAD option I cannot unload this module.