dengqf6 sended a patch to linux netdev to fix this issue: https://lore.kernel.org/netdev/20200414063408.4026-1-dqfext@gmail.com/T/#u
I did some basic testing added two VLAN ontop of a normal network, ping them and remove 1 of the VLANs.
I had no issues.