What I found in early experiments was that removing VLAN 1 from a port would block all traffic on that port regardless of VLAN. Tagged or not doesn't matter. Only membership. My "fix" was to let the driver fake removals of VLAN 1 from any port. Very ugly since it creates a hidden domain with all ports, which can't be disabled.
Now this is a long time ago, and I must admit I haven't followed the development closely. So for all I know this was an issue related to how we configured the SoC at the time, and not relevant anymore.
But magic behaviour related to VLAN 1 (or another ID configured as the switch "management VLAN") is/was very common. So common that some guides recommend staying away from VLAN 1.