VLANs are likely the "right" way to do this, it's just that the traffic needs to be routed between the VLANs.
You'll have to decide what risks you want to take if you want to manage your IoT devices directly.
Many IoT devices provide both on-link access and access through a remote server (MQTT over TLS, or the like). Those that don't have a remote server can be very challenging to "isolate" due to broadcast protocols that inherently don't span multiple networks.
IP- and MAC-based firewalls (as opposed to using the interface and ensuring that all the addresses you see on that interface are "expected") are very weak security. It's trivial to capture a MAC address off the air, and equally trivial to change the MAC or IP address of a "rogue" device.