I had an 200 IQ moment yesterday after my message and moved the cable that comes from the first router. Instead of using the second router's WAN port, I plugged it into LAN1. After all I want it to behave like a switch. After enabling VLAN20 on port 4 (they are swapped in the config) instead of 5 (WAN) it worked fine. I have it running like this since yesterday and everything seems to be going as expected!
I lost an ethernet port, but I can live with that. Should I report this bug somewhere?