It took me some time, but I finally was able to fix the serial connection and unbrick my device. But first of all a list of the mistakes I made:
- Bricking my device (wasn't very helpful )
- Buying the the wrong RS232 to DB9 connector cable, also see https://pinoutguide.com/Motherboard/rs232_header_pinout.shtml . It needs to be the AT/Everex
- Buying the "cheap" and wrong RS232 to USB adapter cable
- Destroying the soldering joint on R603 (R1out) while trying to solder a wire onto it and that was the reason, why it was not possible to send any keyboard input
- Asking the Zyxel Support, about what do I need and how to use the rs232 serial connector
How did I fix it:
- Soldered the right RS232 to DB9 connector cable for AT/Everex pinout
- Bought a Chipi-X10 RS232 to USB adapter (thx for the tip @svanheule )
- Soldered a cable from the U8-Chip to the other end of the destroyed soldering joint on R603 (so it seems like those resistors have 0 Ohm and works like a jumper)
- Connecting via Picocom with the baudrate 115200 and 8N1 to the console
- Starting from the second firmware partition with the following command (Zyxel support said that it's not possible to boot the 2nd partition):
boota 1
And voila the switch was booting properly. Thanks for all the responses, it definitely helped me to figuring it out and most important to learn more about that stuff!