Thank you, it did work.
To summarise, these are the steps I followed to get ubibootmod:
Unscrew, connect serial console as described before. With the router powered-off but the ttyUSB0 serial connected, do:
git clone https://github.com/981213/mtk_uartboot
cd mtk_uartboot
cargo build
./target/debug/mtk_uartboot -s /dev/ttyUSB0 --aarch64 -p bl2-mt7986-ddr4-ram.bin -f openwrt-23.05.3-mediatek-filogic-zyxel_ex5601-t0-ubootmod-bl31-uboot.fip
This will wait for the router to power on. Then power on the router, the custom u-boot should start and the mtk_uartboot command will exit successfully.
Open a terminal emulator (e.g. minicom,screen) with baudrate 115200,
so you can see the output of the custom u-boot.
In the terminal you can see that the uboot tries to tftpboot repeatedly, looking for a specifc ubootmod-initramfs-recovery.itb hosted at 192.168.1.254. You need to connect directly one of the router's 1gig ports to your host. The host's network needs to be setup to have the static address 192.168.1.254 (e.g. if you use systemd-networkd on the host, edit the /etc/systemd/network/ accordingly).
Start a tftp server on the host (e.g. atftp or tftp-hpa), disable your host's firewall (if you have one) and place the correctly-named ubootmod-initramfs-recovery.itb in your /srv/*tftp/.
Eventually the tftpboot should fetch the file from the tftp server. Once this is done, you will be returned to u-boot prompt. Then, issue bootm and openwrt will start running on RAM.
Log into the Openwrt-on-Ram (either via ssh or terminal emulator), and transfer the files necessary to flash the openwrt firmware to the /tmp folder of the router. I used scp -O .... root@192.168.1.1:/tmp for that.
The files you need are:
zyxel_ex5601-t0-ubootmod-bl31-uboot.fip
zyxel_ex5601-t0-ubootmod-initramfs-factory.ubi
zyxel_ex5601-t0-ubootmod-initramfs-recovery.itb
zyxel_ex5601-t0-ubootmod-preloader.bin
zyxel_ex5601-t0-ubootmod-squashfs-sysupgrade.itb
kmod-mtd-rw*_aarch64_cortex-a53.ipk
The kernel module kmod-mtd-rw should match the openwrt version you are currently running on ram; for example for 25.0.3 look into here:
https://downloads.openwrt.org/releases/23.05.3/targets/mediatek/filogic/kmods/
Then inside the router terminal:
cd /tmp
opkg install kmod-mtd-rw*_aarch64_cortex-a53.ipk
insmod mtd-rw.ko i_want_a_brick=1
ubidetach -p /dev/mtd5; ubiformat /dev/mtd5 -y; ubiattach -p /dev/mtd5
ubimkvol /dev/ubi0 -n 0 -N ubootenv -s 128KiB
ubimkvol /dev/ubi0 -n 1 -N ubootenv2 -s 128KiB
ubimkvol /dev/ubi0 -n 2 -N recovery -s 10MiB
ubiupdatevol /dev/ubi0_2 openwrt-mediatek-filogic*-ubootmod-initramfs-recovery.itb
mtd write openwrt-mediatek-filogic-*-ubootmod-preloader.bin bl2
mtd write openwrt-mediatek-filogic-*-ubootmod-bl31-uboot.fip fip
sysupgrade -n openwrt-mediatek-filogic-*ubootmod-squashfs-sysupgrade.itb
The router will then restart, and you should be running openwrt from flash.