OpenVPN TAP, can't ping server's WAN IP while VPN is on

I have 2 sites:
Site A is a LAN with a router running Asus Merlin and OpenVPN server. Router A is connected to internet via PPPoE. It has a WAN public IP (IPv4 only) and a DDNS name.
Site B is a LAN with a router running OpenWRT and OpenVPN client. Router B is connected to internet via cable modem. It has a IPv6 public address and a DS-Lite IPv4 local address.

LAN A and B are connected via OpenVPN in TAP mode. Both LANs appear as one. Each device in each LAN has a static local IPv4 address with gateway/DNS set to closest router (LAN A devices go to router A; LAN B devices go to router B).

What works:
Ping inside each LAN works.
Ping from each LAN to the other LAN works.
Ping from each router to the other LAN works.
Ping from router B (client) to router A (server) WAN address works.
(The other way is not possible; B has no public IPv4 address and router A has no IPv6 address.)

The problem:
When the VPN is connected, ping from LAN B devices to router A WAN address DOES NOT WORK.
Ping from router B (OpenWRT) to router A WAN works, but not from LAN B devices.
It's not just ping. There's absolutely no way to access any services from the WAN-side of router A.
If I turn off the VPN, everything works as expected. If I turn on the VPN, already-established connections continue to work, while new connections fail with timeout.

What I tried:
I looked at the routing table. There's nothing odd there. Default route on router B is via ds-lite interface and there is no special route for router B's WAN IP.
I ran tcpdump. I can see ping packets arrive on the br-lan interface of router B, but I don't see them leaving router B on ds-lite interface or on tap interface. It seems that OpenWRT just drops them.

What else should I check? Is this a bug?
Thank you!

I know I can access services via the tunnel using the LAN IP, but that is not what I want.
Accessing LAN A services via VPN is somewhat slower than using WAN directly and I must change the address in a lot of scripts and devices every time I leave/return home. I prefer to be able to access WAN IP of that router even if the VPN is on.