No connection between Wireguard peers even though tunnel works

I have an internet modem/router on 192.168.178.1 and behind it is an OpenWRT device on 192.168.178.101 with a very simple setup as Wi-Fi AP and Wireguard server.

Here is the full setup:

Internet router on 192.168.178.1
  Static route: 192.168.9.0/24 -> 192.168.178.101
  NAT port forwarding: Wireguard -> 192.168.178.101

OpenWRT device on 192.168.178.101
root@OpenWrt:~# cat /etc/config/network
config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'
config globals 'globals'
        option ula_prefix 'fde1:[...]'
config interface 'lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.178.101'
        option gateway '192.168.178.1'
        list dns '192.168.178.1'
        option device 'br-lan'
config device 'wan_eth0_2_dev'
        option name 'eth0.2'
        option macaddr '[...]'
config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'
config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '2 3 4 5 0t'
config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '1 0t'
config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0.1'
config interface 'vpn'
        option proto 'wireguard'
        option listen_port '[...]'
        list addresses '192.168.9.1/24'
        list addresses 'fdf1:[...]'
config wireguard_vpn 'wgclient'
        option route_allowed_ips '1'
        option description 'Wireguard Client 1'
        list allowed_ips '192.168.9.2/32'
config wireguard_vpn
        option description 'Wireguard Client 2'
        option route_allowed_ips '1'
        list allowed_ips '192.168.9.3/32'

root@OpenWrt:~# cat /etc/config/firewall
config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone 'lan'
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        list network 'vpn'

Example client config:
  [Interface]
  PrivateKey = [...]
  ListenPort = [...]
  Address = 192.168.9.2/24
  DNS = 192.168.9.1

  [Peer]
  PublicKey = [...]
  PresharedKey = [...]
  AllowedIPs = 0.0.0.0/0
  Endpoint = [my domain]:[port]
  PersistentKeepalive = 25

Both clients, 192.168.9.2 and 192.168.9.3, can connect successfully and simultaneously and have access to servers in the 192.168.178.0/24 subnet, as well to the internet via the gateway.

However, they cannot see or ping each other! That is the ping from 192.168.9.2 to 192.168.9.3 fails.

Maybe the problem is restricting the peers to e.g. allowed_ips '192.168.9.3/32'? However, without that, only one peer can connect at all.

What am I missing? Thanks.

No that is not the problem, your router should have a route to route 192.168.9.0/24 via the wg interface already (and also the more specific routes as you enabled route allowed ips).

The problem might be that those devices have their own firewall which does not allow incoming traffic.

Strangely, from the OpenWrt server (192.168.9.1) I can ping the peer (192.168.9.2), but not from the other peer, even though both do have access to the *.178.* LAN.

The configuration looks correct. You could try packet captures (tcpdump) to see if peer-peer pings are being sent out the wireguard interface but not answered by the peer. Also you could look at the wan interface. The packets are encrypted there of course but you can see if an attempt to ping a peer generates an encrypted packet.