The problem here may be that your Wireguard VPN service doesn't "hairpin" a request for your own public IP back to you properly.
Often it is simplest to cheat and install your website's FQDN in your local DNS as the LAN IP of the server. Then the request never leaves the site. If OpenWrt is your local DNS, this is done by adding the name and IP to /etc/hosts.
Use Network-->DHCP and DNS-- Hostnames tab.
This actually installs a config domain into /etc/config/dhcp. The result is the same though, when you look up your public name from a machine on the LAN at home, it will use the LAN IP.
The router is irrelevant as it just routing the packets.
The browser will contact the reverse proxy with a name www.kef.wrt and it will expect to get a trusted certificate including that domain.
If the certificate is self signed, the browser will complain.
If the name is signing another domain, the browser will complain.
You can give examples of fictitious domains, as long as you keep consistency and specify which are self signed.
My certificates for the reverse proxy domains (a.mydomain.com, b.mydomain.com...) are coming from Letsencrypt, that can be the problem ? I don't have any troubles accessing my HTTPS domains except on this configuration.
The thing i don't understand: it's working perfectly when i put the wireguard server on docker, or directly on the server.
What is the topology in both scenarios?
You have OpenWrt router on the edge, then some server in the lan running the reverse proxy and the web servers? The domain names resolve to public IP or private? Which one is it?
Yes, my installation is pretty basic like you said.
The domains are resolving with HTTPS (Letsencrypt certificates generated with NginxProxyManager) on port 443 to the private lan adress of the server (192.168.10.20), to the different ports for each app.
Got NginxProxyManager, AdGuard, SearXNG, etc ... all running in a Docker instance, but i guess it's irrelevant.
This would seem to suggest there is something running on the OpenWRT device that is intercepting the SSL requests. It'd probably be useful to see the contents of /etc/config/firewall and /etc/config/uhttpd as a minimum. /etc/config/network may also be useful.
This error message is so generic that nothing can be known about the nature of the error. NSURL is Apple's SSL stack, so you'll only see this on Apple devices. Do non-Apple operating systems work with your sites on the same network situation?
It's possible that NSURL has flagged the combination of a public name with a RFC1918 IP address as suspicious of a man in the middle attack and refuses to connect. But I don't know much about Apple stuff at all.
No it's not that. The whole point of SSL is so the Internet can be treated as the insecure connection it is with eavesdroppers and men in the middle. So SSL does end to end encryption to protect the application data from attacks, not requiring the Internet channel itself to be secure.