Hi,
- Happy self hosting:
I'm throwing a bottle into the sea today.
Although I no longer had the time to do anything other than updates on my little self-hosted server at home, it was well used, notably by my family, whom I had (with difficulty) managed to convince to switch to my matrix server (with a whatsapp bridge in addition).
That's how it worked before the outage:
Each service runs in an independent unprivileged LXC container.
So I know a lot of services were reached directly by their port when it would have been better to go through the reverse proxy. But I didn't manage to do it for all the services but it worked so, so there you go 
- ISP failure and backup I try to setup:
Unfortunately my internet ADSL line has been cut for 15 days now 
and the ISP will maybe now not repair before the end of the month 
... or later...
A 4G box has been lent to me but obviously it is not possible to use the public IP of this 4G box to reach the server because of the 4G CGNAT.
Since then I've been trying to set up a VPN to use the Internet connection of a third party living nearby. I've managed to install a temporary OpenVPN server at his place and establish the connection from the network behind the 4G box 
The new architecture is this one (what's circled in green is new):
On the left side, my house, the 4G box (A), the previous openWRT router (A1) I want to use as VPN client, the server host (A11), all the services servers inside of LXC unprivilegied containers of this host : for the tests I will use the A112 one on port 3000.
On the right side, the relative house with his Fiber Box (B), a temporary server host (B1), and inside a LXC unprivilegied containers of this host the OpenVPN server (B11).
The idea is to not touch / modify the items under the A1 router so I can easily switch back when my ADSL will be repaired.
The users will connect to the neighbor fiber Box public IP (B) and reach the server on the left side as usual through the VPN tunnel without even knowing it.
I managed to establish a connection between my OpenWRT router (A1) and the OpenVPN server (B11):
- Problems:
I'm not working in IT and English is not my native language so it's been really hard to get some of this architecture starting to work, and I still don't understand all I have done 
Some already helped me thank you, nothing wouldn't work without you.
I'm now stuck for several days, when I "tcpdump" on the service server container (A112) I see the traffic from a user connecting on the internet from the public IP of the Fiber Box (B) but the user only receive a timeout in return.
I can't access internet on the left side (A1, A11, A112) through the Fiber Box (B) connection. I always go out with the 4G box (A) public IP or loose internet access depending of the configuration I setup.
So I'm guessing the timeout error comes from an asymmetric routing?
- Setup : In the LXC OpenVPN server container (B11):
The OpenVPN conf file /etc/openvpn/server.conf (I removed certificates lines):
port 1194
proto udp
dev tun
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
push "route 10.0.4.0 255.255.255.0"
client-config-dir ccd
route 192.168.11.0 255.255.255.0#Not sure if I should do this
#;push "redirect-gateway def1 bypass-dhcp"### If I set this I have no more internet on the left side (A1, A11, A112)
#;push "dhcp-option DNS 208.67.222.222"### Should I uncomment dhcp-option?
#;push "dhcp-option DNS 10.0.4.1"### Should I uncomment dhcp-option and set the temporary server host as DNS server?
client-to-client
keepalive 5 60
cipher AES-256-GCM
max-clients 5
max-routes-per-client 50
stale-routes-check 300
verify-client-cert require
auth SHA512
tls-server
tls-version-min 1.3
remote-cert-tls client
#;push "block-outside-dns"### Should I uncomment this line?
user openvpn
group openvpn
persist-key
persist-tun
verb 3
mute 10
explicit-exit-notify 1
/etc/openvpn/ccd/client :
iroute 192.168.11.0 255.255.255.0
I activated net.ipv4.ip_forward=1
And activated masquerate iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
I don't know if I should do some more? like:
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE
iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -j ACCEPT
- Setup : In the OpenWRT router (A1):
Network-specific OpenVPN client configuration:
client
dev tun
remote zzz.zzz.zzz.zzz 1194 udp
auth SHA512
cipher AES-256-GCM
tls-client
tls-version-min 1.3
remote-cert-tls server
connect-retry 5 600
resolv-retry infinite
nobind
verb 3
mute 10
explicit-exit-notify 1
I followed parts of this guide https://openwrt.org/docs/guide-user/services/vpn/openvpn/client-luci to setup the VPN tunnel on the openWRT (A1) VPN client.
I tried the both possibilities:
LuCi (config 1st try):
Firewall => Wan => Edit => Advanced Settings => Covered devices => tun0
LuCi (config 2nd try):
Interfaces => Add new interface => OpenVPN, Unmanaged, device tun0, Pas de Bring up on boot
I'm not sure what I should do more like:
Firewall => General settings => Zones add => vpn, Input : accept, output : accept, intra zone forward : accept, no masquerating, covered network : OpenVPN,
Allow forward to dest zones : lan
Allow forward from source zones : lan
The default route still go to the 4G Box (A) so I guess this is why I have trouble getting internet from Fiber Box (B).
- Configuration to correct/improve:
I tried to resume all configurations on this architecture:
Here's what I'd like to do:
- The server behind the 4G Box in CGNAT should be accessible from the outside through the VPN.
- Get internet access on the left network when the VPN is connected (via the 4G Box or via the third-party connection, whatever.)
- Don't miss anything about the security of the system
- Understand what I'm doing (yes, because I followed at least 6 tutorials to set up the VPN, with some points still unclear).
Questions:
-
Do I need to do a "MASQUERADE" anything other than the openVPN server (B11)? (The servers' LXC hosts have apparently already a "MASQUERADE" on everything that isn't respectively in 10.0.3.0 and 10.0.4.0)
-
Do I need to pass "net.ipv4.ip_forward=1" elsewhere than on the openVPN server container (B11)? (Apparently this is already the case by default on router A1)
-
On the openVPN server container (B11), does the DNAT on port 3000 have to go to the router (A1) or directly to the host server (A11) or container (A112)? (This seems to work with the server host (A11))
-
On openVPN server configuration (B11):
- Should the "redirect-gateway def1" push be enabled? If so, how do I avoid losing Internet access on the client side?
- Is it necessary to activate the "block-outside-dns" push?
- I've tried to follow parts of this page https://openwrt.org/docs/guide-user/services/vpn/openvpn/client-luci to configure the router (A1) as an openVPN client. Is there anything else I need to do on this router (A1), such as:
- Allow "FORWARD"?
- a "MASQUERADE"?
- Other?
-
Is it possible / recommended to make different paths between input and output? For example, the outside user connects via the fiber box (B), but the server responds via the CGNAT 4G box (A)? (For gain in speed if it doesn't complicate things)
-
Do I need to manually create routes (with ip route ccc.ccc.ccc.0/24 via ddd.ddd.ddd.ddd dev xxx) when configuring the network (interfaces file or eth0.network) already defines a default gateway with this network ddd.ddd.ddd.ddd?
-
How can I make the output to the Internet on the CGNAT side (A1, A11 and A112) pass through the box on the fiber side (B)?
-
What append if the two internet boxes (A) and (B) were both on a 192.168.1.1 (different) network, would it be a problem?
- Thank you:
Anyway, thank you very much for reading this far ^^.
I hope this will help others too
(And sorry for my poor English level)
)