[SOLVED] Gretap Tunnel and MTU-Size Problem

[mm93]

Hi,

I'm trying to bridge a network over two TP Link Archer C7 Routers with LEDE 17.01.4.

The Gretap intergration for bridging Ethernet over IP which works as designed. In a later step I'd like to use IPSec for the encryption.

Now, whenever I'm trying to start web browsing or doing some other things else, I'm having trouble with the MTU-Size. The setup looks like this:

Company Network <-> TP Link 1 <-->GRETAP TUNNEL<--> TP Link 2 <-> PC

When I set the MTU of the PC to 1200, everything works fine but I have to reduce the the MTU Size in one of the TP Link Routers, as there are many different devices in this location.
I tried to change the MSS with iptables but I couldn't get it working.

I'm trying for days now and I'm desperate, please help

Thank in advance!

Best Regards

[lleachii]

Did you enable MSS Clamping on that Firewall Zone?

[mm93]

Yes, I've added the Bridges to the Lan-FW-Zone and enable MSS Clamping.

Not working

[lleachii]

to the Lan-FW-Zone and enable MSS Clamping

What about enabling MSS Clamping on the Zone containing the IPSec tunnel?

[mm93]

I dont even have an ipsec tunnel yet, as it's not even working with pure gretap.
The problem is that the routers not sending any ICMP-Error back to the PC, so the PC continous sends it's 1500 byte packets.

[jeff]

The problem is that a Layer 2 tunnel doesn't "know" where to send "fragmentation needed" packets when the link MTU is exceeded. This is because it deals with Layer 2 and the IP source address is Layer 3. On-router connectivity "works" as the sender is aware of the interface MTU. Off-router connections fail because PMTU is not possible. This is a protocol limitation, not an OpenWRT limitation.

I wasn't able to get /etc/config/network to immediately do what I wanted. It may be possible to do so, but I haven't gone back to trying.

Note that this will only work if both ends of the tunnel "agree" on a sufficiently large MTU. 2304 is overkill for 1500-byte packets plus their Ethernet framing, plus the 802.11 framing.

The solution (OK, hack) I am using is to hook the wireless-interface creation:

--- a/package/kernel/mac80211/files/lib/netifd/wireless/mac80211.sh
+++ b/package/kernel/mac80211/files/lib/netifd/wireless/mac80211.sh
@@ -513,6 +513,8 @@ mac80211_prepare_vif() {
                ip link set dev "$ifname" address "$macaddr"
        fi
 
+       [ -x /etc/mac80211-post-add.sh ] && /etc/mac80211-post-add.sh "$ifname"
+
        json_select ..
 }

and set the MTU (and IP address, as I don't bridge the wireless interface) in a script

jeff@office:~$ cat /etc/mac80211-post-add.sh 
#!/bin/sh

ifname="$1"

if [ "$ifname" = "mesh0" ] ; then
	ip addr add <mesh0-ip-address>/24 dev mesh0
	ip link set dev mesh0 mtu 2304
fi

[mm93]

Good approach! Thank you!!

I was successful with the option "mtu" in /etc/config/network. I've set MTU on the P2P IP Interface and it's working. But with such an high MTU I can't bridge over VDSL

I need something to remove df=1 from every packets getting sent over the bridge or a patch which tells the client to reduce it's MTU size to 1200 f.e.

[jeff]

If you're routing the packets through the OpenWRT devices, then the ingress interface, running at Level 3, should be able to send back the ICMP message. Given that you've got the problem you describe, it sounds like they are bridging the networks at Level 2.

This is a "common" problem with encapsulated Layer 2 bridging that I haven't seen a good solution for. I stopped looking once I was able to get the MTU on my 802.11 link up.

Some interesting references include

http://blog.systemathic.ch/2015/03/05/openstack-mtu-pitfalls-with-tunnels/

the-notorious-mtu.pdf

Resolve IPv4 Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC

[mm93]

Yes, that's exactly the problem: layer 2. I'd have to change the MTU of the remote ip interface at layer 3.

This guy wrote a patch to remove the df flag with iptables:
http://backreference.org/2013/07/23/gre-bridging-ipsec-and-nfqueue/

But I can't use this for openwrt and my skills in c are poor.
I've hoped that this issue has been fixed in a kernel patch -.-

I've tried to set TCPMSS with iptables, but I guess that iptables ignores this traffic and won't touch it. Any other suggestions with iptables?

[mm93]

I figured out a solution now.
For using iptables in bridges the package kmod-br-netfilter is needed and net.bridge.bridge-nf-call-iptables has to be set to '1' in /etc/sysctl.conf.
Now every packet flows through iptables and the mss clamping or set-mss can be used for tcp connections.

[dlakelan]

I wonder how well this would have worked if you were using IPv6? I'm impressed with ipv6 in general and the icmp6 based path mtu discovery stuff is generally a good thing. The fact that there is no possibility to fragment ipv6 at the router means generally the default on firewalls is to properly handle the appropriate icmpv6 messages, and the lack of NAT means the end-to-end icmpv6 messages are routable.

[mm93]

I think it‘s the same problem like in ipv4: pmtu discovery in layer 2 doesn‘t get recognized, so no icmp error 3 or 4 and ‚fragmentation needed‘ packets would als get lost in layer 2 - and wouldn‘t even work as the df bit is set in almost every case.

ipv6 also uses icmp error 2 in the pmtu discovery.

[dlakelan]

Would ebtables and a brouter config help here? basically ebtables looks at the layer 2 packet, sees it's too big, and DENYs it in the brouting table, where it is shuffled out of bridging into routing path... The router then needs to act appropriately... not sure what config you'd need there, depends on details of your network .

[mm93]

Maybe, but I couldn‘t find an option in ebtable for controlling the frame size

[dlakelan]

Yeah, I don't see anything to detect frame size either. It seems to me like the issue is that the routing code has to see the frame so that it can generate the icmp/icmp6 message. So here's a thought:

I'm assuming you're bridging something like an Eth port, a wifi SSID, and the GREtap together on each of the Archer C7's. If in ebtables brouting chain, you see the frame come in on anything other than the GREtap interface, then broute it. The routing code will inspect it, and then route it back out the bridge... if you set the MTU of the bridge to the right value, then the routing code should do the right thing with it right?

Worst case you might be able to bridge a Veth pair into the bridge, and have the router route it to the veth input and have the veth output inject the packet into the bridge. The MTU on the Veth could then be set appropriately.

If the frame comes in on GREtap then just accept it for bridging.

Thoughts?

[dlakelan]

I'm no ebtables expert, but I think what you need is something like:

ebtables -t broute -A BROUTING -p IPv4 -i ! mygre0 -j DROP
ebtables -t broute -A BROUTING -p IPv6 -i ! mygre0 -j DROP

EDIT: I added a protocol match, you can only use the names if you have them defined in /etc/ethertypes

here's the two relevant lines in my debian install:

IPv4            0800    ip ip4          # Internet IP (IPv4)
IPv6            86DD    ip6             # IP version 6

So that everything not coming in from the gre port gets routed, where the routing table then has it sent out the bridge.... I'm not sure if this causes a loop. I think the routed packet then comes in via the bridge "logical interface" so won't match -i but test it with care.