GRETAP over VLAN + TCP problems

[PPousada]

Hello everyone! I'm having some problems with a bit of a weird setup I'm working on, and maybe one of you folk can point me on the right direction. Thanks in advance!

My setup is the next:
I'm connecting several 18.06 OpenWRT devices, with each port pair separated on a different VLAN. Each eth0.X interface is assigned a IP address on the corresponding 192.168.X.0/24 range, so each pair of ports can see each other. I have full connection between each pair of devices, it seems like no problem there.
Then, for each pair of VLAN connected ports, I create a GRETAP tunnel, and I merge the endpoint of those tunnels to the same bridge, configured on a independent subnet to the VLAN ones, so I don't have routing problems, and with STP enabled. The GRE tunnel is established without problems, and sets a 1280 MTU, with the bridge itself going down to 1200. STP negotiates properly, and I have ping between all the different devices.

However:
When I try to open any TCP connection, nothing happens. Going down the old tcpdump tree, the TCP initial package seems to go through the bridge, into the GRE tunnel, into the eth0.X interface, and out the eth0 interface. The MSS shown in the package is 1160, correct VLAN ID IP-MAC pairs both inside and outside the tunnel and all. But on the other device, no trace of the package. I get the ARP and STP traffic, but no TCP, even when snooping on eth0 itself. I tested fragmentation sending huge ICMP packages, and it seems to be working properly, so the problem seems to be on TCP itself.

In the final solution there are other devices between them but, for testing purposes, right now they are directly connected.

Any ideas on how to debug this? Thanks for your help!

Configuration files for reference:

Network:

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config interface 'mgmt'
    option ifname   'eth1'
    option proto    'static'
    option ipaddr   '192.168.80.225'
    option netmask  '255.255.255.0'
    option gateway  '192.168.80.1'
    option dns      '192.168.30.50'

config interface 'master'
    option type   'bridge'
    option stp    '1'
    option ifname 'gre4t-tun239 gre4t-tun138 '
    option proto  'static'
    option ipaddr '192.168.55.121'
    option netmask '255.255.255.0'

config interface 'tun239'
    option proto 'gretap'
    option ipaddr '192.168.239.243'
    option peeraddr '192.168.239.239'

config interface 'vlan239'
    option ifname 'eth0.239'
    option proto 'static'
    option ipaddr '192.168.239.243'
    option netmask '255.255.255.0'

config interface 'tun138'
    option proto 'gretap'
    option ipaddr '192.168.138.242'
    option peeraddr '192.168.138.138'

config interface 'vlan138'
    option ifname 'eth0.138'
    option proto 'static'
    option ipaddr '192.168.138.242'
    option netmask '255.255.255.0'


config interface 'basic'
    option ifname   'eth0.14 wlan1 '
    option type     'bridge'
    option proto    'static'
    option ipaddr   '192.168.1.1'
    option netmask  '255.255.255.0'    
    
config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '0t'


config switch_vlan
        option device 'switch0'
        option vlan '22'
        option vid '239'
        option ports '0t 2'

config switch_vlan
        option device 'switch0'
        option vlan '21'
        option vid '138'
        option ports '0t 1'


config switch_vlan
        option device 'switch0'
        option vlan '14'
        option vid '14'
        option ports '0t 4'

Firewall:

config defaults
	option syn_flood	1
	option input		ACCEPT
	option output		ACCEPT
	option forward		REJECT
# Uncomment this line to disable ipv6 rules
#	option disable_ipv6	1

config zone
	option name		basic
	list   network		'basic'
	option input		ACCEPT
	option output		ACCEPT
	option forward		ACCEPT

config zone
	option name		master
	list   network		'master'
	option input		ACCEPT
	option output		ACCEPT
	option forward		ACCEPT
	option masq		    1
	option mtu_fix		1

config forwarding
	option src		basic
	option dest		master

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
	option name		Allow-DHCP-Renew
	option src		master
	option proto		udp
	option dest_port	68
	option target		ACCEPT
	option family		ipv4

# Allow IPv4 ping
config rule
	option name		Allow-Ping
	option src		master
	option proto		icmp
	option icmp_type	echo-request
	option family		ipv4
	option target		ACCEPT

config rule
	option name		Allow-IGMP
	option src		master
	option proto		igmp
	option family		ipv4
	option target		ACCEPT

# include a file with users custom iptables rules
config include
	option path /etc/firewall.user

# allow IPsec/ESP and ISAKMP passthrough
config rule
	option src		master
	option dest		basic
	option proto		esp
	option target		ACCEPT

config rule
	option src		master
	option dest		basic
	option dest_port	500
	option proto		udp
	option target		ACCEPT

[jeff]

Classic networking problem as the Layer 2 tunnel doesn't know where to send the Layer 3 "fragmentation required" messages. "Local" interfaces work, but "off-router" interfaces fail as soon as the packet size goes up, right? ping works, ssh fails. Not specific to OpenWRT.

There aren't any good solutions, as they typically require cooperation with devices that don't know that there is a Layer 2 path with a reduced MTU. "MSS clamping" can help, but isn't a guarantee at all.

With an 802.11-based link, the MTU can be increased to allow for the GRE headers. If you're trying to tunnel Ethernet over Ethernet, then you're sort of stuck.

[SOLVED] Gretap Tunnel and MTU-Size Problem

The problem is that a Layer 2 tunnel doesn't "know" where to send "fragmentation needed" packets when the link MTU is exceeded. This is because it deals with Layer 2 and the IP source address is Layer 3. On-router connectivity "works" as the sender is aware of the interface MTU. Off-router connections fail because PMTU is not possible. This is a protocol limitation, not an OpenWRT limitation. I wasn't able to get /etc/config/network to immediately do what I wanted. It may be possible to do so, …

[PPousada]

I saw that thread and tried playing with the MTU sizes of the network to no avail. The thing is, if the problem were MTU sizes, a big ping [ping -s 3000 IP ] would fail to be fragmented too right? But those big pings work and are fragmented, so fragmentation must be working properly, unless there is something I don't understand here.