New OpenWrt device does not work behind RFC1483 VDSL bridge?

SvenH · August 1, 2020, 12:12pm

Hi, I am transitioning from a working (release 19.07) OpenWRT device (Zyxel 2812) to a new device, NanoPI R2S. I am using the latest available snapshot for that device.
Situation: OpenWRT device is a router, behind a Draytek VDSL modem in bridge mode (RFC1483).
Problem is that I cannot get WAN connectivity to work. The configuration is pretty much the same; dhcp on the WAN interface with an additional interface ("modem") to allow access to the modem user interface (on 192.168.2.1).
Please see below some outputs. I have masked my public IP address.
Other facts:

The R2S WAN interface works fine it it's in the LAN behind the existing Zyxel.
I can access the modem UI just fine via web browser from the R2S LAN, ping also works.
This happens with latest OpenWRT snapshot for R2S as well as with the most recent dev build by @jayanta525.
The modem can handle switching a device just fine; I tried my Windows laptop, it gets my public IP, and connects to the Internet just fine.
Before I tested that, I tried to copy the MAC address from the working Zyxel, hence you will see 00:11:22:33:44:55 twice below.

Any suggestions on where to look?

R2S output:

root@castor:~# uname -a
Linux castor 5.4.52 #0 SMP PREEMPT Fri Jul 31 21:50:38 2020 aarch64 GNU/Linux
root@castor:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.2/24 brd 192.168.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet XXX.YY.ZZ.200/25 brd XXX.YY.ZZ.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::211:22ff:fe33:4455/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether 46:07:cf:44:7a:f4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fd12:4c15:1f7b::1/60 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::4407:cfff:fe44:7af4/64 scope link
       valid_lft forever preferred_lft forever
root@castor:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         XXX.YY.ZZ.129   0.0.0.0         UG    0      0        0 eth0
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
XXX.YY.ZZ.128   0.0.0.0         255.255.255.128 U     0      0        0 eth0
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@castor:~# ping -c1 -W3 XXX.YY.ZZ.129
PING XXX.YY.ZZ.129 (XXX.YY.ZZ.129): 56 data bytes

--- XXX.YY.ZZ.129 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@castor:~# ping -c1 -W3 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: seq=0 ttl=255 time=0.532 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.532/0.532/0.532 ms
root@castor:~# ping -c1 -W3 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@castor:~#

Zyxel output:

root@castor:~# uname -a
Linux castor 4.14.171 #0 SMP Thu Feb 27 21:05:12 2020 mips GNU/Linux
root@castor:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::211:22ff:fe33:4455/64 scope link
       valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:0c:43:30:62:00 brd ff:ff:ff:ff:ff:ff
4: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b0:b2:dc:10:3e:fb brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet6 fd88:aa2e:255b::1/60 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::b2b2:dcff:fe10:3efb/64 scope link
       valid_lft forever preferred_lft forever
5: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.2/24 brd 192.168.2.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet XXX.YY.ZZ.200/25 brd XXX.YY.ZZ.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::211:22ff:fe33:4455/64 scope link
       valid_lft forever preferred_lft forever
root@castor:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         XXX.YY.ZZ.129   0.0.0.0         UG    0      0        0 eth0.2
10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0.1
XXX.YY.ZZ.128   0.0.0.0         255.255.255.128 U     0      0        0 eth0.2
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0.2
root@castor:~# ping -c1 -W3  XXX.YY.ZZ.129
PING XXX.YY.ZZ.129 (XXX.YY.ZZ.129): 56 data bytes
64 bytes from XXX.YY.ZZ.129: seq=0 ttl=255 time=9.879 ms

--- XXX.YY.ZZ.129 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 9.879/9.879/9.879 ms
root@castor:~# ping -c1 -W3 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=61 time=11.486 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 11.486/11.486/11.486 ms
root@castor:~# ping -c1 -W3 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: seq=0 ttl=255 time=1.154 ms

--- 192.168.2.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.154/1.154/1.154 ms
root@castor:~#

jayanta525 · August 1, 2020, 12:38pm

Do you have the same issue without the additional interface (modem 192.168.2.0/24)?

SvenH · August 1, 2020, 3:02pm

Yes, no difference, I even tried a clean install of a recent image of yours, with no config changes, just dhcp on WAN, and it does not work. IP address is assigned (expected public IP) but traffic does not flow.

jayanta525 · August 1, 2020, 3:33pm

It might be an issue with MAC address, try swapping the interfaces.

Post your /etc/config/network for both the devices.

SvenH · August 1, 2020, 7:07pm

Thanks for thinking with me here. I have tested a bit more.
Swapping LAN/WAN by swapping eth0/eth1 in /etc/config/network (and swapping the cables of course) results in the same behavior.

I ran a tcpdump on the not-working WAN interface and curiously:

There was incoming traffic (random pings and myself over cellular going to port 80) on the interface, which was not responded to by OpenWRT
About every second, OpenWRT was sending out an ARP request for the remote gateway, and got a response.
Sending a ping to the remote gateway does not result in an ICMP packet being picked up by tcpdump (even if I force ping to use the correct interface).

Compared to the working device, there is exactly the same ARP request-response pair happening, except it happens every minute instead of every second.

Here is /etc/config/network from the R2S (before the swap). I think this is the default from a clean image.

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd5d:db9f:f335::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth1'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device 'lan_eth1_dev'
        option name 'eth1'
        option macaddr '8a:0e:aa:45:b1:07'

config interface 'wan'
        option ifname 'eth0'
        option proto 'dhcp'

config device 'wan_eth0_dev'
        option name 'eth0'
        option macaddr '8a:0e:aa:45:b1:06'

config interface 'wan6'
        option ifname 'eth0'
        option proto 'dhcpv6'

And this from the working 2812 (please ignore the obsolete VDSL stuff and the secondary "modem" interface).

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd88:aa2e:255b::/48'

config atm-bridge 'atm'
        option encaps 'llc'
        option payload 'bridged'
        option nameprefix 'dsl'
        option vci '34'
        option vpi '0'

config dsl 'dsl'
        option annex 'a'
        option ds_snr_offset '-5'
        option line_mode 'vdsl'
        option xfer_mode 'ptm'
        option firmware '/lib/firmware/vdsl-575617.bin'

config interface 'lan'
        option ifname 'eth0.1'
        option proto 'static'
        option ipaddr '10.0.0.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device 'lan_eth0_1_dev'
        option name 'eth0.1'
        option macaddr 'b0:b2:dc:10:3e:fb'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'
        list dns '1.0.0.1'
        list dns '1.1.1.1'
        option peerdns '0'

config interface 'modem'
        option ifname 'eth0.2'
        option proto 'static'
        option ipaddr '192.168.2.2'
        option netmask '255.255.255.0'

config device 'wan_dsl0_dev'
        option name 'dsl0'
        option macaddr 'b0:b2:dc:10:3e:f8'

config interface 'wan6'
        option ifname '@wan'
        option proto 'dhcpv6'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '0 1 2 4 6t'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '5 6t'

Any ideas? I am not sure if the arp "chattiness" is an issue or not?

jayanta525 · August 1, 2020, 7:47pm

@SvenH Setting macaddr in the LAN interface is useless. Try setting macaddr in the WAN interface as

config interface 'wan'
        option ifname 'eth0'
        option proto 'dhcp'
        option macaddr '8a:0e:aa:45:b1:06'

config interface 'wan6'
        option ifname 'eth0'
        option proto 'dhcpv6'
        option macaddr '8a:0e:aa:45:b1:06'

slh · August 1, 2020, 7:56pm

I'd (re-)check the modem configuration, please refer to Configuration Issues OpenWRT 19.07.3 and DrayTek Vigor 130

SvenH · August 2, 2020, 5:15am

Hi @slh, thanks, I actually have the Vigor 130 in use. All configuration is set correctly. Another OpenWRT client works fine, as does my Windows laptop. And there is incoming traffic, so I am not suspecting the Vigor at this point.
I will try Jayanta's suggestion with the MAC address config shortly. Also I will experiment with switching the firewall off. Will report here.

SvenH · August 2, 2020, 8:06am

Moving the MAC address statement to the WAN/WAN6 config does not do anything
Switching firewall to LAN does not do anything
Setting WAN to static internal IP and adding a new interface "VDSL" with DHCP: Draytek reachable, VDSL interface gets the public IP, but again no connectivity
The ARP entry for the upstream gateway (.129) does not get populated (status INCOMPLETE). This probably correlates with the ARP request happening every second. Inserting the entry by hand does not work.
Adding a physical switch between R2S and Draytek does not do anything (to rule out the autoswitching/crossover).

Also of note: I tried the August 2 snapshot this morning, clean image, and could not get any network to work (LAN or WAN). I had to hook up the serial console; 'ip addr' would show link state UP but no traffic was going out on either interface.

The fact that the interface works fine communicating with the Draytek is really weird!

I have no idea where to search next.

EnfermeraSexy · August 2, 2020, 5:59pm

Don't set any mac address, delete that line and try.

SvenH · August 3, 2020, 7:09am

Removing mac address statement from config made no difference. In other news, I tried the latest Armbian Focal with Linux 5.4.46. Same issue! Public IP address is assigned but I can't ping anywhere; instead of a timeout, I got a "destination unreachable", and the ARP table shows 'incomplete' for the remote gateway.
Either the 5.4 kernel does not work well with Draytek RFC1483 implementation or it's the R2S hardware/drivers that's causing it.
Later today I will try the (working!) Zyxel 2812 with latest snapshot as that appears to run kernel 5.4 now.