IP breaks using @wan instead of eth0

Connecting to my ISP using PPPoE. OpenWRT is v4.9.152. Machine is RPi 3B+. Connection to ISP is fine, and setting wan6 ifname to eth0 gives me a global IPv6 address as well as IPv4. But when I change the ifname to @wan ('at' wan), I lose ppoe-wan and have no connection to my ISP, either v4 or v6. logread shows that I get as far as "pppd 2.4.7 started by root, uid 0" but no further - I don't get to "PPP session is 33737, Connected to 00:2a:10:c3:6f:53 via interface eth0", etc.

I want to use @wan because otherwise the DHCP6 Solicit messages go out on the eth0 interface rather than the ppoe-wan interface, and get ignored. My config/network is

 config interface 'loopback'
         option ifname 'lo'
         option proto 'static'
         option ipaddr '127.0.0.1'
         option netmask '255.0.0.0'
 
 config globals 'globals'
         option ula_prefix 'fd87:5070:a96b::/48'
 
 config interface 'wan'
         option ifname 'eth0'
         option proto 'pppoe'
         option username 'uid@abc'
         option password 'pwd'
         option ipv6 'auto'
 
 config interface 'lan'
         option proto 'static'
         option ifname 'eth1'
         option ipaddr '192.168.3.254'
         option netmask '255.255.255.0'
         option gateway '192.168.1.254'
         option type 'bridge'
         option ip6assign 64
 
 config interface wan6
         option ifname 'eth0'   # @wan doesn't work
         option proto dhcpv6
         option reqddress 'try'
         option reqprefix '56'

Using eth0, I get the following from ifconfig

 br-lan    Link encap:Ethernet  HWaddr 38:4B:72:F0:26:C3  
           inet addr:192.168.3.254  Bcast:192.168.3.255  Mask:255.255.255.0
           inet6 addr: fe80::3a4b:72ff:fef0:26c3/64 Scope:Link
           inet6 addr: fe80::3a4b:72ff:fef0:26c3/64 Scope:Link
           inet6 addr: fd87:5070:a96b::1/64 Scope:Global
           inet6 addr: 2400:3740:200:d900::1/64 Scope:Global
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:32752 errors:0 dropped:0 overruns:0 frame:0
           TX packets:45510 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:5896225 (5.6 MiB)  TX bytes:44437614 (42.3 MiB)
 
 eth0      Link encap:Ethernet  HWaddr B8:27:EB:10:52:A1  
           inet6 addr: fe80::ba27:ebff:fe10:52a1/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:8228717 errors:0 dropped:0 overruns:0 frame:0
           TX packets:5949636 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:7906096201 (7.3 GiB)  TX bytes:1477797091 (1.3 GiB)
 
 eth1      Link encap:Ethernet  HWaddr 38:4B:72:F0:26:C3  
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:2312634 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2527335 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:408618847 (389.6 MiB)  TX bytes:2398699464 (2.2 GiB)
 
 lo        Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:1463 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1463 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1 
           RX bytes:141028 (137.7 KiB)  TX bytes:141028 (137.7 KiB)
 
 pppoe-wan Link encap:Point-to-Point Protocol  
           inet addr:103.79.105.27  P-t-P:103.247.3.47  Mask:255.255.255.255
           inet6 addr: 2400:3740:100:d9:283b:38c4:3ab1:1444/64 Scope:Global
           inet6 addr: fe80::283b:38c4:3ab1:1444/10 Scope:Link
           UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
           RX packets:44318 errors:0 dropped:0 overruns:0 frame:0
           TX packets:31148 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:3 
           RX bytes:43547583 (41.5 MiB)  TX bytes:5737576 (5.4 MiB)
 
 wlan0     Link encap:Ethernet  HWaddr B8:27:EB:45:07:F4  
           inet6 addr: fe80::ba27:ebff:fe45:7f4/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:35018 errors:0 dropped:0 overruns:0 frame:0
           TX packets:59818 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:2815073 (2.6 MiB)  TX bytes:29010577 (27.6 MiB)

Using @wan I get

 br-lan    Link encap:Ethernet  HWaddr 38:4B:72:F0:26:C3  
           inet addr:192.168.3.254  Bcast:192.168.3.255  Mask:255.255.255.0
           inet6 addr: fe80::3a4b:72ff:fef0:26c3/64 Scope:Link
           inet6 addr: fe80::3a4b:72ff:fef0:26c3/64 Scope:Link
           inet6 addr: fd87:5070:a96b::1/64 Scope:Global
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:107 errors:0 dropped:0 overruns:0 frame:0
           TX packets:70 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:22733 (22.2 KiB)  TX bytes:8592 (8.3 KiB)
 
 eth0      Link encap:Ethernet  HWaddr B8:27:EB:10:52:A1  
           inet6 addr: fe80::ba27:ebff:fe10:52a1/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:8228904 errors:0 dropped:0 overruns:0 frame:0
           TX packets:5949814 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:7906120496 (7.3 GiB)  TX bytes:1477879489 (1.3 GiB)
 
 eth1      Link encap:Ethernet  HWaddr 38:4B:72:F0:26:C3  
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:2312899 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2527522 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:408723192 (389.7 MiB)  TX bytes:2398723830 (2.2 GiB)
 
 lo        Link encap:Local Loopback  
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:65536  Metric:1
           RX packets:1465 errors:0 dropped:0 overruns:0 frame:0
           TX packets:1465 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1 
           RX bytes:141388 (138.0 KiB)  TX bytes:141388 (138.0 KiB)
 
 wlan0     Link encap:Ethernet  HWaddr B8:27:EB:45:07:F4  
           inet6 addr: fe80::ba27:ebff:fe45:7f4/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:35176 errors:0 dropped:0 overruns:0 frame:0
           TX packets:59950 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000 
           RX bytes:2827229 (2.6 MiB)  TX bytes:29039864 (27.6 MiB)

@wan potentially conflicts with Linux naming conventions. Unless you’re intending the @ to have its Linux meaning, a different name may resolve your breakage.

Thanks, I'm basing this on OpenWRT doc: IPv6 which has:

root@lede:/# cat /etc/config/network
config interface wan
        option ipv6 1 # only required for PPP-based protocols
        ...

config interface wan6
        option ifname   eth1 # use same ifname as in wan-section or "@wan"
        option proto    dhcpv6

I just tried changing wan to wan1 throughout, no difference

First of all remove this from LAN

Second I don't understand what is your problem exactly here. With ifname eth0 it works fine. You have IPv6 on pppoe-wan and br-lan.
So what exactly is not working?

2 Likes

There's also that bug:

Thanks @trendy! Removing the gateway option seems to have fixed it.

With the iface set to eth0, the problem was that I was sending out ethernet packets, not PPoE packets. To give more context: I'm trying to do prefix delegation. My ISP tells me I have a /56, so I include the prefix request in the wan6 interface by option reqprefix 56. I run tcpdump into wireshark using
ssh root@192.168.3.254 tcpdump -i eth0 -U -s0 -w - ip6 |sudo wireshark -k -i -
and can see the DHCP6 Solicitation ethernet packets do include the prefix delegation request. That part is working okay.

But I never saw router adverts from my ISP on the eth0 interface. That's because they are coming in on the ppoe-wan interface. They include no mention of a prefix, only the /64 address assigned to my router. I was sending out ethernet packets when I should be sending out PPPoE packets and my ISP was ignoring them. I saw this in Issue 13599 Problems with native IPv6 VDSL connection, where the solution is to set the wlan6 iface to @wan.

Now that @wan is working, I see DHCP6 Solicit XID packets from me on ppoe-wan with a prefix request and DHCP6 Advertise XID packets coming back from the ISP with a prefix assignment. Success - now on to the next part of distributing them :slight_smile:

1 Like

Sorry, it seems inconsistent behaviour fooled me into thinking the solution from @trendy worked. I wish it did!

It seems that openwrt is only occasionally sending out PADI PPPoE Discovery packets. Until it does, there is no PPPoE connection, and no pppoe-wan device. Once a PADI packet is sent out to the ISP's modem it initiiates a PADO/PADR/PADS session and everything is then okay: I get a pppoe-wan device with IPv4 and IPv6 addresses.

My environment now adds a hub between the router and the modem, and I am monitoring all ethernet packets using wireshark on a PC connected to the hub. Also, on the router I have added option pppd_options debug to the wan interface of /etc/config/network. If I don't see a PADI packet on wireshark, I restart the network using /etc/init.d/network restart. According to logread, openwrt sends many PADI packets

Send PPPOE Discovery V1T1 PADI session 0x0 length4

I go through this restart upto 10 times! Wireshark does not see most of those as ethernet packets. When it finally does see one, it also sees the PPoE session initiation as described above.

So where are the PADI packets that pppd is producing going to? pppd is version 2.4.7 and rp-pppoe.so is version 3.8p. OpenWRT is 18.06.2

tcpdump -n -i any ...

Have you tried to change that to 1 ? (or manual in Luci)
You'll then have to create the wan6 interface yourself, rather than having it automatically created by openwrt. Maybe it will you more leverage on setting the options you want.

1 Like

Thanks @trendy, yes I changed it back to one. There was a posting
PPPoE addressing and prefix... which suggested auto which is why I tried it. my current network config is

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd87:5070:a96b::/48'

config interface 'wan'
	option ifname 'eth0'
	option proto 'pppoe'
	option username '...'
	option password '...'
	option ipv6 '1'
	option pppd_options debug

config interface 'lan'
	option proto 'static'
	option ifname 'eth1'
	option ipaddr '192.168.3.254'
	option netmask '255.255.255.0'
	option type 'bridge'
	option ip6assign 64

config interface wan6
	option ifname '@wan'
	option proto dhcpv6
	option reqprefix '56'

I'm trying tcpdump -n -i any, no luck yet with spotting where the (assumed) missing packets are going to.

tcpdump -i eth0 ether[0x0c:2] == 0x8863 or ether[0x0c:2] == 0x8864

This must show you PPPoE packets. 8863 is the ether type for Active Discovery and 8864 is the ether type for PPPoE sessions.

Making progress - at identifying the problem, not found the solution yet!

Running tcpdump -i eth0 ether[0x0c:2] == 0x8863 on the router shows that a PADI packet is sent, but is immediately followed by

tcpdump: pcap_loop: The interface went down

No packet appears on the hub between the router and ISP modem. So I keep restarting the interface /etc/init.d/network restart. Eventually the interface stays up, and then a PADI packet comes out to the modem, the ISP responds with PADO, followed by PADR/PADS and then the pppoe-wan interface is created and everything is fine.

So the question is now, not "where are the PADI packets going to," but "why is the interface going down?"

Notes: each time the interface goes down, tcpdump terminates. So I run it in a shell loop

while true
do
    tcpdump -i eth0 ether[0x0c:2] == 0x8863
done

The output is

...
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:15:43.524301 PPPoE PADI [Service-Name]
tcpdump: pcap_loop: The interface went down
1 packet captured
1 packet received by filter
0 packets dropped by kernel
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:15:49.814308 PPPoE PADI [Service-Name]
tcpdump: pcap_loop: The interface went down
1 packet captured
1 packet received by filter
0 packets dropped by kernel
tcpdump: eth0: That device is not up
tcpdump: eth0: That device is not up
tcpdump: eth0: That device is not up
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:15:52.514365 PPPoE PADI [Service-Name]
09:15:57.519739 PPPoE PADI [Service-Name]
09:15:57.523511 PPPoE PADO [Service-Name] [AC-Name "*********"] [AC-Cookie *******************]
09:15:57.523908 PPPoE PADR [Service-Name] [AC-Cookie **********************]
09:15:57.542984 PPPoE PADS [ses 0xee0c] [Service-Name] [AC-Cookie ***********************]

5 packets captured
5 packets received by filter
0 packets dropped by kernel

Anything in the logs explaining why the interface went down?
Faulty cable?

The cable has been okay so far. I changed it anyway for a new one, and no change to the behaviour. lastlog gives info that can be divided into 2 parts: when the packet fails to be delivered and the interface goes down, and when it does get delivered and the interface stays up. This part is common to both:

Wed Feb 27 09:14:21 2019 daemon.notice netifd: Interface 'wan' is enabled
Wed Feb 27 09:14:21 2019 daemon.notice netifd: Interface 'wan' is setting up now
Wed Feb 27 09:14:21 2019 kern.info kernel: [   76.474986] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
Wed Feb 27 09:14:21 2019 daemon.err insmod: module is already loaded - slhc
Wed Feb 27 09:14:21 2019 daemon.err insmod: module is already loaded - ppp_generic
Wed Feb 27 09:14:21 2019 daemon.err insmod: module is already loaded - pppox
Wed Feb 27 09:14:21 2019 daemon.err insmod: module is already loaded - pppoe
Wed Feb 27 09:14:21 2019 kern.info kernel: [   76.551284] device eth0 entered promiscuous mode
Wed Feb 27 09:14:21 2019 daemon.info pppd[1972]: Plugin rp-pppoe.so loaded.
Wed Feb 27 09:14:21 2019 daemon.info pppd[1972]: RP-PPPoE plugin version 3.8p compiled against pppd 2.4.7
Wed Feb 27 09:14:21 2019 daemon.notice pppd[1972]: pppd 2.4.7 started by root, uid 0
Wed Feb 27 09:14:21 2019 daemon.debug pppd[1972]: Send PPPOE Discovery V1T1 PADI session 0x0 length 4
Wed Feb 27 09:14:21 2019 daemon.debug pppd[1972]:  dst ff:ff:ff:ff:ff:ff  src b8:27:eb:10:52:a1
Wed Feb 27 09:14:21 2019 daemon.debug pppd[1972]:  [service-name]
Wed Feb 27 09:14:22 2019 daemon.notice netifd: Network device 'eth0' link is down
Wed Feb 27 09:14:22 2019 daemon.notice netifd: Interface 'wan' has link connectivity loss
Wed Feb 27 09:14:22 2019 daemon.notice netifd: Network device 'eth0' link is up
Wed Feb 27 09:14:22 2019 daemon.notice netifd: Interface 'wan' has link connectivity
Wed Feb 27 09:14:22 2019 daemon.notice netifd: Interface 'wan' is setting up now
Wed Feb 27 09:14:22 2019 kern.info kernel: [   77.984118] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xD1E1

Then in the first case, after about 5 seconds it goes down:

Wed Feb 27 09:14:27 2019 daemon.notice netifd: Interface 'wan' is now down
Wed Feb 27 09:14:27 2019 kern.info kernel: [   82.596506] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
Wed Feb 27 09:14:27 2019 daemon.notice netifd: Interface 'wan' is disabled
Wed Feb 27 09:14:27 2019 kern.info kernel: [   82.615517] device eth0 left promiscuous mode

This behaviour repeats until it stays up and then attention (correctly) switches to setting up lan:

...
Wed Feb 27 09:14:35 2019 daemon.notice netifd: Interface 'wan' has link connectivity
Wed Feb 27 09:14:35 2019 daemon.notice netifd: Interface 'wan' is setting up now
Wed Feb 27 09:14:35 2019 kern.info kernel: [   90.464081] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xD1E1
CHANGES FROM HERE
Wed Feb 27 09:14:37 2019 daemon.notice netifd: Interface 'lan' is now down
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.743819] br-lan: port 2(wlan0) entered disabled state
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.752349] br-lan: port 1(eth1) entered disabled state
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.769722] device eth1 left promiscuous mode
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.777304] br-lan: port 1(eth1) entered disabled state
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.795002] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.804015] device wlan0 left promiscuous mode
Wed Feb 27 09:14:37 2019 kern.info kernel: [   92.811451] br-lan: port 2(wlan0) entered disabled state
Wed Feb 27 09:14:37 2019 daemon.info dnsmasq[1026]: read /etc/hosts - 4 addresses
Wed Feb 27 09:14:37 2019 daemon.info dnsmasq[1026]: read /tmp/hosts/odhcpd - 0 addresses
Wed Feb 27 09:14:37 2019 daemon.info dnsmasq[1026]: read /tmp/hosts/dhcp.cfg01411c - 2 addresses
ALL OKAY FROM HERE

I don't see anything wrong. Just to be safe before taking any further action, backup the system, test the backup file, try to install the firmware from scratch without saving the configuration, and restore one step at a time the configuration.

If it still gets unsolved, unless someone else has a better idea, I'd suggest to take it to developers for a potential bug.

Thanks @trendy you've been a tremendous help on this. I wouldn't have been able to get so far without your pointers.

It is starting to look like developer issue rather than a configuration issue. One other thing I have noticed so far: when the interface goes down, tcpdump almost immediately can pick it up again. But in the last instance, when the interface restarts and stays up, tcpdump has about 6 attempts to see the device which all fail, but then succeeds. Then the PADI message goes out and the interface stays up. Looks like the kernel taking time to get the interface set up properly - but that is speculation at the moment!

Update. I have two ethernet ports on the RP: the original one eth0, and another one on a USB-to-ethernet adapter as eth1. I have been using eth0 as my WAN connection, eth1 as my LAN. With that setup, I have been getting problems getting PPoE PADI packets out to the WAN. So, I just tried swapping them around: the USB ethernet adapter is now my WAN port, the builtin ethernet is now my LAN port. I've rebooted the RPi at least 6 times now, and each time pppoe-wan has come up immediately, and I can see the IA prefixes coming up okay.

Assuming this behaviour stays consistent, it more and more looks like a issue with the device drivers, perhaps. Yes, I know that on the RPi, ethernet hangs off the USB bus. So two USB-ethernet adapters, one apparently ok, one not. Hmmmm!

2 Likes

Try installing ethtool and see how the ethernet NIC reports to be configured. I had problems in the past with EEE (energy efficient ethernet) settings, some years ago, so probably not your issue but worth looking at as it should be simple

Thanks @moeller0. Installed ethtools and generally got: no link detected, speed 10Mbps, half-duplex, repeated until a connection stayed up and then switched to link detected, 100Mbps, full duplex.

So then I changed to another RPi. Same SSD card, same cables, same configuration. Basically the same behaviour, but with wrinkles.
a) Sometimes the link is alive, and then the interface goes down, with zero packets sent.
b) Sometimes the link is alive and PADI/PADO/PADR get sent and received and then the link goes down
c) Something in between such as PADI gets sent, link goes down.

Looking for reports on RPi ethernet drivers (smsc95xx), I'm not seeing many complimentary posts, but none reporting this kind of behaviour that I can see. I would guess at driver, h/w issues, except that when the PPoE link gets set up, it stays up.