19.07.1 on AR150 not forwarding traffic

mfullerca · February 21, 2020, 4:31pm

Hey, I just flashed OpenWrt 19.07.1 onto my AR150 and it's not forwarding traffic. To be clear, I'm talking the openwrt-19.07.1-ath79-generic-glinet_gl-ar150-squashfs-sysupgrade.bin and zero configuration changes. If this was a regular Linux distro, I'd suspect that net.ipv4.ip_forward wasn't enabled, but I just verified that it is. I've used every major version of OpenWrt since 15.05 on this device w/o issue, so I'm at a loss.

As an example, I can ping the AR150 from the LAN and I can reach the Internet (e.g., 8.8.4.4) from the AR150, but I cannot reach the Internet from the LAN. If I install tcpdump, I can see the traffic reaching the AR150 on br-lan but it never leaves the router. Weird thing is that it is ARPing for 8.8.4.4 on br-lan:

root@OpenWrt:~# tcpdump -i br-lan not port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
16:23:26.165030 ARP, Request who-has dns.google tell 192.168.1.206, length 46
16:23:27.170923 ARP, Request who-has dns.google tell 192.168.1.206, length 46
16:23:28.194936 ARP, Request who-has dns.google tell 192.168.1.206, length 46

as if there's no default route, but there is:

root@OpenWrt:~# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.10.1    0.0.0.0         UG        0 0          0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 br-lan
192.168.10.0    0.0.0.0         255.255.255.0   U         0 0          0 eth1

So ‾\(ツ)/‾

root@OpenWrt:~# dmesg
[    0.000000] Linux version 4.14.167 (builder@buildhost) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r10911-c155900f66)) #0 Wed Jan 29 16:05:35 2020
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019374 (MIPS 24Kc)
[    0.000000] MIPS: machine is GL.iNet GL-AR150
[    0.000000] SoC: Atheros AR9330 rev 1
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 04000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] On node 0 totalpages: 16384
[    0.000000] free_area_init_node: node 0, pgdat 804f9f60, node_mem_map 81006380
[    0.000000]   Normal zone: 128 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 16384 pages, LIFO batch:3
[    0.000000] random: get_random_bytes called from 0x804fd740 with crng_init=0
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16256
[    0.000000] Kernel command line: console=ttyATH0,115200 rootfstype=squashfs,jffs2
[    0.000000] PID hash table entries: 256 (order: -2, 1024 bytes)
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 58336K/65536K available (4027K kernel code, 156K rwdata, 532K rodata, 1228K init, 212K bss, 7200K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 51
[    0.000000] CPU clock: 400.000 MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 9556302233 ns
[    0.000013] sched_clock: 32 bits at 200MHz, resolution 5ns, wraps every 10737418237ns
[    0.007578] Calibrating delay loop... 265.42 BogoMIPS (lpj=1327104)
[    0.092181] pid_max: default: 32768 minimum: 301
[    0.096902] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.102954] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.115400] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.123495] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.129383] pinctrl core: initialized pinctrl subsystem
[    0.137689] NET: Registered protocol family 16
[    0.171136] clocksource: Switched to clocksource MIPS
[    0.176161] NET: Registered protocol family 2
[    0.180230] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.185690] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.191746] TCP: Hash tables configured (established 1024 bind 1024)
[    0.197976] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.203423] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.209656] NET: Registered protocol family 1
[    0.213579] PCI: CLS 0 bytes, default 32
[    0.219210] Crashlog allocated RAM at address 0x3f00000
[    0.224828] workingset: timestamp_bits=30 max_order=14 bucket_order=0
[    0.237447] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.241710] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.265091] io scheduler noop registered
[    0.267461] io scheduler deadline registered (default)
[    0.272971] ar7200-usb-phy usb-phy: phy reset is missing
[    0.278041] pinctrl-single 18040028.pinmux: 64 pins at pa b8040028 size 8
[    0.285320] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.293717] 18020000.uart: ttyATH0 at MMIO 0x18020000 (irq = 9, base_baud = 1562500) is a AR933X UART
[    0.301684] console [ttyATH0] enabled
[    0.308427] bootconsole [early0] disabled
[    0.339072] m25p80 spi0.0: w25q128 (16384 Kbytes)
[    0.342446] 4 fixed-partitions partitions found on MTD device spi0.0
[    0.348661] Creating 4 MTD partitions on "spi0.0":
[    0.353480] 0x000000000000-0x000000040000 : "u-boot"
[    0.359627] 0x000000040000-0x000000050000 : "u-boot-env"
[    0.364888] 0x000000050000-0x000000ff0000 : "firmware"
[    0.372875] 2 uimage-fw partitions found on MTD device firmware
[    0.377347] Creating 2 MTD partitions on "firmware":
[    0.382372] 0x000000000000-0x000000190000 : "kernel"
[    0.388406] 0x000000190000-0x000000fa0000 : "rootfs"
[    0.393333] mtd: device 4 (rootfs) set to be root filesystem
[    0.397868] 1 squashfs-split partitions found on MTD device rootfs
[    0.404105] 0x000000400000-0x000000fa0000 : "rootfs_data"
[    0.410465] 0x000000ff0000-0x000001000000 : "art"
[    0.416830] libphy: Fixed MDIO Bus: probed
[    0.762279] ag71xx 19000000.eth: Could not connect to PHY device. Deferring probe.
[    1.441596] libphy: ag71xx_mdio: probed
[    1.444962] libphy: ar8xxx-mdio: probed
[    1.450820] switch0: Atheros AR724X/AR933X built-in rev. 2 switch registered on mdio-bus.0
[    1.497158] ag71xx 1a000000.eth: connected to PHY at fixed-0:00 [uid=00000000, driver=Generic PHY]
[    1.505643] eth0: Atheros AG71xx at 0xba000000, irq 5, mode: gmii
[    1.514081] NET: Registered protocol family 10
[    1.524643] Segment Routing with IPv6
[    1.526983] NET: Registered protocol family 17
[    1.531451] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.544249] 8021q: 802.1Q VLAN Support v1.8
[    1.882890] ag71xx 19000000.eth: connected to PHY at mdio-bus.0:1f:04 [uid=004dd041, driver=Generic PHY]
[    1.892290] eth1: Atheros AG71xx at 0xb9000000, irq 4, mode: mii
[    1.898121] hctosys: unable to open rtc device (rtc0)
[    1.908689] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.923809] Freeing unused kernel memory: 1228K
[    1.926878] This architecture does not have kernel memory protection.
[    2.551159] random: fast init done
[    2.885308] init: Console is alive
[    2.887585] init: - watchdog -
[    4.055839] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    4.182164] usbcore: registered new interface driver usbfs
[    4.186332] usbcore: registered new interface driver hub
[    4.191695] usbcore: registered new device driver usb
[    4.207145] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    4.217258] chipidea-usb2 1b000000.usb: 1b000000.usb supply vbus not found, using dummy regulator
[    4.227841] ci_hdrc ci_hdrc.0: EHCI Host Controller
[    4.231389] ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1
[    4.261157] ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00
[    4.266282] hub 1-0:1.0: USB hub found
[    4.269433] hub 1-0:1.0: 1 port detected
[    4.275067] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    4.291566] init: - preinit -
[    5.644506] random: jshn: uninitialized urandom read (4 bytes read)
[    5.982054] random: jshn: uninitialized urandom read (4 bytes read)
[    6.082330] random: jshn: uninitialized urandom read (4 bytes read)
[    6.481729] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    7.532312] eth0: link up (1000Mbps/Full duplex)
[    7.535519] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.770956] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[    9.815810] urandom-seed: Seed file not found (/etc/urandom.seed)
[   10.088877] eth0: link down
[   10.108042] procd: - early -
[   10.109618] procd: - watchdog -
[   10.907818] procd: - watchdog -
[   10.909967] procd: - ubus -
[   11.142325] urandom_read: 5 callbacks suppressed
[   11.142339] random: ubusd: uninitialized urandom read (4 bytes read)
[   11.261262] random: ubusd: uninitialized urandom read (4 bytes read)
[   11.268131] procd: - init -
[   12.199437] kmodloader: loading kernel modules from /etc/modules.d/*
[   12.438071] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   12.473322] Loading modules backported from Linux version v4.19.98-0-gd183c8e2647a
[   12.479443] Backport generated by backports.git v4.19.98-1-0-g8204eb99
[   12.504703] ip_tables: (C) 2000-2006 Netfilter Core Team
[   12.529954] nf_conntrack version 0.5.0 (1024 buckets, 4096 max)
[   12.706180] xt_time: kernel timezone is -0000
[   12.865048] PPP generic driver version 2.4.2
[   12.882569] NET: Registered protocol family 24
[   13.000923] ath: EEPROM regdomain: 0x0
[   13.000938] ath: EEPROM indicates default country code should be used
[   13.000943] ath: doing EEPROM country->regdmn map search
[   13.000964] ath: country maps to regdmn code: 0x3a
[   13.000972] ath: Country alpha2 being used: US
[   13.000978] ath: Regpair used: 0x3a
[   13.016101] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   13.018722] ieee80211 phy0: Atheros AR9330 Rev:1 mem=0xb8100000, irq=2
[   13.221740] kmodloader: done loading kernel modules from /etc/modules.d/*
[   13.290831] urngd: v1.0.2 started.
[   13.950282] random: crng init done
[   54.612975] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   54.618454] jffs2_build_filesystem(): unlocking the mtd device... 
[   54.636835] done.
[   54.643484] jffs2_build_filesystem(): erasing all blocks after the end marker... 
[   55.360929] eth0: link up (1000Mbps/Full duplex)
[   55.376114] br-lan: port 1(eth0) entered blocking state
[   55.379896] br-lan: port 1(eth0) entered disabled state
[   55.385656] device eth0 entered promiscuous mode
[   55.435836] br-lan: port 1(eth0) entered blocking state
[   55.439619] br-lan: port 1(eth0) entered forwarding state
[   55.445375] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   55.548601] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   56.411296] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   57.613135] eth1: link up (100Mbps/Full duplex)
[   57.616260] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   94.571444] done.
[   94.571971] jffs2: notice: (1256) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   94.864589] overlayfs: upper fs does not support tmpfile.

ulmwind · February 21, 2020, 6:48pm

Revert settings to default.

trendy · February 21, 2020, 8:26pm

Does that mean that you kept the settings over flashing or flashed without keeping settings (but then restored a backup from an older version) ?
Keeping settings from older major releases has some unpredictable results, hence it is advised to flash without keeping settings and then restoring manually the configs.

mfullerca · February 21, 2020, 8:51pm

Sorry, to be clear that's without keeping the settings (well, initially I'd been more ambitious, but my final test before posting was to reflash entirely with the defaults, wiping my configs). This is why I didn't post my configs - they are all unchanged.

Btw for those unfamiliar w/ the AR150, it's a made for OpenWrt device and very simple/basic hardware: one WAN and one LAN Ethernet and 2.4GHz WiFi.

mfullerca · February 21, 2020, 9:26pm

Also I should have mentioned that the only obvious difference with 19.07.1 vs 18.06.4 is that eth0 and eth1 are swapped (in 18. eth0 is WAN but in 19. eth1 is WAN). However, the physical ports are correct and everything else seems copacetic.

trendy · February 22, 2020, 6:10pm

Could you verify that settings are correct on the lan hosts?
An arp request has local validity only, that means that a host can send an arp request for devices within the same subnet. When a host needs to contact a device outside of its subnet, it needs to know the mac address of the default gateway, hence it will send an arp request for the 192.168.1.1.
However sending arp requests for 8.8.4.4 (if I am not mistaken this is dns.google in your dump) it seems to me that subnet mask is not correct on the host.

bluewavenet · February 22, 2020, 6:38pm

Post the files:
/etc/config/network
and
/etc/config/wireless

mfullerca · February 22, 2020, 10:43pm

sigh Good point @trendy. I was testing using "ping -I" on the LAN host to force it out the correct interface, but that wasn't a valid test. So I think I fubarred something between my first test (a modified config with the AR150 functioning as my actual router) and the later test (the reset config but the AR150 running locally because I can't use it as my actual router w/o a modified config), so whatever problem I initially experienced that led me down this path was with my modified config. I'll do more debugging later and report back, but whatever the problem is, it's probably more mundane.

mfullerca · February 24, 2020, 5:02pm

So I did a little more testing this morning. I hesitate to jump to conclusions, but I've tried this twice now and it appears if I enable WiFi and then reboot, the wired Ethernet LAN will not work after the reboot, and if I disable WiFi, the wired Ethernet LAN comes back (w/o rebooting).

@bluewavenet here you go, but as I said before, the config is largely unmodified (except for a password I added to the WiFi):

root@OpenWrt:/etc/config# cat wireless 

config wifi-device 'radio0'
	option type 'mac80211'
	option channel '11'
	option hwmode '11g'
	option path 'platform/ahb/18100000.wmac'
	option htmode 'HT20'
	option legacy_rates '0'
	option disabled '1'

config wifi-iface 'default_radio0'
	option device 'radio0'
	option network 'lan'
	option mode 'ap'
	option key 'password'
	option ssid 'OpenWrt'
	option encryption 'psk2'
	option disabled '1'

root@OpenWrt:/etc/config# cat network 

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd79:4a86:cd2a::/48'

config interface 'lan'
	option type 'bridge'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wan'
	option ifname 'eth1'
	option proto 'dhcp'

config interface 'wan6'
	option ifname 'eth1'
	option proto 'dhcpv6'

mfullerca · March 3, 2020, 3:34pm

So I haven't heard anything. This problem seems reproducible. Any suggestions for work-arounds to try (e.g., if I had to hazard a guess, I'd say it was related to bridging) or data to gather to file a proper bug report? Thanks.

lleachii · March 3, 2020, 4:03pm

You never did a nslookup on dns.google ...I honestly can't imagine that you didn't add a hostname or configure a DNS server?

mfullerca · March 3, 2020, 10:48pm

@lleachii that is just tcpdump trying to be helpful by doing a reverse lookup on 8.8.8.8, which is what 192.168.1.206 was pinging. 192.168.1.206 was likely ARPing for it on that interface because I'd done a "ping -I" to try and debug, which was a bad test (I should have just used the AR150 "normally", except I didn't want to rely on it as my default router while debugging it). So please ignore the first part of the problem statement and focus on the one from Feb 24, where simply enabling the WiFi causes the problem, as this is reproducible and still consistent with my initial observations.

trendy · March 4, 2020, 12:21am

Is there anything in the logs (logread or dmesg) that gives any hint for this issue?
Can you try to break the bridge and operate the wifi as a separate interface?
Is the power supply adequate?

bluewavenet · March 12, 2020, 7:31am

I just dug out an old ar150 from my junk box and reflashed it with 19.07.1 and could not reproduce this initially.
I changed the power supply to an old Blackberry charger that outputs only 4.2 volts and do indeed get this issue. So @trendy yes - power supply.

mfullerca · March 12, 2020, 10:58pm

Thanks @bluewavenet and @trendy. I discovered 19.07.2 had been released, reflashed to that, "broke the bridge," and the problem still recurred. Then COVID-19 disrupted my schedule so I never got around to posting the logs. Fwiw my power supply is 1.2A vs minimum requirement is <500mA w/o any USB devices attached (my case - per https://forum.gl-inet.com/t/gl-ar150-power-requirements/1171), but sure, it could still be a bad supply. I hypothesize if the power supply is an issue, the WiFi increases power draw enough to cause this problem, but still it's very strange. I'll try switching supplies when I get a chance.

bluewavenet · March 12, 2020, 11:33pm

@mfullerca
It could be the actual usb power supply, the cable, cable connector, socket in the ar150 or even the onboard regulator coincidentally developed a fault.
Maybe time to upgrade to an ar300m or mt300n at half the price.

mfullerca · March 15, 2020, 10:10pm

@bluewavenet and @trendy so I found a known good power supply, and sure enough, the problem doesn't recur. Weird, I've never seen anything like this as usually if the power supply is marginal, other obviously bad things happen vs subtle issues like this.

Btw I do have an AR300M -- the AR150 is my backup and my preferred method of handling major firmware upgrades and other risky config changes is to do them on the AR150 first, get a stable config, and then do the same on the AR300M so I always have a known, good router to switch back to.

mfullerca · March 18, 2020, 11:50pm

BTW @bluewavenet according to https://openwrt.org/toh/start (entry # 491) 19.07.x isn't supported on the AR300M. I flashed a current, non-release build and messed around with it for a few hours and fell into NAND-flash hell. I eventually gave up on that and got it running on NOR-flash but discovered the WAN interface would die under heavy load. I found a page somewhere that says the AR300M won't be supported under 19.07.x and I assume no one wants to help debug this, so I'm back to the AR150.

bluewavenet · March 19, 2020, 7:54am

@mfullerca

That is not completely true. It had not been migrated from ar71xx to ath79 when 19.07 was forked, so is not supported in ath79, but is still to be found in ar71xx. ath71xx only supports NOR.

The 19.07- ar71xx version is NOR only and covers all versions of the ar300m (but you have to use image builder to enable wifi on an ar300m-lite, but that is a different story)

NAND hell is easily averted if you have the older uboot on your device, by first flashing NOR then flashing NAND with the NOR firmware. This breaks the NAND firmware making it fail over to a NOR boot.

If you have the latest uboot version it is a bit more complex. Let me know if you would like instructions.

I have run the snapshot ath79 NAND and the NAND-NOR versions with no problems at all.
I hate to say this but the only times I have seen issues under load have been when the power supply has been failing or inadequate
You could of course have a fault in the device....

EDIT I have updated the toh entry.

tmomas · March 19, 2020, 4:18pm

4 posts were split to a new topic: GL-AR300M dataentry + devicepage update