Weirdly slow networking on FortiGate 50E

NC1 · June 26, 2024, 5:29pm

I continue to amuse myself with running OpenWrt on semi-exotic devices... This week's experiment is FortiGate 50E by Fortinet:

https://openwrt.org/toh/hwdata/fortinet/fortinet_fortigate_50e

I ended up in possession of a FortiGate 50E on which someone already installed OpenWrt. Ran sysupgrade -n on it to restore it to the initial settings, added a few packages here and there... Here's the weird part: LAN-to-WAN throughput is about 20 Mbps (in words: twenty megabits per second). Right now, I don't have first clue about how to diagnose this issue. Any ideas?

Before someone asks, throughput was measured using iperf3 with iperf3 client running on a computer connected to a LAN port and iperf3 server running on the upstream device connected to a WAN port ("a" WAN port, because the device has two).

Speaking of WAN ports, right now, the device has two bridges configured, a LAN bridge (five ports) and a WAN bridge (two ports). I've never seen a WAN bridge before, so I wonder if this could somehow be responsible for the subpar performance...

All ideas are welcome. If a configuration file or a diagnostic output needs to be posted, please ask. Right now, I am not sure what's relevant...

psherman · June 27, 2024, 5:44am

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

brada4 · June 27, 2024, 6:29am

First run "baseline" test -

and post the llink - so that everybody sees we improve, not just change for sake of change.

Lets see some low level things (always after running that test):
Get list of network adapters via ip link (exotic hardware yk)
Then for each adapter:
ethtool -i eth0 -> just driver line
ethtool eth0 -> validate that negotiated link speed is correct ie 1000 duplex
ethtool -S eth0 | grep -v ": 0$" -> post what seems anomalous, say normal is in/out bytes and packets and packet size histogram, any error,fail, over,buffer is bad, thoughg few pop when you restart upstream CPE.

And post what Mr Sherman asks, lets weed out any mistakes you accidentally made

NC1 · June 27, 2024, 4:38pm

All right, let the fun begin!

System board info:

root@F50E:~# ubus call system board
{
        "kernel": "5.15.150",
        "hostname": "F50E",
        "system": "ARMv7 Processor rev 1 (v7l)",
        "model": "Fortinet FortiGate 50E",
        "board_name": "fortinet,fg-50e",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.3",
                "revision": "r23809-234f1a2efa",
                "target": "mvebu/cortexa9",
                "description": "OpenWrt 23.05.3 r23809-234f1a2efa"
        }
}

Network configuration:

root@F50E:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd11:c006:34e3::/48'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'
        list ports 'lan5'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device
        option name 'br-wan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'eth2'

config interface 'wan'
        option device 'br-wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'br-wan'
        option proto 'dhcpv6'

There is no wireless configuration file, as the device is wired-only.

DHCP configuration:

root@F50E:~# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'
        option ednspacket_max '1232'
        option filter_aaaa '0'
        option filter_a '0'
        option confdir '/tmp/dnsmasq.d'
        list server '/mask.icloud.com/'
        list server '/mask-h2.icloud.com/'
        list server '127.0.0.1#5054'
        list server '/use-application-dns.net/'
        list server '127.0.0.1#5053'
        option doh_backup_noresolv '-1'
        option noresolv '1'
        list doh_backup_server '/mask.icloud.com/'
        list doh_backup_server '/mask-h2.icloud.com/'
        list doh_backup_server '127.0.0.1#5054'
        list doh_backup_server '/use-application-dns.net/'
        list doh_backup_server '127.0.0.1#5053'
        list doh_server '127.0.0.1#5053'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        option ra_slaac '1'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

Firewall configuration:

root@F50E:~# cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

Thank you for your willingness to help!

psherman · June 27, 2024, 4:46pm

Your config looks mostly default, but there are two things to look at:

The wan bridge may be responsible for the performance issue. The ports appear to be individually routed (i.e. not part of a switch), so creating a bridge with those puts the burden on the CPU for all traffic moving through that bridge.

Therefore, try deleting br-wan and then assign eth1 to both wan and wan6 in the device section (instead of br-wan).

I see you are using DoH or DoT. It's possible that this is causing some issues. You may want to revert back to regular DNS via dnsmasq if the above doesn't help.

NC1 · June 27, 2024, 4:46pm

OK, here's my (atrocious) bufferbloat test:

Will post the rest shortly...

NC1 · June 27, 2024, 4:59pm

Indeed!

ip link:

root@F50E:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP qlen 1024
    link/ether 90:6c:ac:53:6a:3e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake master br-wan state UP qlen 1024
    link/ether 90:6c:ac:53:6a:3f brd ff:ff:ff:ff:ff:ff
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-wan state DOWN qlen 1024
    link/ether 90:6c:ac:53:6a:40 brd ff:ff:ff:ff:ff:ff
5: lan5@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 90:6c:ac:53:6a:45 brd ff:ff:ff:ff:ff:ff
6: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 90:6c:ac:53:6a:44 brd ff:ff:ff:ff:ff:ff
7: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 90:6c:ac:53:6a:43 brd ff:ff:ff:ff:ff:ff
8: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-lan state LOWERLAYERDOWN qlen 1000
    link/ether 90:6c:ac:53:6a:42 brd ff:ff:ff:ff:ff:ff
9: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether 90:6c:ac:53:6a:41 brd ff:ff:ff:ff:ff:ff
10: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 90:6c:ac:53:6a:41 brd ff:ff:ff:ff:ff:ff
14: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 90:6c:ac:53:6a:3f brd ff:ff:ff:ff:ff:ff
31: ifb4eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc cake state UNKNOWN qlen 32
    link/ether da:6e:b3:55:9f:43 brd ff:ff:ff:ff:ff:ff

All eth* adapters show driver: mvneta. Looks like eth0 has a switch on it, while eth1 and eth2 are the WAN ports...

eth0 and eth1 report Speed: 1000Mb/s; eth2 is not connected to anything, so Speed: Unknown!.

root@F50E:~# ethtool -S eth0 | grep -v ": 0$"
NIC statistics:
     good_octets_received: 94586347
     good_frames_received: 94549
     broadcast_frames_received: 7
     multicast_frames_received: 211
     frames_65_to_127_octets: 66934
     frames_128_to_255_octets: 1708
     frames_256_to_511_octets: 741
     frames_512_to_1023_octets: 3084
     frames_1024_to_max_octets: 117702
     good_octets_sent: 88578354
     good_frames_sent: 95791
     multicast_frames_sent: 42
     broadcast_frames_sent: 7
     p06_in_good_octets: 88580064
     p06_in_unicast: 95742
     p06_in_broadcasts: 7
     p06_in_multicasts: 42
     p06_out_octets: 94586692
     p06_out_unicast: 94331
     p06_out_broadcasts: 8
     p06_out_multicasts: 211
     p06_hist_64bytes: 171
     p06_hist_65_127bytes: 66934
     p06_hist_128_255bytes: 1708
     p06_hist_256_511bytes: 742
     p06_hist_512_1023bytes: 3084
     p06_hist_1024_max_bytes: 117702
     p06_sw_out_filtered: 30255

Before you spend any time on this, let me implement @psherman's suggestion (remove the WAN bridge and configure WAN on a single port) and see if that solves my problem. Will report shortly.

brada4 · June 27, 2024, 5:01pm

Do not touch WAN bridge, ot is default in multi-wan devices.

brada4 · June 27, 2024, 5:04pm

The link speed and stats report needed from connected lan1 port and connected wan port eth1
the eth0 dsa parent is nearly always fine unless you have bad habit of soldering.

brada4 · June 27, 2024, 5:11pm

Also check ethtool --show-eee and ethtool -a for both connected ports - those protocol corners sometimes contribute to radical slowness.

NC1 · June 27, 2024, 5:28pm

Since I made no changes to DNS settings, this made me think that I may have a false memory of running sysupgrade -n. It's possible that I actually ran sysupgrade without the -n. So, with that in mind, I ran sysupgrade -n to ensure any previous settings are purged, and, presto!, problem solved:

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-60.04  sec  6.48 GBytes   927 Mbits/sec
----------------------------------------------------

Thank you for your help, and my apologies for not figuring it out on my own. Now I'm going to go write STOOPID on my forehead in thick black marker and walk around like this all day...

NC1 · June 27, 2024, 5:30pm

Thank you for your help! Problem solved.

brada4 · June 27, 2024, 5:51pm

Your nickname is now expert resetter actual command is firstboot, no need to reflash base squash fs

NC1 · June 27, 2024, 6:01pm

Since I received the device with OpenWrt already installed, I thought I'd make sure that I have only base packages, on an off chance an additional package was installed and misconfigured, and that's what the wart on the ass of progress is. firstboot wouldn't do it, would it?

brada4 · June 27, 2024, 6:03pm

Once you flashed base packages and old config (overlay) was in the way firstboot ; reboot should have sufficed.

NC1 · June 27, 2024, 6:05pm

Thank you, got it!

psherman · June 27, 2024, 6:06pm

firstboot will erase everything that is not part of the original image (i.e. post-flash package additions and configurations). If the image that was installed had any additional packages and/or baked-in configurations that are not standard, firstboot will not be able to change those. This mainly applies to an image that may have been loaded prior to your ownership of the device.

It sounds like you had already flashed a known standard image, but there was no harm in flashing again, even though firstboot would likely have been sufficient.

NC1 · June 27, 2024, 6:08pm

Thank you for the explanation!