A lot of NAT'd Multicast packets are lost

Installing and Using OpenWrt Network and Wireless Configuration
1

I have set up IGMP Proxy and forward allowed 224.0.0.0/4 from wan to lan in the firewall. It seems to work fine overall, but Multicast UDP packets are randomly lost.

The load is not that heavy, CPU usage is almost at 0%.

On router. WAN to Router. without NAT: Good

# iperf3 -c speedtest.uztelecom.uz -u -p 5206 -R -b 50M
Connecting to host speedtest.uztelecom.uz, port 5206
Reverse mode, remote host speedtest.uztelecom.uz is sending
[  5] local 175.198.43.100 port 41721 connected to 195.69.189.215 port 5206
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  5.91 MBytes  49.6 Mbits/sec  0.018 ms  61/4343 (1.4%)
[  5]   1.00-2.00   sec  5.96 MBytes  50.0 Mbits/sec  0.018 ms  0/4316 (0%)
[  5]   2.00-3.00   sec  5.96 MBytes  50.0 Mbits/sec  0.021 ms  0/4316 (0%)
[  5]   3.00-4.00   sec  5.94 MBytes  49.8 Mbits/sec  0.017 ms  19/4317 (0.44%)
[  5]   4.00-5.00   sec  5.87 MBytes  49.2 Mbits/sec  0.008 ms  65/4316 (1.5%)
[  5]   5.00-6.00   sec  5.96 MBytes  50.0 Mbits/sec  0.018 ms  0/4316 (0%)
[  5]   6.00-7.00   sec  5.96 MBytes  50.0 Mbits/sec  0.017 ms  0/4316 (0%)
[  5]   7.00-8.00   sec  5.96 MBytes  50.0 Mbits/sec  0.018 ms  0/4318 (0%)
[  5]   8.00-9.00   sec  5.96 MBytes  50.0 Mbits/sec  0.017 ms  0/4319 (0%)
[  5]   9.00-10.00  sec  5.96 MBytes  50.0 Mbits/sec  0.015 ms  0/4313 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  60.8 MBytes  51.0 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-10.00  sec  59.4 MBytes  49.9 Mbits/sec  0.015 ms  145/43190 (0.34%)  receiver

iperf Done.

Router to PC. without NAT: Good

(right now it's wifi. When I tested it wired I had 0% loss.)

iperf3 -c 192.168.0.11 -u  -R -b 50M
Connecting to host 192.168.0.11, port 5201
Reverse mode, remote host 192.168.0.11 is sending
[  5] local 192.168.0.168 port 61728 connected to 192.168.0.11 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  5.98 MBytes  49.7 Mbits/sec  0.163 ms  32/4324 (0.74%)
[  5]   1.01-2.01   sec  5.89 MBytes  49.2 Mbits/sec  0.256 ms  15/4243 (0.35%)
[  5]   2.01-3.02   sec  5.83 MBytes  48.7 Mbits/sec  0.222 ms  167/4353 (3.8%)
[  5]   3.02-4.00   sec  5.87 MBytes  49.8 Mbits/sec  0.264 ms  16/4230 (0.38%)
[  5]   4.00-5.00   sec  5.88 MBytes  49.5 Mbits/sec  0.240 ms  43/4268 (1%)
[  5]   5.00-6.01   sec  5.92 MBytes  49.4 Mbits/sec  0.519 ms  23/4273 (0.54%)
[  5]   6.01-7.01   sec  5.97 MBytes  50.1 Mbits/sec  0.053 ms  20/4309 (0.46%)
[  5]   7.01-8.01   sec  5.92 MBytes  49.6 Mbits/sec  0.298 ms  31/4282 (0.72%)
[  5]   8.01-9.00   sec  5.91 MBytes  49.9 Mbits/sec  0.310 ms  16/4261 (0.38%)
[  5]   9.00-10.00  sec  5.92 MBytes  49.6 Mbits/sec  0.297 ms  29/4282 (0.68%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  59.7 MBytes  50.0 Mbits/sec  0.000 ms  0/42843 (0%)  sender
[  5]   0.00-10.00  sec  59.1 MBytes  49.5 Mbits/sec  0.297 ms  392/42825 (0.92%)  receiver

iperf Done.

WAN to PC. with NAT: BAD

iperf3 -c speedtest.uztelecom.uz -u -p 5206 -R -b 50M
Connecting to host speedtest.uztelecom.uz, port 5206
Reverse mode, remote host speedtest.uztelecom.uz is sending
[  5] local 192.168.0.168 port 53429 connected to 195.69.189.215 port 5206
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  4.01 MBytes  33.2 Mbits/sec  0.065 ms  1846/4723 (39%)
[  5]   1.01-2.01   sec  3.27 MBytes  27.6 Mbits/sec  0.071 ms  1942/4294 (45%)
[  5]   2.01-3.01   sec  4.98 MBytes  41.9 Mbits/sec  0.029 ms  707/4281 (17%)
[  5]   3.01-4.01   sec  4.34 MBytes  36.3 Mbits/sec  0.095 ms  1116/4232 (26%)
[  5]   4.01-5.01   sec  4.54 MBytes  38.2 Mbits/sec  0.097 ms  1068/4330 (25%)
[  5]   5.01-6.00   sec  4.09 MBytes  34.6 Mbits/sec  0.065 ms  1296/4237 (31%)
[  5]   6.00-7.01   sec  5.04 MBytes  42.0 Mbits/sec  0.081 ms  653/4274 (15%)
[  5]   7.01-8.00   sec  4.96 MBytes  41.9 Mbits/sec  0.098 ms  630/4190 (15%)
[  5]   8.00-9.01   sec  4.66 MBytes  38.7 Mbits/sec  0.086 ms  1042/4392 (24%)
[  5]   9.01-10.01  sec  4.60 MBytes  38.8 Mbits/sec  0.126 ms  860/4166 (21%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  61.4 MBytes  51.5 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-10.01  sec  44.5 MBytes  37.3 Mbits/sec  0.126 ms  11160/43119 (26%)  receiver

iperf Done.

What should I check? Looking at the test results, it seems that the loss is definitely occurring in NAT.

2

Lets check for obvious config glitches (omit wifi if it has same "speed" as nat-wired)

Can you make reports (link to result} via https://www.waveform.com/tools/bufferbloat
For 3 configurations?

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/sqm
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
3

Here are the test results on PC (WAN-Router-PC). These results are already very good. Do you still need results in other configurations?
https://www.waveform.com/tools/bufferbloat?test-id=7df70878-fa2f-471b-91bf-5c72f12ae670

ubus call system board:

{
        "kernel": "6.6.44",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 0",
        "model": "Bananapi BPI-R4",
        "board_name": "bananapi,bpi-r4",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r27100-21a5b3b540",
                "target": "mediatek/filogic",
                "description": "OpenWrt SNAPSHOT r27100-21a5b3b540"
        }
}

/etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '...'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        option igmp_snooping '1'
        option multicast '1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.0.1'
        option netmask '255.255.224.0'
        option ip6assign '60'
        option defaultroute '0'

config device
        option name 'br-wan'
        option type 'bridge'
        list ports 'wan'
        list ports 'eth2'

config interface 'wan'
        option device 'br-wan'
        option proto 'dhcp'
        option dns_metric '10'
        option metric '10'
        option hostname '*'

/etc/config/sqm: cat: can't open '/etc/config/sqm': No such file or directory

AP is non-openwrt. but wired also same result.

/etc/config/dhcp:

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '1'
        option ednspacket_max '1232'
        list server '1.1.1.1#53'
        list server '127.0.0.1#5054'
        list server '/mask.icloud.com/'
        list server '/mask-h2.icloud.com/'
        list server '/use-application-dns.net/'
        list server '127.0.0.1#5053'
        option doh_backup_noresolv '-1'
        option noresolv '1'
        list doh_backup_server '1.1.1.1#53'
        list doh_backup_server '127.0.0.1#5054'
        list doh_backup_server '/mask.icloud.com/'
        list doh_backup_server '/mask-h2.icloud.com/'
        list doh_backup_server '/use-application-dns.net/'
        list doh_backup_server '127.0.0.1#5053'
        list doh_server '127.0.0.1#5053'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

/etc/config/firewall:

config defaults
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option src 'wan'
        option target 'ACCEPT'
        option name 'iptv-multicast'
        list proto 'udp'
        list proto 'igmp'
        option dest 'lan'
        list dest_ip '224.0.0.0/4'

Re-tested result (Wired):

c:\programs> iperf3 -c 192.168.0.11 -u  -R -b 25M
Connecting to host 192.168.0.11, port 5201
Reverse mode, remote host 192.168.0.11 is sending
[  5] local 192.168.0.196 port 49821 connected to 192.168.0.11 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  2.99 MBytes  24.9 Mbits/sec  0.625 ms  0/2145 (0%)
[  5]   1.01-2.02   sec  2.97 MBytes  24.6 Mbits/sec  0.722 ms  0/2135 (0%)
[  5]   2.02-3.01   sec  3.01 MBytes  25.6 Mbits/sec  0.061 ms  1/2164 (0.046%)
[  5]   3.01-4.00   sec  2.97 MBytes  25.0 Mbits/sec  0.154 ms  0/2133 (0%)
[  5]   4.00-5.00   sec  2.97 MBytes  25.0 Mbits/sec  0.186 ms  0/2136 (0%)
[  5]   5.00-6.00   sec  2.97 MBytes  25.0 Mbits/sec  0.288 ms  0/2135 (0%)
[  5]   6.00-7.00   sec  2.99 MBytes  25.1 Mbits/sec  0.227 ms  0/2148 (0%)
[  5]   7.00-8.00   sec  2.97 MBytes  25.0 Mbits/sec  0.178 ms  0/2134 (0%)
[  5]   8.00-9.02   sec  3.03 MBytes  25.0 Mbits/sec  0.198 ms  0/2173 (0%)
[  5]   9.02-10.01  sec  2.97 MBytes  25.0 Mbits/sec  0.223 ms  0/2136 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.02  sec  29.9 MBytes  25.0 Mbits/sec  0.000 ms  0/21439 (0%)  sender
[  5]   0.00-10.01  sec  29.8 MBytes  25.0 Mbits/sec  0.223 ms  1/21439 (0.0047%)  receiver

iperf Done.

c:\programs> iperf3 -c 192.168.0.11 -u  -b 25M
Connecting to host 192.168.0.11, port 5201
[  5] local 192.168.0.196 port 50438 connected to 192.168.0.11 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.01   sec  2.97 MBytes  24.7 Mbits/sec  2135
[  5]   1.01-2.01   sec  2.99 MBytes  25.0 Mbits/sec  2145
[  5]   2.01-3.01   sec  2.96 MBytes  25.0 Mbits/sec  2127
[  5]   3.01-4.01   sec  2.98 MBytes  25.0 Mbits/sec  2141
[  5]   4.01-5.01   sec  2.99 MBytes  25.0 Mbits/sec  2148
[  5]   5.01-6.00   sec  2.95 MBytes  25.0 Mbits/sec  2121
[  5]   6.00-7.01   sec  3.00 MBytes  25.0 Mbits/sec  2152
[  5]   7.01-8.01   sec  3.00 MBytes  25.0 Mbits/sec  2154
[  5]   8.01-9.00   sec  2.95 MBytes  25.0 Mbits/sec  2121
[  5]   9.00-10.01  sec  3.00 MBytes  25.0 Mbits/sec  2154
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  29.8 MBytes  25.0 Mbits/sec  0.000 ms  0/21398 (0%)  sender
[  5]   0.00-10.03  sec  29.8 MBytes  24.9 Mbits/sec  0.051 ms  0/21398 (0%)  receiver

iperf Done.

c:\programs> iperf3 -c speedtest.uztelecom.uz -u -p 5206 -R -b 10M
Connecting to host speedtest.uztelecom.uz, port 5206
Reverse mode, remote host speedtest.uztelecom.uz is sending
[  5] local 192.168.0.196 port 55865 connected to 195.69.189.215 port 5206
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec   867 KBytes  7.05 Mbits/sec  0.063 ms  245/853 (29%)
[  5]   1.01-2.01   sec   814 KBytes  6.64 Mbits/sec  0.063 ms  286/857 (33%)
[  5]   2.01-3.01   sec   880 KBytes  7.18 Mbits/sec  0.037 ms  240/857 (28%)
[  5]   3.01-4.01   sec   897 KBytes  7.41 Mbits/sec  0.115 ms  226/855 (26%)
[  5]   4.01-5.00   sec   764 KBytes  6.28 Mbits/sec  0.132 ms  313/849 (37%)
[  5]   5.00-6.01   sec   831 KBytes  6.77 Mbits/sec  0.125 ms  278/861 (32%)
[  5]   6.01-7.01   sec   960 KBytes  7.83 Mbits/sec  0.034 ms  190/863 (22%)
[  5]   7.01-8.00   sec   808 KBytes  6.68 Mbits/sec  0.113 ms  285/852 (33%)
[  5]   8.00-9.00   sec   806 KBytes  6.61 Mbits/sec  0.171 ms  283/848 (33%)
[  5]   9.00-10.01  sec   753 KBytes  6.11 Mbits/sec  0.118 ms  323/851 (38%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  12.2 MBytes  10.2 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-10.01  sec  8.18 MBytes  6.86 Mbits/sec  0.118 ms  2669/8546 (31%)  receiver

iperf Done.

c:\programs> iperf3 -c speedtest.uztelecom.uz -u -p 5206 -b 10M
Connecting to host speedtest.uztelecom.uz, port 5206
[  5] local 192.168.0.196 port 65129 connected to 195.69.189.215 port 5206
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.18 MBytes  9.86 Mbits/sec  844
[  5]   1.00-2.01   sec  1.20 MBytes  9.99 Mbits/sec  861
[  5]   2.01-3.01   sec  1.20 MBytes  10.0 Mbits/sec  862
[  5]   3.01-4.00   sec  1.18 MBytes  9.99 Mbits/sec  849
[  5]   4.00-5.01   sec  1.20 MBytes  10.0 Mbits/sec  862
[  5]   5.01-6.01   sec  1.20 MBytes  10.0 Mbits/sec  859
[  5]   6.01-7.00   sec  1.18 MBytes  10.0 Mbits/sec  847
[  5]   7.00-8.01   sec  1.20 MBytes  10.0 Mbits/sec  864
[  5]   8.01-9.01   sec  1.19 MBytes  10.0 Mbits/sec  853
[  5]   9.01-10.00  sec  1.18 MBytes  9.99 Mbits/sec  849
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  9.98 Mbits/sec  0.000 ms  0/8550 (0%)  sender
[  5]   0.00-10.00  sec  11.9 MBytes  9.98 Mbits/sec  0.066 ms  0/8550 (0%)  receiver

iperf Done.
4

Probably local iperf server is overloaded. You get 20x better against cloudflare.
If you want to get closer to gigabit you may need firewall soft offload.

5

It works fine on the router, but there is a lot of loss on the PC. So it doesn't seem to be an iperf issue. Also, the real problem is that there is a lot of loss of Multicast UDP packets in IPTV.

If you look at the log, I set the traffic to 10Mbps. Even so, the loss is more than 20~40%. Also, the result is the same whether I turn on or off soft offloading in the firewall.

6

It is properly configured

What you could do (not guaranteed though)
Add a rule setting 244/4 destination to DSCP 0xEF so that it goes in front of other network transmits.

7

DCSP also had no effect.
The traffic was around 10Mbps, so there was no shortage of bandwidth.
(No other devices were connected except the test equipment)

8

The bw is so accurate that it looks like your provider is just creating stream to imitate video - 1Mbps - hd 4Mbps fhd 10Mbps 4k and so on?

http/3 as in waveform test is also UDP and works at normal speed.

9

Yes. iperf transmits at a constant bandwidth without flow control like http/3.

My real case is IPTV.
I need to receive multicast from wan and forward it to iptv.
But in the current state, the video is lost too often, which is not very good.

If http/3 is ok, then for some reason my router can't even handle 10Mbps without flow control?

10

What makes you think packets lost at your router?

11

There is no problem when connecting directly to WAN, but when going through a router, the video breaks up every 1-2 seconds.

Loss is also visible in iperf.

12

Add "drop invalid" checkbox in firewall, the code path is same for all udp streams.

cat /proc/net/softnet_stats # nonzero values in 2nd 3rd field are drops
ethtool wan # where internet arrives
ethtool -S wan | grep -v ": 0$" # another place drops can occur
13

There doesn't seem to be any Err or Drop.

# cat /proc/net/softnet_stat
15db17b5 00000002 0000dbda 00000000 00000000 00000000 00000000 00000000 00000000 091dfe03 00000000 00000000 00000000 00000000 00000000
1e173398 00000012 0000d0fd 00000000 00000000 00000000 00000000 00000000 00000000 0ade3440 00000000 00000000 00000001 00000000 00000000
af86da3a 00001207 000163d1 00000000 00000000 00000000 00000000 00000000 00000000 50460741 00000000 00000000 00000002 00000000 00000000
126c309e 00000000 0000c933 00000000 00000000 00000000 00000000 00000000 00000000 056d29e9 00000000 00000000 00000003 00000000 00000000

# ethtool -S wan
NIC statistics:
     tx_packets: 106579146
     tx_bytes: 22537056640
     rx_packets: 1381975206
     rx_bytes: 1605898112506
     TxDrop: 0
     TxCrcErr: 0
     TxUnicast: 107734761
     TxMulticast: 194532
     TxBroadcast: 968
     TxCollision: 0
     TxSingleCollision: 0
     TxMultipleCollision: 0
     TxDeferred: 1
     TxLateCollision: 0
     TxExcessiveCollistion: 0
     TxPause: 0
     TxPktSz64: 12996861
     TxPktSz65To127: 68430954
     TxPktSz128To255: 11324821
     TxPktSz256To511: 2373886
     TxPktSz512To1023: 4148670
     Tx1024ToMax: 8655069
     TxBytes: 23545322477
     RxDrop: 0
     RxFiltering: 139
     RxUnicast: 187853522
     RxMulticast: 1162953181
     RxBroadcast: 34990314
     RxAlignErr: 0
     RxCrcErr: 1
     RxUnderSizeErr: 0
     RxFragErr: 0
     RxOverSzErr: 0
     RxJabberErr: 0
     RxPause: 0
     RxPktSz64: 39308990
     RxPktSz65To127: 49719277
     RxPktSz128To255: 148059792
     RxPktSz256To511: 7652514
     RxPktSz512To1023: 3207142
     RxPktSz1024ToMax: 1137849303
     RxBytes: 1616826584314
     RxCtrlDrop: 0
     RxIngressDrop: 0
     RxArlDrop: 0

# ethtool -S lan1
NIC statistics:
     tx_packets: 450751176
     tx_bytes: 548802670077
     rx_packets: 25128573
     rx_bytes: 6417497301
     TxDrop: 0
     TxCrcErr: 0
     TxUnicast: 432118761
     TxMulticast: 18633523
     TxBroadcast: 33388
     TxCollision: 0
     TxSingleCollision: 0
     TxMultipleCollision: 0
     TxDeferred: 0
     TxLateCollision: 0
     TxExcessiveCollistion: 0
     TxPause: 0
     TxPktSz64: 3211570
     TxPktSz65To127: 18794107
     TxPktSz128To255: 6515005
     TxPktSz256To511: 47005774
     TxPktSz512To1023: 1215357
     Tx1024ToMax: 374043859
     TxBytes: 550649568064
     RxDrop: 0
     RxFiltering: 0
     RxUnicast: 23033577
     RxMulticast: 732062
     RxBroadcast: 1374994
     RxAlignErr: 0
     RxCrcErr: 0
     RxUnderSizeErr: 0
     RxFragErr: 0
     RxOverSzErr: 0
     RxJabberErr: 0
     RxPause: 0
     RxPktSz64: 3233077
     RxPktSz65To127: 15516434
     RxPktSz128To255: 1265643
     RxPktSz256To511: 2831444
     RxPktSz512To1023: 299206
     RxPktSz1024ToMax: 1994829
     RxBytes: 6521512810
     RxCtrlDrop: 0
     RxIngressDrop: 0
     RxArlDrop: 0
14

It looks like kind of packet aggregation (ethtool -k eth0 sg) OR
The "output" interface is eth0 - dsa master - where losses due to buffets could occur.

     RxMulticast: 1 162 953 181

     TxMulticast: 18 633 523
15

Yes, it is a BPI-R4 board and uses DSA.

ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: on
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp-mangleid-segmentation: on
        tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: on
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
rx-gro-list: on
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

wan and lan1 are one switch. However, both wan - lan1 - iptv and wan - eth1(sfp+) - sw - iptv have the same loss.

16

Stats ethtool -S eth0
The lan1 got very small part of packets....

17 
ethtool -S eth0
NIC statistics:
     tx_bytes: 40102789072
     tx_packets: 37109295
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 57141341245
     rx_packets: 62726773
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 79
     rx_flow_control_packets: 0
     rx_xdp_redirect: 0
     rx_xdp_pass: 0
     rx_xdp_drop: 0
     rx_xdp_tx: 0
     rx_xdp_tx_errors: 0
     tx_xdp_xmit: 0
     tx_xdp_xmit_errors: 0
     rx_pp_alloc_fast: 61748564
     rx_pp_alloc_slow: 208
     rx_pp_alloc_slow_ho: 0
     rx_pp_alloc_empty: 208
     rx_pp_alloc_refill: 980049
     rx_pp_alloc_waive: 0
     rx_pp_recycle_cached: 0
     rx_pp_recycle_cache_full: 0
     rx_pp_recycle_ring: 62717554
     rx_pp_recycle_ring_full: 9219
     rx_pp_recycle_released_ref: 0
     p06_TxDrop: 0
     p06_TxCrcErr: 0
     p06_TxUnicast: 62726852
     p06_TxMulticast: 0
     p06_TxBroadcast: 0
     p06_TxCollision: 0
     p06_TxSingleCollision: 0
     p06_TxMultipleCollision: 0
     p06_TxDeferred: 0
     p06_TxLateCollision: 0
     p06_TxExcessiveCollistion: 0
     p06_TxPause: 0
     p06_TxPktSz64: 0
     p06_TxPktSz65To127: 0
     p06_TxPktSz128To255: 0
     p06_TxPktSz256To511: 0
     p06_TxPktSz512To1023: 0
     p06_Tx1024ToMax: 0
     p06_TxBytes: 57141348519
     p06_RxDrop: 0
     p06_RxFiltering: 16
     p06_RxUnicast: 37109295
     p06_RxMulticast: 0
     p06_RxBroadcast: 0
     p06_RxAlignErr: 0
     p06_RxCrcErr: 0
     p06_RxUnderSizeErr: 0
     p06_RxFragErr: 0
     p06_RxOverSzErr: 0
     p06_RxJabberErr: 0
     p06_RxPause: 4153
     p06_RxPktSz64: 0
     p06_RxPktSz65To127: 0
     p06_RxPktSz128To255: 0
     p06_RxPktSz256To511: 0
     p06_RxPktSz512To1023: 0
     p06_RxPktSz1024ToMax: 0
     p06_RxBytes: 40102789072
     p06_RxCtrlDrop: 0
     p06_RxIngressDrop: 0
     p06_RxArlDrop: 0

There was no change in error/drop/defer/filter before/after the iperf test.

ethtool -S eth0 | grep -E 'drop|err|defer|filter' -i
18

So it is software drops. Seek softnet_stat, apply remedy few times until growth of drops in 2nd 3rd field stop. The magic copypasta needs gawk essentially converts hex to dec.

irqbalance alternative is packet steering, you need to try both for same target.

19

The value of softnet_stat does not increase..

(openwrt) # awk '{for (i=1; i<=NF; i++) printf strtonum("0x" $i) (i==NF?"\n":" ")}' /proc/net/softnet_stat
378921424 2 56949 0 0 0 0 0 0 158480206 0 0 0 0 0
509257735 18 54159 0 0 0 0 0 0 183471235 0 0 1 0 0
2962717195 4615 91715 0 0 0 0 0 0 1357144456 0 0 2 0 0
316748405 0 52090 0 0 0 0 0 0 93528558 0 0 3 0 0

(PC) $ iperf3 -c speedtest.uztelecom.uz -u -p 5207 -R -b20M
Connecting to host speedtest.uztelecom.uz, port 5207
Reverse mode, remote host speedtest.uztelecom.uz is sending
[  5] local 192.168.2.101 port 59301 connected to 195.69.189.215 port 5207
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  1.50 MBytes  12.5 Mbits/sec  0.020 ms  639/1715 (37%)
[  5]   1.01-2.01   sec  1.53 MBytes  12.8 Mbits/sec  0.027 ms  606/1708 (35%)
[  5]   2.01-3.00   sec  1.44 MBytes  12.2 Mbits/sec  0.030 ms  681/1716 (40%)
[  5]   3.00-4.01   sec  1.46 MBytes  12.1 Mbits/sec  0.015 ms  669/1714 (39%)
[  5]   4.01-5.00   sec  1.48 MBytes  12.5 Mbits/sec  0.015 ms  646/1712 (38%)
[  5]   5.00-6.01   sec  1.49 MBytes  12.4 Mbits/sec  0.026 ms  641/1712 (37%)
[  5]   6.01-7.01   sec  1.54 MBytes  12.8 Mbits/sec  0.021 ms  610/1714 (36%)
[  5]   7.01-8.00   sec  1.54 MBytes  13.0 Mbits/sec  0.024 ms  609/1712 (36%)
[  5]   8.00-9.01   sec  1.51 MBytes  12.6 Mbits/sec  0.021 ms  626/1709 (37%)
[  5]   9.01-10.00  sec  1.49 MBytes  12.6 Mbits/sec  0.039 ms  646/1714 (38%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  24.3 MBytes  20.4 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-10.00  sec  15.0 MBytes  12.6 Mbits/sec  0.039 ms  6373/17126 (37%)  receiver

iperf Done.

(PC) $ iperf3 -c speedtest.uztelecom.uz -u -p 5207 -R -b10M
Connecting to host speedtest.uztelecom.uz, port 5207
Reverse mode, remote host speedtest.uztelecom.uz is sending
[  5] local 192.168.2.101 port 55605 connected to 195.69.189.215 port 5207
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.01   sec  1.05 MBytes  8.69 Mbits/sec  0.034 ms  103/857 (12%)
[  5]   1.01-2.00   sec   980 KBytes  8.11 Mbits/sec  0.028 ms  169/856 (20%)
[  5]   2.00-3.01   sec  1.00 MBytes  8.35 Mbits/sec  0.029 ms  137/856 (16%)
[  5]   3.01-4.01   sec  1.02 MBytes  8.54 Mbits/sec  0.028 ms  120/855 (14%)
[  5]   4.01-5.01   sec  1.03 MBytes  8.74 Mbits/sec  0.038 ms  116/858 (14%)
[  5]   5.01-6.01   sec   989 KBytes  8.05 Mbits/sec  0.025 ms  162/856 (19%)
[  5]   6.01-7.00   sec  1.00 MBytes  8.51 Mbits/sec  0.031 ms  134/855 (16%)
[  5]   7.00-8.01   sec  1.06 MBytes  8.81 Mbits/sec  0.029 ms  98/858 (11%)
[  5]   8.01-9.02   sec  1.03 MBytes  8.60 Mbits/sec  0.031 ms  115/856 (13%)
[  5]   9.02-10.01  sec  1.00 MBytes  8.49 Mbits/sec  0.028 ms  136/855 (16%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.01  sec  12.2 MBytes  10.2 Mbits/sec  0.000 ms  0/0 (0%)  sender
[  5]   0.00-10.01  sec  10.1 MBytes  8.49 Mbits/sec  0.028 ms  1290/8562 (15%)  receiver

iperf Done.

(openwrt) ~#  awk '{for (i=1; i<=NF; i++) printf strtonum("0x" $i) (i==NF?"\n":" ")}' /proc/net/softnet_stat
378965242 2 56952 0 0 0 0 0 0 158501678 0 0 0 0 0
509277681 18 54164 0 0 0 0 0 0 183472565 0 0 1 0 0
2962899257 4615 91718 0 0 0 0 0 0 1357261510 0 0 2 0 0
316969278 0 52090 0 0 0 0 0 0 93615815 0 0 3 0 0
20

3rd column just did?