Is your mobile using any kind of proxy that might bypass ISP restrictions?
Can you take a look using tcpdump to check and compare the packets sent/received in respect of desktop and mobile during speed test?
I caught Vodafone UK out at throttling in a way that was circumventable using a VPN - see this video:
Well, was that a TCP or a UDP test, and how many drops did you see in tc -s qdisc output from before and after the iperf test? After all AQMs by default use drops to signal a flow to slow down... (or ECN, but for the the endpoints need to negotiate the use of ECN during the connection setup.)
Ah, great in that case please post the output of ethtool -S eno1´ and ifconfig` just the ethernet statistics section similar to:
RX packets 10789715 bytes 3423971358 (3.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16962288 bytes 1524707199 (1.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
just to see whether there are any issues with the NIC.
Nothing really special in my phone network, I dont have proxy, VPN and anything else on it. The only real difference is that the smartphone connection travels first through the Wireless 5G to the Xiaomi Router in Repeater mode before arrives in the OpenWRT router.
BTW very interesting case that you found with your mobile connection, this is the kind of thing that 99.9% of users will never realize.
Later night I will inspect with tcpdump packets from waveform, for both devices, to see if something interesting appears.
Do you have any hints about these inspections with tcpdump that could help somehow, where to look, some flags?
BTW thanks you all for the help on this topic.
The packet loss tests was using UDP. I will execute again them later and evaluate the tc -s qdisc.
After all AQMs by default use drops to signal a flow to slow down
Interesting, this means that sometimes the packet loss is just a signal that the network is congested. However since I performed the same test on the mobile and just saw 0.0042% of packet loss I don't think that is related to congestion.
See the statistics about the NIC:
ifconfig eno1 Output:
RX packets 64960608 bytes 86847521578 (86.8 GB)
RX errors 0 dropped 13 overruns 0 frame 0
TX packets 69339648 bytes 72488855073 (72.4 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0x55400000-55420000
ethtool -S eno1 Output:
NIC statistics:
rx_packets: 64960109
tx_packets: 69339178
rx_bytes: 86847314699
tx_bytes: 72488777011
rx_broadcast: 91512
tx_broadcast: 872
rx_multicast: 5584
tx_multicast: 8729
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 5584
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 2882335
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_csum_offload_good: 64910441
rx_csum_offload_errors: 6
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 0
dropped_smbus: 0
rx_dma_failed: 0
tx_dma_failed: 0
rx_hwtstamp_cleared: 0
uncorr_ecc_errors: 0
corr_ecc_errors: 0
tx_hwtstamp_timeouts: 0
tx_hwtstamp_skipped: 0
Well 13 of your drops might have come from the ethernet side...
Maybe sample both ifconfig and tc -s qdisc closely before and after your iperf tests to see whether the amount of dropped packets correlates with any..
Well, who knows whether the android UDP stack might not be a tad less aggressive, of android iperf might behave a bit different...
I’d use something like:
tcpdump -i wan host 192.168.1.2 -v
New iperf tests performed. So far nothing seems strange in the comparison of ifconfig and tc -s qdisc before and after iperf. However one thing in the tc output seems strange, the drops in the cake queue seems quite high. (0.79% of dropped packets), is that fine?
qdisc cake 80a6: dev ifb4eth0 refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 78115052920 bytes 61452685 pkt (dropped 461275, overlimits 61909681 requeues 0)
The full result is below:
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-20.00 sec 2.21 GBytes 951 Mbits/sec 0.000 ms 0/1642447 (0%) sender
[ 5] 0.00-20.00 sec 2.21 GBytes 950 Mbits/sec 0.018 ms 2092/1642436 (0.13%) receiver
ifconfig and tc -s qdisc BEFORE
eth2
RX packets:42786393 errors:0 dropped:4 overruns:80 frame:0
TX packets:95487777 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20364017683 (18.9 GiB) TX bytes:118044089903 (109.9 GiB)
eno1
RX packets 82893145 bytes 108048285332 (108.0 GB)
RX errors 0 dropped 13 overruns 0 frame 0
TX packets 78718871 bytes 75508311961 (75.5 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0x55400000-55420000
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 40692Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 11623354747 bytes 25714685 pkt (dropped 41789, overlimits 32399953 requeues 3029)
backlog 0b 0p requeues 3029
memory used: 925Kb of 4Mb
capacity estimate: 14Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 64 / 1522
average network hdr offset: 14
Tin 0
thresh 40692Kbit
target 5ms
interval 100ms
pk_delay 1.2ms
av_delay 795us
sp_delay 3us
backlog 0b
pkts 25756474
bytes 11683381961
way_inds 511098
way_miss 123625
way_cols 0
drops 41789
marks 0
ack_drop 0
sp_flows 5
bk_flows 1
un_flows 0
max_len 44820
quantum 1241
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
Sent 76294328072 bytes 61906378 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth1 root
Sent 160267174103 bytes 142528257 pkt (dropped 69, overlimits 0 requeues 129458)
backlog 0b 0p requeues 129458
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31587149598 bytes 25356238 pkt (dropped 11, overlimits 0 requeues 22245)
backlog 0b 0p requeues 22245
maxpacket 18642 drop_overlimit 0 new_flow_count 14192 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 47346411936 bytes 38759825 pkt (dropped 24, overlimits 0 requeues 40557)
backlog 0b 0p requeues 40557
maxpacket 25738 drop_overlimit 0 new_flow_count 30947 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 45671248789 bytes 35215759 pkt (dropped 25, overlimits 0 requeues 32496)
backlog 0b 0p requeues 32496
maxpacket 22115 drop_overlimit 0 new_flow_count 24981 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 35662363780 bytes 43196435 pkt (dropped 9, overlimits 0 requeues 34160)
backlog 0b 0p requeues 34160
maxpacket 25738 drop_overlimit 0 new_flow_count 23081 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth2 root
Sent 118047424340 bytes 95493983 pkt (dropped 228, overlimits 0 requeues 64023)
backlog 0b 0p requeues 64023
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31313630274 bytes 24705302 pkt (dropped 18, overlimits 0 requeues 12691)
backlog 0b 0p requeues 12691
maxpacket 24378 drop_overlimit 0 new_flow_count 13873 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31199830266 bytes 23845555 pkt (dropped 17, overlimits 0 requeues 12917)
backlog 0b 0p requeues 12917
maxpacket 28766 drop_overlimit 0 new_flow_count 13956 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 26825953096 bytes 23187208 pkt (dropped 23, overlimits 0 requeues 14285)
backlog 0b 0p requeues 14285
maxpacket 24378 drop_overlimit 0 new_flow_count 13185 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 28708010704 bytes 23755918 pkt (dropped 170, overlimits 0 requeues 24130)
backlog 0b 0p requeues 24130
maxpacket 19682 drop_overlimit 0 new_flow_count 23593 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth3 root
Sent 111135417719 bytes 103288228 pkt (dropped 18, overlimits 0 requeues 60254)
backlog 0b 0p requeues 60254
qdisc fq_codel 0: dev eth3 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 26696815582 bytes 24233318 pkt (dropped 4, overlimits 0 requeues 12250)
backlog 0b 0p requeues 12250
maxpacket 44392 drop_overlimit 0 new_flow_count 8721 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 27969009058 bytes 23809027 pkt (dropped 14, overlimits 0 requeues 14349)
backlog 0b 0p requeues 14349
maxpacket 8604 drop_overlimit 0 new_flow_count 8382 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 19475201720 bytes 16275478 pkt (dropped 0, overlimits 0 requeues 13212)
backlog 0b 0p requeues 13212
maxpacket 14340 drop_overlimit 0 new_flow_count 10313 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 36994391359 bytes 38970405 pkt (dropped 0, overlimits 0 requeues 20443)
backlog 0b 0p requeues 20443
maxpacket 11472 drop_overlimit 0 new_flow_count 14407 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 78114305671 bytes 61445103 pkt (dropped 461275, overlimits 61909594 requeues 0)
backlog 0b 0p requeues 0
memory used: 5724344b of 15000000b
capacity estimate: 310Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 68 / 1522
average network hdr offset: 14
Tin 0
thresh 300Mbit
target 5ms
interval 100ms
pk_delay 16us
av_delay 7us
sp_delay 3us
backlog 0b
pkts 61906378
bytes 78802391656
way_inds 561942
way_miss 120184
way_cols 0
drops 461275
marks 0
ack_drop 0
sp_flows 4
bk_flows 1
un_flows 0
max_len 68130
quantum 1514
ifconfig and tc -s qdisc AFTER
eth2
RX packets:42804760 errors:0 dropped:4 overruns:80 frame:0
TX packets:95506906 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:20373729930 (18.9 GiB) TX bytes:118054431297 (109.9 GiB)
eno1
RX packets 83229183 bytes 108514985643 (108.5 GB)
RX errors 0 dropped 13 overruns 0 frame 0
TX packets 80486883 bytes 78043529908 (78.0 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0x55400000-55420000
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 25Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 11623610326 bytes 25716889 pkt (dropped 41789, overlimits 32400180 requeues 3029)
backlog 0b 0p requeues 3029
memory used: 925Kb of 4Mb
capacity estimate: 14Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 64 / 1522
average network hdr offset: 14
Tin 0
thresh 25Mbit
target 5ms
interval 100ms
pk_delay 69us
av_delay 10us
sp_delay 3us
backlog 0b
pkts 25758678
bytes 11683637540
way_inds 511497
way_miss 123779
way_cols 0
drops 41789
marks 0
ack_drop 0
sp_flows 3
bk_flows 1
un_flows 0
max_len 44820
quantum 762
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
Sent 76294965401 bytes 61913960 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth1 root
Sent 160272393927 bytes 142539940 pkt (dropped 69, overlimits 0 requeues 129476)
backlog 0b 0p requeues 129476
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31587207657 bytes 25356472 pkt (dropped 11, overlimits 0 requeues 22245)
backlog 0b 0p requeues 22245
maxpacket 18642 drop_overlimit 0 new_flow_count 14192 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 47351434872 bytes 38770749 pkt (dropped 24, overlimits 0 requeues 40575)
backlog 0b 0p requeues 40575
maxpacket 25738 drop_overlimit 0 new_flow_count 30960 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 45671310684 bytes 35215981 pkt (dropped 25, overlimits 0 requeues 32496)
backlog 0b 0p requeues 32496
maxpacket 22115 drop_overlimit 0 new_flow_count 24981 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 35662440714 bytes 43196738 pkt (dropped 9, overlimits 0 requeues 34160)
backlog 0b 0p requeues 34160
maxpacket 25738 drop_overlimit 0 new_flow_count 23081 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth2 root
Sent 118052725912 bytes 95503743 pkt (dropped 228, overlimits 0 requeues 64079)
backlog 0b 0p requeues 64079
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31318497057 bytes 24710507 pkt (dropped 18, overlimits 0 requeues 12747)
backlog 0b 0p requeues 12747
maxpacket 24378 drop_overlimit 0 new_flow_count 13908 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 31199896024 bytes 23845870 pkt (dropped 17, overlimits 0 requeues 12917)
backlog 0b 0p requeues 12917
maxpacket 28766 drop_overlimit 0 new_flow_count 13956 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 26826273149 bytes 23191190 pkt (dropped 23, overlimits 0 requeues 14285)
backlog 0b 0p requeues 14285
maxpacket 24378 drop_overlimit 0 new_flow_count 13185 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 28708059682 bytes 23756176 pkt (dropped 170, overlimits 0 requeues 24130)
backlog 0b 0p requeues 24130
maxpacket 19682 drop_overlimit 0 new_flow_count 23593 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc mq 0: dev eth3 root
Sent 111140041768 bytes 103293467 pkt (dropped 18, overlimits 0 requeues 60304)
backlog 0b 0p requeues 60304
qdisc fq_codel 0: dev eth3 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 26696927570 bytes 24233706 pkt (dropped 4, overlimits 0 requeues 12252)
backlog 0b 0p requeues 12252
maxpacket 44392 drop_overlimit 0 new_flow_count 8721 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 27969143260 bytes 23809499 pkt (dropped 14, overlimits 0 requeues 14351)
backlog 0b 0p requeues 14351
maxpacket 8604 drop_overlimit 0 new_flow_count 8383 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 19479513895 bytes 16279568 pkt (dropped 0, overlimits 0 requeues 13257)
backlog 0b 0p requeues 13257
maxpacket 14340 drop_overlimit 0 new_flow_count 10363 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth3 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 36994457043 bytes 38970694 pkt (dropped 0, overlimits 0 requeues 20444)
backlog 0b 0p requeues 20444
maxpacket 11472 drop_overlimit 0 new_flow_count 14408 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 78115052920 bytes 61452685 pkt (dropped 461275, overlimits 61909681 requeues 0)
backlog 0b 0p requeues 0
memory used: 5724344b of 15000000b
capacity estimate: 310Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 68 / 1522
average network hdr offset: 14
Tin 0
thresh 300Mbit
target 5ms
interval 100ms
pk_delay 36us
av_delay 7us
sp_delay 2us
backlog 0b
pkts 61913960
bytes 78803138905
way_inds 561942
way_miss 120335
way_cols 0
drops 461275
marks 0
ack_drop 0
sp_flows 1
bk_flows 1
un_flows 0
max_len 68130
quantum 1514
Very interesting thing I just spotted here. Inspired by the iperf comparison, I decided to perform a speedtest comparing tc -s qdisc before and after a Waveform test in Desktop and Mobile. During desktop tests the cake queue dropped 2.2% of the packets, while during mobile tests cake dropped 0.5% of the packets.
What could be the reason for that? Is it just a consequence of the autorate dropping the bandwidth? (disabled autorate but the result persisted)
root@OpenWrt:~# tc -s qdisc | grep cake -A 1
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 25102Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 11983951724 bytes 26533652 pkt (dropped 43709, overlimits 33476837 requeues 3085)
--
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 80322304197 bytes 63166423 pkt (dropped 481164, overlimits 63779453 requeues 0)
root@OpenWrt:~# tc -s qdisc | grep cake -A 1
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 43662Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 12146420990 bytes 26774191 pkt (dropped 44136, overlimits 34023903 requeues 3163)
--
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 81436424293 bytes 64001315 pkt (dropped 485560, overlimits 64643688 requeues 0)
root@OpenWrt:~# tc -s qdisc | grep cake -A 1
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 23761Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 11785863111 bytes 26130959 pkt (dropped 42565, overlimits 32824014 requeues 3045)
--
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 346500Kbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 79157962055 bytes 62292750 pkt (dropped 461977, overlimits 62720775 requeues 0)
root@OpenWrt:~# tc -s qdisc | grep cake -A 1
qdisc cake 80a5: dev eth0 root refcnt 5 bandwidth 43225Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 11983657115 bytes 26531717 pkt (dropped 43709, overlimits 33476583 requeues 3085)
--
qdisc cake 80a6: dev ifb4eth0 root refcnt 2 bandwidth 283276Kbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 80321800853 bytes 63161492 pkt (dropped 481164, overlimits 63779407 requeues 0)
Likely that your mobile iperf somehow is sending less aggressively than your desktop one...
@moeller0, just to make sure, my last post was a comparison before and after waveform test, not iperf. I dont know if you noticed that. The iperf comparison is the other one.
Ah, sorry then, that changes the conclusion slightly, not a difference in iperf then but likely the TCP stacks of ubuntu and android... (different linux kernels, different defaults)
What does cat /proc/sys/net/ipv4/tcp_congestion_control return on the ubuntu host?
And how do the tests/drop rates look if you do sudo sysctl -w net.ipv4.tcp_ecn=1 before?
(ECN signaling should replace most/all drops with CE marks)...
I performed the tests here, and the drops disappear with tcp_ecn = 1 as you highlighted, but the latency/bloat is still quite bad.
cat /proc/sys/net/ipv4/tcp_ecn
2
cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
sysctl net.core.default_qdisc
net.core.default_qdisc = fq_codel
cheetah:/ $ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
cheetah:/ $ sysctl net.ipv4.tcp_ecn
net.ipv4.tcp_ecn = 2
cheetah:/ $ sysctl net.core.default_qdisc
net.core.default_qdisc = pfifo_fast
qdisc cake 80a9: dev eth0 root refcnt 5 bandwidth 25Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 144619685 bytes 328541 pkt (dropped 576, overlimits 654066 requeues 23)
--
qdisc cake 80aa: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 912117035 bytes 690263 pkt (dropped 15600, overlimits 833799 requeues 0)
qdisc cake 80ad: dev eth0 root refcnt 5 bandwidth 41940Kbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 171965986 bytes 468964 pkt (dropped 435, overlimits 943240 requeues 65)
--
qdisc cake 80ae: dev ifb4eth0 root refcnt 2 bandwidth 300Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 22 mpu 64
Sent 980461346 bytes 741193 pkt (dropped 0, overlimits 768243 requeues 0)
Have you checked tcpdump during speed tests to see if there’s a difference between desktop and mobile?
Hello @Lynx !
Yesterday I collected entire dumps from desktop waveform tests and from mobile and analyze them in wireshark. I could not spot any meaningful difference in the packets. See the two samples below:
The only thing that I could see diferent between both platforms were the packets average size, in desktop the AVG size is smaller than mobile.
I also tried several kernel network tunings (tcp rmem, tcp_wmem, tcp_scaling_window, ecn, and etc) to see if something could improve but without success so far.
Desktop
Mobile
Desktop Packets Length
Mobile Packets size
Thanks @niltonvasques. So same host IPs (just different LAN IPs)?
The remaining networking stuff is way outside my comfort zone, but this is well inside @moeller0's - he eats this stuff for breakfast, so hopefully he can advise.
This SRC IPv6 2606:4700::6810:4326 is used in waveform tests, probably some speedtest host. The IPv6 DST are different from both packets, one has the IPV6 address of the smartphone and the other from the Desktop.
maybe a bug in a new version sqm
update last 10 days but not sure
@moeller0 can you help if you want
I am pretty sure this is not counting on-the-wire ethernet frame sizes... could be GRO/GSO type meta packets, or maybe HTTPS payload sizes...
For TCP tests I would look at the statistics -> TCP Stream Graphs -> Time Sequence (tcptrace) graphs for selected flows from the speedtest and compare over Mobile and Desktop...
OK, if the drops go away (and marks in the tc -s qdisc output increase) that part is working...
Please do:
tc -s qdisctc -s qdisc (ideally while the speed test has rund for a few seconds but also is still running, the delay counters age out quite quickly)The wavefront bufferbloat test uses the cloudflare infrastructure, so this lkikely is just a local cloudflare CDN node that is serving you, but for me this is likely different.
While it is possible that there is a bug in that commit (in spite of my testing) I doubt that @niltonvasques did manually install a hot-of-the-press sqm-scripts version manually... That comnit does not fix a real bug, but changes a default (and fixes the set defaults for cake machinery).
Hello @moeller0, here is the tcp statistics using tcptrace chart. I'm not sure how to evaluate this data, but the desktop has more sequence numbers and mobile finalizes the stream sooner, probably because of the greater bandwidth achieved. (BTW there are several streams from cloudflare host, but all looks very similar)
Mobile tcp stream
Desktop Tcp stream
@Dopam-IT_1987, regarding the sqm-script version, this is the one that I'm using:
Package: sqm-scripts
Version: 1.6.0-1
Depends: libc, tc, kmod-sched-cake, kmod-ifb, iptables, iptables-mod-ipopt
Status: install ok installed
Architecture: all
Conffiles:
/etc/config/sqm e250c66bc56921b788e1db345897e1fc57cbff588ad368a7f3afc327489d5d49
/etc/sqm/sqm.conf 9f27a5fa7813cc1f7ca299851274dce139e373e1d3451e12441c26e370520c1a
Installed-Time: 1708317147
I will perform the other test with tc -s qdisc later night.
But one thing that I noticed here: When I perform a waveform test from mobile, the ping 1.1.1.1 executed from the openwrt router do not spikes, but when I perform waveform tests from my desktop computer, the ping latency start to spikes. At least for me it indicates that somehow the traffic from the desktop generates bloats outside the desktop.