Internal Packet loss (first two hops)

I'm currently on RC4 and having a TON of issues. So I would wait.

In regards to packet loss. I would eliminate any router/switches/hubs etc from the lab.
You need hardline test from your internet connection directly into a functioning PC.
I've had packet loss before on DOCSIS and had to switch service providers and contact the FCC.

Goodluck.

I already simplified the connection and replaced every cable. It‘s a rented apartment so I can‘t do anything about the house wiring but we even had a technician measuring directly from the access point in our basement and within our apartment in order to eliminate the possibility that PL might only occur on this section of the line. My ISP measured the connection from their end several times; they told me that there was no packet loss visible for them and that they would know about any systematic problem since I‘m close - I guess on the same DSLAM - to business customers of them that rely on a connection without any packet loss.
Unfortunately, they are the only one offering a DSL connection here via FTTC (50/10 = maximum, the copper connection itself would offer 2/0,5).

Mmmh, let's try a more tricked out configuration here:

config queue 'eth1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option itarget 'auto'
        option etarget 'auto'
        option verbosity '5'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option qdisc_really_really_advanced '1'
        option eqdisc_opts 'nat dual-srchost memlimit 8mb'
        option linklayer 'ethernet'
        option linklayer_advanced '1'
        option tcMTU '2047'
        option tcTSIZE '128'
        option linklayer_adaptation_mechanism 'default'
        option debug_logging '1'
        option iqdisc_opts 'nat dual-dsthost ingress memlimit 8mb'
        option interface 'eth0.2'
        option tcMPU '88'
        option enabled '1'
        option overhead '34'
        option download '37500'
        option upload '10000'

This configures sqm/cake for per-internal-IP-fairness, giving each active host its fair share (but not more) of the available capacity, which often helps when multiple devices are active at the same time.

It also configures cake to allow a higher priority tier which could be used to isolate the gaming traffic fro the rest.

But on a 50/10 link with a 48.342/14.224 sync the shaper setting 37.5/10 might be too conservative. Could you run 3 speedtests over at waveform with SQM disabled and one with your old SQM setting, and post the links to the results here please. The goal is to use these to estimate robust shaper values for sqm.

These values indicate at least 7709 + 358 = 8067 lost packets in downstream direction, so your ISP's claim of no packet loss appears somewhat exaggerated... (mind you that is not a super high number, but clearly more than 'no packet loss', not clear that that is your issue)

I‘m at work right now but I‘ll post the results later this evening.
Just to make sure that I got it right:
A: 3 tests = 1) old sqm settings, 2) without sqm, 3) the sqm settings you posted

OR

B: 3 tests without sqm + 1 with the old settings?

This, I want us to figure out how much throughput you can get reliably without SQM. Since you seem to be based in Germany, you could also in addition run the on-line speedtest at breitbandmessung.de (or use their desktop app) to get a set of measurements at different times (well mostly around the time you actually would like to play your games).

EDIT:
I was confused by the lower rates even without SQM and restarted modem + router.
Now the results are a bit worse again but not what they used to be:

I also realized that my modem is able to handle QoS as well. Should I prefer limiting bandwith there or via OpenWrt?


Ok, so I measured the connection. First of all, I realized that the bufferbloat is much better now compared to when I made the switch to OpenWrt in order to use its SQM. Back then I had about +100 down, +50up, right now there is not much difference (idle + under load). I changed nothing about the setup so maybe there was a potential jamming/noise source (maybe something in another apartment) that has been eliminated since then.

  1. 37.5/10 sqm settings
    https://www.waveform.com/tools/bufferbloat?test-id=7c62ddbc-84df-4721-8942-fe1d912ecf11

  2. without sqm settings

Downstream seems to be a bit low compared to the potential bandwith (xDSL settings) but apart from that, the line looks much better than it did a couple of months until a couple of weeks ago (when I tested the last time and still witnessed much more bufferbloat). Should I nethertheless apply the sqm settings you listed above or rather run another gaming + monitoring run without sqm settings?

I agree that does not look to bad even without SQM.

Yes, typically there is a hierarchy of reasons for that:
a) the DSL sync speed is gross speed while speedtests report net throughput:
for your 48.342/14.224 sync you could measure at best around (assuming IPv4):
48.342 * (64/65) * ((1500-8-20-20)/(1500+26)) = 45.29 Mbps
14.224 * (64/65) * ((1500-8-20-20)/(1500+26)) = 13.33 Mbps
b) most ISPs employ additional traffic shapers as DSLAMs are pretty much dumb switches with bad head of line blocking, so using the sync to throttle users to the contracted rates is pretty terrible.
c) peering between your ISP and the network where the speedtest servers live.

The measured speed is the minimum of the three...

I would very much try these out, yes.

So it looks like even though I put in the new settings, sqm is not active - at least the bandwith is not reduced and there's as much bufferbloat as there is without qos/sqm:

root@OpenWrt:~# cat /etc/config/sqm
config queue 'eth1'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option itarget 'auto'
        option etarget 'auto'
        option verbosity '5'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option qdisc_really_really_advanced '1'
        option eqdisc_opts 'nat dual-srchost memlimit 8mb'
        option linklayer 'ethernet'
        option linklayer_advanced '1'
        option tcMTU '2047'
        option tcTSIZE '128'
        option linklayer_adaptation_mechanism 'default'
        option debug_logging '1'
        option iqdisc_opts 'nat dual-dsthost ingress memlimit 8mb'
        option interface 'eth0.2'
        option tcMPU '88'
        option enabled '1'
        option overhead '34'
        option download '37500'
        option upload '10000'

What does tc -s qdisc return?

After editing the file you need to stop and start sqm like this:

/etc/init.d/sqm stop ; /etc/init.d/sqm start

Aah, this did the trick, thank you! I unchecked/checked via the LuCi interface but it looks like sqm was still inactive afterwards.

Now I got this result: https://www.waveform.com/tools/bufferbloat?test-id=b4141623-8e41-4850-b154-fdb45647958f

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 10519919329 bytes 12590524 pkt (dropped 0, overlimits 0 requeues 258)
 backlog 0b 0p requeues 258
  maxpacket 1514 drop_overlimit 0 new_flow_count 240 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800d: dev eth0.2 root refcnt 2 bandwidth 10Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 88 memlimit 8Mb
 Sent 1095137 bytes 8785 pkt (dropped 3, overlimits 936 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 132Kb of 8Mb
 capacity estimate: 10Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       88 /    1534
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        625Kbit       10Mbit     2500Kbit
  target         29.1ms          5ms       7.29ms
  interval        124ms        100ms        102ms
  pk_delay          0us         65us         20us
  av_delay          0us         17us          1us
  sp_delay          0us         11us          1us
  backlog            0b           0b           0b
  pkts                0         8768           20
  bytes               0      1094223         3876
  way_inds            0            0            0
  way_miss            0          302            6
  way_cols            0            0            0
  drops               0            3            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            2
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514          545
  quantum           300          305          300

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
 Sent 18396986 bytes 13838 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800e: dev ifb4eth0.2 root refcnt 2 bandwidth 37500Kbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 88 memlimit 8Mb
 Sent 18526578 bytes 13794 pkt (dropped 22, overlimits 24295 requeues 0)
 backlog 33308b 22p requeues 0
 memory used: 186Kb of 8Mb
 capacity estimate: 37500Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       88 /    1534
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       2343Kbit    37500Kbit     9375Kbit
  target         7.75ms          5ms          5ms
  interval        103ms        100ms        100ms
  pk_delay          0us       7.76ms        216us
  av_delay          0us       6.49ms          7us
  sp_delay          0us       4.99ms          7us
  backlog            0b       33308b           0b
  pkts                0        13817           21
  bytes               0     18589288         1430
  way_inds            0           13            0
  way_miss            0          303            2
  way_cols            0            0            0
  drops               0           22            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514           70
  quantum           300         1144          300

Edit:

This was after 5 mins of just running around within the training area (still on a game server):

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 10698988156 bytes 12838172 pkt (dropped 0, overlimits 0 requeues 258)
 backlog 0b 0p requeues 258
  maxpacket 1514 drop_overlimit 0 new_flow_count 240 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800d: dev eth0.2 root refcnt 2 bandwidth 10Mbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 88 memlimit 8Mb
 Sent 46712528 bytes 130140 pkt (dropped 39, overlimits 45567 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 132Kb of 8Mb
 capacity estimate: 10Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       88 /    1534
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh        625Kbit       10Mbit     2500Kbit
  target         29.1ms          5ms       7.29ms
  interval        124ms        100ms        102ms
  pk_delay          0us        949us        520us
  av_delay          0us         94us         16us
  sp_delay          0us         13us         11us
  backlog            0b           0b           0b
  pkts                0       130120           59
  bytes               0     46753579        12915
  way_inds            0           79            0
  way_miss            0         1484            9
  way_cols            0            0            0
  drops               0           39            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            3
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514          574
  quantum           300          305          300

qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
 Sent 151984228 bytes 141328 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800e: dev ifb4eth0.2 root refcnt 2 bandwidth 37500Kbit diffserv3 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 34 mpu 88 memlimit 8Mb
 Sent 152032820 bytes 140049 pkt (dropped 1279, overlimits 192150 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 274Kb of 8Mb
 capacity estimate: 37500Kbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       88 /    1534
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       2343Kbit    37500Kbit     9375Kbit
  target         7.75ms          5ms          5ms
  interval        103ms        100ms        100ms
  pk_delay          4us       2.36ms        210us
  av_delay          0us        1.2ms         30us
  sp_delay          0us         12us         10us
  backlog            0b           0b           0b
  pkts                1       141096          231
  bytes              60    153946580        16180
  way_inds            0           33            0
  way_miss            1         1481            5
  way_cols            0            0            0
  drops               0         1279            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            1            7            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len            60         1514          220
  quantum           300         1144          300

root@OpenWrt:~#

corresponding xDSL stats:

Previous 15 minutes time = 15 min 0 sec
FEC:		1487		0
CRC:		4		0
ES:		2		0
SES:		0		0
UAS:		0		0
LOS:		0		0
LOF:		0		0
LOM:		0		0

Mmmh that are a lot of drops for 5 minutes. How are playing the game. On a local computer or via streaming like gforce now? Normal gaming traffic should be so narrow that no packets get dropped. And if nothing else is connected/active in your network it is hard to see why you accumulate > 1000 drops in 5 minutes.

I'm playing on a local computer, connected via LAN. nothing else was running in the background, wifi was turned off as well. I don't understand why these drops are not visible in the xDSL stats.

Well in ingress/download direction packets first need to successfully cross the dsl link before sqm/cake can drop them... I am mors puzzled why cake should drop ~1000 packets in 5 minutes with only light gaming traffic...

I have no idea :(. I made sure that there is nothing else installed apart from firefox, valorant and neccessary drivers so that I could make sure that I'm not factoring in 'external issues'.

Is windows updating in the background

No, I installed every available update when I set up the pc and then paused updates. OpenWrt‘s realtime monitoring showed usage of about 15-20 down and 3-5up which I think should be fine, even with sqm on

If we are talking about Mbps, that seems quite a lot for simple gaming traffic. What game are yo mainly playing and are your streaming via OBS and/or chatting with others via discord while gaming?

I tested only valorant and had no other application running except for firefox (monitoring, no additional tabs, no plugins whatsoever ). the pc was the only device that was connected via LAN, wifi was off.

What unit was the realtime monitor showing Mbps or Kbps? Again 15-20 Mbps would be quite a lot...

Maybe it is time to collect a packet capture on the router and look with wireshark what is eating the bandwidth?