Linux + Cake SQM

[moeller0]

@warchief

It is fixed now but that output I am not sure if it is right because I don't see any info which would benefit here.

That looks massively broken. How did you log into your router in the first place?
here is the output I get from mtr -z 172.217.2.4 (copied from the terminal windw running the ssh session to my openwrt router, but if you have a mac or linux machine in your network you do not need to run mtr on the router itself):

router (AA.BB.CC.DD)                                                                                 2019-12-08T22:35:07+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                           Packets               Pings
 Host                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS6805   loopback1.0003.acln.06.ham.de.net.telefonica.de                              0.0%     6    9.1   9.0   8.9   9.1   0.1
 2. AS6805   bundle-ether10.0002.dbrx.06.ham.de.net.telefonica.de                         0.0%     6    9.2   9.1   8.9   9.2   0.1
 3. AS6805   bundle-ether2.0001.prrx.06.ham.de.net.telefonica.de                          0.0%     6    8.9   9.1   8.9   9.4   0.2
 4. AS15169  72.14.208.60                                                                 0.0%     6    9.4   9.3   8.8  10.1   0.5
 5. AS15169  108.170.253.84                                                               0.0%     6   11.4  12.7   9.1  21.3   4.9
 6. AS15169  108.170.225.178                                                              0.0%     6    9.9  10.2   9.9  10.7   0.3
 7. AS15169  172.253.50.214                                                               0.0%     6   15.2  15.1  14.4  15.7   0.5
 8. AS15169  172.253.66.16                                                                0.0%     6   18.1  17.8  17.5  18.1   0.2
 9. AS15169  209.85.142.167                                                               0.0%     6   23.3  23.3  23.1  23.8   0.3
10. AS15169  172.253.65.176                                                               0.0%     5   91.0  94.4  90.7 108.8   8.0
11. AS15169  216.239.58.254                                                               0.0%     5  106.7 106.9 106.7 107.5   0.4
12. AS15169  72.14.232.70                                                                 0.0%     5  116.0 116.1 116.0 116.3   0.1
13. AS15169  209.85.251.139                                                               0.0%     5  125.5 126.0 125.5 127.1   0.7
14. AS15169  172.253.51.119                                                               0.0%     5  125.8 125.8 125.2 126.7   0.6
15. AS15169  108.170.254.81                                                               0.0%     5  126.3 126.2 126.0 126.4   0.1
16. AS15169  72.14.238.163                                                                0.0%     5  125.1 124.8 124.7 125.1   0.2
17. AS15169  lga15s45-in-f4.1e100.net                                                     0.0%     5  124.9 125.0 124.8 125.3   0.2

You can nicely see the packet loss per hop (typically as long as the end host has no or very little packetloss you can ignore that) as well as current, average, best and worst RTT (as well as standard deviation), often if a network path is congested the RTT will show some sort of a step function, with reasonably low RTTs (low compared to the path length, light in GF gives ~100Km distance per ms RTT) up to a point and increased RTTs to all points after that. Note that is a sign of congestion, but not a precise localization of the place where the congestion happens (see https://archive.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf for the challenges in properly interpreting traceroute/mtr output), if however you have traceroutes/mtr output in both directions, so from your end to the server and from (close to) the server to your router's IP address than you can even localize the congested hop with some confidence (it s still error prone, but you at least have a chance).

@warchief

For the VPS/VPN. It would take some additional digging to see if that would benefit the line but as far as I know, as from what I have seen it should not be an issue.

This is only a work-around if the normal network path from your ISP to the game servers is congested and introduces unwanted delay, if you have no indication for that an VPN/VPS is not going to help much.

@warchief

But like I said the only problem with the RTT is I live ~750 miles from the game server which increases my ping by 30ms

THe rule of thumb is 1ms RTT allows for 100Km of fiber, so 750 miles -> 1,207.008 Km or 12 ms propagation delay, add a few active components and a few detours, 30 ms sounds reasonable, if you get that reliably with low variation/jitter, than digging to deep into traceroute might be a waste of time (except it is a skill that can come in handy when you have network issues).

@warchief

I will be building a server to run OpenWRT on and that should be sufficient to shape and optimize the line.

Assuming there are things on your line that can be improved noticeably (you seem to have done a thorough job with the modem and ISP already).

@warchief

For the tc output this is the traffic control output so I assume the actual network config and shows what it is coded to do and how it is doing it i.e diffserv 3/4/8 tiers

Yes cake reports its configuration pretty complete:

qdisc cake 802a: dev eth0 root refcnt 2 bandwidth 30Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 27

So in your case it is the default diffserv3 "profile" and hence cake also gives you three detal columns for the three priority tins, called Bulk, Best Effort and Voice. If you switch to diffserv4 you get four columns (and 8 for diffserv8).

@warchief

So I think the main thing is the drops but that could be due to anything in the line I assume it is from my DSLAM to me.

No, drops is expected, this is how cake tells individual flows to slow down if the capacity is exhausted (if a flow uses ECN it will see less drops and more marks). So drops are a-OK, just too many drops can be suspicious.

@warchief

So I am getting everything in the BE tin even though I have an AP set up so it should not be dumping there it should be dumping in the Bulk tin(iphones/firestick) But the main thing is my gaming rig is dumping there so not an issue ATM.

Well, on the egress side "qdisc cake 802a: dev eth0" you have a few packets in the voice bin. Not sure about what you expect.

@warchief

ECN, triple isolate, latency target - should I care about those or can I just default and they are fine.

ECN:
Cake does ECN by default, but ECN needs to be negotiated by both end-points of a flow, so if you want it used make sure to configure your end-host appropriately (and the same goes if you want to be sure it is not used).

triple isolate:
IMHO configuring nat dual-dsthost ingress for the ingress traffic and nat dual-srchost for the egress traffic is more predictable (as that should give each computer/IP-address in your home network an equal share of the available bandwidth (but dynamically, only active hosts are counted here, so a single active host will get 100% of the capacity), triple isolate will do something similar (especially if there is a decent mix of active internal and external IP addresses) but without the simple predictability of the dual-xxxhost modes (then again triple isolate does not need to treat ingress and egress differently, this is why triple is the default).
latency target: you can not even directly configure that in cake, it is typically set to 5% of interval (and you can configure interval) interval should be in the right order of magnitude of your typical RTTs, the default 100ms works quite well for many uses and RTTs from 10-200ms or so (there is no hard cut-off).

@warchief

I would like to see the skeleton of the system, how do I see the configuration of the network, like the actual coding, is there a way I can pull that from ssh?

No idea, except for the fact that cake reports its effective configuration in a way you could use as part of a cake invocation:

qdisc cake 8028: dev eth1 root refcnt 2 bandwidth 6Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 27

tc qdisc add root cake bandwidth 6Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 27

You can basically copy and paste the keywords from the tc -s qdisc output to instantiate cake with the same options.

@warchief

I ask because it would make it easier if I screw something up or run into a problem when I begin to code the new system.

Make backups and if possible keep a spare router already configured as a cold-spare so you can avoid downtime if something goes pear-shaped?

[warchief]

I am mtr -z 52.53.224.41 and it is a UDP stream, I will look at some other connections and see if I can get a better reading.

I am looking in through PuTTY as root, I also tried the debian liveboot ssh but same result maybe it is the UDP stream screwing things up. My MTR was downloaded via ssh and through LuCI, it says it is up to date, I think we are seeing 0% packet loss. As a matter of fact it doesn't seem as if I am losing packets or have any stuttering.

The concern is and this is how I first found the issue. I was watching youtube of a gaming stream, now I immediately noticed that stream, which is like video playback and includes any video playback such as ABC, FOX, NBC type of video feed same thing you would see on the news - it seemed like my game window wasn't running at that same bitrate/speed.

Now the streamers game, and it goes for most streams out there - they had no issues with the streaming(uploading) the game and the game connection itself, it ran like any television video feed would run. Now mine, I cannot stream and game at the same time due to speed limits where it takes about 7.5 mb egress to stream and 7.5 egress to game you know confidently. I can only pull 10 here on this vdsl line so I can only do one at a time.

So my game window runs like its choked right? a little buffered or at a slower line rate than say something like monday night football. Now before we start considering video rate lets just assume everything is able to stream at the rate of monday night football and monday night football runs at 200 frames per second. Okay so, my line rate which is apparent in my game window full screen or not fully capable PC fully capable x86 with cake on it feels like it only attains 90% and it feels like the other 10% is just bleeding out somewhere. Now I don't know if it is egress or ingress but I feel like egress is just fire and forget with the DSCP marking, now I feel like it is the ingress side of things and I have some type of iptables or other OpenWRT config issue(pppoe?) which is limiting the video rate causing frames per second slowdowns.

##FIREWALL
iptables -A FORWARD -m conntrack --ctstate NEW,RELATED,ESTABLISHED -j ACCEPT

##GAMINGEGRESS
iptables -A FORWARD -s 192.168.1.2
iptables -A PREROUTING -s 192.168.1.2 -j DSCP --set-dscp-class CS6

##GAMINGINGRESS
iptables -A FORWARD -d 192.168.1.2
iptables -A PREROUTING -d 192.168.1.2 -j DSCP --set-dscp-class CS6

Now as long as I have the correct routing table for the iptables rules then that is ruled out, there shouldn't have to be any changes in the OpenWRT default firewall just the scripts that I added so that is the firewall+scripts, there is not a hardware issue on my side, from what I gather the VDSL should be capable of the same video rate as any other connection up to the speeds we are talking about:

1514 kb max game packet / peaks of 700 kb but 90% is under 150 kb.

I am measuring less than 3 ms of bufferbloat on both egress and ingress I am happy with the sqm config. I am missing the berkeley netalyzer for more in depth network analyzing but even then it did not show the spikes which dslreports does.

@moeller0

you seem to have done a thorough job

Yes it is becoming a money pit. In a sense I am building a formula 1 car to drive daily here. But the issue is the car I am building can do 200 mph, it is only going 40 mph because theres a problem to fix.

@moeller0

the default diffserv3

I am attempting to put all network traffic into Bulk, and my gaming rig into the best effort. Maybe I am missing a iptables rule?

iptables -A -I eth+ -j DSCP --set-dscp 0

Perhaps? But I still have DNS important router connections and such and I don't know if that is a problem so I guess it would be best to keep everything best effort just keep the network egress congestion free(no torrenting) but there have never been any torrents or any heavy traffic besides youtube/netflix going on while I game and that does not create an issue.

At the end of it all, I have a single UDP stream from fortnite's game server transmitting less than 10mb for every game session no matter how long it lasts, and it performs poorly compared to worse set ups. It is not a PC issue, not a router issue, not a modem issue, not a cable issue. So the issue is between the game server and my DSLAM......

nat dual-dsthost ingress
eth0(lan) > Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully

nat dual-srchost
eth1(wan/wan6) > Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully

So that is how I have it set up, but openwrt recommends it should be set up differently when you use 2 sqms(something about it being flipped) if I should flip it let me know.

Again, should the cake sqm be on eth1 (wan,wan6) or pppoe-wan(wan,wan6)

@moeller0

if something goes pear-shaped

Well since I have 2 x86 set ups I don't think it is possible to brick these - it runs off of a samsung ssd, but I also have a usb bootable with the same kernel on it. And when I do begin to code from scratch I will just keep stable configs(as you said) or just wipe it and start over.

EDIT:

UDP Stream

mtr 2

TCP Stream same game

mtr 3

[moeller0]

Now I am retry confused. Let's take a step back and try again from the foundations, as otherwise I am lost.

You managed to install cake/sqm and configure it to your desire, great. Could you please post the output of:

cat /etc/config/sqm
ifstatus wan
tc -s qdisc

again, so I know where we are standing please?

Also, please post a link to a dslteports Speedtest with sqm disabled and one with the current sqm configuration enabled.

Thanks.

[moeller0]

@warchief

I am mtr -z 52.53.224.41 and it is a UDP stream, I will look at some other connections and see if I can get a better reading.

So could you try to copy and paste as text from the putty terminal window running mtr instead of taking a scree-shoot, please?

Looking at the delays if that mtr in the ~400ms range starting at around hop 7, I wonder what kind of host 52.53.224.41 is, and where it is located?

@warchief

My MTR was downloaded via ssh and through LuCI, it says it is up to date, I think we are seeing 0% packet loss.

That is quite curious, as OpenWrt master is at version 0.92 for mtr, which OpenWrt version are you running?

@warchief

The concern is and this is how I first found the issue.

Which issue exactly?

@warchief

I was watching youtube of a gaming stream, now I immediately noticed that stream, which is like video playback and includes any video playback such as ABC, FOX, NBC type of video feed same thing you would see on the news - it seemed like my game window wasn't running at that same bitrate/speed.

Does this mean that your game got disturbed by concurrent video streaming? If yes, were the video streams received on the same computer the game was played on or on another computer (if the later per-internal-IP-isolation might help)

@warchief

Now the streamers game,

Most games only send a relatively small amount of data upstream (FPS type games often send at considerably less than 1Mbps). But if you try to stream a video of yourself playing (like twitch) than this changes and you will put a considerable load on your egress link.

@warchief

Now mine, I cannot stream and game at the same time due to speed limits where it takes about 7.5 mb egress to stream and 7.5 egress to game you know confidently.

7.5 Mbps for streaming sounds about right, but the same 7.5 Mbps for game state update traffic seems quite excessive. BUT you realize that you throttled your uplink to 6 Mbps with your sqm configuration:
qdisc cake 8028: dev eth1 root refcnt 2 bandwidth 6Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 27
That is not going to fly if you need to send at 7.5 Mbps.... (according to the luci-app-sqm screenshot below eth1 is carrying your WAN egress traffic). Also note that:

                   Bulk  Best Effort        Voice
  thresh        375Kbit        6Mbit     1500Kbit

means that high priority traffic is capped at 1.5 Mbps (the overflow will end up in Best Effort) so most of your game streaming packets will not be treated to high priority...

@warchief

I can only pull 10 here on this vdsl line so I can only do one at a time.

But that is what you should do, try to max out your upload (without overloading it ) shaping a 10 Mbps link down to 6Mbps certainly does not help if you want to send at high rates while playing.

@warchief

So my game window runs like its choked right? a little buffered or at a slower line rate than say something like monday night football.

I have no idea what you are talking about here. Are you telling my your ISP also sends IPTV over your internet access link so that the 30/10 is shared with TV traffic? Because for cake to be effective it needs control over all traffic on your internet access link. So neither TV traffic nor wifi traffic can be allowed to directly connect to the internet without having their packets controlled by cake.

@warchief

ow before we start considering video rate lets just assume everything is able to stream at the rate of monday night football and monday night football runs at 200 frames per second.

I am pretty confident it does not run at 200 Hz, it runs at 60 Hz if you are lucky, but I digress.

@warchief

Now I don't know if it is egress or ingress but I feel like egress is just fire and forget with the DSCP marking,

It is time to investigate this instead of relaying on feeling.

You could opkg update ; opkg install iftop and then iftop -i pppoe-wan to look at the instantaneous ingress and egress traffic while playing/streaming. That should allow you to figure out whether your ingress, your egress or both are overloaded.

@warchief

I am measuring less than 3 ms of bufferbloat on both egress and ingress I am happy with the sqm config.

Great!

@warchief

But the issue is the car I am building can do 200 mph, it is only going 40 mph because theres a problem to fix.

Your issue might be that a formula one racer is not built for dirt road racing.

@warchief

I am attempting to put all network traffic into Bulk, and my gaming rig into the best effort. Maybe I am missing a iptables rule?

That seems pretty extreme, as under congestion Bulk only gets 375Kbps on egress....

@warchief

but there have never been any torrents or any heavy traffic besides youtube/netflix going on while I game and that does not create an issue.

Youtube/netflix typically put load on your ingress, not the egress.

@warchief

At the end of it all, I have a single UDP stream from fortnite's game server transmitting less than 10mb for every game session no matter how long it lasts, and it performs poorly compared to worse set ups.

What are you comparing with here exactly? Friends and neighbors with the same ISP close to where you live or people in other parts of the US?

@warchief

So that is how I have it set up, but openwrt recommends it should be set up differently when you use 2 sqms(something about it being flipped) if I should flip it let me know.

I believe the flipping is in regards to Upload and Download speed and only fully effects you if you instantiate sqm's normal ingress instance via IFB, with your one egress instance on each of the WAN and LAN interface will not have that issue. So that config looks about right.

@warchief

Again, should the cake sqm be on eth1 (wan,wan6) or pppoe-wan(wan,wan6)

On a pppoe link I typically recommend to instantiate it on pppoe-wan.

Regarding the mtr movies, it probably is enough to let mtr run for say 100 probes and only copy and paste the final outcome, or use mtr -z 52.53.224.41 -n -c 100 -w to get the static output. (running mtr on-line can be helpful to diagnose transient latency spikes, but for that you need to be able to look a what haapens locally in your network, so sharing videos of that is not going to be too helpful).

[warchief]

@moeller0

OpenWrt version are you running?

X86 version - Powered by LuCI openwrt-18.06 branch (git-19.170.32094-4d6d8bc) / OpenWrt 18.06.4 r7808-ef686b7292

SQM pppoe-wan On, avg 2ms bufferbloat ingress+egress

SQM pppoe-wan Off avg 5ms bufferbloat ingress, 300+ms egress

Switched sqm values to 50MB ingress, 10MB Egress, single sqm on pppoe-wan

root@OpenWrt:~# cat /etc/config/sqm

config queue 'eth1'
        option debug_logging '0'
        option verbosity '5'
        option qdisc 'cake'
        option qdisc_advanced '1'
        option qdisc_really_really_advanced '1'
        option linklayer 'ethernet'
        option squash_dscp '1'
        option squash_ingress '1'
        option script 'layer_cake.qos'
        option egress_ecn 'NOECN'
        option overhead '27'
        option ingress_ecn 'NOECN'
        option itarget 'default'
        option etarget 'default'
        option eqdisc_opts 'nat dual-srchost'
        option iqdisc_opts 'nat dual-dsthost ingress'
        option interface 'pppoe-wan'
        option download '50000'
        option upload '10000'
        option enabled '1'

config queue
        option debug_logging '0'
        option verbosity '5'
        option squash_dscp '1'
        option squash_ingress '1'
        option interface 'eth0'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option qdisc_really_really_advanced '1'
        option linklayer 'ethernet'
        option overhead '27'
        option ingress_ecn 'NOECN'
        option egress_ecn 'ECN'
        option download '0'
        option upload '30000'
        option itarget 'default'
        option etarget 'default'
        option eqdisc_opts 'nat dual-dsthost ingress'
        option enabled '0'

root@OpenWrt:~# ifstatus wan
{
        "up": true,
        "pending": false,
        "available": true,
        "autostart": true,
        "dynamic": false,
        "uptime": 99816,
        "l3_device": "pppoe-wan",
        "proto": "pppoe",
        "device": "eth1",
        "updated": [
                "addresses",
                "routes"
        ],
        "metric": 0,
        "dns_metric": 0,
        "delegation": true,
        "ipv4-address": [
                {
                        "address": "68.69.74.73",
                        "mask": 32,
                        "ptpaddress": "64.251.185.10"
                }
        ],
        "ipv6-address": [
                {
                        "address": "fe80::2d6f:4ffd:8188:74f8",
                        "mask": 128
                }
        ],
        "ipv6-prefix": [

        ],
        "ipv6-prefix-assignment": [

        ],
        "route": [
                {
                        "target": "0.0.0.0",
                        "mask": 0,
                        "nexthop": "64.251.185.10",
                        "source": "0.0.0.0\/0"
                }
        ],
        "dns-server": [
                "8.8.8.8",
                "8.8.4.4"
        ],
        "dns-search": [

        ],
        "inactive": {
                "ipv4-address": [

                ],
                "ipv6-address": [

                ],
                "route": [

                ],
                "dns-server": [
                        "64.251.176.13",
                        "64.251.173.49"
                ],
                "dns-search": [

                ]
        },
        "data": {

        }
}
root@OpenWrt:~# tc -s qdisc

@moeller0

Looking at the delays if that mtr in the ~400ms range starting at around hop 7, I wonder what kind of host 52.53.224.41 is, and where it is located

location san jose,cali, but I do not think my results are coming out correct. That is what I was saying - there is a delay.

@moeller0

if you need to send at 7.5 Mbps

The ingress link needs to send 1080p video right, to download the packets, which are sent through wireshark at ~1514 kb packets.
the egress link 90% are less than 120 kb but can avg 700 kb, and max at 1514 kb but not more than 1514 kb for the udp egress game packets as seen in wireshark

In general I think the games are optimized to send 1 packet at a time confirming your [quote="moeller0, post:18, topic:49694"] it runs at 60 Hz [/quote]

As most, atleast fortnites server is tickrate 64 or something.

1920x1080 144hz in game(dirt racing) and 1080p video stream on youtube(formula 1) do NOT look identical

Now the frames aren't really the issue because in either case, the hardware I have and the link I have is capable of:
1920x1080 144hz egress(requires ~1.5 mb egress)
1920x1080 144hz ingress(requires ~1.5 mb ingress)

Or

1080p egress streaming(requires 7.5 mb egress)

but as stated before not at the same time, in theory I have enough but I am not comfortable with the numbers that close to my 10 MB max.

@moeller0

your ISP also sends IPTV

Now this I am not sure - the cable tv used is just that, cable tv, they had issues and pulled all the zyxel DOCSIS cable modems, probably due to this very issue. The problem is games end-to-end slow just a bit, enough so that it doesn't stream 100%. Not that it is jittery, stuttery, major packet loss.
The games, just the games, some type of tcp/udp issue where it is just choked right? But pure 1080p video from youtube? streams fine zero issues. 1080p video ingress and egress/ingress for gaming? not the same, We see some throtteling here which I thought was the VDSL copper line to DSLAM distance but I sense it is in the network(openwrt) config.

@moeller0

most of your game streaming packets will not be treated to high priority...

What code or mod to solve this for 7.5 MB limit.

@moeller0

iftop -i pppoe-wan

2019 12 09 20 57 39

I installed but it looked too complicated and I do not know if this is the same

@moeller0

That seems pretty extreme, as under congestion Bulk only gets 375Kbps on egress....

I basically gave up trying to set up the diffserv tiers and just send all traffic based on my rigs ip to best effort and do not configure anything else with the 3 iptables rules. this seems to be the least computationally taxing and simplest way to issue a rule for a single rig.

@moeller0

Friends and neighbors with the same ISP close to where you live or people in other parts of the US

generally people with the best connections, the best pings, like I said I do not the believe packet delivery to be a link issue but a coding issue regardless of location within reasonable limits i.e chile.

@moeller0

mtr -z 52.53.224.41 -n -c 100 -w

HOST: OpenWrt Loss% Snt Last Avg Best Wrst StDev

1. AS6580 64.251.185.10 0.0% 100 11.9 11.9 11.1 13.8 0.2
2. AS6580 64.251.187.186 0.0% 100 11.8 12.4 11.7 16.9 0.4
3. AS6580 64.251.163.197 0.0% 100 21.7 21.8 20.9 30.0 1.0
4. AS3356 4.34.41.189 83.0% 100 38.2 38.2 37.7 38.9 0.0
5. AS3356 4.69.203.169 94.0% 100 15261 14600 14002 15261 607.3
6. AS3356 4.16.168.34 0.0% 100 66.1 66.4 65.4 71.1 0.7
   @Not a TXT record
7. AS??? 52.95.54.224 0.0% 100 76.2 77.5 75.9 89.8 1.8
   @Not a TXT record
8. AS??? 52.95.55.141 0.0% 100 75.2 75.4 74.3 86.0 1.4
   @Not a TXT record
9. AS??? 52.95.55.120 0.0% 100 73.3 73.6 72.4 85.1 1.3
   @Not a TXT record
10. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
    @Not a TXT record
11. AS??? 54.239.44.34 0.0% 100 67.2 68.6 66.8 85.2 2.8
    @Not a TXT record
12. AS??? 52.93.131.71 0.0% 100 73.9 74.5 72.8 87.6 2.4
    @Not a TXT record
13. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0
    @Not a TXT record
14. AS??? 54.240.243.20 0.0% 100 74.0 75.1 72.8 97.9 3.5
    @Not a TXT record
15. AS??? 52.93.141.132 0.0% 100 73.0 74.3 72.0 100.0 4.2
    @Not a TXT record
16. AS??? 52.93.141.137 0.0% 100 73.9 75.4 73.6 91.8 2.7
    @Not a TXT record
17. AS??? 54.240.243.67 0.0% 100 75.9 75.8 74.6 81.4 0.9
    @Not a TXT record
18. AS??? 52.93.47.80 0.0% 100 73.4 73.8 72.9 85.5 1.3
    @Not a TXT record
19. AS??? ??? 100.0 100 0.0 0.0 0.0 0.0 0.0

[moeller0]

@warchief

X86 version - Powered by LuCI openwrt-18.06 branch (git-19.170.32094-4d6d8bc) / OpenWrt 18.06.4 r7808-ef686b7292

Okay, probably okay to stick to 18.06 while 19.07 goes through the RC stages, but an update is in order soon.

@warchief

SQM pppoe-wan On, avg 2ms bufferbloat ingress+egress

So what was the /etc/config/sqm for this test? And did you test over wifi or over a wired connection?

Also, please have a look at https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803/19 for how to best configure the dslreport speedtest (especially the comments about high-res bufferbloat probes and increasing the measureent duration after a free registration at dslreports)

@warchief

Switched sqm values to 50MB ingress, 10MB Egress, single sqm on pppoe-wan

That seems dubious from the SQM off test. My rule of thumb is that the gross shaper settings for sqm should be (for the first iteration) pretty much the maximal net-goodput rates you get from the ty[ical speedtests, and the dslreports speedtest does not seem to give you 50/10. IN your case, assuming the dslreports speedtes is reliable 35400/8500 looks like decent starting numbers for sqm, but what results do you get from ookla's speedtest (www.speedtest.net) of netflix's fast.com?

@warchief

root@OpenWrt:~# cat /etc/config/sqm

That looks about right, as does:

@warchief

"l3_device": "pppoe-wan",
"proto": "pppoe",
"device": "eth1",

@warchief

location san jose,cali, but I do not think my results are coming out correct.

Why do you believe the measurements to be off? I agree that ir probably does not reflect pure propagation delay for the distance, but to me it rather indicates congestion somewhere along the line, the challenge now is to figure out where....

@warchief

The ingress link needs to send 1080p video right,

In the ingress, I would say you receive video streams from say netflix, amazon, apple. We should reserve the word send for data you push into the internet (and sure almost all receive connections will also cause some send traffic, but for video streaming that typically is relative low bandwidth).

@warchief

~1514 kb

Nope, typical packets are 1514 Bytes, not Kilo and not bits, jumbo frames can go up to 9KB though, but typically not outside of a carefully managed and configured local network.

@warchief

1920x1080 144hz in game(dirt racing) and 1080p video stream on youtube(formula 1) do NOT look identical

Sure, as far as I can tell youtube currently oly streams at up to 60Hz....

@warchief

1080p egress streaming(requires 7.5 mb egress)

Do you actually use egress streaming?

@warchief

Now this I am not sure - the cable tv used is just that, cable tv,

Fair enough, so you get your cable TV from a different supplier with its own coaxial cabling (and not some weird contraption, where there is a "cable TV" box that connects via coax cable to your TV and vie ethernet to your router)?

@warchief

The problem is games end-to-end slow just a bit, enough so that it doesn't stream 100%. Not that it is jittery, stuttery, major packet loss.

That might indicate some component being maxed out and limiting your game that way, might not even be on the network side.

@warchief

The games, just the games, some type of tcp/udp issue where it is just choked right?

No idea.

@warchief

But pure 1080p video from youtube? streams fine zero issues.

Okay, but video streaming, in spite of the naming, typically is not sending data at a constant rate, but it rather binges and send a batch at maximal achievable rate and fills a buffer, and when the buffer gets emptied below a "watermark" it requests the next batch, so it has some tolerance against variable network conditions and RTT variance. You game does not have that luxury unless it accepts a relative high general delay/dejitter buffer (but that leads to less zippy game reactivity which players detest).

@warchief

What code or mod to solve this for 7.5 MB limit.

For that you will need to either build your own shaper hierarchy with HFSC or HTB or modify the source code of the cake module. For a link that supports ~8.4Mbps egress traffic pushing 7.5 into high priority seems rather unhealthy though. (per internal IP fairness is also not going to help you much, as we two active users each only gets >= 4.2 Mbps, too little for sending 7.5 Mbps). You might want to consider getting a dedicated DSL/cable link for your gaming purposes (or a higher upload link).

@warchief

iftop -i pppoe-wan

The idea is that you look at the iftop output to diagnose things, not that you post videos of that

@warchief

I installed but it looked too complicated and I do not know if this is the same

Nope collectd is a quite helpful tool which allows to periodically sample and store different parameters from your router and the generate fancy plots. It can be quite helpful in finding issues, but let's stick to the tools you already know how to use.

@warchief

I basically gave up trying to set up the diffserv tiers and just send all traffic based on my rigs ip to best effort and do not configure anything else with the 3 iptables rules. this seems to be the least computationally taxing and simplest way to issue a rule for a single rig.

So you disabled these override rules completely?

@warchief

mtr -z 52.53.224.41 -n -c 100 -w

This looks odd in that the final hop does not respond, but the RTTs of most intermediate hops look reasonable to me, at least more reasonable than >=400ms.

[Fil]

@jeff After applying this patch, the following errors appear, how should one proceed?

fil@Owrt-FIl:~/openwrt$ git am 1968.patch
Applying: firmware-utils: ptgen: add GPT support
Applying: grub2: split to grub2 and grub2-efi packages
Applying: x86: add EFI images and make iso images EFI bootable
error: patch failed: scripts/gen_image_generic.sh:30
error: scripts/gen_image_generic.sh: patch does not apply
error: patch failed: target/linux/x86/image/Makefile:43
error: target/linux/x86/image/Makefile: patch does not apply
Patch failed at 0003 x86: add EFI images and make iso images EFI bootable
Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

[jeff]

Looks like there is a conflict with where master has gone. You'd have to resolve the conflicts by editing the partial merge (you'll see the old/new choices in the combined file) and commit to your local branch.

[NixonUey]

And for those of us who have no idea how to do that... are we just SOL?

[NixonUey]

To answer my own question, yes I suppose we are. On that note I took a shot at sorting the two files out and was able to integrate them. I successfully compiled a build but my hardware hasn't arrived yet so I don't have a test bed. If there's a way to attach files to this thread I'd be happy to share.

[warchief]

Google drive? Github? Dropbox? I think it is excellent that you are taking this approach and expanding on it. Taking one idea and furthering our collective knowledge. That is how we make progress!

[NixonUey]

Caveats: I'm not a programmer. I have almost zero experience with GitHub. I have even less experience with Linux. The fact I was able to update these files and successfully build is a bigger surprise to me than to you.

With that out of the way, here we go!

1. Download the modified files from here (Google Drive Link) and save them in a convenient location. I put them in ~/1968updates.
2. Clone the OpenWRT git repository.
3. Create a branch. I called mine x86patch.
4. Pull in the 1968 pull request as a patch and apply it.
5. Replace the two bad files with the files from step 1.
6. Add the new files to git.
7. Continue the patch.
8. You are ready to build.

Yeah I admit that isn't very helpful so here are the commands I used:

> me@computer:~$ git clone https://git.openwrt.org/openwrt/openwrt.git ; cd openwrt
me@computer:~/openwrt$ git checkout -b x86patch
me@computer:~/openwrt$ wget https://github.com/openwrt/openwrt/pull/1968.patch
me@computer:~/openwrt$ git am 1968.patch
me@computer:~/openwrt$ cp ~/1968updates/gen_image_generic.sh ./scripts/gen_image_generic.sh
me@computer:~/openwrt$ cp ~/1968updates/Makefile ./target/linux/x86/image/Makefile
me@computer:~/openwrt$ git add scripts/gen_image_generic.sh
me@computer:~/openwrt$ git add target/linux/x86/image/Makefile
me@computer:~/openwrt$ git am --continue

Now you can continue with your regular build process as if you've just finished cloning the repository.

Let us never speak of this again.

[NixonUey]

@NixonUey

Caveats: I'm not a programmer. I have almost zero experience with GitHub. I have even less experience with Linux. The fact I was able to update these files and successfully build is a bigger surprise to me than to you....

OK that totally didn't work! And I can't edit the post so let's try again! This time with feeling!!*
Caveats: I'm not a programmer, use at your own risk, blah blah blah...

With that out of the way, here we go!

1. Specific Commands follow. If you use the same directory and branch names you can just copy-paste!
2. Download the modified files from here (link corrected) and save them in ~/1968updates.**
3. Clone the OpenWRT git repository.
4. Create a branch called x86patch.
   Here are the commands for steps 3 and 4:

git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git checkout -b x86patch

5. Pull in the 1968 pull request as a patch and apply it.
6. Replace the two bad files with the files from step 2.

wget https://github.com/openwrt/openwrt/pull/1968.patch
patch -p1 < 1968.patch
cp ~/1968updates/gen_image_generic.sh ./scripts/gen_image_generic.sh
cp ~/1968updates/Makefile ./target/linux/x86/image/Makefile

7. Add the new files to git.

git add config/Config-images.in
git add package/base-files/files/lib/upgrade/common.sh
git rm package/boot/grub2/Makefile
git add scripts/gen_image_generic.sh
git add target/linux/x86/base-files/lib/preinit/79_move_config
git add target/linux/x86/base-files/lib/upgrade/platform.sh
git add target/linux/x86/generic/config-4.14
git add target/linux/x86/generic/config-4.19
git add target/linux/x86/image/Makefile
git add target/linux/x86/image/grub-iso.cfg
git add tools/firmware-utils/Makefile
git add tools/firmware-utils/src/ptgen.c
git add package/boot/grub2/common.mk
git add package/boot/grub2/grub2-efi/
git add package/boot/grub2/grub2/

8. You are ready to build.

If you select Target System (x86), Subtarget (x86_64), and Target Profile (Generic x86/64) you'll get a new option under Target Images --->

[*] Build EFI images (Linux x86 or x86_64 host only)

If you see this you are in good shape, and when you compile you'll see efi images like

openwrt-x86-64-combined-ext4-efi.img.gz
openwrt-x86-64-combined-squashfs-efi.img.gz

Final caveats: I haven't had a chance to load this on hardware yet. It looks correct but I won't know for sure until the next day or two when I give it a go.

Footnotes:

• with feeling and more exclamation points.
  ** Sorry the link got messed up. But I don't think the files were quite right anyway. I spent a good chunk of my weekend sorting out what went wrong. I think I might have it now.

@warchief

[ldir]

I rebased, resoved conflicts and did some of the requested tidy ups in the PR, dumping it here https://git.openwrt.org/?p=openwrt/staging/ldir.git;a=shortlog;h=refs/heads/pr/1968

Completely untested. Maybe it will save some of the above work arounds for you.

[NixonUey]

(Talking to myself again...)

Well today I was able to compile a UEFI image and install it on a ODROID-H2! Looks like we are in business

[NixonUey]

@ldir

I rebased, resoved conflicts and did some of the requested tidy ups in the PR, dumping it here https://git.openwrt.org/?p=openwrt/staging/ldir.git;a=shortlog;h=refs/heads/pr/1968

Completely untested. Maybe it will save some of the above work arounds for you.

Oh sure, steal my thunder @ldir. A thunder thief is what you are! All jokes aside thank you for doing this. My workarounds are because I wouldn't know where to begin to do what you did. Now how do we get UEFI support formally tested and merged?

[ldir]

All I did was git-pr 1968 github to pull in the PR branch.
git rebase master to rebase that onto current master state, which produced the conflict warning.
edit the 2 conflicting files to resolve those conflicts (which were very straightforward) and fix up a couple of things that were mentioned in the PR whilst I was there. It's straightforward if you know how of course, a perplexing mystery if you don't.

I've nudged the PR, you never know. :-/

[warchief]

The simplest way to accomplish building a custom firewall is @dlakelan's approach: Retain LuCI and the OpenWRT Kernel. By removing iptables and implementing nftables the cake sqm is fully coded. Simply remove and replace the firewall code.

QoS and nftables ... some findings to share

Hi all, every so often there's discussions on the forum about how to do QoS at a level a little higher than just set-it-and-forget-it that SQM offers (which is a great service thanks @moeller0 and others involved in SQM!). I've been doing custom QoS for several years now and have figured out some stuff I thought I'd share, and we can discuss here. Warning 1 I don't actually use OpenWrt for my router, only for my access points. So some of this will require some help from you to debug how it work…

Things are moving slowly but progress is happening. I will begin with @dlakelan's approach to start things off because this method involves switching things and my method involves learning linux networking commands, and I do not even know where to begin with those - there is no definitive guide on constructing anything like this, it is reserved for the knowledgeable few and seems as though people would like to have it kept that way. We are out to change that. If anyone can point out resources for us to use this topic can serve as a stepping stone for linux networking, the first of its kind.

[warchief]

Alternatively~

Replacing firewall service with "own" iptables

The default is to allow practically everything, and to enable NAT. But this depends upon the services, to be run. Minimum: LAN_IF=br-lan iptables -F iptables -X iptables -P INPUT DROP iptables -P OUTPUT ACCEPT iptables -P FORWARD ACCEPT WAN_IF=uci get network.wan.ifname #Basic protection iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP iptables -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j DROP iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP iptables -A INPUT -i …

[warchief]

Thanks to @moeller0 and @jeff. This is a thread which will continue on the building of a Linux system with implementation of Cake SQM as I build the system, my underlying problem of having latency problems was solved with disabling the OpenWRT firewall and coding a basic firewall with iptables, which is fairly standard. @dlakelan also suggests running nftables which requies some extensive knowledge in the networking area. I built a stable firewall within 16 hours with iptables.

At first I tried to edit /etc/config/firewall with the default rules from fw3 print and just save the file that way. I ran into a ton of issues then figured luci builds the iptables rules from the /etc/config/firewall. This mainly came because I spent an hour fiddeling with the /etc/config/firewall adding iptables to it then something happened, I must have messed up the code and with my own iptables rules in it something happened and the latency issues were gone. I made an addition to the /etc/config/firewall and I couldn't get the firewall to connect. I had a pppoe dial and drew an ip and web access on the interfaces but nothing going through to the internet so I had to start over but I had a general idea of where the lag was in the iptables code of the default firewall.

Thus I set out to code a firewall with iptables. At the beginning of the coding I had trouble finding the script I wanted then adding the things I wanted to make the firewall function properly latency wise. After I got the firewall to function with cake but I had latency troubles which just gave me a sluggish sort of feeling, just like the default firewall, basically I had a custom firewall which didn't perform any better than what I had before and I felt like I wasted 10 hours.

Then I added some rules which broke through the latency issues. I can't tell which one did the trick but if anyone can offer some advice on how and why this happened, I set up the firewall to function as usual, then from my previous trial and error of editing the default firewalls iptables rules I saw these rules and changed them, this is what removed the lag.

iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
iptables -A -o eth1 -j ACCEPT
iptables -A -o pppoe-wan -j ACCEPT
ptables  -A -o br-lan -j ACCEPT
iptables -A FORWARD -j ACCEPT
iptables -A OUTPUT -j ACCEPT

Now one of these worked the magic - but which one?