Would anyone care to assist me in building something like this paired with OpenWRT's Cake SQM? There are users on this forum who have the expertise but do not provide step by step instructions for how to build something like this. I just want that type of setup + dial in using pppoe and apply cake and that's it using a debian box.
I know how to:
install debian on ssd
ssh into OpenWRT and execute 2 commands: opkg update, appinstall-luci
I don't know how to:
begin to build the router config in debian
configure the system to utilize pppoe
draw the cake package, install, and configure from the command line
I run "headless" over SSH, so I don't need a graphical user environment.
(There may be steps missing here that were "obvious" to me)
apt install sudo
usermod -a -G sudo jeff # Use your "normal" user name here
Log out, and log in as your "normal" user
Update the install
sudo apt update
sudo apt upgrade
and get ready to build your first ROM!
sudo apt install build-essential git gitk libncurses5-dev gawk unzip wget curl ccache rsync zlib1g-dev
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git checkout master
./scripts/feeds update -a
./scripts/feeds install -a
make menuconfig
Select your device's target, subtarget, and specific board -- this selects "good" defaults, without LuCI (it should include PPP-related kernel modules automatically).
I select Developer Options, then the entries for
(Hint: The / key will let you search in make menuconfig)
The first build will need to build the toolchain. This takes a while. Figure around an hour for the first build.
make -j12 clean download toolchain/install world
Set the -j12 to the number of cores on your build box or VM, or perhaps one less if you want to do something else compute-intensive on the box (more than watching the output or running [h]top or the like) while it's building.
re-builds are much faster (5-10 min for me), and you generally can just do
make -j12 clean download world
Using ccache makes "clean" not really cost a lot of time, so I do it every time out of habit.
Important: With your own builds, best is to add packages to a new ROM, as the OpenWrt package repos won't be consistent with your build. I have added "simple" packages (no "kmods" needed) on occasion, when I needed something without taking down a device to flash it.
You do that in config, not in the build, unless you need functions that are only in PACKAGE_tc (which would show in the thread there as something like "install tc" or opkg install tc
So the cake qdisc is part of the official upstream kernel since version 4.19, and sqm-scripts can also be used on non-OpenWrt linux distributions (without any GUI though), see https://github.com/tohojo/sqm-scripts/blob/master/Makefile.
But what exactly is your goal here, use a debian box to build OpenWrt images for a home-brew router or do you want to run the home-brew router under debian in the first place?
I wanted to attempt the veth method as described by @dlakelan
I legit put 2k into a pc to test my connection and I am not happy with the performance of my connection. I am going to try perform some magic with linux and just teardown the entire router, build it from the ground up. This is similiar to like MintOS Linux/Archlinux right? the bloated OS vs the bare minimal. I don't want any interference in packet transfer so I can basically make this DSL function like cable in terms of packet latency, I mean were talking ~200 kb packets here it should not take a home brew to get some raw power or pure speed if you will. But none the less this device will be built, and I hope to achieve a desirable outcome since I have come a long way with the help of the community, I would like to continue advancing and build a homebrew using the OpenWRT kernal or "snapshot of master".
Sure that sounds like an interesting learning experience/project. A decent starting pint might be to first try to measure/localize where your current solution fails to meet your requirements so you can tackle the biggest issues first?
Well, what kind of latencies to the first hop do you see with your DSL link and what kind of latency distribution do you expect for a cable link, in terms of mean RTT and variance/jitter?
And beyond that, how are your current RTT/jitter values to the servers that you actually want to connect to?
I ask, because one of the early steps for me would be trying to figure out where the "problematic/undesired" changes to packet delivery happen, as playing with traffic shaping will only really allow you to address issues with your own network and your ISPs closely connected network elements. That is to say, if you problems are related to your ISPs transit/peering with say your preferred game's ISP than no amount of SQM is going to fix that (but in that case finding a better route, e.g. via a VPN might be a viable work-around/route-around).
I guess my point is, unless you have a deep desire to dive into tc, iptables and friends, maybe start with defining the status quo and you r goals and assess whether reaching them is feasible and worth your time.
I have purchased 2 dsl modems and 2 routers and 1 switch I think it is beyond worth my time but at this point I would just like to give back to the community and finish building a bare bones linux "server" or router. I am pushing for a fiber link and they are ~15 miles from my home I will probably get star link before I get fiber but..... okay... so... linux tests I am going to be running through a liveboot because I do not have a rig set up with linux installed. I was running a cable link and dsl link through the past few years. They have both pinged the same and as I am quite a novice with network diagnostics and linux. I can debug but not at an expert level.
Right now my extent is learning how to do tc -s qdisc
Generally I am pinging ~64 east west ~50 to chicago generally on the ingame servers with blizzard's chicago server pinging the closest. I play fortnite which has 2 NA servers and I am not sure where but I assume east server ohio and west server california. I have a zyxel broadcom dsl chip which is good, 100 yards from my dslam, I believe 100 miles to my first hop, 125 miles from there, another 450 to another then off to the races my udp packets are off to fortnites east server.
When I talked to the system admin that's basically how he explained it to me and he said I have no interleaving on my line which is good, I unchecked all the DSL mods. This should give me good stability and performance. I had some sync issues with some of the mods which caused trouble tickets which now my isp hates me and customer service has 1000 tickets and notes which they refer to each time and now know they are dealing with a 'problem' customer who can't be satisfied.
Try running mtr to your game server, while actually playing.
You could use opkg update ; opkg install mtr to install mtr on your router and then just start mtr -z 172.217.2.4 to get a continuous RTT measurement, and after playing you look at the statistics. Better than google would be the game server you want to play on, if that responds at all. Also many game companies allow reverse ping/reverse traceroute (search for looking glass and the name of the game company) to allow you to measure congestion on the reverse path.
In theory PhyR US and PhyR DS (also known as G.INP in ITU standards) can increase the performance, as it can avoid most packet loss by CRC, but your ISP's DSLAM needs to support it. From the short uptime it is onclear whether you see enough CRC errors for this to be relevant to your network experience though.
Bitswap, most users like to enable that as it allows modem/dsl to switch bit allocation (if there is still some reserve) to avoid individually noise frequency bins, without bitswap the modem will be forced to resync more often, leading to larger downtimes. Then again, if you never get unenforced re-syncs this is also not relevant for you.
SRA, allows modem and dslam to change the synchronization rate without a re-sync, which again can avoid unenforced resyncs, but will lead to variable IAS rates, making it hard to track with sqm.
It looks like your shell tried to execute the output of the command. How did you call it? I typically log in via ssh and simply call tc -s qdisc and get nice output. But if that does not work tc -s qdisc > /tmp/tc--output.txt will store the output in a file that you can then get from the router with scp . mtr -z 8.8.8.8 against Google's DNS servers as example will do a continuous 'traceroute' and update the results in the current terminal window, you should be able to copy and paste from there. Alternatively, you could use mtr -z -c 100 -r 8.8.8.8 to let mtr create a textual report for 100 iterations. As before the > /path/to/file redirection should allow you to save the output to a file.
About VPNs, as you saw VPNs are not guaranteed to improve your routing, you need an uncongesten VPN that has a short RTT path to you and your desired target servers. I would naively assume that such a VPN might not be the cheapest, but I have very little experience with VPN services. Another alternative could be to rent a virtual private Server at a well connected holster and build your own VPN between your router and that VPS, assuming the VPS allows enough traffic per month.
Please note that your embedded video seems to be private, on clicking it I get:
"null
If the owner of this video has granted you access, please sign in."
So no idea what this shows and hence no comment on the content.
Do you have a specific question regarding the tc output?
It is fixed now but that output I am not sure if it is right because I don't see any info which would benefit here. It was the same with netstat -a, I was using that to pull the game server ip and then match it with my PC netstat -a ip then mtr that ip address, but there were just hostnames maybe I pulled the wrong IP.
For the VPS/VPN. It would take some additional digging to see if that would benefit the line but as far as I know, as from what I have seen it should not be an issue. From what I understand, once the packets are out on the wire from my DSLAM it is all the same. I believe we are using fiber optic lines from the DSLAM. Although I am using copper wire and going through 2 or 3 line filters I don't see that as being an issue, I did straight wire through the filters using crimp connectors and that is another reason why my ISP hates me. But like I said the only problem with the RTT is I live ~750 miles from the game server which increases my ping by 30ms, and I am willing to live with this I kind of have to if I had the money to change that I would not be on a DSL line and be living east coast. Do I have 2 million dollars? lol no, same goes for the vpn/vps deal but I do not see this as a 2 million dollar issue, I just see a coding issue here that I expect we can fix because in essence, I will be building a server to run OpenWRT on and that should be sufficient to shape and optimize the line.
For the tc output this is the traffic control output so I assume the actual network config and shows what it is coded to do and how it is doing it i.e diffserv 3/4/8 tiers. So I think the main thing is the drops but that could be due to anything in the line I assume it is from my DSLAM to me. Otherwise the only thing I can think of is the Network>Firewall>Firewall Zone Settings>Drop Invalid Packets?
So I am getting everything in the BE tin even though I have an AP set up so it should not be dumping there it should be dumping in the Bulk tin(iphones/firestick) But the main thing is my gaming rig is dumping there so not an issue ATM.
ECN, triple isolate, latency target - should I care about those or can I just default and they are fine.
I would like to see the skeleton of the system, how do I see the configuration of the network, like the actual coding, is there a way I can pull that from ssh? I ask because it would make it easier if I screw something up or run into a problem when I begin to code the new system. I am too lazy to liveboot debian but yeah I was just using putty to call the tc output.
That looks massively broken. How did you log into your router in the first place?
here is the output I get from mtr -z 172.217.2.4 (copied from the terminal windw running the ssh session to my openwrt router, but if you have a mac or linux machine in your network you do not need to run mtr on the router itself):
You can nicely see the packet loss per hop (typically as long as the end host has no or very little packetloss you can ignore that) as well as current, average, best and worst RTT (as well as standard deviation), often if a network path is congested the RTT will show some sort of a step function, with reasonably low RTTs (low compared to the path length, light in GF gives ~100Km distance per ms RTT) up to a point and increased RTTs to all points after that. Note that is a sign of congestion, but not a precise localization of the place where the congestion happens (see https://archive.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf for the challenges in properly interpreting traceroute/mtr output), if however you have traceroutes/mtr output in both directions, so from your end to the server and from (close to) the server to your router's IP address than you can even localize the congested hop with some confidence (it s still error prone, but you at least have a chance).
This is only a work-around if the normal network path from your ISP to the game servers is congested and introduces unwanted delay, if you have no indication for that an VPN/VPS is not going to help much.
THe rule of thumb is 1ms RTT allows for 100Km of fiber, so 750 miles -> 1,207.008 Km or 12 ms propagation delay, add a few active components and a few detours, 30 ms sounds reasonable, if you get that reliably with low variation/jitter, than digging to deep into traceroute might be a waste of time (except it is a skill that can come in handy when you have network issues).
Assuming there are things on your line that can be improved noticeably (you seem to have done a thorough job with the modem and ISP already).
Yes cake reports its configuration pretty complete:
So in your case it is the default diffserv3 "profile" and hence cake also gives you three detal columns for the three priority tins, called Bulk, Best Effort and Voice. If you switch to diffserv4 you get four columns (and 8 for diffserv8).
No, drops is expected, this is how cake tells individual flows to slow down if the capacity is exhausted (if a flow uses ECN it will see less drops and more marks). So drops are a-OK, just too many drops can be suspicious.
Well, on the egress side "qdisc cake 802a: dev eth0" you have a few packets in the voice bin. Not sure about what you expect.
ECN:
Cake does ECN by default, but ECN needs to be negotiated by both end-points of a flow, so if you want it used make sure to configure your end-host appropriately (and the same goes if you want to be sure it is not used).
triple isolate:
IMHO configuring nat dual-dsthost ingress for the ingress traffic and nat dual-srchost for the egress traffic is more predictable (as that should give each computer/IP-address in your home network an equal share of the available bandwidth (but dynamically, only active hosts are counted here, so a single active host will get 100% of the capacity), triple isolate will do something similar (especially if there is a decent mix of active internal and external IP addresses) but without the simple predictability of the dual-xxxhost modes (then again triple isolate does not need to treat ingress and egress differently, this is why triple is the default).
latency target: you can not even directly configure that in cake, it is typically set to 5% of interval (and you can configure interval) interval should be in the right order of magnitude of your typical RTTs, the default 100ms works quite well for many uses and RTTs from 10-200ms or so (there is no hard cut-off).
No idea, except for the fact that cake reports its effective configuration in a way you could use as part of a cake invocation:
I am mtr -z 52.53.224.41 and it is a UDP stream, I will look at some other connections and see if I can get a better reading.
I am looking in through PuTTY as root, I also tried the debian liveboot ssh but same result maybe it is the UDP stream screwing things up. My MTR was downloaded via ssh and through LuCI, it says it is up to date, I think we are seeing 0% packet loss. As a matter of fact it doesn't seem as if I am losing packets or have any stuttering.
The concern is and this is how I first found the issue. I was watching youtube of a gaming stream, now I immediately noticed that stream, which is like video playback and includes any video playback such as ABC, FOX, NBC type of video feed same thing you would see on the news - it seemed like my game window wasn't running at that same bitrate/speed.
Now the streamers game, and it goes for most streams out there - they had no issues with the streaming(uploading) the game and the game connection itself, it ran like any television video feed would run. Now mine, I cannot stream and game at the same time due to speed limits where it takes about 7.5 mb egress to stream and 7.5 egress to game you know confidently. I can only pull 10 here on this vdsl line so I can only do one at a time.
So my game window runs like its choked right? a little buffered or at a slower line rate than say something like monday night football. Now before we start considering video rate lets just assume everything is able to stream at the rate of monday night football and monday night football runs at 200 frames per second. Okay so, my line rate which is apparent in my game window full screen or not fully capable PC fully capable x86 with cake on it feels like it only attains 90% and it feels like the other 10% is just bleeding out somewhere. Now I don't know if it is egress or ingress but I feel like egress is just fire and forget with the DSCP marking, now I feel like it is the ingress side of things and I have some type of iptables or other OpenWRT config issue(pppoe?) which is limiting the video rate causing frames per second slowdowns.
##FIREWALL
iptables -A FORWARD -m conntrack --ctstate NEW,RELATED,ESTABLISHED -j ACCEPT
##GAMINGEGRESS
iptables -A FORWARD -s 192.168.1.2
iptables -A PREROUTING -s 192.168.1.2 -j DSCP --set-dscp-class CS6
##GAMINGINGRESS
iptables -A FORWARD -d 192.168.1.2
iptables -A PREROUTING -d 192.168.1.2 -j DSCP --set-dscp-class CS6
Now as long as I have the correct routing table for the iptables rules then that is ruled out, there shouldn't have to be any changes in the OpenWRT default firewall just the scripts that I added so that is the firewall+scripts, there is not a hardware issue on my side, from what I gather the VDSL should be capable of the same video rate as any other connection up to the speeds we are talking about:
1514 kb max game packet / peaks of 700 kb but 90% is under 150 kb.
I am measuring less than 3 ms of bufferbloat on both egress and ingress I am happy with the sqm config. I am missing the berkeley netalyzer for more in depth network analyzing but even then it did not show the spikes which dslreports does.
Yes it is becoming a money pit. In a sense I am building a formula 1 car to drive daily here. But the issue is the car I am building can do 200 mph, it is only going 40 mph because theres a problem to fix.
I am attempting to put all network traffic into Bulk, and my gaming rig into the best effort. Maybe I am missing a iptables rule?
iptables -A -I eth+ -j DSCP --set-dscp 0
Perhaps? But I still have DNS important router connections and such and I don't know if that is a problem so I guess it would be best to keep everything best effort just keep the network egress congestion free(no torrenting) but there have never been any torrents or any heavy traffic besides youtube/netflix going on while I game and that does not create an issue.
At the end of it all, I have a single UDP stream from fortnite's game server transmitting less than 10mb for every game session no matter how long it lasts, and it performs poorly compared to worse set ups. It is not a PC issue, not a router issue, not a modem issue, not a cable issue. So the issue is between the game server and my DSLAM......
nat dual-dsthost ingress
eth0(lan) > Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully
nat dual-srchost
eth1(wan/wan6) > Advanced option string to pass to the egress queueing disciplines; no error checking, use very carefully
So that is how I have it set up, but openwrt recommends it should be set up differently when you use 2 sqms(something about it being flipped) if I should flip it let me know.
Again, should the cake sqm be on eth1 (wan,wan6) or pppoe-wan(wan,wan6)
Well since I have 2 x86 set ups I don't think it is possible to brick these - it runs off of a samsung ssd, but I also have a usb bootable with the same kernel on it. And when I do begin to code from scratch I will just keep stable configs(as you said) or just wipe it and start over.
So could you try to copy and paste as text from the putty terminal window running mtr instead of taking a scree-shoot, please?
Looking at the delays if that mtr in the ~400ms range starting at around hop 7, I wonder what kind of host 52.53.224.41 is, and where it is located?
That is quite curious, as OpenWrt master is at version 0.92 for mtr, which OpenWrt version are you running?
Which issue exactly?
Does this mean that your game got disturbed by concurrent video streaming? If yes, were the video streams received on the same computer the game was played on or on another computer (if the later per-internal-IP-isolation might help)
Most games only send a relatively small amount of data upstream (FPS type games often send at considerably less than 1Mbps). But if you try to stream a video of yourself playing (like twitch) than this changes and you will put a considerable load on your egress link.
7.5 Mbps for streaming sounds about right, but the same 7.5 Mbps for game state update traffic seems quite excessive. BUT you realize that you throttled your uplink to 6 Mbps with your sqm configuration:
qdisc cake 8028: dev eth1 root refcnt 2 bandwidth 6Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 27
That is not going to fly if you need to send at 7.5 Mbps.... (according to the luci-app-sqm screenshot below eth1 is carrying your WAN egress traffic). Also note that:
Bulk Best Effort Voice
thresh 375Kbit 6Mbit 1500Kbit
means that high priority traffic is capped at 1.5 Mbps (the overflow will end up in Best Effort) so most of your game streaming packets will not be treated to high priority...
But that is what you should do, try to max out your upload (without overloading it ) shaping a 10 Mbps link down to 6Mbps certainly does not help if you want to send at high rates while playing.
I have no idea what you are talking about here. Are you telling my your ISP also sends IPTV over your internet access link so that the 30/10 is shared with TV traffic? Because for cake to be effective it needs control over all traffic on your internet access link. So neither TV traffic nor wifi traffic can be allowed to directly connect to the internet without having their packets controlled by cake.
I am pretty confident it does not run at 200 Hz, it runs at 60 Hz if you are lucky, but I digress.
It is time to investigate this instead of relaying on feeling.
You could opkg update ; opkg install iftop and then iftop -i pppoe-wan to look at the instantaneous ingress and egress traffic while playing/streaming. That should allow you to figure out whether your ingress, your egress or both are overloaded.
Great!
Your issue might be that a formula one racer is not built for dirt road racing.
That seems pretty extreme, as under congestion Bulk only gets 375Kbps on egress....
Youtube/netflix typically put load on your ingress, not the egress.
What are you comparing with here exactly? Friends and neighbors with the same ISP close to where you live or people in other parts of the US?
I believe the flipping is in regards to Upload and Download speed and only fully effects you if you instantiate sqm's normal ingress instance via IFB, with your one egress instance on each of the WAN and LAN interface will not have that issue. So that config looks about right.
On a pppoe link I typically recommend to instantiate it on pppoe-wan.
Regarding the mtr movies, it probably is enough to let mtr run for say 100 probes and only copy and paste the final outcome, or use mtr -z 52.53.224.41 -n -c 100 -w to get the static output. (running mtr on-line can be helpful to diagnose transient latency spikes, but for that you need to be able to look a what haapens locally in your network, so sharing videos of that is not going to be too helpful).
location san jose,cali, but I do not think my results are coming out correct. That is what I was saying - there is a delay.
The ingress link needs to send 1080p video right, to download the packets, which are sent through wireshark at ~1514 kb packets.
the egress link 90% are less than 120 kb but can avg 700 kb, and max at 1514 kb but not more than 1514 kb for the udp egress game packets as seen in wireshark
In general I think the games are optimized to send 1 packet at a time confirming your [quote="moeller0, post:18, topic:49694"] it runs at 60 Hz [/quote]
As most, atleast fortnites server is tickrate 64 or something.
1920x1080 144hz in game(dirt racing) and 1080p video stream on youtube(formula 1) do NOT look identical
Now the frames aren't really the issue because in either case, the hardware I have and the link I have is capable of:
1920x1080 144hz egress(requires ~1.5 mb egress)
1920x1080 144hz ingress(requires ~1.5 mb ingress)
Or
1080p egress streaming(requires 7.5 mb egress)
but as stated before not at the same time, in theory I have enough but I am not comfortable with the numbers that close to my 10 MB max.
Now this I am not sure - the cable tv used is just that, cable tv, they had issues and pulled all the zyxel DOCSIS cable modems, probably due to this very issue. The problem is games end-to-end slow just a bit, enough so that it doesn't stream 100%. Not that it is jittery, stuttery, major packet loss.
The games, just the games, some type of tcp/udp issue where it is just choked right? But pure 1080p video from youtube? streams fine zero issues. 1080p video ingress and egress/ingress for gaming? not the same, We see some throtteling here which I thought was the VDSL copper line to DSLAM distance but I sense it is in the network(openwrt) config.
I installed but it looked too complicated and I do not know if this is the same
I basically gave up trying to set up the diffserv tiers and just send all traffic based on my rigs ip to best effort and do not configure anything else with the 3 iptables rules. this seems to be the least computationally taxing and simplest way to issue a rule for a single rig.
generally people with the best connections, the best pings, like I said I do not the believe packet delivery to be a link issue but a coding issue regardless of location within reasonable limits i.e chile.
That seems dubious from the SQM off test. My rule of thumb is that the gross shaper settings for sqm should be (for the first iteration) pretty much the maximal net-goodput rates you get from the ty[ical speedtests, and the dslreports speedtest does not seem to give you 50/10. IN your case, assuming the dslreports speedtes is reliable 35400/8500 looks like decent starting numbers for sqm, but what results do you get from ookla's speedtest (www.speedtest.net) of netflix's fast.com?
That looks about right, as does:
Why do you believe the measurements to be off? I agree that ir probably does not reflect pure propagation delay for the distance, but to me it rather indicates congestion somewhere along the line, the challenge now is to figure out where....
In the ingress, I would say you receive video streams from say netflix, amazon, apple. We should reserve the word send for data you push into the internet (and sure almost all receive connections will also cause some send traffic, but for video streaming that typically is relative low bandwidth).
Nope, typical packets are 1514 Bytes, not Kilo and not bits, jumbo frames can go up to 9KB though, but typically not outside of a carefully managed and configured local network.
Sure, as far as I can tell youtube currently oly streams at up to 60Hz....
Do you actually use egress streaming?
Fair enough, so you get your cable TV from a different supplier with its own coaxial cabling (and not some weird contraption, where there is a "cable TV" box that connects via coax cable to your TV and vie ethernet to your router)?
That might indicate some component being maxed out and limiting your game that way, might not even be on the network side.
No idea.
Okay, but video streaming, in spite of the naming, typically is not sending data at a constant rate, but it rather binges and send a batch at maximal achievable rate and fills a buffer, and when the buffer gets emptied below a "watermark" it requests the next batch, so it has some tolerance against variable network conditions and RTT variance. You game does not have that luxury unless it accepts a relative high general delay/dejitter buffer (but that leads to less zippy game reactivity which players detest).
For that you will need to either build your own shaper hierarchy with HFSC or HTB or modify the source code of the cake module. For a link that supports ~8.4Mbps egress traffic pushing 7.5 into high priority seems rather unhealthy though. (per internal IP fairness is also not going to help you much, as we two active users each only gets >= 4.2 Mbps, too little for sending 7.5 Mbps). You might want to consider getting a dedicated DSL/cable link for your gaming purposes (or a higher upload link).
The idea is that you look at the iftop output to diagnose things, not that you post videos of that
Nope collectd is a quite helpful tool which allows to periodically sample and store different parameters from your router and the generate fancy plots. It can be quite helpful in finding issues, but let's stick to the tools you already know how to use.
So you disabled these override rules completely?
This looks odd in that the final hop does not respond, but the RTTs of most intermediate hops look reasonable to me, at least more reasonable than >=400ms.
(you seem to have done a thorough job with the modem and ISP already).
