Prioritising relatively large Shadow PC stream

Installing and Using OpenWrt Network and Wireless Configuration
21

There are a number of things going on here. First off, STREAM sounds like a genuine continuous, non-bursty stream (ahem) of packets. Netflix video 'streaming' isn't streaming in a continuous sense but is instead bursty. In other words 'send data as fast as you can for a bit, pause till video playback buffer reaches a low water mark, send data as fast as you can for a bit...'. So those 'as fast as you can' sections could well be stealing bandwidth from your genuine STREAM stream.

CAKE could potentially help here by using classification to guarantee a minimum bandwidth to 'video streaming' flows and allowing 'netflix best effort' flows to take up the spare space.

The problem with that as has been mentioned is that typically CAKE only gets to see packets on ingress before iptables and any classification rules (eg DSCP marks) have been applied.

A 'neat' workaround to that problem would be to use firewall connmarks to mark the connection as a particular category and have that category restored as part of a tc action at ingress time, just before CAKE gets to see packets.

I wrote act_ctinfo (in the kernel 5.3/4) and iptables connmark --setdscp (sadly not in kernel because kernel wants this in nftables form as well) for expressly this sort of thing: Basically on egress use iptables mangle table to choose as DSCP, use setdscp to store that in the connmark, use tc act_ctinfo to restore the DSCP to incoming packets from the value store in connmark.

2 Likes
22

Thanks for your reply @ldir!

Yes, the Shadow stream is quite continuous, as it is a video game stream that reacts to my inputs. It never passes a certain bandwidth. Netflix just takes all it can, especially when buffering a new episode. When fully buffered, it still seems to load in bursts, but these do not seem to take as much bandwidth as initial buffering. Overall, using QoS or SQM, a Netflix video playing does not make my Shadow stream stutter, but initial buffering does.

This sounds really good, and when I read this forum, I think I see more people that would like to set classes while using SQM, is that correct? Like traditional priority based QoS inside Cake/SQM?

I can surely imagine the method you describe. But how to apply these settings I do not have a clue.

Which makes me wonder if you think having only traditional QoS would also work for me. I know everyone here loves SQM and Cake, but for the sake of keeping it simple: just putting Netflix on low priority or setting a bandwidth limit maybe would already suffice. My network is really low on traffic. Large downloads are really occasional. It is mostly Netflix, Spotify, browsing. My network must be shocked to see so much traffic since I started using Shadow. There aren't a lot of traffic scenario's, and the only one I am having issues with is Netflix loading while I am using Shadow.

23

Here I would just accept any source IP, so that everything to your oculus rig gets into AC_VI initially, that way you can first test whether that helps at all before tightening the rules (especially since IPs of cloud devices often can change anytime).

Here I wonder about the value of doing that at the router? If the Oculus Quest is running on a specific internal PC maybe set the dscp in widows instead? See https://forum.openwrt.org/t/sqm-cake-and-being-a-twitch-streamer-upload-direction/62474/3?u=moeller0 for how that might work under windows 10.

Well, that is probably a consequence of ingress shaping being somewhat approximate, so unless you can establish a better traffic shaper at the ISP end of your access link (unlikely) your gaming stays sensitive to netflix starting (or any other similar traffic source starting).

That might be true, but then this is partly because that is what people where/are used to do from other QoS/AQM solutions. SQM's explicit goal is not being the one QoS/AQM solution that exposes all toggles and controls to power users, but rather doing the right thing for most users with the least requirements for twiddling and tweaking. But you can always fork the *.qos scripts and put in everything you like, so even for the tweakers SQM should at least offer a decent starting point and a reasonable framework to quickly test and change QoS configurations. That said, @ldir's great addition is someting I would like to pull into sqm scripts somehow, but I keep failing to free enough time to actually get this working and tested...

Well if you come up with a rue that allows you to unambiguously identify netflix traffic, you can always add an extra TBF instance to only throttle netflix (or try nft-qos/luci-app-nft-qos in addition to SQM)?

24

Good one.

The Oculus Quest is a standalone device. It runs Android. It runs its own apps/games internally, but it also has the Virtual Desktop app to wirelessly connect to your own pc or to a cloud pc in (for example) Shadow datacenter. Using Virtual Desktop to get PC-grade VR onto the Oculus Quest is kind of a workaround. Shadow is working on their own Oculus Quest app actually.

Hmm, that would be a bummer. The stuttering when Netflix is doing initial buffering is exactly what I am trying to solve here... So I should give up? What are we trying to improve here then? Just optimising latency in general?

I see the SQM benefit, and I also love it. Great work you guys are keeping on improving it. You guys really rock!

Because we are almost always streaming Netflix to a Chromecast, could the rule be 'throttle everything to Chromecast? I will read into TBF en nft-qos. These are new words for me.

25

Ah, okay, then I think setting DSCPs on the oculus is not going to be an option...

No, but I just want to prepare you for a potential outcome that this might not fully fixable in your router.

In this thread? We are trying to find a solution to your issue, but as above there might not be an optimal solution.

Well, that feature is @ldir's invention, and I concur, he rocks!

That would probably work, in the luci GUI you can set fixed DHCP configs by MAC address, I would do this for the chromwcast(s) that way you will have known internal IP addresses for that netflix shaper...
TBF is just a "simple" traffic shaper that might work well to throttle the chromecast(s), it seems to have all the features you need, ntf-qos is its own fancy bandwidth by IP-address allotment scheme, whith a nice GUI that might allow to throttle the chromecast from the GUI. But I have never tried nft-qos and do not know how well it plays with SQM (I assume it should work well, but I have no data to base this optimism on).

26

Conceptually it's pretty simple. Within sqm-scripts where we instantiate an egress cake instance we also (optionally) instantiate a tc action ctinfo. I did this to avoid the need for every egress packet to go through my iptables classification rules, a 'set once and forget' operation, instead the connmark stored DSCP is 'restored' to the egress packet for that connection.

    # Put an action on the egress interface to set DSCP from the stored connmark.
    # This seems counter intuitive but it ensures once the mark is set that all
    # subsequent egress packets have the same stored DSCP avoiding the need to have
    # iptables rules mark every packet.

    $TC filter add dev $IFACE protocol all u32 \
        match u32 0 0 \
        action ctinfo dscp 0xfc000000 0x02000000

ingress makes use of the existing ifb4interface construct and adds a ctinfo action to restore DSCP from the connmark before the re-direct (tc mirred action) to the ifb interface and hence before CAKE gets to see it:

    # redirect all IP packets arriving in $IFACE to ifb0
    # set DSCP from conntrack mark
    $TC filter add dev $IFACE parent ffff: protocol all u32 \
        match u32 0 0 \
        action ctinfo dscp 0xfc000000 0x02000000 \
        action mirred egress redirect dev $DEV

Storing a DSCP (and saying we stored it there so ctinfo knows to restore it) is achieved by using (a hacked) ip/6/tables connmark rule:

# store the decided DSCP into connmark for later restoration by ctinfo
ipt -A QOS_MARK_F_${IFACE} -t mangle -j CONNMARK --set-dscpmark 0xfc000000/0x02000000

The above simply makes DSCPs 'stick' across egress & (crucially) ingress so CAKE's diffserv awareness can be exercised.

The actual classification process of identifying and then 'twiddling DSCP bits' to suit flows is a continually evolving challenge. So far I'm using a combination of ipsets (filled by dnsmasq) and ports but it's pretty low tech.

A deep packet inspection method could prove better but definitely at the cost of CPU. I'm (very slowly) writing a program to interface with the ndpi 'netifyd' and its JSON stream, the idea being it could take netifyd's flow identification, decide on a suitable DSCP and interface to conntrack to store that DSCP into the connmark ready for tc's act_ctinfo to use it.

I think netify already have something along those lines (well a firewall interface) but it's written in Python3 which is a bit too heavy for embedded type devices.

If a proper programmer wishes to join me then that'd be great. I'm using it as an excuse to improve my 'c' and learn about sockets etc, 'cos I've never dealt with that before. I've got as far as reading the netify socket and parsing the JSON stream that's coming from it.

2 Likes
27

What can I do about this missing kernel?

Screenshot_20200522-181714
Screenshot_20200522-181714949×1152 190 KB

28

Update. I flashed DD-WRT info the other bank. Enabled QoS using fq_codel, and set the Chromecast to Bulk Priority and the Oculus Quest to Maximum Priority. I think I read this means only 5% of bandwidth for the Chromecast when the Oculus needs it.

My problem was instantly solved. Big initial Netflix buffering causes only a small 5ms latency spike while using Shadow on the Oculus Quest. I see the spike on the latency counter but cannot even notice it when I am not looking. So I am happy.

What would be the easiest way to get this rule in place in OpenWRT?