Prioritize RTMP Stream Packets

ZombieBest · July 1, 2020, 11:30am

Hello all!
In my current setup I have a Fritzbox modem configured with DMZ to the OpenWrt router (which WAN is a DHCP Client of the Fritzbox). Nothing else is attached to the fritzbox other than the VoIP phone.

My VDSL connection is pretty slow, with 27.000Kbps download and 6.700Kbps upload, but it has enough upload to do live-streaming at 1080p with 4500Kbps bitrate. The only issue is that I need to prioritize that traffic, otherwise if another devices uploads something, I start to loose frames.

To do that I configured this traffic rule on Firewall settings:

And then I configured SQM on WAN interface like this:

But it does not seem to work. I've checked with Wireshark and packets that gets sent have "Destination Port 1935", with protocol TCP.

For testing I just tried to run a Speedtest on my phone: when it starts the upload test, stream speed decreases from 4500Kbps to about 800Kbps, so something is clearly wrong.

Any advice?
Thanks as always

mpa · July 1, 2020, 12:09pm

Check which cake tin the traffic was assigned to:
tc -s qdisc ls dev eth0.835

ZombieBest · July 1, 2020, 12:15pm

Thanks for your answer!

This is the output I get... while streaming the "bytes" row of the "Best Effort" column increase a lot. Not sure how to interpret it.

qdisc cake 8011: root refcnt 2 bandwidth 6500Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 34
 Sent 21749665 bytes 16738 pkt (dropped 84, overlimits 27138 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 125312b of 4Mb
 capacity estimate: 6500Kbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       62 /    1526
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh      406248bit     6500Kbit     1625Kbit
  target         44.8ms        5.0ms       11.2ms
  interval      139.8ms      100.0ms      106.2ms
  pk_delay          0us       19.6ms        736us
  av_delay          0us        8.4ms         24us
  sp_delay          0us        795us         24us
  backlog            0b           0b           0b
  pkts                0        16806           16
  bytes               0     21866564         2114
  way_inds            0            1            0
  way_miss            0          131            6
  way_cols            0            0            0
  drops               0           84            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        31626          269
  quantum           300          300          300

mpa · July 1, 2020, 12:20pm

Now do the same check for the speed test.
Is it also assigned to the "Best Effort" tin? If so, then it has the same priority as the streaming.

ZombieBest · July 1, 2020, 12:34pm

Yes, exactly. I don't think that RTMP traffic should be in the "Best Effort" tin, right?

mpa · July 1, 2020, 12:52pm

Right, but the higher-priority Voice tin appears to be limited to 1625Kbit.
I don't know if this limit can be changed.

ZombieBest · July 1, 2020, 1:16pm

I've done a little bit more researching, and found out that Tins are decided by the "diffserv" setting. Default seems to be diffserv3, I set it to diffserv8 by editing the /usr/lib/sqm/defaults.shfile and recreating the sqm interface.

Now I can see 8 tins... will test and let you know! Thanks for pointing this out to me!

moeller0 · July 1, 2020, 1:19pm

Not without editing the cake kernel code and recompilation. But you can much easier change simple.qos, there the three implemented priority tiers are completely constructed/defined in /usr/lib/sqm/simple.qos.

Just make a copy of simple.qos and simple.qos.help to say my_simple.qos and my_simple.qos.help and edit both. They then will appear in the list of queue scripts....

moeller0 · July 1, 2020, 1:20pm

You still can not manually set the min and max bandwidth for the tins in cake...

moeller0 · July 1, 2020, 1:25pm

And there is your issue, SQM/cake default to per flow fairness (with potential per-internal-IP-fairness on top) and 4.5 Mbps simply is more than the fair share of your bandwidth. These easiest way around that is to created a modified copy of /usr/lib/sqm/simple.qos. Have a look at:
https://forum.openwrt.org/t/sqm-cake-and-being-a-twitch-streamer-upload-direction/62474/13?u=moeller0
For something similar...

ZombieBest · July 1, 2020, 1:46pm

Thanks for your answer!
I tried to follow the tutorial linked, but I can't make it to work.

I copied simple.qos to a new file (twitch.qos) and edited the PRIO_RATE value on the egress() function to PRIO_RATE=5000.

I have then selected the new qos from the list (keeping "cake" in the queue discipline), but it's not working at all.

Running tc -s qdisc ls dev eth0.835 returns:

qdisc cake 8052: root refcnt 2 bandwidth 6300Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 34
 Sent 109434075 bytes 91331 pkt (dropped 846, overlimits 129150 requeues 0)
 backlog 1506b 1p requeues 0
 memory used: 197760b of 4Mb
 capacity estimate: 6300Kbit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:       62 /    1526
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh      393744bit     6300Kbit     1575Kbit
  target         46.3ms        5.0ms       11.6ms
  interval      141.3ms      100.0ms      106.6ms
  pk_delay          0us       27.3ms        1.9ms
  av_delay          0us       10.2ms        204us
  sp_delay          0us        342us         46us
  backlog            0b        1506b           0b
  pkts                0        92081           99
  bytes               0    110665793        10794
  way_inds            0            1            0
  way_miss            0          270           20
  way_cols            0            0            0
  drops               0          846            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            2
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        31626         1392
  quantum           300          300          300

qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 23365994 bytes 67326 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

and stream traffic does not get prioritized. I get better results by setting differv8 (still some drops as not all packets gets top priority but just some of them).
Any idea on what I'm doing wrong or missing?

moeller0 · July 1, 2020, 1:59pm

Well, sorry for not mentioning that, you need to use fq_codel here, this is not set up for cake as leaf qdisc...

ZombieBest · July 1, 2020, 2:12pm

Still no luck... my speedtest takes over 4.5Mbps causing the stream to drop a lot of frames.

I did change the DSCP classification to 46 as per the tutorial (Note: I've also tried EF):

And set the options like this on SQM:

This is the edited part of the egress() function on twitch.qos:

Thanks

moeller0 · July 1, 2020, 2:16pm

What is the output of:
tc -s qdisc
tc -d qdisc
cat /etc/config/sqm

Also, I believe that you will need to adjust the divisors for the other classes up, so that the sum of PRIO_RATE, BE_RATE and BK_RATE stays below ceiling...

moeller0 · July 1, 2020, 2:23pm

Time for packet captures try to generate some traffic that should be marked 46 or EF and take a packet capture on your router while the streaming runs. Make sue just capture for a short while or attach some additional storage to the router, as most routers are not build for storing megabytes of data. Then look at the capture file with, say, wireshark and conform that the packets are correctly dscp marked....

ZombieBest · July 1, 2020, 2:26pm

root@Main-Router:/usr/lib/sqm# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 3047824377 bytes 3430210 pkt (dropped 0, overlimits 0 requeues 360)
 backlog 0b 0p requeues 360
  maxpacket 1514 drop_overlimit 0 new_flow_count 221 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 1: dev br-app root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 direct_qlen 1000
 Sent 12781 bytes 124 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev br-app parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 1636 bytes 37 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 124 drop_overlimit 0 new_flow_count 2 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 130: dev br-app parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev br-app parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 11145 bytes 87 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 135 drop_overlimit 0 new_flow_count 67 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc ingress ffff: dev br-app parent ffff:fff1 ----------------
 Sent 10669 bytes 169 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.835 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 68 direct_qlen 1000
 Sent 833876889 bytes 675446 pkt (dropped 2925, overlimits 1020588 requeues 0)
 backlog 4532b 2p requeues 0
qdisc fq_codel 120: dev eth0.835 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 833433333 bytes 674168 pkt (dropped 2925, overlimits 0 requeues 0)
 backlog 4532b 2p requeues 0
  maxpacket 34936 drop_overlimit 0 new_flow_count 26703 ecn_mark 0 memory_used 7392
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 130: dev eth0.835 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.835 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 156220 bytes 1050 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1426 drop_overlimit 0 new_flow_count 723 ecn_mark 0
  new_flows_len 1 old_flows_len 7
qdisc ingress ffff: dev eth0.835 parent ffff:fff1 ----------------
 Sent 95147180 bytes 461160 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 105505 bytes 1254 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev tap0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 2635747 bytes 16536 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.835 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
 Sent 117242820 bytes 461134 pkt (dropped 26, overlimits 74680 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.835 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 117242820 bytes 461134 pkt (dropped 26, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1540 drop_overlimit 0 new_flow_count 32290 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc htb 1: dev ifb4br-app root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
 Sent 13035 bytes 169 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4br-app parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
 Sent 13035 bytes 169 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 130 drop_overlimit 0 new_flow_count 160 ecn_mark 0
  new_flows_len 1 old_flows_len 0

root@Main-Router:/usr/lib/sqm# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc htb 1: dev br-app root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
qdisc fq_codel 120: dev br-app parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 130: dev br-app parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 110: dev br-app parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc ingress ffff: dev br-app parent ffff:fff1 ----------------
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc htb 1: dev eth0.835 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 68 ver 3.17 direct_qlen 1000
 linklayer ethernet overhead 34
qdisc fq_codel 120: dev eth0.835 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 130: dev eth0.835 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 110: dev eth0.835 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc ingress ffff: dev eth0.835 parent ffff:fff1 ----------------
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc fq_codel 0: dev tap0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc noqueue 0: dev wlan1-1 root refcnt 2
qdisc htb 1: dev ifb4eth0.835 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
 linklayer ethernet overhead 34
qdisc fq_codel 110: dev ifb4eth0.835 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc htb 1: dev ifb4br-app root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4br-app parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn

root@Main-Router:/usr/lib/sqm# cat /etc/config/sqm

config queue 'eth1'
        option qdisc 'fq_codel'
        option script 'simple.qos'
        option qdisc_advanced '0'
        option linklayer 'none'
        option interface 'br-app'
        option download '3400'
        option upload '8200'
        option debug_logging '0'
        option verbosity '5'
        option enabled '1'

config queue
        option debug_logging '0'
        option verbosity '5'
        option enabled '1'
        option interface 'eth0.835'
        option download '26000'
        option qdisc_advanced '0'
        option linklayer 'ethernet'
        option overhead '34'
        option upload '6000'
        option qdisc 'fq_codel'
        option script 'twitch.qos'

ZombieBest · July 1, 2020, 2:51pm

I've installed tcpdump-mini on the router, and trying to do what you asked.

I found this command online, but no packet gets captured.
tcpdump -nni br-lan -v 'ip[1] & 0xfc == 46'

By listening normally with tcpdump -n -i br-lan

I get these packets:

16:50:12.190771 IP 192.168.12.110.60999 > 52.223.200.13.1935: Flags [.], seq 493344:494796, ack 17, win 64108, length 1452
16:50:12.190951 IP 192.168.12.110.60999 > 52.223.200.13.1935: Flags [P.], seq 494796:496600, ack 17, win 64108, length 1804

Not sure how to check if the DSCP flag is set.
Thanks for your patience

ZombieBest · July 1, 2020, 4:53pm

Ok, found a way to export to file, analyzed it with Wireshark and... nope. DSCP Value is AF43 for those packets, and not EF

I tried everything... seems that all my settings to get DSCP tagging are ignored

moeller0 · July 1, 2020, 5:08pm

Well, are the marked packets directed to the marked port? Maybe you could try to set the dscps on your twitch streaming machine first.

ZombieBest · July 1, 2020, 5:27pm

It seems so

I did tried to set up dscp on my machine, but seems it's not working either... I've used this command on a superuser powershell:

New-NetQosPolicy -Name "twitch" -AppPathNameMatchCondition "obs64.exe" -PolicyStore ActiveStore -NetworkProfile All -DSCPAction 46