Ultimate SQM settings: Layer_cake + DSCP marks


#482

Why does all isp's here use MTU=1480 and not 1500 ?
http://n2.netalyzr.icsi.berkeley.edu/summary/id=36a470be-31160-6e4f1570-a314-4a48-a11c#tcp_connectivity


#483

It might be because of something like pppoE ?


#484

But if you see some ISP's use 1492 or 1500(jumbo frame) as MTU


#485

That could be an IPv4 tunnel, as the IPv4's header length is 20 bytes...


#486

1492 indicates PPPoE, as that header uses 8 Bytes. 1500 bytes indicates no obvious tunnel. Now, the thing to look at is the MTU/MSS into the internet, not only the MTU into the ISP's network.


#487

Thanks, internet MTU is 1480 too.
but today MTU is worse


#488

The results indicate use of timestamps in the IP headers. There might be reasons to drop packets whose timestamp is too old, particularly in networks where congestion and delay are common.

Edit https://stackoverflow.com/questions/7880383/what-benefit-is-conferred-by-tcp-timestamp has more to do with large MSS and wrapping of a 16 bit counter. It's a TCP specific feature to prevent excessive retransmits etc


#489

Ah, TCP timestamps take 12 Bytes, so maybe PPPoE plus timestamps?


#490

This link does not show your results, but will be remeasured for each access, so I just see my own data there..


#491

@dlakelan
I'm always have problems with TCP on all ISP's when browsing sometimes i get errors like:

error_TCP_reset
error_TCP_timeout
tcp_network_error
Network dropped connection on reset
sometimes when i open some sites i saw the spining circle near the page name on the page, it's just spins then stop then spin and page never load!

This is my test results:

Summary

« SpeedGuide.net TCP Analyzer Results »
Tested on: 2019.01.10 10:01
IP address: 185.51.xxx.xxx
Client OS/browser: Windows 10 (Chrome 71.0.3578.98)

TCP options string: 020405780402080a008f89c900000000
MSS: 1400
MTU: 1440
TCP Window: 64240 (not multiple of MSS)
RWIN Scaling: 0 bits
Unscaled RWIN : 64240
Recommended RWINs: 64400, 128800, 257600, 515200, 1030400
BDP limit (200ms): 2570kbps (321KBytes/s)
BDP limit (500ms): 1028kbps (128KBytes/s)
MTU Discovery: ON
TTL: 44
Timestamps: ON
SACKs: ON
IP ToS: 00000010 (2)
Precedence: 000 (routine)
Delay: 0 (normal delay)
Throughput: 0 (normal throughput)
Reliability: 0 (normal reliability)
Cost: 1 (low cost)
Check bit: 0 (correct)
DSCP (DiffServ): CS0 000000 (0) - class 0, default traffic (RFC 2474).

So TCP timestamps 12+ pppoe 8+ ????=at least 1480 but why it's 1440
Can i have a look at your test? (hide important info)
also how it's possible to have 1492 or 1500 on some isp's ?

ifconfig output seems fine:

Summary

pppoe-wan Link encap:Point-to-Point Protocol
inet addr:10.54.106.50 P-t-P:10.0.0.10 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1480 Metric:1
RX packets:6245456 errors:0 dropped:0 overruns:0 frame:0
TX packets:5057702 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:6394391429 (5.9 GiB) TX bytes:1175822788 (1.0 GiB)

root@OpenWrt:~# ifconfig
br-lan Link encap:Ethernet HWaddr C0:4A:00:E7:23:46
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fd80:b841:8de6::1/60 Scope:Global
inet6 addr: fe80::c24a:ff:fee7:2346/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:320846 errors:0 dropped:3 overruns:0 frame:0
TX packets:263257 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:67901368 (64.7 MiB) TX bytes:151799892 (144.7 MiB)

eth0 Link encap:Ethernet HWaddr C0:4A:00:E7:23:46
inet6 addr: fe80::c24a:ff:fee7:2346/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6560721 errors:0 dropped:0 overruns:0 frame:0
TX packets:5083954 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2343733299 (2.1 GiB) TX bytes:1330843850 (1.2 GiB)

eth0.1 Link encap:Ethernet HWaddr C0:4A:00:E7:23:46
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:19722 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1270029 (1.2 MiB)

eth0.4 Link encap:Ethernet HWaddr C0:4A:00:E7:23:46
inet addr:10.10.20.21 Bcast:10.10.20.255 Mask:255.255.255.0
inet6 addr: fe80::c24a:ff:fee7:2346/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6560711 errors:0 dropped:110870 overruns:0 frame:0
TX packets:5064216 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6468108881 (6.0 GiB) TX bytes:1288741553 (1.2 GiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3748 (3.6 KiB) TX bytes:3748 (3.6 KiB)

pppoe-wan Link encap:Point-to-Point Protocol
inet addr:10.54.106.50 P-t-P:10.0.0.10 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1480 Metric:1
RX packets:6252374 errors:0 dropped:0 overruns:0 frame:0
TX packets:5061633 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:6400979740 (5.9 GiB) TX bytes:1177299962 (1.0 GiB)

veth0 Link encap:Ethernet HWaddr 16:DA:D1:25:D2:5E
inet6 addr: fe80::14da:d1ff:fe25:d25e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4724948 errors:0 dropped:3 overruns:0 frame:0
TX packets:6066176 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1169025488 (1.0 GiB) TX bytes:6459975362 (6.0 GiB)

veth1 Link encap:Ethernet HWaddr 82:64:6F:9D:6B:4C
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:6066164 errors:0 dropped:0 overruns:0 frame:0
TX packets:4724940 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6459960876 (6.0 GiB) TX bytes:1169019427 (1.0 GiB)

wlan0 Link encap:Ethernet HWaddr C0:4A:00:E7:23:48
inet6 addr: fe80::c24a:ff:fee7:2348/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4307006 errors:0 dropped:0 overruns:0 frame:0
TX packets:5513918 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1047341137 (998.8 MiB) TX bytes:6037823762 (5.6 GiB)

wlan1 Link encap:Ethernet HWaddr 78:44:76:BE:90:39
inet6 addr: fe80::7a44:76ff:febe:9039/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:719887 errors:0 dropped:0 overruns:0 frame:0
TX packets:809715 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:193034842 (184.0 MiB) TX bytes:677871570 (646.4 MiB)


#492

Well the 12 Bytes timestamps seem confirmed to me, but about PPPoE I am unsure. For the rest, I would recommend to redo the TCP analyzer test with a fresh browser after not using the browser for a while. I sometimes see lower values for MTU and MSS that disappear after a while, I assume this to be path MTU discovery in action...

Sure:

« SpeedGuide.net TCP Analyzer Results » 
Tested on: 2019.01.10 10:36 
IP address: 77.189.xxx.xx 
Client OS/browser: Mac OS (Safari 605.1.15) 
 
TCP options string: 020405ac010303050101080a3ba74dc60000000004020000 
MSS: 1452 
MTU: 1492 
TCP Window: 132480 (not multiple of MSS) 
RWIN Scaling: 5 bits (2^5=32) 
Unscaled RWIN : 4140 
Recommended RWINs: 63888, 127776, 255552, 511104, 1022208 
BDP limit (200ms): 5299kbps (662KBytes/s)
BDP limit (500ms): 2120kbps (265KBytes/s) 
MTU Discovery: ON 
TTL: 52 
Timestamps: ON 
SACKs: ON 
IP ToS: 00000000 (0) 

Pretty much as I expect, as I am using PPPoE and configured my computer to use RFC 1323 TCP timestamps.


#493

I have a great connection in general, and I get 1440 MTU as well. I wonder if some ISPs are using ipv4 in ipv6 tunnels to combat ipv4 exhaustion or other similar voodoo? It probably has more to do with something between you and the speedguide.net site than something right at your end of the connection.

I wonder if your DSCP rules about ACK are causing packet reordering and this is causing some of your problems?


#494

@moeller0
I did a test on my android phone and got the same results.

Nah, ISP's here is always doing stupid things, then we have a big and great infrastructure.
They have a ton's of ipv4 but they never give subscriber an ip(they say we give public ipv4 and even ipv6 only for business, or if you get a band from them, blah blah).

Let me check,...


#495

I just disabled the script and did a restart my router and my pc, same results!

It's not affecting anything at the moment, and the problems that i have exits before i start using SQM and DSCP marks!.

One more thing, when i use the https version of dslreports, bufferbloat(F) is too bad and ping is very bad too(above 400ms), but http is fine, dslreports is using TCP to measure ping!


#496

Well, at least it's good to have checked the DSCP script.

Do your problems vary in frequency throughout the day? Like does it happen a lot when other people are home and trying to stream their favorite videos etc? Does it happen during highest temperatures of the day (thermal issues in electronics devices?).


#497

No, it's not affected throughout the day.
At home i have only 2 phones using youtube at the same time(youtube is isolated from main download/upload), we can't stream videos with 1mbps(youtube is the only choice ATM).
ISP is providing iptv via a webpage(tv.box) and m3u8 file can be used on tv or video player.
It's not affected by high or low temperatures, usually i keep electronics cool in the summer!
If you want the truth, most people here is complain a lot about then internet and they have a lot of problems,
but only me have a few(negligible) problems cause i'm using openwrt!!! :smiling_face_with_three_hearts:


#498

Hey guys, finally got a nice setup working!

Inbound and outbound packet DSCP QoS

And VPN ingress over veth as well, and VPN to redirect only certain games through the VPN to optimize routing for low latency!

I think i should tag DSCP on OpenVPN ports now too? Inbound and outbound?

Als curious about network layer size, i assume I should keep 18 bytes?


#499

If your tagging happens in prerouting or postrouting it will affect all interfaces already. Since you use VPN for only games, you may want to tag all the encrypted packets high prio


#500

Currently I have this:

IPT="iptables"

## set up a 2 pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth

## get new veth interfaces up
ip link set veth0 up
ip link set veth1 up

## trun on promisc mode,sometimes it's needed to make bridge work
ip link set veth1 promisc on

## add veth1,veth3 to bridge
brctl addif br-lan veth1

## just to make sure there's nothing inside those 2 tables
ip rule del priority 100
ip rule del priority 101
ip route flush table 100

##ipset for bulk. They are bening filled by dnsmasq
ipset create bulk hash:ip

## flush mangle table
$IPT -t mangle -F PREROUTING

## add routing for veth0 this will handle all slow traffic
ip route add default dev veth0 table 100
ip rule add iif eth1.2 table 100 priority 100
ip rule add iif tun0 table 100 priority 101 ;For routing tun0 vpn traffic through veth as well.

# *          Latency Sensitive  (CS7, CS6, EF, VA, CS5, CS4)
# *          Streaming Media    (AF4x, AF3x, CS3, AF2x, TOS4, CS2, TOS1)
# *          Best Effort        (CS0, AF1x, TOS2, and those not specified)
# *          Background Traffic (CS1)

#Clear interface dscp marks, we don't trust ISP/VPN marks(also to use our own marks) so basically becomes best effort.
$IPT -t mangle -A PREROUTING -i eth1.2 -j DSCP --set-dscp 0
$IPT -t mangle -A PREROUTING -i tun0 -j DSCP --set-dscp 0

#Others
$IPT -t mangle -A PREROUTING -p icmp -j DSCP --set-dscp-class CS6

#VPN Traffic (For gaming)
$IPT -t mangle -A PREROUTING -p all --dport 443 -j DSCP --set-dscp-class EF
$IPT -t mangle -A PREROUTING -p all --sport 443 -j DSCP --set-dscp-class EF

#MWO specific (ingress UDP)
$IPT -t mangle -A PREROUTING -p udp -m iprange --src-range 5.135.129.0-5.135.129.255 -j DSCP --set-dscp-class EF
$IPT -t mangle -A PREROUTING -p udp -m iprange --src-range 198.27.85.0-198.27.85.255 -j DSCP --set-dscp-class EF

#Clear all outgoing (egress) packets from anything but 192.168.1.100
$IPT -t mangle -A PREROUTING -p all -m iprange --src-range 192.168.1.101-192.168.1.255 -j DSCP --set-dscp 0

$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 80,853,3455,8080 -j DSCP --set-dscp-class CS3
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 80,853,3455,8080 -j DSCP --set-dscp-class CS3

$IPT -t mangle -A PREROUTING -m set --match-set bulk src -j DSCP --set-dscp-class CS1 ##set dscp tag for our bulk ipset
$IPT -t mangle -A PREROUTING -p tcp -m multiport --ports 21,80,8080 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
$IPT -t mangle -A PREROUTING -p udp -m multiport --ports 21,80,8080 -m connbytes --connbytes 1048576: --connbytes-dir both --connbytes-mode bytes -j DSCP --set-dscp-class CS1
$IPT -t mangle -A PREROUTING -p udp -m multiport --dport 11132 -j DSCP --set-dscp-class CS1


vpn-policy-routing service currently routes all connections have EF (46) DSCP to the VPN (So my games pretty much)

All works fine, but I cannot see EF (46) marks on my client end with Wireshark, they are in fact Unknown.
Is this because I tag the VPN on port 443, but only encapsulated in tunnel? As in... once they get decrypted and send on their way to my local PC they are 'reset'? I guess they are in fact QoS'ed at the router level?

Otherwise something weird is going on. If i set EF (46) on firefox.exe and restart it, when i do a speedtest I can clearly see the bandwidth capped to what I set on veth0. So data definately flowing through there.

Perhaps I should use this?

**–passtos**

Set the TOS field of the tunnel packet to what the payload’s TOS is.

in openvpn so it copies tags, but im not sure if that requires server side support.


#501

I'm not quite sure what you're seeing, or asking here, but remember that the mangle table operates on your router so if you capture on your PC you'll see incoming packets tagged by your router, but outgoing ones will not be tagged at the point they're captured they get tagged only after the router receives them and prepares to forward them.

Is that helpful?