my results are not as good, spikes to 150+ms once or twice. on cake it is rock-solid on < 10ms.
How are you measuring it (details please)? I would be interested to try it on mine.
oh, this is just on dslreports speedtest
I am not seeing a plot of bufferbloat vs time. Can you share a result and point me to the section of page? The histograms at the top can be really misleading. What are the error bars? StdDev? StdErr? Just Min/max values? We really need to see all individual samples plotted in a distribution with some corresponding statistics.
Still FW issues here, devices on wifi can hit my forward to internal website fine.
But devices on ethernet can't.
Traffic should be forwarded like this:
Internal device > external ip on router > Internal webserver
@Ansuel Wifi drops seems to be gone tho', so I think you fixed that.
Lots to follow here. After I build nss code openwrt, I have to modprobe and run tc commands and then ip link command? Can this all just go into a startup script?
I just have it in a startup script. Makes it much easier.
What is your script? Is the order of commands right like @darksky posted?
What values to use for the tc commands? Do I take my max values from speed test and multiple by 0.95 like the wiki says and use them? How can I set a different value once I run the command?
My connection is 250 down so I try:
tc qdisc add dev nssifb root handle 1: nsstbl rate 250Mbit burst 1Mb
But I want to adjust to a lower number like 225 and see if I get better bufferbloat but I get file exists error:
tc qdisc add dev nssifb root handle 1: nsstbl rate 225Mbit burst 1Mb
RTNETLINK answers: File exists
@msilletti - You need a minor modification: substitute add
to change
:
tc qdisc change dev nssifb root handle 1: nsstbl rate 225Mbit burst 1Mb
You can see the rules you have setup for the device like this:
tc qdisc show dev nssifb
so i assume this is not as easy as adding a new file in /usr/lib/sqm..
i looked inside one of other .qos file and i see that concepts like "nssfq_codel", "nsstbl" and "nssifb" are - obviously - unknown.
sadly, i don't know how to go on with this..
Ok, i'm trying using the kong's .qos script.
i have this list of errors:
Mon Nov 23 14:38:51 2020 user.notice SQM: Starting SQM script: nss.qos on eth0, in: 900000 Kbps, out: 90000 Kbps
Mon Nov 23 14:38:51 2020 user.notice SQM: Using generic sqm_start_default function.
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /usr/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 0 u32 match ip protocol 1 0xff flowid 1:13
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /usr/sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 match ip protocol 1 0xff flowid 1:13
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel
Mon Nov 23 14:38:52 2020 user.notice SQM: WARNING: sqm_start_default: nss.qos lacks an egress() function
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /usr/sbin/tc qdisc add dev nssifb root handle 1: nsstbl rate 900000kbit burst 1Mb
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Cannot find device "nssifb"
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /usr/sbin/tc qdisc add dev nssifb parent 1: handle 10: nssfq_codel limit 4096 flows 1024 quantum 1514 target 5ms interval 100ms set_default
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Cannot find device "nssifb"
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: ip: FAILURE (1): /sbin/ip link set up nssifb
Mon Nov 23 14:38:52 2020 user.notice SQM: ERROR: cmd_wrapper: ip: LAST ERROR: Cannot find device "nssifb"
Mon Nov 23 14:38:52 2020 user.notice SQM: WARNING: sqm_start_default: nss.qos lacks an ingress() function
Mon Nov 23 14:38:53 2020 user.notice SQM: nss.qos was started on eth0 successfully
i fixed the missing nssifb simply modprobing it, but what about the errors in the egress function?
thanks
@ACwifidude I uploaded your latest image on my 7800 on Saturday morning minutes after you uploaded it. This image has run for 2 days w/o reboot! Thanks!
Got a link to the QoS script?
well, it was easier than i used to think
i just downloaded the Kong's 7800 NSS build and opened it with 7-zip.. too easy!
then root -> usr -> lib -> sqm -> nss.qos
While reviewing the logs for clues of the latest router crash, I've found the following in the kernel log:
Nov 21 18:26:34 owrt kernel: [198122.051642] 8<--- cut here ---
Nov 21 18:26:34 owrt kernel: [198122.051665] Unable to handle kernel NULL pointer dereference at virtual address 00000172
Nov 21 18:26:34 owrt kernel: [198122.053605] pgd = 7d0dcf02
Nov 21 18:26:34 owrt kernel: [198122.061988] [00000172] *pgd=00000000
Nov 21 18:26:35 owrt kernel: [198122.064542] Internal error: Oops: 17 [#1] SMP ARM
Nov 21 18:26:35 owrt kernel: [198122.068338] Modules linked in: ecm pppoe ppp_async ath10k_pci ath10k_core ath pppox ppp_generic nf_conntrack_netlink mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_quota xt_policy xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY slhc sch_cake nfnetlink nf_reject_ipv4 nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conncount iptable_raw iptable_nat iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables exfat crc_ccitt compat fuse sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred qca_nss_qdisc ledtrig_usbport ledtrig_heartbeat ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT nf_log_ipv6
Nov 21 18:26:35 owrt kernel: [198122.068436] nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 msdos ifb ip6_vti ip_vti sit qca_nss_drv ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 ip6_tunnel qca_nss_gmac tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat hfsplus cifs nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 sha1_generic md5 md4 echainiv ecb des_generic libdes cbc authenc arc4 usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug ext4 mbcache jbd2 crc32c_generic
Nov 21 18:26:35 owrt kernel: [198122.202446] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.77 #0
Nov 21 18:26:35 owrt kernel: [198122.224677] Hardware name: Generic DT based system
Nov 21 18:26:35 owrt kernel: [198122.231009] PC is at ieee80211_tx_dequeue+0xcc/0xed0 [mac80211]
Nov 21 18:26:35 owrt kernel: [198122.235560] LR is at ieee80211_tx_dequeue+0x9c/0xed0 [mac80211]
Nov 21 18:26:35 owrt kernel: [198122.241777] pc : [<bf6d6768>] lr : [<bf6d6738>] psr: 60000113
Nov 21 18:26:35 owrt kernel: [198122.247767] sp : c0c01b40 ip : 00005dc0 fp : d64c80a4
Nov 21 18:26:35 owrt kernel: [198122.254102] r10: d64c80a4 r9 : 00000000 r8 : 00000000
Nov 21 18:26:35 owrt kernel: [198122.259397] r7 : da7a0c40 r6 : da7a0c40 r5 : 00000001 r4 : d64c80a4
Nov 21 18:26:35 owrt kernel: [198122.264694] r3 : 0000000c r2 : 0000000a r1 : 00000000 r0 : da7a0cec
Nov 21 18:26:35 owrt kernel: [198122.271032] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none
Nov 21 18:26:35 owrt kernel: [198122.277630] Control: 10c5787d Table: 5b5b406a DAC: 00000051
Nov 21 18:26:35 owrt kernel: [198122.284920] Process swapper/0 (pid: 0, stack limit = 0x16442ebe)
Nov 21 18:26:35 owrt kernel: [198122.290736] Stack: (0xc0c01b40 to 0xc0c02000)
Nov 21 18:26:35 owrt kernel: [198122.296907] 1b40: 00000000 5bc40ac0 d64c8000 d64c8008 d64c8070 5cdc06c8 d9982ba0 c0c30000
Nov 21 18:26:35 owrt kernel: [198122.301253] 1b60: 0000000a da7a0cec 58413874 ba2e8ba3 bf70f27c d64c8084 d6bcbcc0 bf7952fc
Nov 21 18:26:35 owrt kernel: [198122.309500] 1b80: 00000000 00000000 da756d60 da756d60 00000001 eafad78f da7a0c40 0000fffe
Nov 21 18:26:35 owrt kernel: [198122.317745] 1ba0: c0c04e48 00000000 00000000 d64c80a4 da7a2f24 da7a0c40 da7a1ea0 00000000
Nov 21 18:26:35 owrt kernel: [198122.325993] 1bc0: da756d60 da7a2e70 d64c96a4 bf77b5c0 00000002 da7a2f24 da7a1ea0 da7a0d34
Nov 21 18:26:35 owrt kernel: [198122.334240] 1be0: d64c80a4 da7a0c40 d64c80a4 d64c80a4 d64c80a4 d9982b7c da7a0ce4 da7a0cec
Nov 21 18:26:35 owrt kernel: [198122.342486] 1c00: da7a0ce0 bf77ba64 da7a0c40 00000000 d6bcb180 bf6d56d8 00000000 00000000
Nov 21 18:26:35 owrt kernel: [198122.350732] 1c20: da756000 d6bcb180 d64c9000 da7564c0 00000001 c0c01ca0 da756000 d9657580
Nov 21 18:26:35 owrt kernel: [198122.358978] 1c40: da7a0c40 bf6dae50 00000000 00000000 da756d60 00000000 80000193 c0c57ac0
Nov 21 18:26:35 owrt kernel: [198122.367225] 1c60: ffffffff ffffffff dd98fa80 c0b3fa80 c0c04f7c 1ce50000 00000000 c0c08880
Nov 21 18:26:35 owrt kernel: [198122.375472] 1c80: c0c01c80 c0c01c80 d64c9000 e8e8fac0 05fb8fb7 b6ed9438 0000e782 c0363d3c
Nov 21 18:26:35 owrt kernel: [198122.383719] 1ca0: c0c01ca0 c0c01ca0 00000000 00000000 da7a0c40 da7564c0 d64c9000 d81b8000
Nov 21 18:26:35 owrt kernel: [198122.391965] 1cc0: 0000b430 00000002 00000000 d6bcb180 d6bcb180 d64141c6 da756000 da756000
Nov 21 18:26:35 owrt kernel: [198122.400211] 1ce0: bf357040 00000059 00000002 bf6db1f4 00000000 c03151d0 cff4904f ddce22d0
Nov 21 18:26:35 owrt kernel: [198122.408457] 1d00: 00006008 d6bcb180 00000000 00000000 da756000 c077dd94 00000002 d6bcb180
Nov 21 18:26:35 owrt kernel: [198122.416705] 1d20: bf357388 000000d2 dc281040 da756000 bf357040 00000059 00000002 bf310ec8
Nov 21 18:26:35 owrt kernel: [198122.424951] 1d40: 00000000 dd44ac80 bf357610 0000007f dc281040 bf357058 00000002 00000001
Nov 21 18:26:35 owrt kernel: [198122.433197] 1d60: dc231000 00000001 00000002 0000001e bf357600 00000001 dc231004 000000f0
Nov 21 18:26:35 owrt kernel: [198122.441444] 1d80: dd402a40 bf357610 00000040 bf357040 bf364000 00000000 00000040 bf337db4
Nov 21 18:26:35 owrt kernel: [198122.449691] 1da0: bf357600 bf312074 c0c03080 00000002 c0b3f60c 00000002 c0c08880 00000001
Nov 21 18:26:35 owrt kernel: [198122.457937] 1dc0: bf357610 00000040 c0c01e0f c0c01e10 1ce50000 00000100 c0c03d00 c0781620
Nov 21 18:26:35 owrt kernel: [198122.466184] 1de0: bf357610 012dda20 0000012c dd990400 c0c01e10 c07818c4 c0b40400 c0c2b508
Nov 21 18:26:35 owrt kernel: [198122.474431] 1e00: c0a3b190 c0a31200 00000008 00000000 c0c01e10 c0c01e10 c0c01e18 c0c01e18
Nov 21 18:26:35 owrt kernel: [198122.482676] 1e20: bf357610 00000000 00000003 c0c0308c c0c03080 40000003 ffffe000 00000100
Nov 21 18:26:35 owrt kernel: [198122.490923] 1e40: c0c03080 c0302298 00000001 dd52f000 00000001 c0b39208 c0b3f640 0000000a
Nov 21 18:26:35 owrt kernel: [198122.499169] 1e60: c0b39194 012dda1f c0c03d00 c09a4cc8 00200002 c09a4c90 dd52f068 ffffe000
Nov 21 18:26:35 owrt kernel: [198122.507417] 1e80: 00000000 00000001 00000000 dd410000 de803000 e2824340 e281db80 c0322bec
Nov 21 18:26:35 owrt kernel: [198122.515662] 1ea0: c0b3f60c c036f370 c0c05250 c0c18dd4 de80200c de802000 c0c01ee0 c05d30e0
Nov 21 18:26:35 owrt kernel: [198122.523910] 1ec0: c07243b4 80000013 ffffffff c0c01f14 0000b430 c0c00000 e2824340 c0301a8c
Nov 21 18:26:35 owrt kernel: [198122.532155] 1ee0: 00000000 0000b430 1ce50000 dd98fa80 dd7eb000 00000000 dd98ee30 0000b430
Nov 21 18:26:35 owrt kernel: [198122.540402] 1f00: 0000b430 00000000 e2824340 e281db80 00000015 c0c01f30 c07243b0 c07243b4
Nov 21 18:26:35 owrt kernel: [198122.548648] 1f20: 80000013 ffffffff 00000051 00000000 c0c01f6c c08faf24 dd7eb000 dd98f10c
Nov 21 18:26:35 owrt kernel: [198122.556896] 1f40: dd98ee30 77359400 c0c01f8f dd98ee30 dd7eb000 00000000 00000001 dd98ee30
Nov 21 18:26:35 owrt kernel: [198122.565142] 1f60: c0b3ee28 c0c0cc40 dd7eb000 c07247fc ffffe000 c0c04e68 c0c04ea8 c034abe4
Nov 21 18:26:35 owrt kernel: [198122.573389] 1f80: 00000001 00000000 10c5387d 01000000 00000000 000000ce c0c04e40 00000000
Nov 21 18:26:35 owrt kernel: [198122.581636] 1fa0: c0c3202c 00000001 c0b2da38 10c5387d 00000000 c034aef4 c0c32044 c0b00e5c
Nov 21 18:26:35 owrt kernel: [198122.589880] 1fc0: ffffffff ffffffff 00000000 c0b0058c c0b2da38 eaffd3c2 00000000 c0b00330
Nov 21 18:26:35 owrt kernel: [198122.598128] 1fe0: 00000051 10c0387d 0000136c 42ee8848 512f04d0 00000000 00000000 00000000
Nov 21 18:26:35 owrt kernel: [198122.606513] [<bf6d6768>] (ieee80211_tx_dequeue [mac80211]) from [<bf77b5c0>] (ath10k_mac_tx_push_txq+0x74/0x354 [ath10k_core])
Nov 21 18:26:35 owrt kernel: [198122.614676] [<bf77b5c0>] (ath10k_mac_tx_push_txq [ath10k_core]) from [<bf77ba64>] (ath10k_mac_op_wake_tx_queue+0xcc/0xd8 [ath10k_core])
Nov 21 18:26:35 owrt kernel: [198122.626042] [<bf77ba64>] (ath10k_mac_op_wake_tx_queue [ath10k_core]) from [<bf6d56d8>] (ieee80211_reserve_tid+0xd38/0xf88 [mac80211])
Nov 21 18:26:35 owrt kernel: [198122.638552] [<bf6d56d8>] (ieee80211_reserve_tid [mac80211]) from [<bf6dae50>] (__ieee80211_subif_start_xmit+0x7e0/0xaf4 [mac80211])
Nov 21 18:26:35 owrt kernel: [198122.650444] [<bf6dae50>] (__ieee80211_subif_start_xmit [mac80211]) from [<bf6db1f4>] (ieee80211_subif_start_xmit+0x90/0x37c [mac80211])
Nov 21 18:26:35 owrt kernel: [198122.662578] [<bf6db1f4>] (ieee80211_subif_start_xmit [mac80211]) from [<bf310ec8>] (nss_core_send_buffer+0x21b4/0x249c [qca_nss_drv])
Nov 21 18:26:35 owrt kernel: [198122.674790] [<bf310ec8>] (nss_core_send_buffer [qca_nss_drv]) from [<bf312074>] (nss_core_handle_napi+0x198/0x248 [qca_nss_drv])
Nov 21 18:26:35 owrt kernel: [198122.686632] [<bf312074>] (nss_core_handle_napi [qca_nss_drv]) from [<c0781620>] (__napi_poll+0x34/0x120)
Nov 21 18:26:35 owrt kernel: [198122.698296] [<c0781620>] (__napi_poll) from [<c07818c4>] (net_rx_action+0x108/0x2a0)
Nov 21 18:26:35 owrt kernel: [198122.707844] [<c07818c4>] (net_rx_action) from [<c0302298>] (__do_softirq+0x130/0x2d4)
Nov 21 18:26:35 owrt kernel: [198122.715656] [<c0302298>] (__do_softirq) from [<c0322bec>] (irq_exit+0xbc/0xe0)
Nov 21 18:26:35 owrt kernel: [198122.723467] [<c0322bec>] (irq_exit) from [<c036f370>] (__handle_domain_irq+0x6c/0xd0)
Nov 21 18:26:35 owrt kernel: [198122.730675] [<c036f370>] (__handle_domain_irq) from [<c05d30e0>] (gic_handle_irq+0x5c/0xb8)
Nov 21 18:26:35 owrt kernel: [198122.738660] [<c05d30e0>] (gic_handle_irq) from [<c0301a8c>] (__irq_svc+0x6c/0x90)
Nov 21 18:26:35 owrt kernel: [198122.747248] Exception stack(0xc0c01ee0 to 0xc0c01f28)
Nov 21 18:26:35 owrt kernel: [198122.754634] 1ee0: 00000000 0000b430 1ce50000 dd98fa80 dd7eb000 00000000 dd98ee30 0000b430
Nov 21 18:26:35 owrt kernel: [198122.759760] 1f00: 0000b430 00000000 e2824340 e281db80 00000015 c0c01f30 c07243b0 c07243b4
Nov 21 18:26:35 owrt kernel: [198122.767998] 1f20: 80000013 ffffffff
Nov 21 18:26:35 owrt kernel: [198122.776250] [<c0301a8c>] (__irq_svc) from [<c07243b4>] (cpuidle_enter_state+0x94/0x498)
Nov 21 18:26:35 owrt kernel: [198122.779981] [<c07243b4>] (cpuidle_enter_state) from [<c07247fc>] (cpuidle_enter+0x30/0x4c)
Nov 21 18:26:35 owrt kernel: [198122.788053] [<c07247fc>] (cpuidle_enter) from [<c034abe4>] (do_idle+0x1d8/0x240)
Nov 21 18:26:35 owrt kernel: [198122.796123] [<c034abe4>] (do_idle) from [<c034aef4>] (cpu_startup_entry+0x1c/0x20)
Nov 21 18:26:35 owrt kernel: [198122.803770] [<c034aef4>] (cpu_startup_entry) from [<c0b00e5c>] (start_kernel+0x4dc/0x4ec)
Nov 21 18:26:35 owrt kernel: [198122.811232] Code: e2033007 e7923103 e59d2020 e0823003 (e5d33166)
Nov 21 18:26:35 owrt kernel: [198122.819667] ---[ end trace 81b4af63237ac96f ]---
Nov 21 18:26:35 owrt kernel: [198122.825787] Kernel panic - not syncing: Fatal exception in interrupt
@Ansuel, does this mean anything to you ?
Thanks for writing this. I used it and have a working image.
What is the status of of the upstream nss driver? I see it is version 1.0 from a few years ago.
**********************************************************
* Driver :NSS GMAC Driver - RTL v(3.72a)
* Version :1.0
* Copyright :Copyright (c) 2013-2018 The Linux Foundation. All rights reserved.
**********************************************************
Does that mean the development of it has halted? Is there newer code we can use?
@quarky. It seems that OpenVPN module developer will not support AES-CBC-SHA-HMAC mode. I'm explaining to him the reason why we need in the openvpn-dev mailling list. If you have interest, welcome to join the discussion!
From my understanding, adding other crypto algo should not be difficult for us once the kernel drivers are done. Now, if they plan to deprecate AES-CBC-SHA-HMAC for future versions of OpenVPN, then we won’t have any server or client to connect to that we can ‘accelerate’ even if we can add AES-CBC-SHA-HMAC support.
Let’s see how it goes.
Btw, OpenSSL uses the cryptodev or Linux ALG to tap into h/w crypto engines from user space. OpenVPN kernel driver will not use OpenSSL.
If OpenVPN is in kernel, it’ll use the kernel’s crypto code. If a h/w accelerated crypto algo driver is installed, OpenVPN will use it automatically, if the driver priority is highest, which usually is for such drivers.
Version like all the other things in qcom house is confused ahahha
actually the gmac version is latest commit of qsdk 10... not that old honestly. And after all its a ethernet driver, changes are required only if there are present bugs.