If you based your build of the backports branch then then feed is included by default, otherwise I have no idea what you did
btw, JFYI, but consider it as my self-memo-note to don't forget about it (as for now I've not enough spare time to debug further):
Looks like qca-nss-clients fails to build with CONFIG_BONDING enabled:
/home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1/bridge/nss_bridge_mgr.c: In function 'nss_bridge_mgr_add_bond_slave':
/home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1/bridge/nss_bridge_mgr.c:266:18: error: implicit declaration of function 'bond_get_id'; did you mean 'find_get_pid'? [-Werror=implicit-function-declaration]
266 | bondid = bond_get_id(bond_master);
| ^~~~~~~~~~~
| find_get_pid
cc1: all warnings being treated as errors
make[5]: *** [scripts/Makefile.build:280: /home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1/bridge/nss_bridge_mgr.o] Error 1
make[4]: *** [scripts/Makefile.build:497: /home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1/bridge] Error 2
make[3]: *** [Makefile:1822: /home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1] Error 2
make[3]: Leaving directory '/home/mva/.vcs_repos/openwrt/ax3600-stuff/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/linux-5.10.79'
make[2]: *** [Makefile:103: /home/mva/.vcs_repos/openwrt/ax3600/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-clients-2021-04-29-b93c72c1/.built] Error 2
make[2]: Leaving directory '/home/mva/.vcs_repos/openwrt/ax3600-stuff/feeds/nss_packages/qca/qca-nss-clients'
There is mentioned code hidden inside #ifdev BONDIG_SUPPORT, and Makefile defines this if it found CONFIG_BONDING.
Probably, it misses some headers in #includes, but for now I'm ready to investigate this.
Here's a fix for your issue:
and also
2 Likes
@robimarko your latest build was checked on AX6 and it works fine. I did not have enough time to check if the ETSI radar crap is really resolved, will get back to you about that.
MOD: loaded back a BDF where my country is back in ETSI reg domain, and it seems it works. IT applies the 600 sec CAC but enables the 160MHz channel with proper power after the CAC passes.
One more thing: it seems NSS.HK.11.4.0.5-6-R is out: link
Dont get mislead by the readme.txt, if you look inside the files, you will find the correct version string. Compared to ours, this is one bugfix release more recent. So I loaded it out of curiosity, works fine so far.
@robimarko @Ansuel and all the other devs and testers involved, you guys are legends, hats off to you gents!!
I built the IPQ807x-5.10-backports branch yesterday (just before your latest fixes), had to add a few patches to it (my bonding patch posted above, the sysupgrade on the same rootfs patch and the RX decap offload pull request, mainly), and I flashed it on rootfs_1 on an AX3600.
I'm seeing incredible results:
RAM fluctuates between 61-68%, never goes above, it's been running for about 12h now.
I'm using all the radios with multiple AP's on each:
I'm using both IPv4 and IPv6 (the IPv6 NSS bug is fixed, no more kernel panics!!), WireGuard site-to-site, split-routing, mwan3, NAT + NAT6 etc. I have around 10 devices on the Wi-Fi all the time (most have AX cards), 4 are IoT/persistent on the 2.4G IoT radio/AP and some IoT stuff plugged in on all the physical ports in their IoT/dedicated "VLANs" (Philips Hue, PiHole etc.)
I'll keep an eye on the RAM, and post updates, but so far, I think we have a WINNER!
Btw, WireGuard speeds are amazing now, I think I'm hitting the maximum that Mullvad can handle:
On the guest APs (connected to WAN), I'm hitting around 700mbps (my WAN is 1Gbps).
Pure awesomeness!
I'm curious about 2 things though:
- Is it normal for the builds to take SO LONG? I'm building in a Docker container on a Gentoo VM, I gave it 4 AMD Ryzen 5900HX cores and it literally took.. 14-15h. How can I shorten this time in case I wanna make changes (or for example, test your latest commits), ccache and obviously more cores don't necessarily seem to help? The longest part of the builds seem to be the ipkg-build phases for each package, they're not running parallelized, and it literally takes minutes+ for some packages (especially the kernel modules)..
- How are things looking for the AX9000? Should I expect similar results?
2 Likes
Something is really wrong if it takes 14+ hours to build, I am building on a 5950X so even from a scratch is under and hour, and once the toolchain is compiled it will be way faster.
AX9000 is in a rough state, most noticeably ath11k PCI support is not working so most of the 5GHz spectrum is unusable
I am compiling the source in a VM with 4 cores on top of a 2600x, but I dont think it takes an hour even if I rebuild it from a complete clean state. More like 40-50 minutes. Most of the build time is the toolchain for sure. What I would recommend is to do a "make download" before make, as that speeds up the process, also it is less prone to failures during multi-thread make. Previously I had these kind of issues when the compiler thrown a weird error, and at the second attempt with single thread, it passed.
The toolchain and the tools took about 10-15m max. I did make download separately, literally followed the github workflow. As I mentioned, ipkg-build takes ages and it doesn't seem to run in parallel, it's literally doing package by package sequentially and something is stalling it, maybe Docker is interfering somehow?
Sounds like I'll just spin up a Debian VM for build purposes alone and be done with it..
Here's my .config as a reference for replicating my results, the final image is 44MB (heh, the Chinese QSDK 128MB partition layout helps )
OpenWRT AX3600 .config - Pastebin.com
Btw, I just remembered, another thing that might help or not, I have some "non-standard" sysctl' values which I put together throughout the years for OpenWRT devices (picked up a few from Xiaomi's stock OS for NSS), some related to the buffers:
# Fine-tuning
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.ip_forward = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.igmp_max_memberships = 100
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_dsack = 1
# Buffers
net.core.rmem_default = 256960
net.core.rmem_max = 513920
net.core.wmem_default = 256960
net.core.wmem_max = 513920
net.core.netdev_max_backlog = 2000
net.core.somaxconn = 2048
net.core.optmem_max = 81920
net.ipv4.tcp_mem = 131072 262144 524288
net.ipv4.tcp_rmem = 8760 256960 4088000
net.ipv4.tcp_wmem = 8760 256960 4088000
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 2048
# WireGuard
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
# Low latency
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_mtu_probing = 1
# QCA-NSS-ECM
net.netfilter.nf_conntrack_tcp_no_window_check = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 1
net.bridge.bridge-nf-filter-vlan-tagged = 1
#net.netfilter.nf_conntrack_max = 1048576
2 Likes
@bitthief
Do you use it as a dump AP or as your main router?
If you use it as your main router, i'm very interested in your image.
Would you make it available for me and others to try? Which flash manual did you use to flash?
@robimarko / @Ansuel iv given up investigating the ath11k memory usage for now (as far as I can tell it's most likely how it's meant to work ?)
In any case, what's needs cleaning up so we can get the ipq8074 target into mainline openwrt trees ?
Is the latest LTS kernel still causing issues with PCI / ath init ?
Well, the whole target essentially needs to be redone from scratch so that its clean, as currently there i stuff added and then removed etc.
I haven't tried 5.15 in a while but as far as I remember it wasn't working, 2.0 port wasn't working for sure.
That is my idea... they engineered the driver with workqueue and memory is freed only with spike/actual utilization. One of the reason we have leak only on idle/with device connected doing nothing.
There is probably a buffer limit or something when it gets flushed, otherwise it just keeps the memory reserved
BTW @Ansuel I am scratching my head, finally bothered to cleanup ath11k PCI and add the required kernel modules instead of enabling it built-in so that it can be merged into OpenWrt but this QCA6391 card is driving me nuts.
I constantly have BAR no memory errors, first on IPQ4019 and now I moved on to Armada 7040 and its the same.
Armada 7040 has 15MB of PCI MEM space allocated.
yes that... buffer full -> ring is freed -> memory not reserved anymore.
Think reducing the memory available to ath11k seems to be the only way
@robimarko can you post the pci node in the dts?
Sure:
CP11X_LABEL(pcie2): pcie@CP11X_PCIE2_BASE {
compatible = "marvell,armada8k-pcie", "snps,dw-pcie";
reg = <0 ADDRESSIFY(CP11X_PCIE2_BASE) 0 0x10000>,
<0 CP11X_PCIEx_CONF_BASE(2) 0 0x80000>;
reg-names = "ctrl", "config";
#address-cells = <3>;
#size-cells = <2>;
#interrupt-cells = <1>;
device_type = "pci";
dma-coherent;
msi-parent = <&gic_v2m0>;
bus-range = <0 0xff>;
/* non-prefetchable memory */
ranges = <0x82000000 0 CP11X_PCIEx_MEM_BASE(2) 0 CP11X_PCIEx_MEM_BASE(2) 0 CP11X_PCIEx_MEM_SIZE(2)>;
interrupt-map-mask = <0 0 0 0>;
interrupt-map = <0 0 0 0 &CP11X_LABEL(icu_nsr) 23 IRQ_TYPE_LEVEL_HIGH>;
interrupts = <23 IRQ_TYPE_LEVEL_HIGH>;
num-lanes = <1>;
clock-names = "core", "reg";
clocks = <&CP11X_LABEL(clk) 1 12>, <&CP11X_LABEL(clk) 1 14>;
status = "disabled";
};
This is the Armada one, IPQ4019 is below.
It only has 1M of PCI MEM space.
pcie0: pci@40000000 {
compatible = "qcom,pcie-ipq4019", "snps,dw-pcie";
reg = <0x40000000 0xf1d
0x40000f20 0xa8
0x80000 0x2000
0x40100000 0x1000>;
reg-names = "dbi", "elbi", "parf", "config";
device_type = "pci";
linux,pci-domain = <0>;
bus-range = <0x00 0xff>;
num-lanes = <1>;
#address-cells = <3>;
#size-cells = <2>;
ranges = <0x81000000 0 0x40200000 0x40200000 0 0x00100000>,
<0x82000000 0 0x40300000 0x40300000 0 0x00d00000>;
interrupts = <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "msi";
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0x7>;
interrupt-map = <0 0 0 1 &intc 0 142 IRQ_TYPE_LEVEL_HIGH>, /* int_a */
<0 0 0 2 &intc 0 143 IRQ_TYPE_LEVEL_HIGH>, /* int_b */
<0 0 0 3 &intc 0 144 IRQ_TYPE_LEVEL_HIGH>, /* int_c */
<0 0 0 4 &intc 0 145 IRQ_TYPE_LEVEL_HIGH>; /* int_d */
clocks = <&gcc GCC_PCIE_AHB_CLK>,
<&gcc GCC_PCIE_AXI_M_CLK>,
<&gcc GCC_PCIE_AXI_S_CLK>;
clock-names = "aux",
"master_bus",
"slave_bus";
resets = <&gcc PCIE_AXI_M_ARES>,
<&gcc PCIE_AXI_S_ARES>,
<&gcc PCIE_PIPE_ARES>,
<&gcc PCIE_AXI_M_VMIDMT_ARES>,
<&gcc PCIE_AXI_S_XPU_ARES>,
<&gcc PCIE_PARF_XPU_ARES>,
<&gcc PCIE_PHY_ARES>,
<&gcc PCIE_AXI_M_STICKY_ARES>,
<&gcc PCIE_PIPE_STICKY_ARES>,
<&gcc PCIE_PWR_ARES>,
<&gcc PCIE_AHB_ARES>,
<&gcc PCIE_PHY_AHB_ARES>;
reset-names = "axi_m",
"axi_s",
"pipe",
"axi_m_vmid",
"axi_s_xpu",
"parf",
"phy",
"axi_m_sticky",
"pipe_sticky",
"pwr",
"ahb",
"phy_ahb";
status = "disabled";
};
I mean did you try playing with the ranges binding?
We had something like that with ath10k with ipq806x as in 5.10 they added more check about memory allocation
Well not as its not like you can just extend the range as you feel like.
ATF allocates memory like this for PCIe 2:
/* PEX2_X1 window */
{0x00000000f8000000, 0x1000000, PEX2_TID},
{0x00000000c0000000, 0x30000000, PEX2_TID},
{0x0000000800000000, 0x100000000, PEX2_TID},
true but it was pointed that a very big size was very strange (x86 target doesn't have that space) and probably the driver doesn't use it in reality.
Wonder if we can try reduce the size from the driver?
Is 5.15 required for mainline openwrt now ornis 5.10 still supported ?
The issue is that the card wants to allocate a 16MB BAR which I think only Armada 3k has since they extended it to 128MB.
@Apache14 5.10 is fine as it will be in the next OpenWrt release, they are refusing to even merge 5.15 until the code is branched