Well, the affected device (AFAIK) is the EA6350v3, not the other two that this build support (XX8300 or Convexa-X). I do not own the XX8300, but the EA6350v3 does have an amplifier for the 2.4GHz band (as opposed to the XX8300 which only have one for 5 GHz, and the Convexa-S which have none), so this can be the direct solution to the problem.
I need to check the OEM source code and probably test some GPIOs, but at this point I am more interested in why the kernel is crashing without a reason. Candela Technologies was also informed about the issue and their newest beta firmware seems to improve the wireless performance a lot. Hopefully, with the subsequent activation of the amplifier it can perform similar to the OEM firmware.
I am not going to submit this to OpenWrt, anyway, but as per the license anyone may submit my code in my name (with proper attribution) if this works, and all users can benefit from all the findings in this ROM.
I have seen this on multiple devices and targets (at least ipq806x/ nbg6817, lantiq/ bthub5 and ath79/ tl-wdr4300 - the later is ath9k-only), with (most severely on the tl-wdr4300)- and without PPPoE being involved, I don't think these issues are specific to ipq40xx or ath10k{,-ct}.
Sorry for the misunderstanding, I was trying to say that Candela was informed about clients being kicked-out aggressively, dropping of frames and so on. They are working on a permanent solution, which hopefully will land on the official firmware some day.
Anyway, my MT7621 build does not crash, however this IPQ40xx build crashes from oops (null pointer de-reference) on either the DMA subsystem or the ath10k driver. The crash affects all supported devices for this. Thankfully, I have RAMOOPS support so I can see where and when it crashes as it happens. I don't understand why, anyway.
Thanks for pointing out this valuable piece of information, I am going to keep you informed of the findings.
PS: while you are here, you probably can take a look at this, hopefully you can find it interesting.
I removed V2.06 with the V2.04 hot fix and loaded V2.04 Plus the hot fix. Wireless range and connection stability was good. After 3 hours uptime the router rebooted.
When I was running V1.12 I don't remember it having the random reboot problem that we are experiencing with the newer versions. It seemed stable but as I recall it had poor wireless performance. My question is could V1.12 be used with the hotfix from V2.04 to improve wireless performance?
Thanks for the report, I am going to remove it from GitHub too. I am currently testing the new one, after deleting everything and cloning again, and it have not crashed. Anyway, I will wait until the weekend.
I don't know if you read the conversation with the other developer, but it looks like the device needs an amplifier (or two) to be enabled and after that, the wireless performance will be definitely fixed. But first things first, we need a kernel that does not crash. Keep tuned about my findings, I think that enabling the amplifier is the answer and solution to the problem.
Answering your question: yes, probably updating the new Candela firmware will help in older version. Sadly, I can no test it right now because I am waiting for the device to crash (hopefully it doesn't).
I loaded V1.12 plus the hotfix from V2.04. The router carried 14 to 16 connections and it ran for 18 hours and then rebooted. During this 18 hour time period there were 8 to 9 wireless 2.4 Ghz connections which were stable over that time span.
Other user and I, we are testing the next build. Currently, my two devices have been running without crashing for 5 days. I will wait the reports from the other tester at day 7, and at that moment I will declare it stable under the v2.06 version and upload it again. After that, the v2.10 (I will name I differently this time) will have the wireless amplifiers enabled and it should be working with a performance similar to the factory firmware.
What router do you have that has MT7621 and what Openwrt FW load are you using? I have an EA7300V1 that I believe has a MT7621A. I am currently running factory FW on it but I would like an Openwrt load if one is available that is stable.
It's likely the EA6350v3 would already have the amplifiers enabled since the 5Ghz chip is the only one with an enable pin. Seems they just pull it high on this board from what you've reported. The 2.4Ghz chip can only be switched off by a power pin, so it would't make any sense at all for the designers to do that(waste a gpio and transistor). Have a couple coming in the mail so I can actually check in a couple days.
Hello, I was expecting my other tester to reach me, but he didn't. Anyway, I have good results at this point. My Convexa-S haven't crashed in more than 8 days. My EA6350v3 either, but there was a power shortage, and I can't be 100% sure about it. My Convexa-S didn't reboot because it is connected to a UPS, white the EA6350v3 is on the second floor (without a UPS).
Anyway, I have uploaded the files to GitHub under the v2.06, so anyone interested can check it out.
If what @fdm says is true (I haven't done the research yet), it is actually quite sad to know that the device is likely unfixable. I didn't want to give up, but I see no further room for improvement at this point if that's true. Maybe an ath10k guru can help, but it's certainly beyond what I can do or know.
I got my two EA6350v3 routers in, flashed one with this firmware, and compared them against an AC66U I have laying around. The router I flashed outperformed both routers on 2.4Ghz and preformed worse on 5Ghz. Really impressed so far, though I still need to do actual testing.
If the PA were disabled, I'd expect to be able to pick up signal right next to router and lose it after ~1ft of separation.
That sounds right. Researching in the OEM source code I did not found something suggesting that they need to be enabled.
I don't know if it was a kernel or a firmware issue, and I cannot pinpoint where it started or stopped working correctly. But what is for sure is that this:
Outperforms stock OpenWrt
Outperforms the very first versions of itself
It is a real, tangible and measurable improvement but I lost the track of what was the exact cause.
Since v2.06 is working with minor issues, I'll come back with another public build by Feb 2021 (if I still alive, hopefully yes) and I would love to hear feedback during that time.
P.S.: Happy new year to everyone and thank you very much for your effort, patience and support. This would be impossible without all of the people involved, so enjoy our work and I wish you a better 2021.
Why sysupgrade cannot keep the settings after upgrade from a previous build, like stable and other master? What is so different? Are the memory optimizations?
Is it still possible to avoid entering manually all settings when upgrading and get reconnected easily?
Willing to test / try the new build, but as these days everyone is so glued to the online, youngsters and the lady don't easily tolerate any remotely longer internet downtime.
Thanks for your work. Keep it safe, breathe free and chill.
It is intentional and by design. The only difference, in reality, is that I decided it to be like that. There is no technical reason about that, for better or for worst. But you can preserve the settings as follows:
You can download a backup, flash the firmware and restore the backup. It may sound harder or more steps, but is actually a good practice to backup before an update, anyway.
Most settings are preserved, anyway, there are others that will revert, like the default IP and DNS server settings. All other things are not touched at all.
Thank you for your work throughout the year. I have ported the 2 .bin files from your hotfixes in github onto Openwrt 19.07.5 and can see a huge improvement on 2.4Ghz wireless, but it is still not on par (multiply streaming) as my GL-AR300M. Maybe Candela can optimise the code further?
In the meantime, how do we up shift to Openwrt so that everyone can benefits from your hard work?
I notice after sometime have passed, some of the clients cannot go above 3mbps on speed test. Was this expected?
I loaded the revised version 2.06 on my EA6350V3 and ran it for 4 hours and the router rebooted itself while streaming a netflix movie. I thought that there must be something wrong with my router because I have suffered random reboots with all of the V2.xx versions that I have tried. So I thought that I would try Openwrt version 19.07.5 to see if that version would suffer from random reboots as well. I have now been running 19.07.5 for 2 days, 1 hour and 47 minutes with 14 to 15 devices connected with no reboots experienced thus far. My network is very stable and I plan to run 19.07.5 for a few more days to further test its stability.
The V2.xx versions all have kernel versions 5.4.xx while Openwrt version 19.07.5 has kernel 4.14.209. Could the kernel be the root cause of the random reboot problem?
Okay, I would need your help (I asked you before, but you didn't take it seriously, probably).
I think that the kernel is, for some unknown reason, doing a oops to you. I would recommend trying to disable the panic_on_oops by SSH-ing into the device and performing the following command:
printf 0 > /proc/sys/kernel/panic_on_oops
After a couple of days, I would love to have your logs. You can send me the recollected logs by pstore by downloading the /bootfs/pstore folder and, of course, the logs of the running system. You can make them zip and send them to my email.
Since it is not longer crashing for me, I really need your logs to make a proper bug report (if needed).
