R7500v2 kernel 4.19 test

Example for tftpboot initramfs images via tty serial on the r7500v2 (i.e. for testing an initramfs image loaded to ram without having to flash an image to nand):

This example is based on the instructions provided by @quarky for the r7800 here.

Prerequisites:

  1. Compile and install the usb to tty device drivers (in this case provided by hiletgo):
$ make
$ sudo cp cp210x.ko /lib/modules/`uname -r`/kernel/drivers/usb/serial
$ sudo insmod /lib/modules/`uname -r`/kernel/drivers/usb/serial/usbserial.ko
$ sudo insmod /lib/modules/`uname -r`/kernel/drivers/usb/serial/cp210x.ko
  1. Install tftp server "tftpd-hpa" and "screen" on ubuntu. If using the "ufw" firewall, allow tftp (udp port 69):
$ sudo apt update
$ sudo apt install screen tftp-hpa tftpd-hpa
$ sudo service tftpd-hpa status
$ sudo service tftpd-hpa start
$ sudo ufw allow tftp
  1. Copy an initramfs image to the tftpd directory. In my case, I've already built, flashed (via the r7800 TFTP method here which works perfectly on the r7500v2) and tested an openwrt "factory" image on the router; however, it is likely not necessary to have an openwrt image already flashed to nand to tftpboot an initramfs from ram. I used the initramfs automatically generated during the build of a known working image to make sure the initramfs image will work.
$ sudo cp ~/openwrt/bin/targets/ipq806x/generic/openwrt-ipq806x-netgear_r7500v2-initramfs-uImage /var/lib/tftpboot/
  1. Connect the computer ethernet port to a LAN port on the router, set up and enable the computer ethernet as static ipv4 address/netmask: 192.168.1.10/24, gateway: 192.168.1.1. Connect the usb tty serial device to the router and computer (a usb extension cable to the usb tty adapter works for me).
  2. Start a screen session. I use:
sudo screen -h 1000 -L -Logfile ~/r7500v2-`date +"%Y%m%d-%H%M"`.log /dev/ttyUSB0 115200
  1. Power on the router.
  2. At this point you should see the router booting up in screen session started above. If not, you'll need to trouble shoot the usb tty serial setup - it took me a few tries... When you see something similar too:
U-Boot 2012.07 [local,local] (May 29 2015 - 19:03:53)

U-boot 2012.07 dni1 V1.5 for DNI HW ID: 29764958 NOR flash 0MB NAND flash 128MB RAM 512MB 1st Radio 3x3 2nd Radio 4x4
smem ram ptable found: ver: 0 len: 5
DRAM:  491 MiB
NAND:  SF: Unsupported manufacturer 00
ipq_spi: SPI Flash not found (bus/cs/speed/mode) = (0/0/48000000/0)
128 MiB
MMC:   
*** Warning - bad CRC, using default environment

PCI0 Link Intialized
PCI1 Link Intialized
In:    serial
Out:   serial
Err:   serial
 131072 bytes read: OK
cdp: get part failed for 0:HLOS
Net:   MAC1 addr:XX:XX:XX:XX:XX:XX
athrs17_reg_init: complete
athrs17_vlan_config ...done
S17c init  done
MAC2 addr:XX:XX:XX:XX:XX:XX
eth0, eth1
Hit any key to stop autoboot:  1
(IPQ) # 

Interrupt U-Boot by pressing any key (in the serial console) when prompted. You have only 2-3 seconds before U-Boot proceeds to boot from the NAND flash.

  1. Find the ram load address to use with the uboot "tftpboot" command from the uboot command "printenv":
(IPQ) # printenv
baudrate=115200
bootargs=console=ttyHSL1,115200n8
bootcmd=sleep 2;   nmrp;  if loadn_dniimg 0 0x1480000 0x44000000 && chk_dniimg 0x44000000; then bootipq2; else fw_recovery; fi
bootdelay=2
config_ubi_prepare=mtdparts default; ubi part dnidata
ethact=eth0
ipaddr=192.168.1.1
language_ubi_prepare=mtdparts default; ubi part language
loadaddr=0x42000000
machid=1260
modelid=R7500v2
serverip=192.168.1.10
stderr=serial
stdin=serial
stdout=serial
updateloader=ipq_nand sbl && nand erase 0x00c80000 0x00580000 && imgaddr=0x42000000 && source $imgaddr:script

Environment size: 585/262140 bytes

In my case, the ram load address is "44000000" and is shown in the "bootcmd" environment variable as 0x44000000. Using the loadaddr environment variable "0x42000000" did not work (I think this is the nand address to load an image from).

  1. test that you can see the tftp server from uboot:
(IPQ) # ping 192.168.1.10
Using eth1 device
host 192.168.1.10 is alive
  1. Load the initramfs image via "tftpboot":
(IPQ) # tftpboot 44000000 192.168.1.10:openwrt-ipq806x-netgear_r7500v2-initramfs-uImage
Using eth1 device
TFTP from server 192.168.1.10; our IP address is 192.168.1.1
Filename 'openwrt-ipq806x-netgear_r7500v2-initramfs-uImage'.
Load address: 0x44000000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         #################################################################
         ###############################################################
done
Bytes transferred = 5683228 (56b81c hex)
  1. boot the image:
(IPQ) # bootm
   Image Name:   ARM OpenWrt Linux-4.19.57
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    5683164 Bytes = 5.4 MiB
   Load Address: 42208000
   Entry Point:  42208000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK 
mtdparts variable not set, see 'help mtdparts'
no partitions defined

defaults:
mtdids  : nand0=msm_nand
mtdparts: mtdparts=msm_nand:3584K@0x7980000(language),3M@0x7d00000(dnidata)
info: "mtdparts" not set
Using machid 0x1260 from environment

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.19.57 (n@E6410) (gcc version 7.4.0 (OpenWrt GCC 7.4.0 r10459+4-1174b94bc9)) #0 SMP Tue Jul 9 23:20:21 2019
[    0.000000] CPU: ARMv7 Processor [512f04d0] revision 0 (ARMv7), cr=10c5787d 
...

When the kernel log messages stop, hit any key to get a console and go from there. Note, I could not see my "overlay" files from my previous nand flashed openwrt image; however, typing "reboot" from the serial console and not interupting uboot as described above got me back to my nand flashed environment. Nice.

a picture of the r7500v2 with tty usb connection...

That is correct rx/ tx are always labeled from the point of view of the individual devices, meaning the r7500v2 listens on its rx pin for transmissions from the tx pin of your connector - and transmits back over its tx pin, which is listened to by the rx pin of your USB2serial adapter.

cpuidle update: it helps to be able to see boot messages via the serial console and to quickly compile, load and test kernel images.

I confirmed what @Ansuel observed in prior posts. For kernel 4.19 in spm.c, spm_dev_probe is never called (or pr_err fails silently from here) and qcom_cpuidle_init will always return -ENXIO. However, state_count does get to 2, so it looks like the purpose of calling qcom_cpuidle_init is to set fns and per_cpu(qcom_idle_ops, cpu) = fns.

Whats new (for me) is that this also happens in 4.14.

So I'm back to trying to understand why 4.14 calls ret = cpuidle_register_driver(drv) before skipping the ENXIO error and 4.19 will exit on ENXIO before calling this function... @Ansuel observed that calling this function earlier results in a boot loop and my own attempts skipping over the error also results in a boot loop... seems broken to me and I don't think the answer is in this patch set (but who am I to judge).

My other clues are:
qcom-ipq8064.dtsi:

saw0: regulator@2089000 {                                       
                        compatible = "qcom,saw2", "syscon";

"syscon" (system controller?) does not show up in the qcom-ipq8064.dtsi vanilla kernel sources - so I'll try to understand why its here...

and

@Ansuel's suggestion to look at tscr

1 Like

yes all is correct

tscr is the only one that look for compatible syscon so i think syscon = tscr BUT STILL... in tscr there is nothing about cpuidle... it's all about usb inizialization

IMO the following does nothing more than continue to demonstrate my lack of understanding... but it does get

cat /sys/devices/system/cpu/cpuidle/current_driver

return "arm_idle" on 4.19 after booting.

I made the following hackish edits to 4.19 cpuidle-arm.c to make it "work" like 4.14:

	/*                                                                      
         * Allow the initialization to continue for other CPUs, if the reported
         * failure is a HW misconfiguration/breakage (-ENXIO).                  
         */
        if (ret) {
                //pr_err("CPU %d failed to init idle CPU ops\n", cpu);
                ret = ret == -ENXIO ? 0 : ret;
                //goto out_kfree_drv;                                           
        }

        ret = cpuidle_register_driver(drv);
        if (ret) {
                if (ret != -EBUSY)
                        pr_err("Failed to register cpuidle driver\n");
                goto out_kfree_drv;
        }
        /*                                                                      
         *dev = kzalloc(sizeof(*dev), GFP_KERNEL);                                
         *if (!dev) {                                                             
         *        ret = -ENOMEM;                                                  
         *        goto out_unregister_drv;                                        
         *}                                                                       
         *dev->cpu = cpu;                                                         
         *                                                                        
         *ret = cpuidle_register_device(dev);                                     
         *if (ret) {                                                              
         *        pr_err("Failed to register cpuidle device for CPU %d\n",        
         *               cpu);                                                    
         *        goto out_kfree_dev;                                             
         *}                                                                       
        */
        return 0;

I'm away until Monday. I'd really like to get some kind of spec sheet for ipq8064. Are "spm" and "saw" even used?

EDIT: reference this and this regarding cpuidle, qcom-scm driver (not relevant?), and the transition from 4.4 to 4.9.

This way driver gets register but not the device... So it was actually broken in 4.14? Some way to test cpuidle in 4.14?

that's how i see edits/patches evolving in the linux-arm mailing list...

idk but I think its possible. Still contemplating how to test cpuidle. Also, I'd like to know if its useful - its my understanding that having arm cpuidle should use less power and enhance "performance" but I have not seen this proven.

Before I go for the weekend, I'm creating a cleaned up patch and will post a link to it in my "k419" github branch.

EDIT: a less hackish patch on github here.

A Power meter should be sufficient
Our arm cpu doesn't support suspend mode that actually disable cores but it does support the disabling of some part to save power so...
About performance and cpu s along we have another driver. Anyway will try to check about tsense problem this night

Your pretty close now i think @anon98444528

I managed to squeeze out these, although they are likely just "missing stuff" related;

[    1.593968] Speed bin: 0
[    1.598317] PVS bin: 5
[    1.602797] DT idle-states: Parsing idle state node /cpus/idle-states/spc failed with err -19
[    1.807413] cpufreq: cpufreq_online: CPU1: Running at unlisted freq: 387500 KHz
[    1.812917] cpufreq: cpufreq_online: CPU1: Unlisted initial frequency changed to: 600000 KHz

Interestingly, independently also wound up at "syscon" as you did....

Was essentially trying various compatible, saw, and acc options from;
https://linux-arm-kernel.infradead.narkive.com/roIsCrjt/patch-v9-3-9-arm-dts-qcom-add-power-controller-device-node-for-8974-krait-cpus

( seemed that maybe.... "saw" needs to be changed maybe to saw-vXX , perhaps just saw is not 4.19 capable? )

cpuidle update: progress of sorts.

I want to see if "arm-idle" does something and there are sysfs entries set up to show some usage stats as described here... only they didn't show up. This is due the fact I took out the dev setup in cpuidle-arm.c (up to now leaving it in causes a boot loop).

I put the dev reg stuff back into cpuidle-arm.c and worked out that the boot loop is caused by trying to call the "qcom,idle-state-spc" via the spm driver (which for reasons I still don't understand) isn't used/probed (perhaps its not supposed to be for ipq806x).

I neutered the qcom,idle-state-spc functionality (comment out the body of qcom_cpu_spc in spm.c and have it return 0, i also changed

use_scm_power_down = false

spm.c qcom_cpuidle_init but I don't know if this is necessary) and the boot loop is gone.

Now

/sys/devices/system/cpu/cpu0/cpuidle/
/sys/devices/system/cpu/cpu1/cpuidle/

shows up. For me cpu[0|1]/cpuidle/state1 corresponds to "spc" as opposed to WFI (I think WFI is "arm-idle") so I

echo 1 > cpu[0|1]/cpuidle/state1/disable

to disable spc for good measure. Now I have something to look at... not sure if it means anything yet. Also I'm not sure if I can compare "none" vs "arm-idle" this way.

FWIW it is also possible to change the govenor via the cpuidle_sysfs_switch boot option as described in the cpuidle sysfs docs linked in this post without all the hacking above.

I'll update my patches in a bit if anyone want to look or try it.

1 Like

cpuidle update:

I've got the spm driver working by adding:

        { .compatible = "qcom,ipq8064-saw2-v1.1-cpu",
          .data = &spm_reg_8064_cpu },

in spm.c and editing qcom-ipq8064.dtsi:

		saw0: regulator@2089000 {
                        compatible = "qcom,saw2",
                                     "qcom,ipq8064-saw2-v1.1-cpu",
			             "syscon";
                        reg = <0x02089000 0x1000>, <0x02009000 0x1000>;
                        regulator;
		};

                saw1: regulator@2099000 {
                        compatible = "qcom,saw2",
                                     "qcom,ipq8064-saw2-v1.1-cpu",
		                     "syscon";
			reg = <0x02099000 0x1000>, <0x02009000 0x1000>;
		        regulator;
                };

I tried a similar change under saw_l2. I observed (via pr_err statements) spm_dev_prob starts for this but (gracefully) does not finish so I've left it out.

WARNING: I don't know if this approach will get the spm driver working on ipq8065 (r7800) systems. Since spm.c adjusts "power" to a physical device (the cpu) any prospective tester should consider if they can afford a new router before trying this. The data hard coded in &spm_reg_8064_cpu in spm.c appears to be necessary (router won't boot for me without it). I don't know if this data will work for ipq8065 systems.

I can also get both idle states "arm-idle" and "qcom,idle-state-spc" working as shown by

/sys/devices/system/cpu/cpu[0|1]/cpuidle/state[0|1]/usage

Note there is a cleaner way to disable "qcom,idle-state-spc" than by the spm.c hackery in my prior post. Turns out adding status = "disabled":

                idle-states {                                                   
                        CPU_SPC: spc {                                          
                                compatible = "qcom,idle-state-spc",             
                                                "arm,idle-state";
                                status = "disabled"
                                entry-latency-us = <400>;                       
                                exit-latency-us = <900>;                        
                                min-residency-us = <3000>;                      
                        };                                                      

in qcom-ipq8064.dtsi will disable only "qcom,idle-state-spc" and leave "arm,idle-state" functional. The "dev" reg code in cpuidle-arm.c is still necessary to get the additional sysfs info. Remove status = "disabled" to get the spc idle state. I observe after ~40 min up with no load, the spc driver gets used the most.

I have not yet posted patches to my github k419 branch but I plan to do so and will post a link once its ready.

1 Like

didn't fry my 8065... usb init froze tho' ( EDIT: see edit 1 ) ( likely my hacky DTS ) ..... see if same happens on other "known semi-well-configured-for-usb" 806x systems...

did get two found operations without error tho'

[    5.816223] qcom-tsens 900000.thermal-sensor: can't request region for resource [mem 0x00900000-0x0090367f]                            
[    5.821936] qcom-tsens 900000.thermal-sensor: tsens init failed                                           
[    5.831637] qcom-tsens: probe of 900000.thermal-sensor rejects match -19   
[    5.887099] Speed bin: 0                                                                                  
[    5.895006] PVS bin: 5                                                                                    
[    5.899124] Registering platform device 'cpufreq-dt.0'. Parent at platform                                
[    5.899596] device: 'cpufreq-dt.0': device_add                                                            
[    5.906550] bus: 'platform': add device cpufreq-dt.0                                                      
[    5.911138] cpuidle: enable-method property 'qcom,kpss-acc-v1' found operations                           
[    5.916280] cpuidle: enable-method property 'qcom,kpss-acc-v1' found operations                           
[ 

EDIT1: ( maybe usb was a guesstimate it's pre-internal mmc-init.... )

[    1.772518 ] cpu cpu1: Linked as a consumer to regulat
HANGSHERE
[    1.801974 ] mmc0: new high speed MMC card at address 0001
1 Like

i neglected to mention that I changed qcom,kpss-acc-v1 to qcom,kpss-acc-v2 in qcom-ipq8064.dtsi for my testing above (just to see...).

i'm changing it back to v1 now and will test as i suspect it does not matter

EDIT 0: i don't see an effect of v1 vs v2 for cpuidle or usb... (i'll need to do a full flash to fully test usb...)

EDIT 1: my github k419 commits contains edits for usb in qcom-ipq8064.dtsi specific to my edits in qcom-ipq8064-r7500v2.dts . This can cause usb issues for r7800 users using these patches - I'll conform to whatever the community decides as a standard.

1 Like

EDIT: It looks like a few changes to the qcom-ipq8064.dtsi file is all that is needed to get cpuidle working on the r7500v2 and the R7800.

To enable cpuidle on the R7800, see this post below to edit qcom-ipq8064.dtsi. See also the WARNING for R7800 testers in this post above - @anon50098793 has tested the concept on the R7800 and no "smoke" reported yet.

WARNING My patch "ipq806x: k419 cpuidle fix" contains edits for usb in qcom-ipq8064.dtsi specific to my edits in qcom-ipq8064-r7500v2.dts (also on my github site as "ipq806x: r7500v2 k419 extend overlay and usb fix") R7800 users using my patch may have usb issues.

It would be good to have a common "ipq806x k419 usb fix" that will work on all devices that use qcom-ipq8064.dtsi. I'll follow the community... See @Ansuel's post to get started adjusting the dtsi and dts files for usb on the R7800.

Lastly @Ansuel and/or @anon50098793 , assuming everything works out after sufficient testing, would you mind submitting the cpuidle pull request(s)? For personnel reasons related to privacy, I choose not to identify myself in public online forums; consequently, I can't submit the pull request per openwrt identity requirements (a policy I fully support BTW). I'm not bothered about attribution to me - a link to this thread is more than sufficient.

1 Like

i'm currently away from the router :frowning:

Anyway i will test and publish a commit ASAP...

1 Like

a quick thankyou to invisiblek (@invisiblek in the forum here?) and karlp on irc for helping me think through the "workflow" when building initramfs images for testing.

I ran into some errors after make distclean followed by make target/linux/{clean,prepare} V=s and make target/linux/{compile,install} V=s. The short of it is that one can not expect a functional initramfs image to be built from these commands without first building a full flash image... you need things beyond just the kernel of course.

thank you

by "test" I hope you mean demonstrate a benefit and stability resulting from these changes (reduced power at low cpu usage, better performance under high cpu usage, no crashing). See my edit above starting with "Case 3 is problematic" i.e. there is no point in submitting a PR if there is no benefit.

I'll take a shot at demonstrating a benefit and showing stability as well...

Personally I can test stability as it's my main router

After some thought, I've updated my github k419 branch in favor of a simple solution that does not involve patching the kernel drivers. Its likely a few changes to the qcom-ipq8064.dtsi file is all that is needed and is a flexible solution (I'll explain below).

To get cpuidle working (the wfi, spc, and the additional sysfs information), add status = "okay" and "qcom,apq8064-saw2-v1.1-cpu" to the qcom-ipq8064.dtsi along the lines of:

...
                idle-states {                                                   
                        CPU_SPC: spc {                                          
                                compatible = "qcom,idle-state-spc",             
                                                "arm,idle-state";
                                status = "okay"
                                entry-latency-us = <400>;                       
                                exit-latency-us = <900>;                        
                                min-residency-us = <3000>;                      
                        };    
...
		saw0: regulator@2089000 {
                        compatible = "qcom,saw2",
                                    "qcom,apq8064-saw2-v1.1-cpu",
			             "syscon";
                        reg = <0x02089000 0x1000>, <0x02009000 0x1000>;
                        regulator;
		};

                saw1: regulator@2099000 {
                        compatible = "qcom,saw2",
                                     "qcom,apq8064-saw2-v1.1-cpu",
		                     "syscon";
			reg = <0x02099000 0x1000>, <0x02009000 0x1000>;
		        regulator;
                };
...

note the a (not i) in "qcom,apq8064-saw2-v1.1-cpu"

I'd like to keep the status = "okay" line as one could use status = "disabled" to turn off spc at compile time (this does not disable the wfi idle state).

The spc idle state can be disabled/enabled at run time via:

echo 1 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable
echo 1 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable

to disable spc, or

echo 0 > /sys/devices/system/cpu/cpu0/cpuidle/state1/disable
echo 0 > /sys/devices/system/cpu/cpu1/cpuidle/state1/disable

to enable spc. For me state1 is spc, state0 is wfi. I can not disable wfi with this method (even with spc enabled).

This solution resolves the issue and is flexible; consequently I think it has a better chance of acceptance (in openwrt master) than would patching kernel drivers.

Testing for stability only should be fine in this case.

Your approach to "do some magic in the dts" is correct...