OpenWrt 21.02.0 - EspressoBin Ultra - Bugs

I get some Kernel Panic :

[ 2550.977166] ------------[ cut here ]------------                                                                           
[ 2550.981954] Kernel BUG at 0xffffffc010080aa4 [verbose debug info unavailable]                                              
[ 2550.989315] Internal error: Oops - BUG: 0 [#1] SMP                                                                         
[ 2550.994250] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOg
[ 2551.071161] CPU: 1 PID: 3093 Comm: tar Not tainted 5.4.143 #0                                                              
[ 2551.077082] Hardware name: Globalscale Marvell ESPRESSOBin Ultra Board (DT)                                                
[ 2551.084258] pstate: 00400085 (nzcv daIf +PAN -UAO)                                                                         
[ 2551.089193] pc : 0xffffffc010080aa4                                                                                        
[ 2551.092784] lr : 0xffffffc010080844                                                                                        
[ 2551.096374] sp : ffffffc0130238a0                                                                                          
[ 2551.099785] x29: ffffffc0130238a0 x28: ffffff80043af080                                                                    
[ 2551.105259] x27: ffffffffffffc005 x26: ffffffc013023b18                                                                    
[ 2551.110733] x25: 0000000000000060 x24: 0000000000000000                                                                    
[ 2551.116207] x23: 0000000060400085 x22: 0000000000000080                                                                    
[ 2551.121681] x21: 0000000097fffd42 x20: ffffffc01099e0f0                                                                    
[ 2551.127154] x19: ffffffc0130238e0 x18: 0000000000000000                                                                    
[ 2551.132628] x17: 0000000000000000 x16: 0000000000000000                                                                    
[ 2551.138102] x15: 0000000000000000 x14: 17fffe67b5ffcd8c                                                                    
[ 2551.143576] x13: 00000000008b7000 x12: 0000000000000000                                                                    
[ 2551.149050] x11: ffffffc01099e000 x10: 0000000000000060                                                                    
[ 2551.154524] x9 : ffffffc01099e000 x8 : 0000000000000000                                                                    
[ 2551.159998] x7 : ffffff803c6cf000 x6 : ffffffc0130238e0                                                                    
[ 2551.165471] x5 : 0000000000000000 x4 : ffffffc01099e100                                                                    
[ 2551.170945] x3 : 00000000d5300000 x2 : ffffffc01093fe68                                                                    
[ 2551.176418] x1 : ffffffc01099e100 x0 : 0000000060400085                                                                    
[ 2551.181893] Call trace:                                                                                                    
[ 2551.184409]  0xffffffc010080aa4                                                                                            
[ 2551.187641]  0xffffffc010082b50                                                                                            
[ 2551.190873]  0xffffffc010774598                                                                                            
[ 2551.194105]  0xffffffc010174344                                                                                            
[ 2551.197336]  0xffffffc010174540                                                                                            
[ 2551.200569]  0xffffffc010174704                                                                                            
[ 2551.203801]  0xffffffc01017519c                                                                                            
[ 2551.207033]  0xffffffc010286d28                                                                                            
[ 2551.210266]  0xffffffc0101719b8                                                                                            
[ 2551.213497]  0xffffffc010173f8c                                                                                            
[ 2551.216729]  0xffffffc010274b84                                                                                            
[ 2551.219961]  0xffffffc0101db438                                                                                            
[ 2551.223193]  0xffffffc0101db4cc                                                                                            
[ 2551.226425]  0xffffffc0101dcee0                                                                                            
[ 2551.229658]  0xffffffc0101dd184                                                                                            
[ 2551.232890]  0xffffffc0101dd218                                                                                            
[ 2551.236122]  0xffffffc010094fac                                                                                            
[ 2551.239354]  0xffffffc010083748                                                                                            
[ 2551.242591] Code: 17fffff1 d2800c01 d50342df 17ffffcb (d4210000)                                                           
[ 2551.248872] ---[ end trace ab5336dab5bd79f5 ]---
[   10.764365] Unable to handle kernel paging request at virtual address 00000421f9400c01                                     
[   10.772540] Mem abort info:                                                                                                
[   10.775507]   ESR = 0x96000021                                                                                             
[   10.778661]   EC = 0x25: DABT (current EL), IL = 32 bits                                                                   
[   10.784201]   SET = 0, FnV = 0                                                                                             
[   10.787349]   EA = 0, S1PTW = 0                                                                                            
[   10.790583] Data abort info:                                                                                               
[   10.793565]   ISV = 0, ISS = 0x00000021                                                                                    
[   10.797518]   CM = 0, WnR = 0                                                                                              
[   10.800575] [00000421f9400c01] address between user and kernel address ranges                                              
[   10.807968] Internal error: Oops: 96000021 [#1] SMP                                                                        
[   10.813002] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOg
[   10.889925] CPU: 1 PID: 821 Comm: kworker/1:2 Not tainted 5.4.143 #0                                                       
[   10.896475] Hardware name: Globalscale Marvell ESPRESSOBin Ultra Board (DT)                                                
[   10.903662] Workqueue: events_power_efficient 0xffffffc010510740                                                           
[   10.909857] pstate: 60400005 (nZCv daif +PAN -UAO)                                                                         
[   10.914794] pc : 0xffffffc010782264                                                                                        
[   10.918387] lr : 0xffffffc01051be4c                                                                                        
[   10.921979] sp : ffffffc010b3bbe0                                                                                          
[   10.925391] x29: ffffffc010b3bbe0 x28: 0000000000000000                                                                    
[   10.930867] x27: ffffff803c985b38 x26: ffffffc0107fe050                                                                    
[   10.936343] x25: 0000000000000000 x24: 52802000f9000c01                                                                    
[   10.941819] x23: 91000421f9400c01 x22: b50000c1f9401001                                                                    
[   10.947294] x21: 5400006b3101683f x20: 0000000000000000                                                                    
[   10.952769] x19: 97ffff6954fffe01 x18: 0000000000000000                                                                    
[   10.958244] x17: 0000000000000000 x16: 0000000000000000                                                                    
[   10.963719] x15: 0000000000000000 x14: 0000000000000000                                                                    
[   10.969194] x13: 0000000000000000 x12: 0000000000000038                                                                    
[   10.974669] x11: 0000000000000038 x10: 0000000000000800                                                                    
[   10.980144] x9 : ffffffc010b3b890 x8 : ffffff803e2a40a0                                                                    
[   10.985619] x7 : ffffff803fde0738 x6 : ffffff803e314458                                                                    
[   10.991095] x5 : 0000000000000000 x4 : 0000000000000000                                                                    
[   10.996571] x3 : ffffff803e2a3840 x2 : 0000000000000000                                                                    
[   11.002046] x1 : ffffff803e2a3840 x0 : 91000421f9400c01                                                                    
[   11.007521] Call trace:                                                                                                    
[   11.010039]  0xffffffc010782264                                                                                            
[   11.013275]  0xffffffc01051be4c                                                                                            
[   11.016511]  0x540009cc7100029f                                                                                            
[   11.019753] Code: d2800002 d5384101 910003fd f9800011 (c85f7c03)                                                           
[   11.026054] ---[ end trace e7820a91a7c35e7c ]---

How can I analyse these ?

 [ 2370.354520] Unable to handle kernel paging request at virtual address ffffffc01092b668                                    
[ 2370.362695] Mem abort info:                                                                                                
[ 2370.365574]   ESR = 0x96000007                                                                                             
[ 2370.368722]   EC = 0x25: DABT (current EL), IL = 32 bits                                                                   
[ 2370.374196]   SET = 0, FnV = 0                                                                                             
[ 2370.377340]   EA = 0, S1PTW = 0                                                                                            
[ 2370.380570] Data abort info:                                                                                               
[ 2370.383533]   ISV = 0, ISS = 0x00000007                                                                                    
[ 2370.387482]   CM = 0, WnR = 0                                                                                              
[ 2370.390538] swapper pgtable: 4k pages, 39-bit VAs, pgdp=00000000008b8000                                                   
[ 2370.397445] [ffffffc01092b668] pgd=000000003ffff003, pud=000000003ffff003, pmd=000000003fffc003, pte=0000000000000000      
[ 2370.408392] Internal error: Oops: 96000007 [#1] SMP                                                                        
[ 2370.413418] Modules linked in: pppoe ppp_async iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOg
[ 2370.490329] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.143 #0                                                           
[ 2370.496518] Hardware name: Globalscale Marvell ESPRESSOBin Ultra Board (DT)                                                
[ 2370.503695] pstate: 80400085 (Nzcv daIf +PAN -UAO)                                                                         
[ 2370.508632] pc : 0xffffffc0100de654                                                                                        
[ 2370.512224] lr : 0xffffffc0100f2638                                                                                        
[ 2370.515813] sp : ffffffc010003f90                                                                                          
[ 2370.519224] x29: ffffffc010003f90 x28: ffffffc01093e500                                                                    
[ 2370.524699] x27: 0000000000000000 x26: ffffffc010004000                                                                    
[ 2370.530172] x25: ffffffc010000000 x24: 00000000000000e0                                                                    
[ 2370.535646] x23: ffffff8004215000 x22: 0000000000000000                                                                    
[ 2370.541119] x21: 0000000000000000 x20: 0000000000000001                                                                    
[ 2370.546592] x19: ffffffc01092afb0 x18: 0000000000000000                                                                    
[ 2370.552066] x17: 0000000000000000 x16: 0000000000000000                                                                    
[ 2370.557539] x15: 0000000000000000 x14: 0000000000000000                                                                    
[ 2370.563013] x13: 0000000000000000 x12: 0000000000000000                                                                    
[ 2370.568486] x11: 0000000000000001 x10: 0000000000000040                                                                    
[ 2370.573960] x9 : ffffffc010946f40 x8 : ffffff803fdd0700                                                                    
[ 2370.579434] x7 : ffffff803fdd0738 x6 : 00000006ea604e8f                                                                    
[ 2370.584908] x5 : 00ffffffffffffff x4 : 002e255767482e3e                                                                    
[ 2370.590381] x3 : 0000000003333333 x2 : 0000000000001fff                                                                    
[ 2370.595855] x1 : 000000000000001e x0 : 0000000000000000                                                                    
[ 2370.601329] Call trace:                                                                                                    
[ 2370.603846]  0xffffffc0100de654                                                                                            
[ 2370.607079]  0xffffffc0100810ac                                                                                            
[ 2370.610310]  0xffffffc010082df0                                                                                            
[ 2370.613543]  0xffffffc0100864f0                                                                                            
[ 2370.616776]  0xffffffc0100d2004                                                                                            
[ 2370.620007]  0xffffffc0100d2188                                                                                            
[ 2370.623240]  0xffffffc01077f608                                                                                            
[ 2370.626473]  0xffffffc0108c086c                                                                                            
[ 2370.629704]  0xffffffc0108c0d28                                                                                            
[ 2370.632943] Code: d2800015 a90363f7 a90573fb 5280001b (f9435e78)                                                           
[ 2370.639224] ---[ end trace 625026192905179a ]---

High cpu usage get theses crashes :tired_face:
I look to change the CPU governor, but EBIN-ULTRA do not get the needed path !?

on Espressobin (not ultra) I get :

root@LPM:~# ls /sys/devices/system/cpu/cpufreq/policy0/scaling_governor 
/sys/devices/system/cpu/cpufreq/policy0/scaling_governor
root@LPM:~# cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
200000
root@LPM:~# cat /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq
200000

On EspressoBin-Ultra :

root@ULTRA-CLOUD:/# ls /sys/devices/system/cpu/cpufreq/

empty folder !!!

root@ULTRA-CLOUD:/# cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq                                                 
cat: can't open '/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq': No such file or directory                            
root@ULTRA-CLOUD:/# cat /sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq                                                 
cat: can't open '/sys/devices/system/cpu/cpu1/cpufreq/cpuinfo_cur_freq': No such file or directory

same with snapshot :

root@ULTRA-CLOUD:/# ls /sys/devices/system/cpu/cpufreq/                                                                                                                               
root@ULTRA-CLOUD:/# uname -ar                                                                                                                                                         
Linux ULTRA-CLOUD 5.10.64 #0 SMP Wed Sep 15 20:47:32 2021 aarch64 GNU/Linux

https://bugs.openwrt.org/index.php?do=details&task_id=4041

There is no CPUFreq for 1.2G model, take a look at the commit description.

PS, this board is so buggy that its insane.
I have some patches for it out of tree that I will start pushing soon, but frequency scaling is broken because of Marvell.

1 Like

You mean the Ultra ?
The EBINv7 (not-ULTRA) is pretty stable... but your right that I get many problems with ULTRA...
Fell free to propose patch for testing if you want to ! :wink:

:+1:

I have contact Marvell on their extranet, and open a support request giving your commit as link...

Yes, the Ultras as I have to support these at work

1 Like

Good to know ! Thanks...
Ready for beta tests. :wink:

Has Marvell maybe replied?
We have been trying to get them to provide some info and help for months

Still nothing, sorry...

I will try to get in touch with old contacts !

Because of some out office, will have to wait a little, but I still hope of a contact at Marvell SOC support...

@robimarko

I opened issues on Marvell GitHub with the commit you give as reference...

Here some answers:

Great, hopefully they will provide some useful feedback instead of patch looks ok so far

1 Like

Hope so !
I also made attempts in "private" to friendly personal contacts...

Thanks, as they are just trying to ignore the issue completely