Adding OpenWrt support for ws-ap3825i

i have this device, can i help ?

Yes, for sure! I'm hearing that the engineers are almost done building the dump for both devices.

I will keep you posted; my plan is to post the dump once I get it.

1 Like

I have received my copy of the dumps. Here they are: https://downloads.laboratoryb.org/dumps/

I plan to create a git repository for each of these, where I apply these dumps as a branch on top of the most relevant version of Linux / u-boot.

1 Like

I'm not sure of these dumps: the board calls itself P1020RDB, not P1020SKU. I would be interested to see:

  • The copy of Linux in this GPL dump actually booting on the AP3825i
  • A vbindiff of the compiled U-boot with one from the boards themselves.

... but after some thought, I don't think that any of that is the reason for this board not booting. I don't think AP3825i can boot using initramfs for AP3710i; there is probably plenty of setup required for the AP3825i specifically, like having a platform file specific to AP3825i (for AP3710i, it is target/linux/mpc85xx/files/arch/powerpc/platforms/85xx/ws-ap3710i.c).

We should closely copy 16b01fb1b9c99513c318109bef96a1a3545c57a0 in this respect.

Edit: Now that I am essentially finished with the AP370 port, I will turn to the ws-ap3825i. I have only one of these boards, but I will soon order more for testing once I get Linux booting on the first ...

2021-11-27: Addendum: I still have not been able to get the serial console to come up to find what is crashing the board on early boot. However, I looked closely into the standard boot script and have found a possible way to view the content of the kernel ring buffer after a soft reset triggered by the watchdog timer (WDT). The stock U-boot uses the WDT to detect whether the watchdog caused the last reboot; it can do this because neither u-boot nor the BootROM reset the content of RAM upon reboot, and the WDT sets the value of a small patch of memory (through some special-purpose registers) upon reboot.

All I'd need to do is use the System.map (https://support.xilinx.com/s/question/0D52E00006hpihR/linux-kernel-hangs-after-allocatting-dma-contiguous-pool-on-zynq-7000?language=en_US) to find the memory location from which to read the kernel message log.

Here are the details of how the existing boot script works:

:
bootcmd=run boot_flash
boot_flash=source boot_kernel
:

boot_kernel is a script stored in u-boot which sets up the watchdog timer (WDT) and then boots an image from JFFS2; it also describes the mechanism for how it sets up the WDT and checks, upon each boot, whether the last reboot was caused by the WDT.

Some other necessary variables to understand the script:

BOOT_KERNEL=primary
DEFAULT_SETTING=0
LOGHDRRREASON=0xfffec24
MOSTRECENTKERNEL=0
REBOOT_PATTERN_WDG=0x5A5A5A5A
WATCHDOG_COUNT=0x00000001
WATCHDOG_LIMIT=3

And the script itself:

boot_kernel
# U-boot script start
bSaveEnvironment=0
saved_ver=$ver
# The watchdog registers are actually SPR's.
# These are hardcoded in the chenv command as 0xffffffff for TSR and 0xfffffffe for TCR.
# NOTE 1: test command is non-intuitive, for environment variables try to do the -eq check for 0, not 1
#         if the variable is not present, -eq 1 will return true
reg_tsr=0xffffffff
reg_tcr=0xfffffffe
if test $DEFAULT_SETTING -eq 1; then
    bSaveEnvironment=1
    setenv DEFAULT_SETTING 0
    setenv CURR_VER $ver
    echo first time using environment, will save at the end...
# The application has informed us which kernel is more recent in MOSTRECENTKERNEL.
if test $MOSTRECENTKERNEL -eq 0; then
    newer="primary"
    older="secondary"
else
    newer="secondary"
    older="primary"
# This is the default order of images to boot.
if test $USE_ALT_DTB -eq 0; then
    smp_amp="-"
    dtbnum=1
    order="${newer} ${older}"
else
    smp_amp="+"
    dtbnum=2
    order="${newer} ${older} ${older}"
if test $WATCHDOG_LIMIT -ne 0; then
    echo check for watchdog, limit=$WATCHDOG_LIMIT...
    if test $WATCHDOG_COUNT -ge $WATCHDOG_LIMIT; then
        echo watchdog count hit previously, running older image...
        order="${older}"
    else        
        echo check for reset by watchdog...
        rebootReason=$LOGHDRRREASON
        if itest.l *${rebootReason} -eq $REBOOT_PATTERN_WDG; then
            if test ${intWd} -ne 1; then
                echo last reset caused by watchdog...
                chenv WATCHDOG_COUNT + 1
                bSaveEnvironment=1
            fi
            if test $WATCHDOG_COUNT -ge $WATCHDOG_LIMIT; then
                echo watchdog limit reached, cnt=$WATCHDOG_COUNT...
                echo switching to older image...
                order="${older}"
            else
                echo updating watchdog count, cnt=$WATCHDOG_COUNT...
            fi
       else
            echo no watchdog...
       fi
    fi
else
    echo skipping all watchdog checks...
echo bSaveEnviron=${bSaveEnvironment} image_order=${order}
if test ${bSaveEnvironment} -eq 1; then
    saveenv
    saveenv
chenv REBOOT_PATTERN_WDG save.l $LOGHDRRREASON
# Set watchdog to about 1 minute.
# If WATCHDOG_DISABLE is not set, then the test command returns 0.
if test $WATCHDOG_DISABLE -eq 0; then
    setenv TCR 0xA4100000
    chenv TCR save.l ${reg_tcr}
    setenv TSR 0xE0000000
for image in ${order}
    # Kick the watchdog.
    if test $WATCHDOG_DISABLE -eq 0; then
        chenv TSR save.l ${reg_tsr}
    fi
    
    if fsload 0x0A000000 "${image}.gz.uImage"; then
        setenv bootargs "$mtdparts BOOT_BOOTROM=\""${saved_ver}"\"" BOOT_KERNEL=${image} $static_bootargs
        echo ready to boot kernel... [DTB ${dtbnum}]
        # Reset system when watchdog occurs (starts the final countdown).
        if test $WATCHDOG_DISABLE -eq 0; then
            chenv TSR save.l ${reg_tsr}
        fi
        # If this command fails, then the image is corrupt. If it succeeds, then we are running the kernel.
        bootm 0x0A000000 - ${smp_amp}
        echo boot failed...
        # For SMP/AMP images, we trying running with the 2nd DTB, for both the newer then the
        # older image. However, if that doesn't work, then try running the older image again with the
        # 1st DTB.
        if test ${image} = ${older}; then
            smp_amp="-"
            dtbnum=1
        fi
    fi
done
# If we are here, there is a problem with the image(s) we were trying to run.
setenv ZERO 0
chenv ZERO save.l $LOGHDRRREASON
setenv ZERO
echo ERROR: Cannot boot either kernel image, dropping to interactive shell (watchdog might trigger)

So a script to set the WDT and boot might be:

setenv ipaddr 10.0.7.2
setenv serverip 10.0.7.1
tftpboot 0xa000000 openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-initramfs-kernel.bin
tftpboot 0x6000000 openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-squashfs-fdt.bin;

reg_tsr=0xffffffff
reg_tcr=0xfffffffe

chenv REBOOT_PATTERN_WDG save.l $LOGHDRRREASON
setenv TCR 0xA4100000

chenv TCR save.l ${reg_tcr}
setenv TSR 0xE0000000

chenv TSR save.l ${reg_tsr}
bootm 0xa000000 - 0x6000000

My System.map compiles out __log_buf to 0xc0a88e84. An md to that address crashes u-boot, I realize now because that's virtual memory. Apparently CONFIG_KERNEL_START is usually 0xC0000000 on PowerPC. So I just need to do a md a88e84.

This should all work, but unfortunately md shows there is nothing in any of the buffers, so I'm forced to assume the kernel just never launches ...

Edit: I have managed to capture a couple of lines of an oops. Not sure why these lines are so incomplete, though: https://paste.c-net.org/DaisyDearest

Edit: I disabled SMP and got are more robust output: there's probably something going on where output gets destroyed by SMP being active: https://paste.c-net.org/BauerFlippers

2 Likes

Hey @hurricos nice work so far on this. I've been following your progress on IRC for the past two months, and just wanted to toss you a nod of encouragement.

I picked up one of these a couple years ago for cheap and had been planning to force it into standalone AP mode, but could never get it working, so it got set aside and forgotten. I have all kinds of hardware for reading and writing flash chips and EEPROMs, so if you need a hand testing anything, I'm happy to set stuff up. I know one of my buddies had some of these units around in a box as well, so I could borrow a few for unit testing. These units are like $8.00/piece in volume on eBay, so it's definitely worth it to get them going. I'm not much of a programmer once you get past microcontrollers, but I can at least help on the hardware side.

Thanks again, and good luck!

P.S. if you have a link for PayPal etc, I'd be happy to send you a few bucks to help support any needed beer and snacks during development hahaha

Hey @Gadorach ! Thanks for the encouragement, and thanks for reviving the thread (I was coming up against the 3-post limit).

I was ultimately able to pull the full kernel oops: https://paste.c-net.org/LakhiToner


?BUG: Unable to handle kernel data access on read at 0xc9ffa008
?Faulting instruction address: 0xc033c920
?Thread overran stack, or stack corrupted
?Oops: Kernel access of bad area, sig: 11 #1
?BE PAGE_SIZE=4K 
?Modules linked in:
?CPU: 0 PID: 0 Comm: swapper Not tainted 5.10.82 #0
?NIP:  c033c920 LR: c033ca50 CTR: c000e938
?REGS: c0c01e28 TRAP: 0300   Not tainted  (5.10.82)
?	MSR:  00021000 <CE,ME>  CR: 28248044  XER: 200000000 
GPR00: c033ccdc c0c01ee0 c0b7ba40 c9ffa000 00000000 00000004 40000000 60000000 
GPR08: c9ffa000 fffffff8 00000000 00a3a348 88242044 00a3a2a0 00000000 

At point, the problem is that the kernel is given a somewhat high address for the device tree blob (starting at 0x09ffa000 in physical memory / 0xc9ffa000 in ppc32 virtual memory). Upon accessing it the kernel panics, apparently due to a memory restriction violation.

The AP3710i, a working OpenWrt board which is extremely similar but which does not relocate the device tree blob, does not panic.

If I can figure out how to tell u-boot on the AP3825i to not relocate the dtb, I am confident booting will succeed.

One part of that will be figuring out through which region of memory the Linux kernel is informed of the location of the device tree blob.

Edit: I looked up one of the variables, CONFIG_SYS_BOOTMAPSZ, from the u-boot source dump whose value helps determine where the flattened device tree (fdt) gets relocated to, and found a patch (mind you, for ARM) which actually describes the situation we're seeing where the fdt can be relocated high enough that the kernel panics.

This bootlin presentation helps me answer my above question:

One part of that will be figuring out through which region of memory the Linux kernel is informed of the location of the device tree blob.

... the answer is on slide 18:

The DTB address is passed through a dedicated CPU register to the kernel: r2 on ARM32

I also see for the u-boot Beaglebone documentation a variable (fdt_high) which can be used to tell u-boot not to relocate too high to solve the issue we're having -- but I worry our u-boot is too old (and I bet the u-boot derivative used on the AP3825i squashes any attempt to mess with the device tree's relocation address).

1 Like

Hmm, that's interesting. Surely there's some kind of variable being set in uboot to facilitate this. Any luck decompiling the original to see where it sends uboot configuration options for DTB setup? Probably more trouble than it's worth if there's a chance the documentation provides a good answer.

Was just doing some reading, and maybe this thread would help us here?

Figured I'd take a look myself before digging into the posted uboot source. Considering the uboot version seems to be a late 2010 build (December-ish) on my WS-AP3825i, hopefully it's new enough to support the configuration options shown in that thread.

Yes, this is exactly the issue.

The WS-AP3825i DOES care about the value of bootm_size -- just confirmed that you can bound the highest address of the fdt / dtb relocation by setting e.g. setenv bootm_size 0x2000000. Testing now whether it will boot my build.

Edit: And success!

My new boot script:

setenv bootm_size 0x3000000; saveenv;

setenv ipaddr 10.0.7.2; setenv serverip 10.0.7.1; tftpboot 0xa000000 openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-initramfs-kernel.bin

# I set the TCR differently here, starting with E instead of A, to shorten watchdog reboot 
# See page 16 of https://www.nxp.com/docs/en/application-note/AN2817.pdf
reg_tsr=0xffffffff; reg_tcr=0xfffffffe; chenv REBOOT_PATTERN_WDG save.l $LOGHDRRREASON; setenv TCR 0xE4100000;

chenv TCR save.l ${reg_tcr}; setenv TSR 0xE0000000; chenv TSR save.l ${reg_tsr}; bootm 0xa000000

And the log:

## Booting kernel from FIT Image at 0a000000 ...
   Using 'config-1' configuration
   Trying 'kernel-1' kernel subimage
     Description:  POWERPC OpenWrt Linux-5.10.82
     Type:         Kernel Image
     Compression:  lzma compressed
     Data Start:   0x0a0000ec
     Data Size:    5936715 Bytes = 5.7 MiB
     Architecture: PowerPC
     OS:           Linux
     Load Address: 0x00000000
     Entry Point:  0x00000000
     Hash algo:    crc32
     Hash value:   8f615dc7
     Hash algo:    sha1
     Hash value:   95df388d3e750fad97a2f3ebeac07a09011212f7
   Verifying Hash Integrity ... crc32+ sha1+ OK
## Flattened Device Tree from FIT Image at 0a000000
   Using 'config-1' configuration
   Trying 'fdt-1' FDT blob subimage
     Description:  POWERPC OpenWrt extreme-networks_ws-ap3825i device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x0a5a9888
     Data Size:    10777 Bytes = 10.5 KiB
     Architecture: PowerPC
     Hash algo:    crc32
     Hash value:   1e89c312
     Hash algo:    sha1
     Hash value:   5bb5087e0f3ffc198070148abefeb94a4b3f3ae5
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0xa5a9888
   Uncompressing Kernel Image ... OK
   Loading Device Tree to 02ffa000, end 02fffa18 ... OK
ft_fixup_l2cache: FDT_ERR_NOTFOUND
[    0.000000] Memory CAM mapping: 16/16/16 Mb, residual: 0Mb
[    0.000000] Linux version 5.10.82 (labby@lobon) (powerpc-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r18216+1-a662d8550f) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 Sun Nov 28 02:05:00 2021
[    0.000000] ##### ws_ap3825i_probe: PASSED: L 76
[    0.000000] Using P1020 RDB machine description
[    0.000000] ioremap() called early from find_legacy_serial_ports+0x5ec/0x6f8. Use early_ioremap() instead
[    0.000000] printk: bootconsole [udbg0] enabled
[    0.000000] -----------------------------------------------------
[    0.000000] phys_mem_size     = 0x3000000
[    0.000000] dcache_bsize      = 0x20
[    0.000000] icache_bsize      = 0x20
[    0.000000] cpu_features      = 0x0000000010010128
[    0.000000]   possible        = 0x0000000010010128
[    0.000000]   always          = 0x0000000010010128
[    0.000000] cpu_user_features = 0x84e08000 0x08000000
[    0.000000] mmu_features      = 0x00020010
[    0.000000] -----------------------------------------------------
[    0.000000] ##### ws_ap3825i_setup_arch: PASSED: L 58
ws_ap3825i_setup_arch()
[    0.000000] WS-AP3825i board from Extreme Networks
[    0.000000] barrier-nospec: using isync; sync as speculation barrier
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000002ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000002ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000002ffffff]
[    0.000000] MMU: Allocated 1088 bytes of context maps for 255 contexts
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 12180
[    0.000000] Kernel command line: console=ttyS0,115200
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 35664K/49152K available (6296K kernel code, 644K rwdata, 1284K rodata, 4444K init, 217K bss, 13488K reserved, 0K cma-reserved)
[    0.000000] Kernel virtual memory layout:
[    0.000000]   * 0xffbdf000..0xfffff000  : fixmap
[    0.000000]   * 0xffbdd000..0xffbdf000  : early ioremap
[    0.000000]   * 0xc4000000..0xffbdd000  : vmalloc & ioremap
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[    0.000000] mpic: Setting up MPIC " OpenPIC  " version 1.2 at ffe40000, max 1 CPUs
[    0.000000] mpic: ISU size: 256, shift: 8, mask: ff
[    0.000000] ##### ws_ap3825i_pic_init: PASSED: L 48
[    0.000000] mpic: Initializing for 256 sources
[    0.000000] random: get_random_u32 called from start_kernel+0x33c/0x4e0 with crng_init=0
[    0.000012] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
[    0.010240] clocksource: timebase mult[14000000] shift[24] registered
[    0.016704] pid_max: default: 32768 minimum: 301
[    0.021390] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.028628] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.037432] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build
[    0.047041] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.056814] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.064136] NET: Registered protocol family 16
             
[    0.077460] Found FSL PCI host bridge at 0x00000000ffe09000. Firmware bus number: 0->1
[    0.085326] PCI host bridge /pcie@ffe09000 (primary) ranges:
[    0.090943]  MEM 0x00000000a0000000..0x00000000bfffffff -> 0x00000000a0000000 
[    0.098146]   IO 0x00000000ffc10000..0x00000000ffc1ffff -> 0x0000000000000000
[    0.105289] /pcie@ffe09000: PCICSRBAR @ 0xfff00000
[    0.110035] setup_pci_atmu: end of DRAM 3000000
[    0.114549] /pcie@ffe09000: Setting PCI inbound window greater than memory size
[    0.122331] Found FSL PCI host bridge at 0x00000000ffe0a000. Firmware bus number: 0->1
[    0.130159] PCI host bridge /pcie@ffe0a000  ranges:
[    0.135039]  MEM 0x0000000080000000..0x000000009fffffff -> 0x0000000080000000 
[    0.142222]   IO 0x00000000ffc00000..0x00000000ffc0ffff -> 0x0000000000000000
[    0.149369] /pcie@ffe0a000: PCICSRBAR @ 0xfff00000
[    0.154111] setup_pci_atmu: end of DRAM 3000000
[    0.158630] /pcie@ffe0a000: Setting PCI inbound window greater than memory size
[    0.166894] PCI: Probing PCI hardware
[    0.170564] fsl-pci ffe09000.pcie: PCI host bridge to bus 9000:00
[    0.176596] pci_bus 9000:00: root bus resource [io  0x0000-0xffff]
[    0.182729] pci_bus 9000:00: root bus resource [mem 0xa0000000-0xbfffffff]
[    0.189592] pci_bus 9000:00: root bus resource [bus 00-ff]
[    0.195057] pci_bus 9000:00: busn_res: [bus 00-ff] end is updated to ff
[    0.201692] pci 9000:00:00.0: [1957:0100] type 01 class 0x060400
[    0.207651] pci 9000:00:00.0: reg 0x10: [mem 0xfff00000-0xffffffff]
[    0.213952] pci 9000:00:00.0: supports D1 D2
[    0.218143] pci 9000:00:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.225803] pci 9000:01:00.0: [168c:0033] type 00 class 0x028000
[    0.231740] pci 9000:01:00.0: reg 0x10: [mem 0xa0000000-0xa001ffff 64bit]
[    0.238537] pci 9000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.245255] pci 9000:01:00.0: supports D1
[    0.249179] pci 9000:01:00.0: PME# supported from D0 D1 D3hot
[    0.280075] pci 9000:00:00.0: PCI bridge to [bus 01-ff]
[    0.285222] pci 9000:00:00.0:   bridge window [io  0x0000-0x0fff]
[    0.291306] pci 9000:00:00.0:   bridge window [mem 0xa0000000-0xa00fffff]
[    0.298079] pci_bus 9000:01: busn_res: [bus 01-ff] end is updated to 01
[    0.304672] pci_bus 9000:00: busn_res: [bus 00-ff] end is updated to 01
[    0.311483] fsl-pci ffe0a000.pcie: PCI host bridge to bus a000:02
[    0.317502] pci_bus a000:02: root bus resource [io  0x20000-0x2ffff] (bus address [0x0000-0xffff])
[    0.326427] pci_bus a000:02: root bus resource [mem 0x80000000-0x9fffffff]
[    0.333290] pci_bus a000:02: root bus resource [bus 02-ff]
[    0.338754] pci_bus a000:02: busn_res: [bus 02-ff] end is updated to ff
[    0.345386] pci a000:02:00.0: [1957:0100] type 01 class 0x060400
[    0.351348] pci a000:02:00.0: reg 0x10: [mem 0xfff00000-0xffffffff]
[    0.357648] pci a000:02:00.0: supports D1 D2
[    0.361841] pci a000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.369372] pci a000:02:00.0: Primary bus is hard wired to 0
[    0.374943] pci a000:02:00.0: bridge configuration invalid ([bus 01-01]), reconfiguring
[    0.383063] pci a000:03:00.0: [168c:003c] type 00 class 0x028000
[    0.388998] pci a000:03:00.0: reg 0x10: [mem 0x80000000-0x801fffff 64bit]
[    0.395793] pci a000:03:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.402523] pci a000:03:00.0: supports D1 D2
[    0.437570] pci a000:02:00.0: PCI bridge to [bus 03-ff]
[    0.442717] pci a000:02:00.0:   bridge window [io  0x20000-0x20fff]
[    0.448974] pci a000:02:00.0:   bridge window [mem 0x80000000-0x801fffff]
[    0.455746] pci_bus a000:03: busn_res: [bus 03-ff] end is updated to 03
[    0.462339] pci_bus a000:02: busn_res: [bus 02-ff] end is updated to 03
[    0.469037] PCI: Cannot allocate resource region 0 of device 9000:00:00.0, will remap
[    0.476790] PCI: Cannot allocate resource region 0 of device a000:02:00.0, will remap
[    0.484631] pci 9000:00:00.0: BAR 0: no space for [mem size 0x00100000]
[    0.491191] pci 9000:00:00.0: BAR 0: failed to assign [mem size 0x00100000]
[    0.498136] pci 9000:01:00.0: BAR 6: assigned [mem 0xa0020000-0xa002ffff pref]
[    0.505339] pci 9000:00:00.0: PCI bridge to [bus 01]
[    0.510281] pci 9000:00:00.0:   bridge window [io  0x0000-0xffff]
[    0.516362] pci 9000:00:00.0:   bridge window [mem 0xa0000000-0xbfffffff]
[    0.523127] pci_bus 9000:00: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.532158] pci_bus 9000:00: resource 4 [io  0x0000-0xffff]
[    0.537709] pci_bus 9000:00: resource 5 [mem 0xa0000000-0xbfffffff]
[    0.543963] pci_bus 9000:01: resource 0 [io  0x0000-0xffff]
[    0.549513] pci_bus 9000:01: resource 1 [mem 0xa0000000-0xbfffffff]
[    0.555775] pci a000:02:00.0: BAR 0: no space for [mem size 0x00100000]
[    0.562360] pci a000:02:00.0: BAR 0: failed to assign [mem size 0x00100000]
[    0.569314] pci a000:03:00.0: BAR 6: assigned [mem 0x80200000-0x8020ffff pref]
[    0.576508] pci a000:02:00.0: PCI bridge to [bus 03]
[    0.581462] pci a000:02:00.0:   bridge window [io  0x20000-0x2ffff]
[    0.587706] pci a000:02:00.0:   bridge window [mem 0x80000000-0x9fffffff]
[    0.594481] pci_bus a000:02: Some PCI device resources are unassigned, try booting with pci=realloc
[    0.603503] pci_bus a000:02: resource 4 [io  0x20000-0x2ffff]
[    0.609236] pci_bus a000:02: resource 5 [mem 0x80000000-0x9fffffff]
[    0.615481] pci_bus a000:03: resource 0 [io  0x20000-0x2ffff]
[    0.621214] pci_bus a000:03: resource 1 [mem 0x80000000-0x9fffffff]
[    0.627576] /soc@ffe00000/timer@41100: cannot get timer frequency.
[    0.633729] /soc@ffe00000/timer@42100: cannot get timer frequency.
[    0.653630] clocksource: Switched to clocksource timebase
[    0.659615] NET: Registered protocol family 2
[    0.664008] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.671468] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.679785] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.687386] TCP bind hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.694413] TCP: Hash tables configured (established 1024 bind 1024)
[    0.700819] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.707283] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.714358] NET: Registered protocol family 1
[    0.718684] PCI: CLS 32 bytes, default 32
[    4.630350] workingset: timestamp_bits=14 max_order=14 bucket_order=0
[    4.639954] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    4.645755] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    4.656773] pcieport 9000:00:00.0: enabling device (0106 -> 0107)
[    4.662952] pcieport a000:02:00.0: enabling device (0106 -> 0107)
[    4.686153] Serial: 8250/16550 driver, 2 ports, IRQ sharing enabled
[    4.692949] printk: console [ttyS0] disabled
[    4.697210] serial8250.0: ttyS0 at MMIO 0xffe04500 (irq = 42, base_baud = 24999999) is a 16550A
[    4.705850] printk: console [ttyS0] enabled
[    4.705850] printk: console [ttyS0] enabled
[    4.714157] printk: bootconsole [udbg0] disabled
[    4.714157] printk: bootconsole [udbg0] disabled
[    4.723779] serial8250.0: ttyS1 at MMIO 0xffe04600 (irq = 42, base_baud = 24999999) is a 16550A
[    4.733147] printk: console [ttyS0] disabled
[    4.737548] printk: console [ttyS0] enabled
[    4.742279] ffe04600.serial: ttyS1 at MMIO 0xffe04600 (irq = 42, base_baud = 24999999) is a 16550
[    4.752472] physmap-flash ec000000.nor: physmap platform flash device: [mem 0xec000000-0xefffffff]
[    4.761529] ec000000.nor: Found 1 x16 devices at 0x0 in 16-bit bank. Manufacturer ID 0x000089 Chip ID 0x00227e
[    4.771549] Amd/Fujitsu Extended Query Table at 0x0040
[    4.776699]   Amd/Fujitsu Extended Query version 1.3.
[    4.781747] number of CFI chips: 1
[    4.869431] 6 fixed-partitions partitions found on MTD device ec000000.nor
[    4.876355] Creating 6 MTD partitions on "ec000000.nor":
[    4.881667] 0x000000000000-0x000003d60000 : "firmware"
[    4.891443] read error in "firmware" at offset 0x3d60000
[    4.896803] 0x000003d60000-0x000003d80000 : "calib"
[    4.902193] 0x000003d80000-0x000003e00000 : "u-boot"
[    4.908365] 0x000003e00000-0x000003f00000 : "nvram"
[    4.913784] 0x000003f00000-0x000003f20000 : "cfg2"
[    4.919715] 0x000003f20000-0x000003f40000 : "cfg1"
[    4.929534] fsl_espi ffe07000.spi: irq = 59
[    4.935030] libphy: Fixed MDIO Bus: probed
[    4.939744] libphy: Freescale PowerQUICC MII Bus: probed
[    4.945278] mdio_bus mdio@ffe24000: MDIO device at address 1 is missing.
[    4.952133] mdio_bus mdio@ffe24000: MDIO device at address 2 is missing.
[    4.959445] fsl-gianfar soc@ffe00000:ethernet@b0000: enabled errata workarounds, flags: 0x4
[    4.978398] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: mac: 20:b3:99:e4:83:57
[    4.985814] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: Running with NAPI enabled
[    4.993466] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: RX BD ring size for Q[0]: 256
[    5.001472] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: RX BD ring size for Q[1]: 256
[    5.009477] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: TX BD ring size for Q[0]: 256
[    5.017483] fsl-gianfar soc@ffe00000:ethernet@b0000 eth0: TX BD ring size for Q[1]: 256
[    5.025896] fsl-gianfar soc@ffe00000:ethernet@b2000: Using random MAC address: 82:1e:cb:f0:13:46
[    5.034709] fsl-gianfar soc@ffe00000:ethernet@b2000: enabled errata workarounds, flags: 0x4
[    5.053489] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: mac: 82:1e:cb:f0:13:46
[    5.060904] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: Running with NAPI enabled
[    5.068571] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: RX BD ring size for Q[0]: 256
[    5.076578] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: RX BD ring size for Q[1]: 256
[    5.084583] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: TX BD ring size for Q[0]: 256
[    5.092580] fsl-gianfar soc@ffe00000:ethernet@b2000 eth1: TX BD ring size for Q[1]: 256
[    5.100724] ucc_geth_driver: QE UCC Gigabit Ethernet Controller
[    5.106866] i2c /dev entries driver
[    5.110546] mpc-i2c ffe03000.i2c: timeout 1000000 us
[    5.115758] mpc-i2c ffe03100.i2c: timeout 1000000 us
[    5.120974] booke_wdt: powerpc book-e watchdog driver loaded
[    5.127250] NET: Registered protocol family 10
[    5.141948] Segment Routing with IPv6
[    5.145714] NET: Registered protocol family 17
[    5.150192] 8021q: 802.1Q VLAN Support v1.8
[    5.154434] drmem: No dynamic reconfiguration memory found
[    5.173376] Freeing unused kernel memory: 4444K
[    5.178297] Run /init as init process
[    5.390518] cp invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[    5.399702] CPU: 0 PID: 859 Comm: cp Not tainted 5.10.82 #0
[    5.405274] Call Trace:
[    5.407723] [c2653a68] [c00f9e9c] dump_header+0x40/0x1b0 (unreliable)
[    5.414172] [c2653a88] [c00fa6d8] oom_kill_process+0x1b4/0x1b8
[    5.420002] [c2653aa8] [c00fb0d0] out_of_memory+0x214/0x38c
[    5.425580] [c2653ad8] [c013d7bc] __alloc_pages_nodemask+0x8f4/0xbb4
[    5.431947] [c2653b88] [c010ea7c] shmem_getpage_gfp.constprop.0+0x2d4/0x7f8
[    5.438930] [c2653c58] [c00f333c] generic_perform_write+0xc0/0x1e4
[    5.445120] [c2653ca8] [c00f6194] __generic_file_write_iter+0x1c4/0x2c0
[    5.451732] [c2653ce8] [c00f6318] generic_file_write_iter+0x88/0x104
[    5.458089] [c2653d08] [c0160eec] do_iter_readv_writev+0x1f0/0x208
[    5.464273] [c2653d78] [c0162284] do_iter_write+0xa0/0x288
[    5.469757] [c2653da8] [c019b8a8] iter_file_splice_write+0x29c/0x428
[    5.476131] [c2653e48] [c019a978] splice_direct_to_actor+0x114/0x334
[    5.482482] [c2653e98] [c019ac34] do_splice_direct+0x9c/0xfc
[    5.488144] [c2653ee8] [c0161e1c] do_sendfile+0x17c/0x544
[    5.493546] [c2653f38] [c000f1c8] ret_from_syscall+0x0/0x38
[    5.499127] --- interrupt: c01 at 0xb7c46b18
[    5.499127]     LR = 0x10007d60
[    5.506525] Mem-Info:
[    5.508801] active_anon:3461 inactive_anon:3860 isolated_anon:0
[    5.508801]  active_file:0 inactive_file:0 isolated_file:0
[    5.508801]  unevictable:0 dirty:0 writeback:0
[    5.508801]  slab_reclaimable:116 slab_unreclaimable:777
[    5.508801]  mapped:304 shmem:7284 pagetables:11 bounce:0
[    5.508801]  free:1212 free_pcp:0 free_cma:0
[    5.539555] Node 0 active_anon:13844kB inactive_anon:15440kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1216kB dirty:0kB writeback:0kB shmem:29136kB writeback_tmp:0kB kernel_stack:184kB all_unreclaimable? yes
[    5.562586] Normal free:4848kB min:4848kB low:5036kB high:5224kB reserved_highatomic:0KB active_anon:13844kB inactive_anon:15440kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:49152kB managed:40108kB mlocked:0kB pagetables:44kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[    5.589768] lowmem_reserve[]: 0 0
[    5.593083] Normal: 0*4kB 2*8kB (UM) 0*16kB 1*32kB (M) 11*64kB (UM) 12*128kB (U) 6*256kB (UM) 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 4848kB
[    5.605819] 7284 total pagecache pages
[    5.609561] 0 pages in swap cache
[    5.612868] Swap cache stats: add 0, delete 0, find 0/0
[    5.618092] Free swap  = 0kB
[    5.620964] Total swap = 0kB
[    5.623841] 12288 pages RAM
[    5.626627] 0 pages HighMem/MovableOnly
[    5.630453] 2261 pages reserved
[    5.633585] Tasks state (memory values in pages):
[    5.638286] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[    5.646901] [    859]     0   859      412      300    24576        0             0 cp
[    5.654818] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),global_oom,task_memcg=/,task=cp,pid=859,uid=0
[    5.664757] Out of memory: Killed process 859 (cp) total-vm:1648kB, anon-rss:112kB, file-rss:4kB, shmem-rss:1084kB, UID:0 pgtables:24kB oom_score_adj:0
[    5.678653] oom_reaper: reaped process 859 (cp), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Killed
Error loading shared libr[    5.743863] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
[    5.751599] CPU: 0 PID: 1 Comm: init Not tainted 5.10.82 #0
[    5.757162] Call Trace:
[    5.759628] [c102be58] [c002ca0c] panic+0x120/0x2e0 (unreliable)
[    5.765636] [c102beb8] [c002f968] do_exit+0x878/0x8a4
[    5.770683] [c102bf08] [c002fa10] do_group_exit+0x40/0x98
[    5.776077] [c102bf28] [c002fa7c] __wake_up_parent+0x0/0x1c
[    5.781650] [c102bf38] [c000f1c8] ret_from_syscall+0x0/0x38
[    5.787219] --- interrupt: c01 at 0xb7f0d514
[    5.787219]     LR = 0xb7f7a86c
[    5.794610] Rebooting in 1 seconds..

Here the kernel crashes for lack of RAM; let me see if either

  • I can change the relocation address with other u-boot variables that won't restrict the kernel to just 0x3000000 of RAM
  • if picking a different address would let CAM mapping pull all of the RAM it needs
  • if increasing the number of CAM mapping layers in the kernel config would be enough to get all of the RAM and not just some of it
2 Likes

Nice! Just need to solve the OOM error and it might just start luci. That's exciting! Glad I could help find that config explanation.

Once you have a build that starts successfully, I can try it on a handful of units and see what happens if we want to see if there are any differences between units. I picked up a few from my buddy's storage just in case.

Thanks again for all your work getting this supported!

We're not out of the weeds yet. bootm_size is the only option I've seen work, but it definitely looks like it's also forcing Linux as a whole to limit to only so much RAM.

Edit: I've traced the source and confirmed that it won't be possible to coerce the fdt to relocate at a specific location. The memory management (lmb) in u-boot is bound by:

#define CONFIG_SYS_BOOTMAPSZ (160 << 20) /* Initial Memory map for Linux */

It is possible to step through manually and skip relocation, but you don't want this because a lot of cleanup and block-alignment and spacing goes into the relocation on the AP3825i.

Edit: I booted with the following options giving me enough memory (64M) not to OoM:

setenv bootm_low 0x0;
setenv bootm_size 0x4000000;
setenv bootm_mapsize 0x4000000;

From OpenWrt, I pulled /sys/firmware/fdt and decompiled (dtc -I dtb 64.fdt -o ap3825i_64mb.dts) the following dts:

https://paste.c-net.org/GrowlingWhitney

Looks like U-boot is shimming in this particular piece, which hampers us:

	memory {
		reg = <0x00 0x00 0x00 0x4000000>;
		device_type = "memory";
	};

Of course, but going from completely non-booting to starting but hitting OOM errors from some config options that just need to be played with to make it happy is definitely a nice change. Definitely easier to debug when you aren't trying to pull logs from a buffer triggered by a watchdog reset routine I'm sure.

Just glad you got to the first booting milestone!

You were right! I was way closer than I thought to the final answer.

So, long story short, the solution is:

  • Store your FIT image low -- think tftpboot or nand read to 0x2000000, not 0xa000000.
  • Use u-boot commands to unpack the FIT image. Instead of:
bootm 0x2000000;

do:

interrupts off; bootm start 0x2000000; bootm loados; bootm ramdisk;
fdt resize; fdt boardsetup; fdt chosen; bootm prep; bootm go

This just skips the far-away default relocation.

Testing sysupgrade now with a completely clean build.

1 Like

Nice! I figured it wouldn't be much longer. Congrats! I'm assuming you picked up that absolute pallet of WS-AP3825i units on eBay earlier hahaha

I'm looking forward to the sysupgrade bins when they're ready, and again, thanks for all your work!

Out of curiosity, how did the sysupgrade go? Any difficulties?

Also, are you planning to submit a PR for this at some point?

Mostly asking as I'd love to be able to test the units I have here to see if they even work properly. My buddy mentioned that he'd sell me his box of them (8 units) for a few bucks if I want them. If you could post a bin for testing, I'd be pretty thankful, as I don't have a build environment set up for these at the moment to apply your changes and compile it myself.

Sysupgrade went fine.

Go get the units, they work. I still need to

  • finish fixing LEDs
  • cleaning up device tree
  • cleaning up commits
    • adding instructions to commits
    • testing with more devices
    • verify instructions don't give any problems as kernel grows larger
  • potentially drop ath10k-ct for ath10k because wave1 chipsets have poorer ath10k-ct support

Tree: https://github.com/Hurricos/openwrt/tree/add_ap3825i_rebase

Builds: https://downloads.laboratoryb.org/releases/snapshot/targets/mpc85xx/p1020/

2 Likes

Alright! I was able to tftpboot into the kernel just fine on my testing unit and it booted successfully. I noticed it didn't have luci embedded in it, but that's no issue for the moment.

I was feeling adventurous, so as you mentioned that sysupgrade was working, I figured it wouldn't hurt to try flashing it through the command line. As it turns out, that did not work hahaha

I was able to load the sysupgrade image onto the device and run the command to start the sysupgrade process inside the tftpboot launched openwrt kernel, which claimed to succeed, but on reboot it just panics about the magic bits being invalid on the decompressed kernel image, then it drops into a recovery shell. I can still easily launch OpenWRT over tftpboot in Uboot after rebooting the unit, so it's not an issue, just figured I'd report. Knowing my lack of knowledge about how uboot works, I probably just flashed it incorrectly.

The rest of the units all boot up just fine over tftpboot, so I stopped there with them. I figure in just doing something wrong, so I'll wait for a better explanation on that and read up more. I'm satisfied just knowing that they all worked.

Thanks again, and great job on this! As I said, I'm happy to help test if you want to send me some things to try. I'm not worried about bricking this unit haha

1 Like

It might be that I'm not cleanly padding the kernel split, it automatically creates the partition but I probably should pad the kernel to a clean eraseblock length.

Okay, so just to make sure I'm clarifying properly, here's the steps and commands I took to flash the sysupgrade file.

-------
Setup
-------

setenv ipaddr 192.168.2.45
setenv serverip 192.168.2.30
tftpboot 0x2000000 openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-initramfs-kernel.bin
interrupts off; bootm start 0x2000000; bootm loados; bootm ramdisk; fdt resize; fdt boardsetup; fdt chosen; bootm prep; bootm go


-------
Network Config
-------

uci set network.lan.ipaddr="192.168.2.45"
uci set network.lan.gateway="192.168.2.1"
uci set network.lan.dns="192.168.2.1"
uci commit
/etc/init.d/network restart

-------
Flash
-------

mkdir /tmp/sysupgrade
cd /tmp/sysupgrade
wget https://downloads.laboratoryb.org/releases/snapshot/targets/mpc85xx/p1020/openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-squashfs-sysupgrade.bin
sysupgrade -v /tmp/sysupgrade/openwrt-mpc85xx-p1020-extreme-networks_ws-ap3825i-squashfs-sysupgrade.bin

From here, the system completes the process and then reboots. Here's my complete log starting from power-up to booting from tftp to running sysupgrade to the reboot and subsequent boot fail.

https://paste.c-net.org/MergerAmmonia

As I said, I can reboot and relaunch from uboot, but a regular reboot doesn't work right now for some reason. As I said, could be that I'm not doing the sysupgrade correctly, but that seems about right from my research anyway. Let me know your thoughts.

You also need to set the bootcmd correctly. This one will work:

setenv bootcmd 'cp.b 0xEC000000 0x2000000 0x2000000; interrupts off; bootm start 0x2000000; bootm loados; bootm ramdisk; fdt resize; fdt boardsetup; fdt chosen; bootm prep; bootm go;'
1 Like