Yeah, it was ISE Webpack, what a horrible piece of SW was that.
You are correct, they are mapping the CS0 NOR to 0x0 on SparX-5:
It boots in 24 bit mode for 3 byte addressing by default.
You can switch it to 32bit addressing and access up to 1G directly as MMIO.
I have asked, they claim there is not BootROM at all
That's interesting, but I suppose only possible for devices that 'only' support mapped storage. E.g. NAND controllers need quite a bit of setup to get all parameters right, forget about USB and UART isn't useful. If you want to be able to have 'fallback' behavior, you need a BOOTROM. BTW, your memory map is not clear if it is offset to 0x0000 (reset vector) of if there is SOME offset (SI) and those chips start at +1, +2, +3. If you know how it is mapped after boot I suppose you'll know.
The OMAP2 from TI has an interesting bootrom, in that it actually features support for all of the above, but also ethernet (DHCP + tftp) boot support. For factory control, that's really cool; as you take an empty board, and force it to netboot to provision it 
I personally really like this approach much much more, no room for exploits inside the chip; no GPL violations even possible within the CHIP, no proprietary behavior inside the chip.
I wonder if our chip internally maps the QSPI also. Though I think it has NAND boot support too; but there's strapping pins for that. So maybe it also is BootROM less. MUCH less need on fixed-function devices like these. To bad we don't have the datasheet that shares with us this information, but the fact that the flash is exposed on 0xb4000000 makes me think no. I'll try to look at the address space again, and where MIPS puts its reset vector by default, that should give us some hints.
Edit: According to wikipedia for mips, the reset vector is normally at virtual address 0xBFC00000. Interesting, and I'm sure this can be changed when 'compiling'. Learned something new, that it is not always at 0x0000 
Yes, SparX-5 is unique that it only support booting from SPI-NOR as rhat is simplest, I checked the memory map and yes, NOR CS0 is mapped at 0x0 and the rest are offset by 16MB if 3B addressing is used.
There is no NAND or eMMC booting support, nor any built-in recovery, there is literaly no burned in SW/BootROM.
Everything is set-up by U-boot, first you only have 64K of SRAM to use before DRAM is trained and you configure the console.
While this is great, means that you have to use an external programmer if U-boot gets broken.
Personally, I would have liked an BootROM like Marvells that allows recovery at least via UART.
To me it looks like RTL is also memory mapping the NOR
It does, to 0xb4000000, but not the reset vector sadly it seems.
Very true, however, you'd still need to know HOW to send data, the protocol; and most vendors don't give this up, or have weird tooling that you'd need to use. But yes, having a fallaback to UART would be great. Putting the pinout for a socket or something similar and just use that during development would be greater 
I am lucky that this is a dev board and there is dedicates flash programming connector so I just hooked up an FT232 breakout and used flashrom.
They connected the system reset to the MCP2211 that is used for UART, so I can toggle the system reset via its GPIO.
BTW, just check the frequency that it used to load from NOR, and its 20-30kHz when it starts loading.
Marvells Armada 7k and later BootROM is quite simple, you spam it with an escape sequence on boot and then just send the image using xmodem that gets booted after loading
Hi Janh,
Excellent work! I'm trying this on a hpe_1920-24g and I can't seem to get the initramfs to fully boot. It looks like the HPE bootware tries to boot it but then it stops and the regular boot process begins. This is what I see when I'm in console.
[ 17.508437] rtl83xx_mdio_probe port -1 has SDS
[ 17.523257] rtl83xx_mdio_probe found port 23
[ 17.537506] rtl83xx_mdio_probe port 23 has phandle
[ 17.553460] rtl83xx_mdio_probe port -1 has SDS
[ 17.568277] rtl83xx_mdio_probe found port 24
[ 17.582527] rtl83xx_mdio_probe port 24 has phandle
[ 17.598486] rtl83xx_mdio_probe port -1 has SDS
[ 17.613303] rtl83xx_mdio_probe found port 25
[ 17.627550] rtl83xx_mdio_probe port 25 has phandle
[ 17.643504] rtl83xx_mdio_probe port -1 has SDS
[ 17.658321] rtl83xx_mdio_probe found port 26
System is starting...
Press Ctrl+D to access BASIC-BOOTWARE MENU
Booting Normal Extend BootWare
The Extend BootWare is self-decompressing......................Done!
Any insight on what settings I need to change to have the initramfs fully boot? Thanks for the great work!
This looks like another case of the watchdog being triggered because the ping worker doesn't get a chance to run. I guess the likelihood of this specific issue increases with the number of ports, as I am sporadically seeing the same issue on a 1920-48G. Unfortunately, I haven't found a solution yet.
Could you hog GPIO0 to an input? That should float the WDI line for the PT7A7514, which would just disable the external watchdog.
I already tried that, and it doesn't work (the watchdog gets triggered). And as the external watchdog is needed to reliably reboot these devices, just leaving GPIO0 in the default "blinking" state is also not an option.
Usually, you can instruct the watchdog with how long it should wait for pings. Have we reached the max in our watchdog driver?
https://github.com/torvalds/linux/blob/master/drivers/watchdog/realtek_otto_wdt.c#L9 indicates we should be able to increase the timeout.
There is at least one kernel parameter that you could try from u-boot (bootargs): open_timeout though I'm not familiar with that one, it seems like this tells the kernel, to wait until X seconds before someone opens /dev/watchdogN to actually ping the watchdog. So setting this to 30s is not horrible.
BUT I do not know if our kernel driver actually supports this feature. Reason I say this, is that if U-Boot turns on the watchdog and sends its final ping before jumping to the kernel; the timer starts to expire, so this setting doesn't do anything in this case, because the timer is already running. If our current bootloader doesn't turn on and ping the watchdog however, we'd be good (though bad design, you want to enable the wdt as early as possible to ensure you do not get stuck hardware 
In a previous project, I used an OMAP, which had an additional flag (should all really be standard required flags imo ...) timer_margin set to something nice and big; so that the kernel would load up and (re)configure the wdt, which probably/would trigger a ping (good). Might need early_enable to make sure the kernel actually loads and enables the wdt though.
I don't see any docs for out wdt, so I suppose we only support the basics; but this would be a fix. But then, I have no idea how all our platforms do their resets and if it is even tied at all to our wdt
In this case, there is an external watchdog (PT7A7514) which is non-configurable. It has a fixed timeout of 1.6 seconds (typical, min 1.2 s, max 2.25 s).
After adding a call to cond_resched at the beginning of the for_each_node_by_name loop in rtl83xx_mdio_probe I haven't seen the issue again.
Interestingly, that loop only takes 2+ seconds because of the pr_info calls. If I remove them, the entire loop finishes practically instantly.
Sounds like a good reason for a patch, at least demoting some things to pr_debug() (or better still: dev_debug()). Anyone who wants to debug something, will probably have to build their own images anyway.
Maybe the internal pull-down on GPIO0 is to blame for that. Probably still leaking enough current for WDI to sense it. Would explain why I've seen that watchdog chip with an external tri-state buffer on other boards.
The pictures of the 1920-8G don't seem to have much more than some passives around the watchdog chip (U7).
Ah, thanks for confirming the issue! I thought I had missed something in your instructions.
how do we ping it? by toggeling a GPIO? I take it we use https://elixir.bootlin.com/linux/latest/source/drivers/watchdog/gpio_wdt.c? It doesn't have the early_init flag, but there should be some way to make sure it has a very high priority, one of the first drivers to actually load. Do we use devicetree to instantiate this wdt? or is it fully software driven? What device do you use again? what's your devicetree?
While it's a good test methodology, I would rely on these kinds of solutions, as the problem just shifts elsewhere, it remains a racy thing.
The cond_resched sounds like a much better solution. Was the driver before fully blocking? With a single core that could be an issue yeah ...
I just tried this and it's alive! Thanks for sharing! Have you noticed any issues yet with leaving cond_resched() in that loop?
The watchdog is connected to the system LED output of the SoC, and the bootloader configures the system LED to blink via the LED_GLB_CTRL register. So the watchdog is satisfied even without handling it in any way from the Linux side (as long as that register isn't touched). What actually made me implement it properly is that the watchdog appears to provide the only reliable way to reboot (on my HPE 1920-16G rebooting worked without it, but the 1920-8G would just hang).
The device tree sets the always-running property to make sure the watchdog is toggled the entire time from the driver being loaded until reboot. But it doesn't matter when the driver is loaded as long as the pinmux configuration happens at the same time (which the kernel should already take care of).
It looks like the driver is actually loaded about 5 seconds in (I added a log message to the watchdog ping function during debugging, and this is the timestamp of the kernel log when the message shows up for the first time). From previous testing without the watchdog driver I know that manually changing the pinmux for the system LED to GPIO mode actually triggers a reset within the specified timeout of the watchdog. So it's pretty clear that the load time of the driver is not an issue.
This is the code in question: https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=target/linux/realtek/files-5.10/drivers/net/dsa/rtl83xx/common.c;h=fd6019ec62e19bf1d7b75e4df45957155cb3ae04;hb=HEAD#l316
Hah, that is so funny, so HP abuses the early led blink to ping the watchdog. E.g. the sys-led. I suppose the 'real' sys-led is connected to something else. Guess we need a wdt-realtek-sysled.c driver But that's a horrible design, because that basically means there is no real watchdog, until you configure the pin as GPIO and 'take over'. So wdt-gpio would be fine; but only 'activate' it, once user-space is up and running. Your solution does offer the kernel to handle it 'earlier' but might still be racey? not sure yet.
Thanks for sharing!