Fortigate 30E flash ends up in recovery (initramfs) mode

Moshe · August 30, 2024, 9:38am

Flashed Fortinet Fortigate 30E from the product page https://openwrt.org/toh/hwdata/fortinet/fortinet_fortigate_30e

ends up with no luci, so I installed it manually, then got on the UI:

System running in recovery (initramfs) mode.

No changes to settings will be stored and are lost after rebooting. This mode should only be used to install a firmware upgrade

Tried to install the initramfs and the upgrade on top of it, but this didn't work. Either broke the machine or ended up in the same place.

What am I missing?
The device has 1GB of SD.
thanks

frollic · August 30, 2024, 9:45am

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=b7c12a9bfc01067f167ff805acdc1411d798b977 ?

not 100% clear which image was used for the 2nd part.

if everything else fails, try serial.

Moshe · August 30, 2024, 11:08am

Thanks @frolic for your answer!!
Here is a better explanation where I get stuck:

steps 1-9 (deploying the initramfs, loading the 2nd image (fg-30e-squashfs-sysupgrade.bin) and sysupgrade it work.

But then the system boots itself into void (as can be seen in the logs):
No dtb image in there. Try more
Initializing firewall...

here is what I did:

root@OpenWrt:~# 
root@OpenWrt:~# wget https://downloads.openwrt.org/snapshots/targets/mvebu/corte
xa9/openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin
Downloading 'https://downloads.openwrt.org/snapshots/targets/mvebu/cortexa9/openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin'
Connecting to 199.232.82.132:443
Writing to 'openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin'
openwrt-mvebu-cortex 100% |*******************************|  6270k  0:00:00 ETA
Download completed (6420794 bytes)
root@OpenWrt:~# 
root@OpenWrt:~# ls -l
-rw-r--r--    1 root     root       6420794 Aug 30 11:02 openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin
root@OpenWrt:~# 
root@OpenWrt:~# sysupgrade ./openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sys
upgrade.bin 
Fri Aug 30 11:03:08 UTC 2024 upgrade: Image not in /tmp, copying...
Cannot save config while running from ramdisk.
Fri Aug 30 11:03:09 UTC 2024 upgrade: Commencing upgrade. Closing all shell sessions.
Hangup
-ash: can't set tty process group: Not a tty
[1]+  Hangup                     sysupgrade ./openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin
root@OpenWrt:~# Watchdog handover: fd=3
- watchdog -
Watchdog does not have CARDRESET support
Fri Aug 30 11:03:09 UTC 2024 upgrade: Sending TERM to remaining processes ...
Fri Aug 30 11:03:09 UTC 2024 upgrade: Sending signal TERM to netifd (1631)
Fri Aug 30 11:03:13 UTC 2024 upgrade: Sending KILL to remaining processes ...
Fri Aug 30 11:03:13 UTC 2024 upgrade: Sending signal KILL to netifd (1631)
[  391.900341] stage2 (3462)op_caches: Fri Aug 30 11:03:21 UTC 2024 upgrade: Switching to ramdisk...
Fri Aug 30 11:03:22 UTC 2024 upgrade: Performing system upgrade...
fwinfo: offset-> 0x184, blocks-> 0x1ee8 (len: 0x003dce83)
1+0 records in
1+0 records out
fwinfo: offset-> 0x18c, blocks-> 0x1201 (len: 0x00240004)
1+0 records in
1+0 records out
Unlocking kernel ...

Writing from <stdin> to kernel ...     
Unlocking rootfs ...

Writing from <stdin> to rootfs ...     
Appending jffs2 data from /tmp/sysupgrade.tgz to rootfs..
.File /tmp/sysupgrade.tgz does not exist
    
Fri Aug 30 11:03:56 UTC 2024 upgrade: Upgrade completed
Fri Aug 30 11:03[  427.263441] reboot: Restg syst

FortiGate-30E (12:44-07.08.2016)
Ver:05000014
Serial number: FGT30E3U16022597
CPU(00): 1332MHz
Total RAM: 1GB
Initializing boot device...
Initializing MAC... egiga0
Please wait for OS to boot, or press any key to display configuration menu..........

Booting OS...

Reading boot image... 4050944 bytes.
No dtb image in there. Try more
Initializing firewall...

overclockrj · August 30, 2024, 11:53am

Same here. Stuck in Initializing firewall...

frollic · August 30, 2024, 12:01pm

@musashino it's your commit, can you provide assistance?

dimoni · August 30, 2024, 12:07pm

you stay on root folder, them you do cd /tmp and after wget & sysupgrade
cheers

frollic · August 30, 2024, 12:08pm

I noticed it too, but that's why it says copying ... ?

dimoni · August 30, 2024, 12:12pm

like that

musashino · August 30, 2024, 1:36pm

Please try to switch partition for booting.

open the bootmenu
select [B]: Boot with backup firmware and set as default.

Note:

and

are not related to this issue.

The mtd partitions mounted to / and others will be unmounted and those are unusable while upgrading by sysupgrade, so the image will be copied to /tmp/ before upgrading.

github.com

openwrt/openwrt/blob/c6f04f718ec89561f2261f2388857baa126eaa50/package/base-files/files/sbin/sysupgrade#L354-L365


      
          case "$IMAGE" in
          	'')
          		echo "Image file not found." >&2
          		exit 1
          		;;
          	/tmp/*)	;;
          	*)
          		v "Image not in /tmp, copying..."
          		cp -f "$IMAGE" /tmp/sysupgrade.img
          		IMAGE=/tmp/sysupgrade.img
          		;;
          esac

And, user configurations won't be kept on sysupgrade while running initramfs image and /tmp/sysupgrade.tgz won't be generated. So that error will be showed.

overclockrj · August 30, 2024, 2:40pm

first of all, thanks for your time @musashino

I tried the way you sent, but still the same. After apply the openwrt-mvebu-cortexa9-fortinet_fg-30e-squashfs-sysupgrade.bin image, it boots itself and then stuck in initializing firewall.

Using R or B the result is the same.

Captura de tela 2024-08-30 111743

Moshe · August 31, 2024, 9:12am

Thank you musashino for your replay. Hitting B says there is no backup.

What am I missing?

musashino · September 1, 2024, 5:30am

Please check the baudrate of console:

bootmenu ---> [I]: System information. ---> [S]: Set serial port baudrate.

If it is set to anything other than 9600bps, set it to 9600bps. Only that baudrate is supported on OpenWrt for FortiGate/FortiWiFi devices.

And, don't use D or B on Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]? prompt. initramfs/sysupgrade images are not for direct flashing.

BTW, I tried the steps in the commit on my FG-30E with the official firmware images, but the problem in this topic did not reoccur.

overclockrj · September 3, 2024, 11:07am

So, I think the problem is my fortigate box.

I did everything as you said and my fortigate stopped in initializing firewall again.

Baudrate is 9600
image initramfs applied using R option on Save as Default firmware/Backup firmware/Run image without saving:[D/B/R]
image squashfs-sysupgrade applied

after boot, stucks in Initializing firewall...

I quit.

By the way, thanks again for your time.

miminashi · September 11, 2024, 8:05pm

Same problem on FortiGate 50E.
I bought 10 FortiGate 50E devices, and 4 devices have this problem.

I still haven’t solved the problem, but I’ve left my work notes.

frollic · September 12, 2024, 6:25am

then it should be possible to compare the working and the non working ones ?

miminashi · September 15, 2024, 3:03pm

Maybe, yes.
However, I'm a novice at this kind of work.
To start, I'll compare the contents of mtd0 to mtd4.
I'd appreciate any advice you may have.

musashino · September 15, 2024, 3:19pm

Could you provide a dump of firmware-info partition?

hexdump -n $((0x200)) -C /dev/mtdblock1

A note about that partition:

https://memo205.hatenablog.jp/entry/2024/09/12/203303

miminashi · September 15, 2024, 4:40pm

Here is the result of the hexdump.
I have 3 non-booting devices on hand at now, but there were no differences in the hexdump results for any of the non-booting devices.

booting one:

root@OpenWrt:/# hexdump -n $((0x200)) -C /dev/mtdblock1
00000000  00 00 00 00 00 00 11 00  00 00 00 00 ff 00 aa 55  |...............U|
00000010  46 47 54 35 30 45 2d 35  2e 30 34 2d 46 57 2d 62  |FGT50E-5.04-FW-b|
00000020  75 69 6c 64 31 31 36 35  2d 31 37 31 30 31 38 2d  |uild1165-171018-|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000120  00 00 00 00 00 00 00 00  8d 04 00 00 00 00 00 00  |................|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000160  46 53 4f 43 00 00 20 00  00 00 00 0f e0 0f 00 00  |FSOC.. .........|
00000170  00 00 16 00 80 0e 00 00  00 0e 00 00 b0 01 00 00  |................|
00000180  00 10 00 00 4f 1b 00 00  00 40 00 00 01 10 00 00  |....O....@......|
00000190  00 00 01 00 00 00 00 00  00 30 01 00 00 00 00 00  |.........0......|
000001a0  00 f0 01 00 00 90 00 00  20 01 00 08 00 00 00 00  |........ .......|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 80 e6  |................|
000001c0  20 07 83 32 28 0a 00 f0  01 00 00 90 00 00 00 32  | ..2(..........2|
000001d0  29 0a 83 7d 31 0c 00 80  02 00 00 90 00 00 00 7d  |)..}1..........}|
000001e0  32 0c 83 51 01 10 00 10  03 00 00 f0 00 00 00 00  |2..Q............|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

not booting one:

root@OpenWrt:/# hexdump -n $((0x200)) -C /dev/mtdblock1
00000000  00 00 00 00 ff ff ff 0f  00 00 00 00 ff 00 aa 55  |...............U|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000160  46 53 4f 43 00 00 20 00  00 00 00 0f e0 0f 00 00  |FSOC.. .........|
00000170  00 00 00 00 80 0e 00 00  00 0e 00 00 b0 01 00 00  |................|
00000180  00 20 00 00 4f 1b 00 00  00 60 00 00 01 10 00 00  |. ..O....`......|
00000190  00 20 01 00 00 00 00 00  00 60 01 00 00 00 00 00  |. .......`......|
000001a0  00 20 02 00 00 80 00 00  00 01 00 00 00 00 00 00  |. ..............|
000001b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 80 00  |................|
000001c0  05 00 83 00 3b 05 00 20  02 00 00 80 00 00 00 00  |....;.. ........|
000001d0  3c 05 83 fc 33 09 00 a0  02 00 00 80 00 00 00 fc  |<...3...........|
000001e0  34 09 83 f9 2b 0e 01 00  10 00 00 00 80 00 00 00  |4...+...........|
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

miminashi · September 15, 2024, 5:48pm

I found a workaround.
Since I'm not sure if it's safe, please do not try it lightly.

Procedure:

Boot the device that can boot from sysupgrade image and make it ready for SSH connection

Dump /dev/mtd1 from the device that can boot from sysupgrade image

Execute on your Linux machine

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.1 'dd if=/dev/mtd1' > mtd1_firmware-info

Boot the non-bootable device using initramfs from TFTP and make it ready for SSH connection

Write the dumped image to /dev/mtd1 on the non-bootable device.

Execute on your Linux machine

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@192.168.1.1 'cat > /tmp/mtd1_firmware-info' < mtd1_firmware-info

Execute on the 50E device booted with initramfs

mtd write /tmp/mtd1_firmware-info /dev/mtd1
reboot

Then, the sysupgrade image will be bootable.

musashino · September 16, 2024, 9:39am

I tried to calculate length/offset values of 2x images on your devices, the 2nd one has the wrong offsets for image1 and image2.

Booting

image1:

kernel: 0x369E00@0x200000
rootfs: 0x200200@0x800000

image2 (empty):

kernel: 0x0@0x2000000
rootfs: 0x0@0x2600000

Not booting

image1:

kernel: 0x369E00@0x400000
rootfs: 0x200200@0xC00000

image2 (empty):

kernel: 0x0@0x2400000
rootfs: 0x0@0x2C00000

Note: length and offsets are aligned by 0x200 (512 bytes)

Partition definitions in DeviceTree

			partition@200000 {
				reg = <0x200000 0x600000>;
				label = "kernel";
			};

			partition@800000 {
				reg = <0x800000 0x1800000>;
				label = "rootfs";
			};

			partition@2000000 {
				reg = <0x2000000 0x600000>;
				label = "kn2";
				read-only;
			};

			partition@2600000 {
				reg = <0x2600000 0x1800000>;
				label = "rfs2";
				read-only;
			};

So maybe the kernel data is being read from the wrong location.

I'll try to improve sysupgrade script (fortinet.sh) to handle offsets in addition to length.