My MIPS-based Lantiq xrx200 router (o2 Box 6431) keeps crashing randomly and I really would like to see why. Syslog over the network doesn't capture anything, the messages just stop and then the router reboots.
The Linux kernel messages on bootup have one really intriguing entry which looks like it might help:
This message seems to be specific to OpenWrt. Does anybody know what that feature does exactly and/or how to read crash logs from previous boots? Reading the patch for that feature helped a bit, but I'm not sure how crash log storage gets triggered.
@nbd: You wrote that patch. Would you please share some insight about it?
It is supposed to end up in /sys/kernel/debug/crashlog after a crash triggering a reboot. Whether it actually works in practise mostly depends on the bootloader and some other circumstances.
For the crash dump to be recoverable, the RAM allocated for it must not be cleared by the bootloader or the kernel after a reset.
Thanks for the info. I just tested this on my Lantiq xrx200 o2 Box 6431 by forcing a kernel panic with echo c > /proc/sysrq-trigger
and after the box came up again, /sys/kernel/debug/crashlog contained the last 16 kB of kernel messages including the crash message.
This should be documented on the wiki (and possibly merged upstream) because it is a really helpful tool. I wish I had known of this earlier. @nbd Thank you very much for writing this patch.
[edit]As @jow pointed out, @nbd wrote the patch. My apologies for the confusion.[/edit]
I should really send a patch which modifies crashlog_copy() to print a message "Crashlog found from previous boot, stored in /sys/kernel/debug/crashlog".
Debugging a random crash on my Buffalo WZR-600DHP. Just want to confirm crashlog seems to work on ar71xx + OpenWrt 19.07.2 r10947-65030d81f3 (4.14.171) + kernel sysrq trigger-a-crash. It looks like this:
# echo c > /proc/sysrq-trigger
<ssh session drops here>
On restarting router:
# cat /sys/kernel/debug/crashlog
Time: 1586907412.328379
Modules: ath9k@87680000+17590 ath9k_common@877f8000+2be0 pppoe@87678000+23a0 ppp_async@8766e000+1aa0 ath9k_hw@87000000+53da0 ath@87670000+4740 pppox@877fd000+490 ppp_generic@87668000+5710 nf_nat_pptp@87737000+6e0 nf_conntrack_pptp@877ed000+e80 nf_conntrack_ipv6@8773c000+1530 mac80211@87780000+6c160 iptable_nat@8773e000+2a0 ipt_REJECT@8765f000+3a0 ipt_MASQUERADE@8765a000+280 cfg80211@87700000+36230 xt_time@8765b000+6c0 xt_tcpudp@8765d000+6c0 xt_state@87660000+2a0 xt_nat@8765e000+5e0 xt_multiport@87656000+4c0 xt_mark@87654000+280 xt_mac@87652000+240 xt_limit@87655000+480xt_conntrack@87647000+8e0 xt_comment@87648000+1c0 xt_TCPMSS@87649000+aa0 xt_REDIRECT@87645000+2a0 xt_NFQUEUE@87646000+d20 xt_LOG@8763d000+2e0 xt_FLOWOFFLOAD@875d7000+bd0 xt_CT@8763b000+a40 ts_fsm@875fa000+aa0 ts_bm@87633000+560 slhc@8764a000+1360 nfnetlink_queue@8764c000+2660 nfnetlink@87638000+1020 nf_reject_ipv4@87642000+820 nf_nat_tftp@87632000+200 nf_nat_snmp_basic@87640000+1890 nf_nat_sip@8763e000+1e80 nf_nat_redirect@87631000+580 nf_nat_proto_gre@875d2000+330nf_nat_masquerade_ipv4@875f9000+630 nf_nat_irc@875fe000+3c0 nf_conntrack_ipv4@87636000+13c0 nf_nat_ipv4@87627000+ef0 nf_nat_h323@8762c000+14a0 nf_nat_amanda@875fd000+2e0 nf_nat@87628000+2730 nf_log_ipv4@875b5000+ce0 nf_flow_table_hw@875cb000+820 nf_flow_table@875dc000+3350 nf_defrag_ipv6@875d4000+1260 nf_defrag_ipv4@875be000+420 nf_conntrack_tftp@875c9000+ac0nf_conntrack_snmp@875ef000+2d0 nf_conntrack_sip@87620000+4840 nf_conntrack_rtcache@875ff000+a20 nf_conntrack_proto_gre@875cf000+a40 nf_conntrack_irc@875bb000+b30 nf_conntrack_h323@875f0000+8a40 nf_conntrack_broadcast@875a8000+360 ts_kmp@875c1000+500 nf_conntrack_amanda@875bf000+6a0 nf_conntrack@875e0000+eb80 iptable_raw@8759d000+240 iptable_mangle@87599000+380 iptable_filter@8759c000+260 ip_tables@875b0000+2950 crc_ccitt@875a4000+3c0 compat@875cc000+1700 sch_teql@87595000+de0 sch_sfq@875a0000+2460 sch_red@8759e000+1560 sch_prio@8758e000+d80 sch_pie@8759a000+1080 sch_multiq@8758f000+d40 sch_gred@87596000+1c00 sch_fq@87590000+1a40 sch_dsmark@8758a000+fe0 sch_codel@8756e000+1760 em_text@87589000+560 em_nbyte@8757f000+2c0 em_meta@87586000+1540em_cmp@8756c000+2e0 act_simple@87585000+750 act_police@87569000+d90 act_pedit@87582000+f70 act_ipt@87580000+e10 act_gact@8755f000+8d0 act_csum@8757c000+13b0 libcrc32c@8756a000+230 sch_tbf@87564000+16a0 sch_ingress@87fe1000+4a0 sch_htb@87578000+36b0sch_hfsc@87574000+39c0 em_u32@8755d000+200 cls_u32@87570000+21d0 cls_tcindex@87560000+14c0 cls_route@87fe2000+1400 cls_matchall@8743e000+9a0 cls_fw@87ffe000+ec0 cls_flow@87562000+15c0 cls_basic@8755c000+be0 act_skbedit@87436000+970 act_mirred@87555000+dd0 ledtrig_usbport@8743f000+8f0 nf_log_ipv6@87553000+fa0 nf_log_common@87430000+a20 ip6table_mangle@87559000+4a0 ip6table_filter@87fe7000+260 ip6_tables@87fec000+28e0 ip6t_REJECT@87556000+3e0 x_tables@87fe8000+3170nf_reject_ipv6@87fff000+980 vfat@87fe4000+2180 fat@87ff0000+c910 nls_utf8@8743a000+320 nls_iso8859_1@8743b000+b00 nls_cp437@8743c000+1100 usb_storage@87540000+9e60 sd_mod@87518000+71d0 scsi_mod@87520000+15430 crc32c_generic@874d4000+540 crypto_hash@874f4000+21c0 ehci_platform@87f9e000+10a0 ehci_hcd@87f80000+89f0 gpio_button_hotplug@87f90000+1400 usbcore@87fc0000+1f650 nls_base@87f70000+12e0 usb_common@87f50000+540
<5>[ 3.268037] 0x000000450000-0x000001ff0000 : "rootfs_data"
<5>[ 3.275652] 0x000001ff0000-0x000002000000 : "user_property"
<6>[ 3.488332] ar8316: Using port 4 as switch port
<6>[ 3.533010] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.0:00 [uid=004dd041, driver=Atheros AR8216/AR8236/AR8316]
<6>[ 3.544128] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode: rgmii
<6>[ 3.888328] ar8316: Using port 4 as PHY
<6>[ 4.949072] ag71xx ag71xx.1: connected to PHY at ag71xx-mdio.0:04 [uid=004dd041, driver=Atheros AR8216/AR8236/AR8316]
<6>[ 4.960186] eth1: Atheros AG71xx at 0xba000000, irq 5, mode: rgmii
<6>[ 4.967783] NET: Registered protocol family 10
<6>[ 4.975748] Segment Routing with IPv6
<6>[ 4.979559] NET: Registered protocol family 17
<6>[ 4.984041] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
<6>[ 4.997452] 8021q: 802.1Q VLAN Support v1.8
<6>[ 5.002986] hctosys: unable to open rtc device (rtc0)
<6>[ 5.013242] VFS: Mounted root (squashfs filesystem) readonly on device 31:7.
<6>[ 5.021333] Freeing unused kernel memory: 328K
<4>[ 5.025764] This architecture does not have kernel memory protection.
<14>[ 5.731337] init: Console is alive
<14>[ 5.734939] init: - watchdog -
<14>[ 6.769173] kmodloader: loading kernel modules from /etc/modules-boot.d/*
<6>[ 6.828384] usbcore: registered new interface driver usbfs
<6>[ 6.833938] usbcore: registered new interface driver hub
<6>[ 6.839373] usbcore: registered new device driver usb
<6>[ 6.848824] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
<6>[ 6.856756] ehci-platform: EHCI generic platform driver
<6>[ 6.862114] ehci-platform ehci-platform: EHCI Host Controller
<6>[ 6.867909] ehci-platform ehci-platform: new USB bus registered, assigned bus number 1
<6>[ 6.875909] ehci-platform ehci-platform: irq 3, io mem 0x1b000000
<6>[ 6.906992] ehci-platform ehci-platform: USB 2.0 started, EHCI 1.00
<6>[ 6.913978] hub 1-0:1.0: USB hub found
<6>[ 6.918080] hub 1-0:1.0: 2 ports detected
<14>[ 6.923082] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
<14>[ 6.940756] init: - preinit -
<6>[ 7.447602] usb 1-1: new high-speed USB device number 2 using ehci-platform
<6>[ 7.526540] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<5>[ 7.553909] random: procd: uninitialized urandom read (4 bytes read)
<7>[ 8.587335] ar71xx: pll_reg 0xb8050010: 0x110000
<6>[ 8.587349] eth0: link up (1000Mbps/Full duplex)
<6>[ 8.592012] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<5>[ 11.132737] jffs2: notice: (438) jffs2_build_xattr_subsystem: complete building xattr subsystem, 12 of xdatum (3 unchecked, 9 orphan) and 76 of xref (9 dead, 3 orphan) found.
<14>[ 11.176750] mount_root: loading kmods from internal overlay
<14>[ 12.158098] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
<5>[ 12.284447] SCSI subsystem initialized
<6>[ 12.329671] usb-storage 1-1:1.0: USB Mass Storage device detected
<6>[ 12.336550] scsi host0: usb-storage 1-1:1.0
<6>[ 12.341209] usbcore: registered new interface driver usb-storage
<14>[ 12.351005] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
<14>[ 12.881631] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
<14>[ 12.900076] block: extroot: not configured
<14>[ 12.905250] mount_root: switching to jffs2 overlay
<4>[ 12.932729] overlayfs: upper fs does not support tmpfile.
<12>[ 12.945006] urandom-seed: Seeding with /etc/urandom.seed
<6>[ 13.066097] eth0: link down
<14>[ 13.092891] procd: - early -
<14>[ 13.095869] procd: - watchdog -
<5>[ 13.387951] scsi 0:0:0:0: Direct-Access CBM Flash Disk 5.00 PQ: 0 ANSI: 2
<5>[ 13.398894] sd 0:0:0:0: [sda] 2009088 512-byte logical blocks: (1.03 GB/981 MiB)
<5>[ 13.411904] sd 0:0:0:0: [sda] Write Protect is off
<7>[ 13.416730] sd 0:0:0:0: [sda] Mode Sense: 0b 00 00 08
<3>[ 13.423064] sd 0:0:0:0: [sda] No Caching mode page found
<3>[ 13.428443] sd 0:0:0:0: [sda] Assuming drive cache: write through
<6>[ 13.470382] sda: sda1 sda2
<5>[ 13.482177] sd 0:0:0:0: [sda] Attached SCSI removable disk
<14>[ 13.978780] procd: - watchdog -
<14>[ 13.982196] procd: - ubus -
<5>[ 14.102648] random: ubusd: uninitialized urandom read (4 bytes read)
<5>[ 14.111614] random: ubusd: uninitialized urandom read (4 bytes read)
<5>[ 14.118538] random: ubusd: uninitialized urandom read (4 bytes read)
<14>[ 14.126065] procd: - init -
<14>[ 15.375453] kmodloader: loading kernel modules from /etc/modules.d/*
<14>[ 15.666161] urngd: v1.0.2 started.
<6>[ 15.849473] ip6_tables: (C) 2000-2006 Netfilter Core Team
<6>[ 15.928258] Mirror/redirect action on
<5>[ 15.952079] random: crng init done
<5>[ 15.955487] random: 6 urandom warning(s) missed due to ratelimiting
<6>[ 16.022213] u32 classifier
<6>[ 16.024921] input device check on
<6>[ 16.028623] Actions configured
<6>[ 16.240243] GACT probability on
<6>[ 16.251008] Simple TC action Loaded
<6>[ 16.474613] Loading modules backported from Linux version v4.19.98-0-gd183c8e2647a
<6>[ 16.482220] Backport generated by backports.git v4.19.98-1-0-g8204eb99
<6>[ 16.519797] ip_tables: (C) 2000-2006 Netfilter Core Team
<6>[ 16.547529] nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
<6>[ 16.958626] Netfilter messages via NETLINK v0.30.
<6>[ 17.021629] xt_time: kernel timezone is -0000
<6>[ 17.145906] PPP generic driver version 2.4.2
<6>[ 17.168032] NET: Registered protocol family 24
<6>[ 17.229609] pci 0000:00:11.0: using irq 40 for pin 1
<4>[ 17.234583] PCI: Enabling device 0000:00:11.0 (0000 -> 0002)
<6>[ 17.245133] ath: phy0: Ignoring endianness difference in EEPROM magic bytes.
<7>[ 17.253731] ath: EEPROM regdomain: 0x0
<7>[ 17.253736] ath: EEPROM indicates default country code should be used
<7>[ 17.253739] ath: doing EEPROM country->regdmn map search
<7>[ 17.253750] ath: country maps to regdmn code: 0x3a
<7>[ 17.253755] ath: Country alpha2 being used: US
<7>[ 17.253758] ath: Regpair used: 0x3a
<7>[ 17.268206] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
<6>[ 17.269987] ieee80211 phy0: Atheros AR9280 Rev:2 mem=0xb0000000, irq=40
<6>[ 17.276697] pci 0000:00:12.0: using irq 41 for pin 1
<4>[ 17.281719] PCI: Enabling device 0000:00:12.0 (0000 -> 0002)
<6>[ 17.292220] ath: phy1: Ignoring endianness difference in EEPROM magic bytes.
<7>[ 17.300823] ath: EEPROM regdomain: 0x0
<7>[ 17.300827] ath: EEPROM indicates default country code should be used
<7>[ 17.300830] ath: doing EEPROM country->regdmn map search
<7>[ 17.300840] ath: country maps to regdmn code: 0x3a
<7>[ 17.300845] ath: Country alpha2 being used: US
<7>[ 17.300848] ath: Regpair used: 0x3a
<7>[ 17.317332] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
<6>[ 17.319227] ieee80211 phy1: Atheros AR9280 Rev:2 mem=0xb0010000, irq=41
<14>[ 17.547984] kmodloader: done loading kernel modules from /etc/modules.d/*
<6>[ 28.453812] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
<7>[ 28.460135] ar71xx: pll_reg 0xb8050010: 0x110000
<6>[ 28.464745] eth0: link up (1000Mbps/Full duplex)
<6>[ 28.480027] br-lan: port 1(eth0.1) entered blocking state
<6>[ 28.485424] br-lan: port 1(eth0.1) entered disabled state
<6>[ 28.491155] device eth0.1 entered promiscuous mode
<6>[ 28.495939] device eth0 entered promiscuous mode
<6>[ 28.507051] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
<6>[ 28.571860] br-lan: port 1(eth0.1) entered blocking state
<6>[ 28.577309] br-lan: port 1(eth0.1) entered forwarding state
<6>[ 28.583007] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
<6>[ 28.668885] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
<6>[ 29.467084] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
<6>[ 32.020350] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
<6>[ 32.094530] br-lan: port 2(wlan0) entered blocking state
<6>[ 32.099898] br-lan: port 2(wlan0) entered disabled state
<6>[ 32.105481] device wlan0 entered promiscuous mode
<7>[ 32.907660] ar71xx: pll_reg 0xb8050014: 0x110000
<6>[ 32.912284] eth1: link up (1000Mbps/Full duplex)
<6>[ 32.935654] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
<6>[ 34.102232] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
<6>[ 34.108798] br-lan: port 2(wlan0) entered blocking state
<6>[ 34.114125] br-lan: port 2(wlan0) entered forwarding state
<6>[ 684.441078] nf_conntrack: default automatic helper assignment has been turned off for security reasons and CT-based firewall rule not found. Use the iptables CT target to attach helpers instead.
<6>[ 980.913462] device br-lan entered promiscuous mode
<6>[ 1724.662877] device br-lan left promiscuous mode
<6>[ 1725.562480] sysrq: SysRq : Trigger a crash
<1>[ 1725.566778] CPU 0 Unable to handle kernel paging request at virtual address 00000000, epc == 80267fe0, ra == 802683e4
<4>[ 1725.577574] Oops[#1]:
<4>[ 1725.579857] CPU: 0 PID: 2462 Comm: ash Not tainted 4.14.171 #0
<4>[ 1725.585671] task: 87c97840 task.stack: 87392000
<4>[ 1725.590188] $ 0 : 00000000 00000001 00000001 804d0000
<4>[ 1725.595408] $ 4 : 00000063 804d1d88 804d90f0 00004480
<4>[ 1725.600629] $ 8 : 00000000 00000000 00000007 00000000
<4>[ 1725.605851] $12 : 000000f2 804d0000 000000f1 00000000
<4>[ 1725.611071] $16 : 804dcf90 804d0000 00000063 00000000
<4>[ 1725.616293] $20 : 00000008 804dd100 77f2edb0 00000000
<4>[ 1725.621515] $24 : 00000001 8026ec84
<4>[ 1725.626736] $28 : 87392000 87393df0 00000000 802683e4
<4>[ 1725.631958] Hi : 00000000
<4>[ 1725.634824] Lo : 19d60000
<4>[ 1725.637694] epc : 80267fe0 0x80267fe0
<4>[ 1725.641519] ra : 802683e4 0x802683e4
<4>[ 1725.645344] Status: 1000fc03 KERNEL EXL IE
<4>[ 1725.649519] Cause : 8080000c (ExcCode 03)
<4>[ 1725.653518] BadVA : 00000000
<4>[ 1725.656385] PrId : 00019374 (MIPS 24Kc)
<4>[ 1725.660297] Modules linked in: ath9k ath9k_common pppoe ppp_async ath9k_hw ath pppox ppp_generic nf_nat_pptp nf_conntrack_pptp nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_state xt_nat xt_multiport xt_mark xt_mac xt_limit xt_conntrack xt_comment xt_TCPMSS xt_REDIRECT xt_NFQUEUE xt_LOG xt_FLOWOFFLOAD xt_CT ts_fsm ts_bm slhc nfnetlink_queue nfnetlink nf_reject_ipv4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_redirect nf_nat_proto_gre nf_nat_masquerade_ipv4 nf_nat_irc nf_conntrack_ipv4 nf_nat_ipv4 nf_nat_h323 nf_nat_amanda nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_rtcache nf_conntrack_proto_gre nf_conntrack_irc nf_conntrack_h323 nf_conntrack_broadcast
<4>[ 1725.731386] ts_kmp nf_conntrack_amanda nf_conntrack iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt compat sch_teql sch_sfq sch_red sch_prio sch_pie sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_police act_pedit act_ipt act_gact act_csum libcrc32c sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred ledtrig_usbport nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 vfat fat nls_utf8 nls_iso8859_1 nls_cp437 usb_storage sd_mod scsi_mod crc32c_generic crypto_hash ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common
<4>[ 1725.795735] Process ash (pid: 2462, threadinfo=87392000, task=87c97840, tls=77f35efc)
<4>[ 1725.803540] Stack : 87c158d0 80487fc8 00000000 00004450 00000002 fffffffb 00000002 77f2b2d0
<4>[ 1725.811886] 77f08d88 77f34000 77f2edb0 8026853c 00000000 0000000b 87cd8f00 00000020
<4>[ 1725.820233] 87cba580 80196c7c 00000400 80160250 0000000d 12cffae0 00000003 00000000
<4>[ 1725.828581] 00000000 87445be0 87393f08 80140570 77f2edb0 00000000 00000000 80089518
<4>[ 1725.836928] 00000000 87393e78 87af2640 7fd3afb4 00000003 00000007 00000000 00000000
<4>[ 1725.845275] ...
<4>[ 1725.847718] Call Trace:
<4>[ 1725.847763] [<8026853c>] 0x8026853c
<4>[ 1725.853706] [<80196c7c>] 0x80196c7c
<4>[ 1725.857184] [<80160250>] 0x80160250
<4>[ 1725.860692] [<80140570>] 0x80140570
<4>[ 1725.864172] [<80089518>] 0x80089518
<4>[ 1725.867693] [<80087904>] 0x80087904
<4>[ 1725.871207] [<80140898>] 0x80140898
<4>[ 1725.874710] [<80160738>] 0x80160738
<4>[ 1725.878217] [<80140afc>] 0x80140afc
<4>[ 1725.881699] [<8007148c>] 0x8007148c
<4>[ 1725.885182] [<8008958c>] 0x8008958c
<4>[ 1725.888662]
<4>[ 1725.890145] Code: 24020001 ac62e288 0000000f <03e00008> a0020000 27bdffe0 24020007 afb10018 afb00014
<4>[ 1725.899887]
<4>[ 1725.902399] ---[ end trace 97497014ac44ecb9 ]---
FYI, upstream supports a RAM-backed pstore ("persistent store") that works in a pretty similar way, IIUC. I believe it differs from crashlog in that one (e.g., the bootloader) has to specifically reserve a pstore region for use, whereas crashlog opportunistically guessed at some fixed offsets it could use. In practice, this likely means that you'll have a tougher time getting it working on an arbitrary OpenWRT system, where the DTS probably doesn't describe a pstore region. But you might be able to guess at a useful memory range, edit your own DTS, and enable the PSTORE drivers yourself.
Not exactly true w.r.t defining a region. You can just as well reserve a region for pstore that is not clobbered by a bootloader of particular device, even picking the guess of crashlog, in the device tree. Been there, done that.
I hope that maintainers accept patches adding such regions in future.
Of course, this works best if you can reserve memory in, say, U-boot specifically for this using
CONFIG_PRAM, or through environment variable "pram" if writable environment is supported, or if you can compile your own bootloader from source, but this is rarely a viable solution for end user.
