I've used 18.06.2, 18.06.4, 18.06.5 and 19.07.0 on this router and have never seen this before. I installed 19.07.2 on Sat March 7th. The router has locked up twice now. Once on Monday March 9th and again this morning. I can't even login via 192.168.1.1. Stupid me didn't think to try ssh each time before power cycling router. I'll definitely try ssh if it happens again.
What's the best way to troubleshoot if this happens again? I was hoping there would be something in the logs but the system log just has crap from the 7th and then entries from after this mornings reboot.
Sat Mar 7 11:36:17 2020 kern.info kernel: [ 25.014205] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1-1: link becomes ready
Sat Mar 7 11:36:17 2020 daemon.info procd: - init complete -
Sat Mar 7 11:36:17 2020 daemon.notice netifd: Network device 'wlan1' link is up
Sat Mar 7 11:36:17 2020 daemon.notice netifd: Network device 'wlan1-1' link is up
Wed Mar 11 07:55:50 2020 daemon.err procd: unable to find /sbin/ujail: No such file or directory (-1)
Wed Mar 11 07:55:50 2020 daemon.info dnsmasq[1529]: exiting on receipt of SIGTERM
Wed Mar 11 07:55:50 2020 daemon.info dnsmasq[3223]: started, version 2.80 cachesize 150
Wed Mar 11 07:55:50 2020 daemon.info dnsmasq[3223]: DNS service limited to local subnets
Since the flash memory in routers is small and has finite write cycles, by default your logs are stored in-memory. So when your router crashes, or is power-cycled you lose all your previous log messages.
The march 7th log entries are during boot, and when your network connection is established the NTP client fetches the current date/time. Hence the rest of the log entries then showing the correct timestamp.
You have a few options regarding storing system logs on a permanent basis. You can flush your logs periodically to a USB drive, you can setup a remote syslog server on the network or simply send raw log data to a port on a remote machine: https://openwrt.org/docs/guide-user/base-system/log.essentials
Another very simple, dumb way would be to have an SSH session established with the logread -f command running. But I'm sure once your router hits a brick wall that connection will be the first casualty, and you probably won't get any interesting data.
Going even one step further would be to open the device up, hook up a serial connection and monitor messages that way.
Thank you for the explanation. I read the doc you linked and none of the options are good for me. 1) RAM (as-is now) which loses log entries on crash 2) local file with flash write and space limits as you mentioned or 3) remote system. I supposed I could set up remote logging to a spare Linux box but that seems extreme.
I do have a USB hard drive that is mounted at boot vi /etc/rc.local. I realize that there is timing issue here in that log messages are generated before the drive is mounted. Is there a way to switch logging from RAM to a file on the USB drive after the drive has been mounted?