Kernel timestamps are 6 days behind on reboot

I cannot figure out how to synchronize timestamps messages from kernel on the router reboot. The following log shows that before reboot the last timestamp was 2022-10-17T14:19. The first log message after reboot is around 6 days earlier: 2022-10-11T18:41, and after boot sequence completes the timestamps come back to normal.
Is there a secrete to make timestamps of log messages synchronized to the current time?

2022-10-17T14:19:37+09:00 myHostName dnsmasq-dhcp[1]: DHCPACK(br-lan) 192
2022-10-11T18:41:01+09:00 myHostName syslog-ng[1520]: syslog-ng starting
2022-10-11T18:41:01+09:00 myHostName kernel: [ 0.000000] Booting Linux
...
2022-10-11T18:41:03+09:00 myHostName dnsmasq-dhcp[1]: DHCPACK(br-lan)
2022-10-17T15:23:00+09:00 myHostName avahi-daemon[1708]: Leaving mDNS

Your device does not have a real-time clock and cannot maintain time across a reboot. What OpenWrt will do is find the most recently modified file, and use the date and time from that file early in the boot cycle.

Then, when the network stack is alive in the boot sequence, the ntp (network time protocol) client will launch and it will synchronize with time servers in the internet.

The result is that you will see inaccurate time stamps as the beginning of the sequence, and then it will be corrected as booting progresses and finishes.

7 Likes

Can I create a random file, e.g. /random-file.txt before reboot to get the timestamps closer?

Sure. touch /etc/banner ; reboot will do it.

2 Likes

Do you think /etc/init.d/sysfixtime should accommodate hardware without clock?
The stop() function can touch /etc/banner if hardware clock is missing.

What you asked would be impossible without Network Time. That command specifically queries a hardware clock. See:

Since the Kernel already is configured to get a time, I'm not sure what you believe such a code modification would change/fix in your use case or others?

I'm not sure how that's relevant, but yes, touch will use the Kernel time (whatsoever it is). It was explained how the Kernel chooses the time before network up with no RTC.

1 Like

As @lleachii re-iterated. Without a Real Time Clock you want to set current time on a file in /etc/ immediately before boot given your situation.

1 Like

The stop() function should be run when a service gets stopped (hopefully also in a managed shutdown). It could be modified to touch a file in /etc if there is no RTC, and that could help storing the current time.

(similarly as start() and boot() get used for the service start)

But that stop() would naturally not help with crashes etc.
I am not even sure, how well it would behave in a sysupgrade.

1 Like

Now that's quite relevant!

Cool, that makes sense!

Yeah, STOP priorities are defined in some init scripts (like 90 in sysfixtime) for defining an orderly process termination, but in general OpenWrt pretty much assumes that the router runs forever and gets rebooted only in a crash. So the "graceful shutdown" is pretty rare for apps.

And I think that e.g. sysupgrade pretty rudely stops/kills all services with no regard to the priority defined in the init script.

Ps.
Personally I used to manually touch /etc/urandom.seed before sysupgrade, but lately I have been saving LuCI stats into flash before a sysupgrade and that creates a suitable timestamp into /etc.

For the OP... is there a reason that the behavior I described is problematic? I know it is not ideal, but does it actually cause issues for your situation? If you're just looking at logs (stored on external media or a syslog server), you can typically figure out when the system went down simply by looking at the date and time prior to the 'jump backwards' event (or by looking at the lines prior to the boot sequence). If you need to know how long a system was down, you can establish that by evaluating the total duration of the boot sequence prior to time sync, and then looking at the difference as follows:

ntp_synced_time - ( previously_valid_timestamp + pre_ntp_boot_sqeuence_duration )

(and for practical purposes, the pre_ntp_boot_sequence_duration is likely to be in the order of <1 minute, so it can be omitted unless you need that level of precision.)

Exactly. As well as abends, power outage, power switch pressed, inadvertently pulling the power plug, . . . isn't going to provide a current timestamp on restore until ntpd fires.

But, the OP will be scratching his head wondering why the touch command in the sysfixtime STOP function didn't work when he sees the timestamps out of sync again in such situations.

@ psherman The OP for all intents and purposes is initiating the reboot purposely.

Looking at my latest time stamp in /etc, I am looking at sysctl.conf from 4 days ago which is exactly the issue he is facing. I think touch /etc/<filename> ; reboot is the better solution than modding sysfixtime STOP function when initiating a reboot.

Fair enough. So then the calculation of the down time may be unnecessary. But I'm still not sure what actual problem this creates except for early boot-cycle logs.

You are right, nothing more than a confusion in timestamps. But it is really confusing if you have persistent /var/log/messages that spans e.g. a week, and in the middle of this file you have "unexplained" timestamps inconsistencies. Of course it is not difficult to calculate a boot time if the correct time is somewhere in the log, but what a pain it is to scroll a couple of pages and do all the math in your head, especially if one is in a rush.

I will just create a cron job that touches /etc/timekeeper file every minute. It will be my solution to cover both unexpected and planned reboots.

This could be a really bad idea. You will likely wear out your flash memory... obviously it depends on the write cycle ratings for the specific memory chips in your device, but some devices are rated for low thousands to tens of thousands of writes... in a year, you would write 525600 times, which may degrade the storage and cause your router to become e-waste.

1 Like

I will have it only for ExtRoot on a USB flash drive. It is really cheap, I can replace it easily if it goes bad.

2 Likes