Sysntpd failing out of the blue

Installing and Using OpenWrt
1

My collectd server started throwing a flood of errors that the sender's time was wrong:

Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/load/load; value time = 1731683507.040; last cache update = 1731683530.516;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/df-mnt-sdc2/df_complex-used; value time = 1731683507.040; last cache update = 1731683530.515;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/df-mnt-sdc2/df_complex-reserved; value time = 1731683507.040; last cache update = 1731683530.515;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/memory/memory-used; value time = 1731683507.040; last cache update = 1731683530.516;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/memory/memory-buffered; value time = 1731683507.040; last cache update = 1731683530.516;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/memory/memory-cached; value time = 1731683507.040; last cache update = 1731683530.516;
Fri Nov 15 10:13:21 2024 daemon.err collectd[13844]: uc_update: Value too old: name = PrePro/memory/memory-free; value time = 1731683507.040; last cache update = 1731683530.516;

On the senders syslog, I see this:

Fri Nov 15 10:21:07 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:12 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:17 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:22 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:27 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:32 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:21:32 2024 daemon.info procd: Instance sysntpd::instance1 s in a crash loop 6 crashes, 0 seconds since last crash

Any time I try to sync time via Luci > System > System > Sync with NTP-Server, I see another log entry for the jail error. I can sync with browser, but that doesn't keep time, over time.

I'm not sure what I need to do to further troubleshoot. Full reinstall of the system isn't ideal, and I have tried a opkg install procd-ujail --force-reinstall which has not solved the jail dependency errors.

I do keep a similar system running the same 23.05.5, and all the packages are up-to-date on them both, so I'm not sure where one would start failing and the other would not.

2

I'm wondering if your issue is that the sysntpd service failed to start on the last reboot (appears to happen when network access is not ready when the first try occurs).

Maybe try restarting the ntpd service manually, then checking if time stays up-to-date? If that works, there are some threads on similar problems, I think they put a sleep nn ... restart into rc.local or something on that order.

$ service sysntpd restart
3

Start with sysupgrade to same version, what is your device as in

ubus call system board
4

The service shows running:

root@PrePro:~# service sysntpd status

running

But on a restart of the service:

root@PrePro:~# service sysntpd restart
root@PrePro:~# logread
...
Fri Nov 15 11:59:17 2024 user.err : jail: failed to load dependencies
Fri Nov 15 11:59:22 2024 user.err : jail: failed to load dependencies

I thought, hmm, it's losing time, maybe the battery is dead, but that shouldn't prevent the service from running.

5

sysupgrade is gonna require an image that has all the packages rolled into it, though, isn't it?

system that isn't displaying the issue:


root@Nemo-Gateway:~# ubus call system board
{
	"kernel": "5.15.167",
	"hostname": "Nemo-Gateway",
	"system": "AMD GX-412TC SOC",
	"model": "PC Engines apu2",
	"board_name": "pc-engines-apu2",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.5",
		"revision": "r24106-10cc5fcd00",
		"target": "x86/64",
		"description": "OpenWrt 23.05.5 r24106-10cc5fcd00"
	}
}

system that's failing:

root@PrePro:~# ubus call system board
{
	"kernel": "5.15.167",
	"hostname": "PrePro",
	"system": "AMD GX-412TC SOC",
	"model": "PC Engines apu2",
	"board_name": "pc-engines-apu2",
	"rootfs_type": "ext4",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.5",
		"revision": "r24106-10cc5fcd00",
		"target": "x86/64",
		"description": "OpenWrt 23.05.5 r24106-10cc5fcd00"
	}
}
6

(Good to see a fellow apu2 user. :+1:t3:)

It would, but auc will solve that for you: builds a new image with all your packages, installs that and restores the config.

There haven't been any changes to 23.05.5 that would affect this, but maybe some permission got mangled on the busted router, so yeah, try re-install?

opkg update && opkg install auc
auc
1 Like
7

Ok, maybe try this first, see if it runs properly:

/usr/sbin/ntpd -dd -n $(printf "-p %s " $(uci get system.ntp.server)) -q

Should spit out all the time source info and set the clock.

8

I spoke too soon; the primary is also having the same issue. It is the NTP server for my network, whereas everything else is pointing at it to find their time.

root@Nemo-Gateway:/etc/init.d# logread | grep 'failed to load dependencies'

Fri Nov 15 09:50:00 2024 user.err : jail: failed to load dependencies
Fri Nov 15 09:50:05 2024 user.err : jail: failed to load dependencies
Fri Nov 15 09:50:10 2024 user.err : jail: failed to load dependencies
Fri Nov 15 09:50:15 2024 user.err : jail: failed to load dependencies
Fri Nov 15 09:50:20 2024 user.err : jail: failed to load dependencies
Fri Nov 15 09:50:25 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:17 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:22 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:27 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:32 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:37 2024 user.err : jail: failed to load dependencies
Fri Nov 15 10:02:42 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:17:54 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:17:59 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:18:04 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:18:09 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:18:14 2024 user.err : jail: failed to load dependencies
Fri Nov 15 12:18:19 2024 user.err : jail: failed to load dependencies

As for ntpd output:

root@Nemo-Gateway:/etc/init.d# /usr/sbin/ntpd -dd -n $(printf "-p %s " $(uci get system.ntp.server)) -q
-ash: /usr/sbin/ntpd: not found
root@Nemo-Gateway:/etc/init.d# which ntpd
/sbin/ntpd
root@Nemo-Gateway:/etc/init.d# /sbin/ntpd -dd -n $(printf "-p %s " $(uci get system.ntp.server)) -q
ntpd error:  only one pidfile option allowed
ntpd - NTP daemon program - Ver. 4.2.8p17
Usage:  ntpd [ -<flag> [<val>] | --<name>[{=| }<val>] ]... \
		[ <server1> ... <serverN> ]
Try 'ntpd --help' for more information.
9

auc fails on an out-of-space error. I run the x86 ext4 image, on a 30gb ssd, whereas the default is something like <200mb for the root filesystem. I think I ran into this issue when I first created the image; I ended up having to remove a bunch of packages from the firmware builder list, then install, expand the fs, and then install the rest of the packages.

GUI auc:

Error building the firmware image
Server response: Error: Selected packages exceed device storage

Please report the error message and request

Abbreviated output from the CLI run (too long for a reply in this forum):

root@PrePro:~# auc
auc/0.3.2-1
Server:    https://sysupgrade.openwrt.org
Running:   23.05.5 r24106-10cc5fcd00 on x86/64 (generic)
Available: 23.05.5 r24106-10cc5fcd00
Requesting package lists...
Nothing to be updated. Use '-f' to force.
root@PrePro:~# auc -f
auc/0.3.2-1
Server:    https://sysupgrade.openwrt.org
Running:   23.05.5 r24106-10cc5fcd00 on x86/64 (generic)
Available: 23.05.5 r24106-10cc5fcd00
Requesting package lists...
Are you sure you want to continue the upgrade process? [N/y] y
Requesting build (position in queue: 0)...................................................Checking 'true'... ok.
Checking 'false'... ok.
Checking 'working-make'... ok.
Checking 'case-sensitive-fs'... ok.
Checking 'proper-umask'... ok.
Checking 'perl-data-dumper'... ok.
Checking 'perl-findbin'... ok.
Checking 'perl-file-copy'... ok.
Checking 'perl-file-compare'... ok.
Checking 'perl-thread-queue'... ok.
Checking 'tar'... ok.
Checking 'find'... ok.
Checking 'bash'... ok.
Checking 'xargs'... ok.
Checking 'patch'... ok.
Checking 'diff'... ok.
Checking 'cp'... ok.
Checking 'seq'... ok.
Checking 'awk'... ok.
Checking 'grep'... ok.
Checking 'egrep'... ok.
Checking 'getopt'... ok.
Checking 'realpath'... ok.
Checking 'stat'... ok.
Checking 'gzip'... ok.
Checking 'unzip'... ok.
Checking 'bzip2'... ok.
Checking 'wget'... ok.
Checking 'install'... ok.
Checking 'perl'... ok.
Checking 'python'... ok.
Checking 'python3'... ok.
Checking 'python3-distutils'... ok.
Checking 'python3-stdlib'... ok.
Checking 'file'... ok.
Checking 'which'... ok.
Checking 'ldconfig-stub'... ok.
Building images for x86 - Generic x86/64
Packages: luci-app-mosquitto luci-proto-wireguard openssh-sftp-server kmod-usb-net-cdc-subset smcroute luci-app-statistics luci-lib-base luci-app-nlbwmon mkf2fs banip opkg luci-app-opkg luci-app-vnstat2 kmod-ixgbe diffutils busybox luci-lib-ip nano-full kmod-usb-net-cdc-eem luci-mod-system openssl-util telnet-bsd resize2fs kmod-crypto-hw-ccp dropbearconvert luci-theme-bootstrap luci-lua-runtime collectd-mod-sensors procd-ujail base-files mosquitto-client-ssl partx-utils libustream-mbedtls netifd luci-app-uhttpd kmod-r8169 firewall4 pciutils procd usbutils luci-proto-bonding qrencode vnstat2 luci-app-banip screen luci-mod-status lsblk kmod-usb-net-rndis block-mount kmod-fs-vfat luci-app-ddns kmod-usb2 ntpd luci-app-firewall grub2-efi ss collectd-mod-wireless coreutils-who grub2-bios-setup tcpdump ethtool-full odhcp6c fstools uclient-fetch uci luci-ssl nmap losetup dropbear logrotate luci-compat mtd odhcpd-ipv6only procd-seccomp luci-app-sqm parted urandom-seed ipset lscpu usbmuxd luci-proto-ppp luci-mod-admin-full ppp collectd-mod-write-graphite cfdisk luci-base exfat-mkfs atftp logd kmod-tg3 ddns-scripts-cloudflare rsync openssh-keygen kmod-usb-net-cdc-ncm luci-proto-ipv6 kmod-pcengines-apuv2 luci-app-attendedsysupgrade iputils-ping tmux irqbalance nftables-json e2fsprogs dmidecode kmod-e1000 rng-tools kmod-usb-net-ipheth kmod-nft-offload luci-app-wol luci-app-samba4 collectd-mod-snmp python3-speedtest-cli auc bmon htop atftpd luci-lib-nixio ca-bundle kmod-forcedeth fping luci-lib-jsonc collectd-mod-nut kmod-amd-xgbe bind-dig luci fdisk luci-light libgcc vnstati2 luci-app-pbr kmod-amazon-ena kmod-e1000e blkid ntpdate luci-mod-network dtc kmod-button-hotplug lldpd mount-utils iperf3 kmod-igb kmod-igc collectd-mod-thermal kmod-bnx2 gdisk collectd-mod-chrony urngd iftop rpcd-mod-rrdns ppp-mod-pppoe collectd-mod-df dnsmasq-full base-files busybox ca-bundle dropbear e2fsprogs firewall4 fstools grub2-bios-setup kernel kmod-amazon-ena kmod-amd-xgbe kmod-bnx2 kmod-button-hotplug kmod-e1000 kmod-e1000e kmod-forcedeth kmod-fs-vfat kmod-igb kmod-igc kmod-ixgbe kmod-nft-offload kmod-r8169 kmod-tg3 libgcc libustream-mbedtls logd mkf2fs mtd netifd odhcp6c odhcpd-ipv6only opkg partx-utils ppp ppp-mod-pppoe procd procd-seccomp procd-ujail uci uclient-fetch urandom-seed urngd


Installing packages...
Installing libc (1.2.4-4) to root...
Installing libgcc1 (12.3.0-4) to root...
Copying /builder/dl/libgcc1_12.3.0-4_x86_64.ipk.
Configuring libgcc1.
Configuring libc.
Installing kernel (5.15.167-1-59d1431675acc6823a33c7eb2323daeb) to root...
Configuring kernel.
Installing luci-app-mosquitto (git-24.313.38121-76e4eca) to root...
...
Package procd (2023-06-25-2db83655-2) installed in root is up to date.
Package procd-seccomp (2023-06-25-2db83655-2) installed in root is up to date.
Package procd-ujail (2023-06-25-2db83655-2) installed in root is up to date.
Package uci (2023-08-10-5781664d-1) installed in root is up to date.
Package uclient-fetch (2023-04-13-007d9454-1) installed in root is up to date.
Package urandom-seed (3) installed in root is up to date.
Package urngd (2023-11-01-44365eb1-1) installed in root is up to date.
Configuring libubox20230523.
...
Configuring iftop.
Configuring ppp-mod-pppoe.
Configuring collectd-mod-df.
Configuring dnsmasq-full.

Finalizing root filesystem...
Enabling atftpd
...
Enabling ntpd
Enabling ntpdate
...

Building images...
Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on /builder/build_dir/target-x86_64_musl/linux-x86_64/root.squashfs, block size 262144.
[=============================================================\] 3785/3785 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
	compressed data, compressed metadata, compressed fragments,
	no xattrs, compressed ids
	duplicates are removed
Filesystem size 33147.62 Kbytes (32.37 Mbytes)
	28.14% of uncompressed filesystem size (117800.02 Kbytes)
Inode table size 29988 bytes (29.29 Kbytes)
	21.16% of uncompressed inode table size (141709 bytes)
Directory table size 40428 bytes (39.48 Kbytes)
	41.69% of uncompressed directory table size (96981 bytes)
Number of duplicate files found 438
Number of inodes 4264
Number of files 3570
Number of fragments 217
Number of symbolic links 448
Number of device nodes 1
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 245
Number of hard-links 0
Number of ids (unique uids + gids) 1
Number of uids 1
	unknown (0)
Number of gids 1
	unknown (0)
Creating filesystem with parameters:
    Size: 109051904
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 6656
    Inode size: 256
    Journal blocks: 1024
    Label: rootfs
    Blocks: 26624
    Block groups: 1
    Reserved blocks: 0
    Reserved block group size: 7
Package list missing or not up-to-date, generating it.

Building package index...
Downloading https://downloads.openwrt.org/releases/23.05.5/targets/x86/64/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_core
Downloading https://downloads.openwrt.org/releases/23.05.5/targets/x86/64/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/base/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_base
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/luci/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_luci
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_packages
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/routing/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_routing
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/telephony/Packages.gz
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/openwrt_telephony
Downloading https://downloads.openwrt.org/releases/23.05.5/packages/x86_64/telephony/Packages.sig
Signature check passed.
Downloading file:packages/Packages
Updated list of available packages in /builder/build_dir/target-x86_64_musl/root-x86/../../../../builder/dl/imagebuilder
Downloading file:packages/Packages.sig
Signature check passed.
/builder/build_dir/target-x86_64_musl/root-x86/etc/init.d/ntpdate: line 7: uci: command not found
./etc/init.d/ntpdate: line 7: uci: command not found
Pseudo file "dev" exists in source filesystem "/builder/build_dir/target-x86_64_musl/root-x86/dev".
Ignoring, exclude it (-e/-ef) to override.
error: ext4_allocate_best_fit_partial: failed to allocate 39 blocks, out of space?
make[3]: *** [/builder/include/image.mk:350: /builder/build_dir/target-x86_64_musl/linux-x86_64/root.ext4] Error 1
make[2]: *** [Makefile:208: build_image] Error 2
make[1]: *** [Makefile:146: _call_image] Error 2
make: *** [Makefile:262: image] Error 2
Error: Selected packages exceed device storage
Bad message (74)

also:

# opkg list-installed | grep ntp
ntp-utils - 4.2.8p17-1
ntpd - 4.2.8p17-1
ntpdate - 4.2.8p17-1

am I failing myself having the ntpd packages in addition to the default ntp server/client sysntpd?

10

Does the default image have working sysntpd on what is more or less like normal PC?

11

Ah, so you've installed the ntpd package and are not using the default busybox applet... That might be the cause of the jail issues, not sure how the standard service deals with a non-standard ntpd.

Yes, that would be my guess.

Ouch, those are huge, so not surprised the imagebuilder is dying. My solution in 23.05 would be to use a local imagebuilder, with make image ROOTFS_PARTSIZE="1024" ... so rootfs is way bigger than ever needed. (In 24.10, you'll have owut replacing auc and it lets you configure rootfs size from the command line.)

1 Like
12

ntp servers race, first come first served.

13

Yeah, this is what I'm thinking is happening. I had to jump some hoops, but I got a clean, no-extra-packages image loaded on the test system (no ntpd, ntp-utils, ntpdate). I see no errors in the syslog following a service reload, or syncing to server in the browser. On my production router, with those packages loaded, I see the jail errors. On both devices, running netstat -peanut, both show port 123 as belonging to ntpd, even though the explicit package isn't installed on the clean image.

On the production router, stopping and disabling the ntpd service was not enough. I had to uninstall the packages (and restart system and sysntpd services for good measure). I think things are good now.

1 Like
14

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.