Issues reaching NTP service on openwrt, but ONLY when bound to an interface?

[aleks-mariusz]

I am trying to setup the NTP server on OpenWRT (the one you can turn on by going to System->System->Time Syncronization->Provide NTP Server - checked ON) on a dedicated network interface (the reasons are at the end of the post, but tl;dr, it's so i can set up forwarding for ANY of my other zones to access/redirect NTP (and any other services).. hence these shared-services zone name is named "svc")

However my clients' on another vlans cannot reach it only when I set Bind NTP server to a specific IP in another VLAN:

16:46:09.774040 d0:xx:yy:zz:e1:b8 > 60:xx:yy:zz:bd:10, ethertype 802.1Q (0x8100), length 94: vlan 2220, p 0, ethertype IPv4 (0x0800), 10.22.20.10.47342 > 10.21.23.123.123: NTPv4, Client, length 48
16:46:09.774090 60:xx:yy:zz:bd:10 > d0:xx:yy:zz:e1:b8, ethertype 802.1Q (0x8100), length 122: vlan 2220, p 0, ethertype IPv4 (0x0800), 10.21.23.123 > 10.22.20.10: ICMP 10.21.23.123 udp port 123 unreachable, length 84

What is strange is that they can however reach that same IP if i don't specify an interface to bind to (so the same destination IP now suddenly works):

17:14:57.613789 d0:xx:yy:zz:e1:b8 > 60:xx:yy:zz:bd:10, ethertype 802.1Q (0x8100), length 94: vlan 2220, p 0, ethertype IPv4 (0x0800), 10.22.20.10.50233 > 10.21.23.123.123: NTPv4, Client, length 48
17:14:57.613863 60:xx:yy:zz:bd:10 > d0:xx:yy:zz:e1:b8, ethertype 802.1Q (0x8100), length 94: vlan 2220, p 0, ethertype IPv4 (0x0800), 10.21.23.123.123 > 10.22.20.10.50233: NTPv4, Server, length 48

Things i've looked at:

- making sure I can ping the service-IP from the client-host

16:39:36.146395 d0:xx:yy:zz:e1:b8 > 60:xx:yy:zz:bd:10, ethertype 802.1Q (0x8100), length 102: vlan 2220, p 0, ethertype IPv4 (0x0800), 172.22.20.10 > 172.21.23.123: ICMP echo request, id 25180, seq 1, length 64
16:39:36.146436 60:xx:yy:zz:bd:10 > d0:xx:yy:zz:e1:b8, ethertype 802.1Q (0x8100), length 102: vlan 2220, p 0, ethertype IPv4 (0x0800), 172.21.23.123 > 172.22.20.10: ICMP echo reply, id 25180, seq 1, length 64

- making sure the service is up running & bound correctly using netstat

# netstat -nlup | grep -E '(Active|123)'
Active Internet connections (only servers)
udp        0      0 :::123                  :::*                                21799/ntpd

(curiously, the netstat output is the same whether i bound it on an interface or not)

- verified in /etc/config/firewall that forwarding is enabled between my lan and the svc zone

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'DROP'
	option drop_invalid '1'
	option flow_offloading '1'
	option synflood_protect '1'

config zone
	option name 'svc'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'DROP'
	list network 'svc'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config forwarding
	option src 'lan'
	option dest 'wan'

config forwarding
	option src 'lan'
	option dest 'svc'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option family 'ipv4'
	option target 'ACCEPT'
	list icmp_type 'echo-request'
	option ipset 'allowed_to_ping'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src '*'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src '*'
	option proto 'icmp'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	list icmp_type 'bad-header'
	list icmp_type 'destination-unreachable'
	list icmp_type 'echo-reply'
	list icmp_type 'echo-request'
	list icmp_type 'neighbour-advertisement'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'packet-too-big'
	list icmp_type 'router-advertisement'
	list icmp_type 'router-solicitation'
	list icmp_type 'time-exceeded'
	list icmp_type 'unknown-header-type'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Block-DHCP-DNS-on-WAN'
	option src 'wan'
	option dest_port '53 67 68'
	option target 'DROP'

config rule
	option name 'Allow-DHCP-DNS-ICMP'
	option src '*'
	option dest_port '53 67 68'
	option target 'ACCEPT'
	list proto 'tcp'
	list proto 'udp'
	list proto 'icmp'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec NAT-Traversal'
	option src 'wan'
	option dest_port '4500'
	option proto 'udp'
	option target 'ACCEPT'

config zone
	option name 'iot'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'DROP'
	list network 'iot'

config forwarding
	option src 'iot'
	option dest 'wan'

config forwarding
	option src 'iot'
	option dest 'svc'

Some more backstory on this particular config - I have an IOT access-point on another VLAN, and i initially wanted to redirect outgoing NTP traffic to this NTP service instead, but i couldn't get that to work and ended up getting the exact same udp port 123 unreachable so in order to "simplify" the config, I removed that rule and thought i should get this working first but seems i ran into this issue now which i'm just baffled by

[psherman]

What is the source zone for the NTP requests? Is the input rule set to drop or reject? That would cause the problem you're describing.

Also, please post your configs here rather than on 3rd party sites...

Please use the "Preformatted text </>" button for logs, scripts, configs and general console output.

Please edit your post accordingly. Thank you!

[aleks-mariusz]

Done, i've updated with direct content instead of the pastebin site...

Also I've verified everything is set to ACCEPT (both the defaults as well as the svc zone.. i'm grasping at straws here)

But I don't see why this with the exact same configs works/breaks if i don't have

config timeserver 'ntp'
        list server '0.openwrt.pool.ntp.org'
        list server '1.openwrt.pool.ntp.org'
        list server '2.openwrt.pool.ntp.org'
        list server '3.openwrt.pool.ntp.org'
        option enable_server '1'
        option use_dhcp '0'
        option interface 'svc'

that last line in there or not.

[aleks-mariusz]

@aleks-mariusz

making sure the service is up running & bound correctly using netstat

So after a bit of digging, what I suspect now is happening is that when you specify an interface, for some reason ntpd is actually only binding on the IPv6 socket for udp-port 123, and not on the IPv4 socket as i had assumed it would..

i confirmed this using lsof:

# lsof -np 10925
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
ntpd    10925  ntp  cwd    DIR    0,25      160    1 /
ntpd    10925  ntp  rtd    DIR    0,25      160    1 /
ntpd    10925  ntp  txt    REG    0,21   393237    5 /usr/sbin/ntpd
ntpd    10925  ntp  mem    REG   254,0             5 /usr/sbin/ntpd (path dev=0,21)
ntpd    10925  ntp  mem    REG   254,0           781 /lib/libgcc_s.so.1 (path dev=0,21)
ntpd    10925  ntp  mem    REG   254,0           779 /lib/ld-musl-armhf.so.1 (path dev=0,21)
ntpd    10925  ntp    0r   CHR     1,3      0t0   38 /dev/null
ntpd    10925  ntp    1w   CHR     1,3      0t0   38 /dev/null
ntpd    10925  ntp    2w   CHR     1,3      0t0   38 /dev/null
ntpd    10925  ntp    3u  IPv6 2074064      0t0  UDP *:ntp

..though the process has the -I ifacename flag:

10925 ntp       1000 S    /usr/sbin/ntpd -n -N -l -I br-lan.2123 -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org

...and i have both an IPv4 and IPv6 address on the br-lan.2123 interface :-/

Any clues as to why the selective inet/addr family only chosing IPv6?

[aleks-mariusz]

@aleks-mariusz

Any clues as to why the selective inet/addr family only chosing IPv6?

Actually that seems was not the problem/cause, per this thread - also confirmed since when i do listen on a specific interface, i can't even reach it over IPv6 either :-/

[mk24]

Bind to an IP is an archaic process that is not very useful for fine-grained security. For that, use the firewall.

[lleachii]

NTP, I'm doing something wrong

First, NTP is UDP, not TCP. This explains why some of your tests ended up in no responses. Next: I have the same behavior, I haven't noticed a change since version 19 or 20. On a client: ntpdate -q 192.168.1.1 This works for me, which made me wonder why. The answer is IPv4-mapped IPv6 addresses. See: https://lwn.net/Articles/688462/ https://unix.stackexchange.com/questions/152612/netstat-why-are-ipv4-daemons-listening-to-ports-listed-only-in-a-inet6 root@OpenWrt:~# sysctl net.ipv6.bin…

@aleks-mariusz

also confirmed since when i do listen on a specific interface, i can't even reach it over IPv6 either :-/

How did you actually bind?

Perhaps I missed it in the thread.

[aleks-mariusz]

@lleachii

How did you actually bind?

this goes into /etc/config/system under the config timeserver 'ntp' block:

@aleks-mariusz

        option interface 'svc'

[NPeca75]

Hi @aleks-mariusz

i have a fleet of DumpAPs without firewall, only bare minimum to fit in small flash
so i went to similar route as you, binding snmp/telnet/ssh/ntp etc ... to specific interfaces
after many try&error with rc.d and startup order, ended up writing hotplug script which will fire up when VlanX mgmn interface is going up
that is only way to be sure that v4/v6 address is available when try to bind service to specific IP

cat /etc/hotplug.d/iface/01-services 

[ "$ACTION" = "ifup" ] && [ "$INTERFACE" = "vlanX" ] && {
    sleep 05 && your_cmdline_here 2>&1 &

[aleks-mariusz]

thanks for that idea @NPeca75 - it looks like an interesting approach, however i don't think this will affect/fix that NTPd seems to be not receiving traffic properly (or not able to respond properly).

i've tried to use nft tracing of the packet, but it honestly looks identical, whether the ntpd service is either told to listen on the other vlans' interface only or allowed to listen on all interfaces (this is the only change).

i've also tried adding setting all fw policies to 'accept' (both for input, as well as forward) on both the originating vlan's fw zone as well as the vlan service's fw zone (where i'm trying to get it to listen on), as well as adding the forwarding to/from the source/svc zones as well..

so strange, when not listening on a specific interface (in another vlan) - i see from tcpdump (listening on the 'any' pseudo-interface, so packets appear multiple times):

20:09:37.491337 lan1  In  ifindex 6 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 96: 172.22.20.10.55922 > 172.21.23.123.123: NTPv4, Client, length 48
20:09:37.491337 br-lan In  ifindex 12 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 96: 172.22.20.10.55922 > 172.21.23.123.123: NTPv4, Client, length 48
20:09:37.491337 br-lan.2220 In  ifindex 14 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 96: 172.22.20.10.55922 > 172.21.23.123.123: NTPv4, Client, length 48
20:09:37.491423 br-lan.2220 Out ifindex 14 60:xx:yy:zz:bd:10 ethertype IPv4 (0x0800), length 96: 172.21.23.123.123 > 172.22.20.10.55922: NTPv4, Server, length 48
20:09:37.491427 br-lan Out ifindex 12 60:xx:yy:zz:bd:10 ethertype IPv4 (0x0800), length 96: 172.21.23.123.123 > 172.22.20.10.55922: NTPv4, Server, length 48

But as soon as i set it to listen on the vlan 2123's interface that has 172.21.23.123, the response never appears as if the packet was null-routed or something:

20:05:46.685195 lan1  In  ifindex 6 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 62: 172.22.20.10.59979 > 172.21.23.123.123: NTPv0, unspecified, length 1
20:05:46.685195 br-lan In  ifindex 12 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 62: 172.22.20.10.59979 > 172.21.23.123.123: NTPv0, unspecified, length 1
20:05:46.685195 br-lan.2220 In  ifindex 14 d0:xx:yy:zz:e1:b8 ethertype IPv4 (0x0800), length 62: 172.22.20.10.59979 > 172.21.23.123.123: NTPv0, unspecified, length 1

[NPeca75]

Hi

i was curious as i never used DumbAP as NTP server, but lets try

tr '\x00' '\x20' < cmdline 
/sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/jshn.sh -- /usr/sbin/ntpd -n -N -l -I switch.255 -S /usr/sbin/ntpd-hotplug -p fdff:255::30 -p fd00:2:255::1 

so, clearly it have capital "I" letter and IF is "switch.255" which is vlan255 with 169.254.2.250 address

and to test it from buntu on vlan255

sudo ntpdate 169.254.2.250
 9 Nov 21:40:00 ntpdate[1918746]: adjust time server 169.254.2.250 offset +0.000681 sec

so it is working

tested on

BusyBox v1.36.1 (2023-10-09 21:45:35 UTC) multi-call binary.
DISTRIB_DESCRIPTION='OpenWrt 23.05.0 r23497-6637af95aa'

edit: i have no any firewall on DumpAp ...

[vgaetera]

Binding services to interfaces is tricky since the result depends on a specific service implementation as it may just ignore requests coming from a different interface.

[NPeca75]

My experience is very positive

Dropbear: OK
Mactelnet: OK
Telnet: OK
SNMPD: OK
LLDPD: OK
an now NTPD: OK
they are all bound to specific IF and/or specific v4/v6 address

[vgaetera]

My experience with interface binding is not limited to OpenWrt and is generally rather negative as it can lead to race conditions resulting in deadlocks or crashes at startup and failures to rebind when restarting the interface the service is bound to, i.e. this is a potential source of transient and difficult to troubleshoot issues I sincerely recommend to avoid.

[NPeca75]

@vgaetera

failures to rebind when restarting the interface

yes, that is why i using hotplug scripts on OWRT to catch UP/DOWN events and start/stop services

but yes, generally, you are right