Https-dns-proxy Crash Loop

Hi all,

I've just set up a new Belkin RT3200, installed 23.05.0, and everything seems to be working fine except https-dns-proxy, which keeps crashing. I can't quite figure out why, and various things I've tried have not worked (changing the upstream resolver, removing the IPv6 listen address, changing the listen ports, removing the Run As User and Run As Group, and some other things).

Here's what syslog shows:

Fri Oct 20 09:14:03 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 965 crashes, 11 seconds since last crash
Fri Oct 20 09:14:03 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 965 crashes, 11 seconds since last crash
Fri Oct 20 09:14:03 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓
Fri Oct 20 09:14:07 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 966 crashes, 4 seconds since last crash
Fri Oct 20 09:14:07 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 966 crashes, 4 seconds since last crash
Fri Oct 20 09:14:07 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓
Fri Oct 20 09:15:05 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 967 crashes, 58 seconds since last crash
Fri Oct 20 09:15:05 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 967 crashes, 58 seconds since last crash
Fri Oct 20 09:15:05 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓
Fri Oct 20 09:15:09 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 968 crashes, 3 seconds since last crash
Fri Oct 20 09:15:09 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 968 crashes, 3 seconds since last crash
Fri Oct 20 09:15:09 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓

The output from each troubleshooting command at https://openwrt.org/docs/guide-user/services/dns/doh_dnsmasq_https-dns-proxy#troubleshooting is here:

service log restart; service dnsmasq restart; service https-dns-proxy restart

udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: no lease, failing
Starting https-dns-proxy 2023-05-25-7 instances ✓✓

logread -e dnsmasq; netstat -l -n -p | grep -e dnsmasq

Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: started, version 2.89 cachesize 150
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-nftset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.5.100 -- 192.168.5.249, lease time 12h
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using nameserver ::1#5054
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using nameserver 127.0.0.1#5054
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: read /etc/hosts - 12 names
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 9 names
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 28 names
Fri Oct 20 09:19:54 2023 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
tcp        0      0 69.253.[censored].[censored]:53       0.0.0.0:*               LISTEN      17898/dnsmasq
tcp        0      0 192.168.5.1:53          0.0.0.0:*               LISTEN      17898/dnsmasq
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      17898/dnsmasq
tcp        0      0 10.14.0.1:53            0.0.0.0:*               LISTEN      17898/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599b:53 :::*                    LISTEN      17898/dnsmasq
tcp        0      0 2601:41:[censored]:[censored]::1:53  :::*                    LISTEN      17898/dnsmasq
tcp        0      0 ::1:53                  :::*                    LISTEN      17898/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599e:53 :::*                    LISTEN      17898/dnsmasq
tcp        0      0 fd9f:5b3c:8f49::1:53    :::*                    LISTEN      17898/dnsmasq
tcp        0      0 2001:558:6027:86:c11b:d34c:[censored]:[censored]:53 :::*                    LISTEN      17898/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                    LISTEN      17898/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                    LISTEN      17898/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599d:53 :::*                    LISTEN      17898/dnsmasq
udp        0      0 127.0.0.1:53            0.0.0.0:*                           17898/dnsmasq
udp        0      0 69.253.[censored].[censored]:53       0.0.0.0:*                           17898/dnsmasq
udp        0      0 192.168.5.1:53          0.0.0.0:*                           17898/dnsmasq
udp        0      0 10.14.0.1:53            0.0.0.0:*                           17898/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           17898/dnsmasq
udp        0      0 ::1:53                  :::*                                17898/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                                17898/dnsmasq
udp        0      0 2001:558:6027:86:c11b:d34c:[censored]:[censored]:53 :::*                                17898/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599b:53 :::*                                17898/dnsmasq
udp        0      0 2601:41:[censored]:[censored]::1:53  :::*                                17898/dnsmasq
udp        0      0 fd9f:5b3c:8f49::1:53    :::*                                17898/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                                17898/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599d:53 :::*                                17898/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599e:53 :::*                                17898/dnsmasq

logread -e https-dns; netstat -l -n -p | grep -e https-dns

Fri Oct 20 09:19:54 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 969 crashes, 284 seconds since last crash
Fri Oct 20 09:19:54 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 969 crashes, 284 seconds since last crash
Fri Oct 20 09:19:54 2023 user.notice https-dns-proxy: Starting service instances ✓✓
udp        0      0 127.0.0.1:5054          0.0.0.0:*                           18010/https-dns-pro
udp        0      0 ::1:5054                :::*                                18009/https-dns-pro

pgrep -f -a dnsmasq; pgrep -f -a https-dns

Fri Oct 20 09:19:54 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 969 crashes, 284 seconds since last crash
Fri Oct 20 09:19:54 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 969 crashes, 284 seconds since last crash
Fri Oct 20 09:19:54 2023 user.notice https-dns-proxy: Starting service instances ✓✓
udp        0      0 127.0.0.1:5054          0.0.0.0:*                           18010/https-dns-pro
udp        0      0 ::1:5054                :::*                                18009/https-dns-pro
root@OpenWrt:~# pgrep -f -a dnsmasq; pgrep -f -a https-dns
17892 /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq.d -r /tmp/hosts -r /usr/bin/jshn -r /usr/lib/dnsmasq/dhcp-script.sh -r /usr/share/dnsmasq/dhcpbogushostname.conf -r /usr/share/dnsmasq/rfc6761.conf -r /usr/share/dnsmasq/trust-anchors.conf -r /usr/share/libubox/jshn.sh -r /var/etc/dnsmasq.conf.cfg01411c -w /var/run/dnsmasq/ -- /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
17898 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
18580 /usr/sbin/https-dns-proxy -r https://cloudflare-dns.com/dns-query -a ::1 -p 5054 -b 1.1.1.1,1.0.0.1 -4 -u nobody -g nogroup
18581 /usr/sbin/https-dns-proxy -r https://cloudflare-dns.com/dns-query -a 127.0.0.1 -p 5054 -b 1.1.1.1,1.0.0.1 -4 -u nobody -g nogroup

head -v -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv./

==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf.d <==
head: /tmp/resolv.conf.d: I/O error

==> /tmp/resolv.conf.d/resolv.conf.auto <==
# Interface wan
nameserver 75.75.75.75
nameserver 75.75.76.76
search hsd1.pa.comcast.net.
# Interface wan6
nameserver 2001:558:feed::1
nameserver 2001:558:feed::2

uci show dhcp; uci show https-dns-proxy

dhcp.@dnsmasq[0]=dnsmasq
dhcp.@dnsmasq[0].domainneeded='1'
dhcp.@dnsmasq[0].localise_queries='1'
dhcp.@dnsmasq[0].rebind_protection='1'
dhcp.@dnsmasq[0].local='/lan/'
dhcp.@dnsmasq[0].domain='lan'
dhcp.@dnsmasq[0].expandhosts='1'
dhcp.@dnsmasq[0].authoritative='1'
dhcp.@dnsmasq[0].readethers='1'
dhcp.@dnsmasq[0].leasefile='/tmp/dhcp.leases'
dhcp.@dnsmasq[0].localservice='1'
dhcp.@dnsmasq[0].ednspacket_max='1232'
dhcp.@dnsmasq[0].quietdhcp='1'
dhcp.@dnsmasq[0].filterwin2k='1'
dhcp.@dnsmasq[0].server='::1#5054' '127.0.0.1#5054'
dhcp.@dnsmasq[0].doh_backup_noresolv='-1'
dhcp.@dnsmasq[0].noresolv='1'
dhcp.@dnsmasq[0].doh_backup_server='::1#5054' '127.0.0.1#5054'
dhcp.@dnsmasq[0].doh_server='::1#5054' '127.0.0.1#5054'
dhcp.lan=dhcp
dhcp.lan.interface='lan'
dhcp.lan.start='100'
dhcp.lan.limit='150'
dhcp.lan.leasetime='12h'
dhcp.lan.dhcpv4='server'
dhcp.lan.dhcpv6='server'
dhcp.lan.ra='server'
dhcp.lan.ra_flags='managed-config' 'other-config'
dhcp.wan=dhcp
dhcp.wan.interface='wan'
dhcp.wan.ignore='1'
dhcp.odhcpd=odhcpd
dhcp.odhcpd.maindhcp='0'
dhcp.odhcpd.leasefile='/tmp/hosts/odhcpd'
dhcp.odhcpd.leasetrigger='/usr/sbin/odhcpd-update'
dhcp.odhcpd.loglevel='4'
https-dns-proxy.config=main
https-dns-proxy.config.dnsmasq_config_update='*'
https-dns-proxy.config.force_dns='0'
https-dns-proxy.config.force_dns_port='53' '853'
https-dns-proxy.config.procd_trigger_wan6='0'
https-dns-proxy.@https-dns-proxy[0]=https-dns-proxy
https-dns-proxy.@https-dns-proxy[0].bootstrap_dns='1.1.1.1,1.0.0.1'
https-dns-proxy.@https-dns-proxy[0].resolver_url='https://cloudflare-dns.com/dns-query'
https-dns-proxy.@https-dns-proxy[0].listen_addr='::1'
https-dns-proxy.@https-dns-proxy[0].listen_port='5054'
https-dns-proxy.@https-dns-proxy[0].user='nobody'
https-dns-proxy.@https-dns-proxy[0].group='nogroup'
https-dns-proxy.@https-dns-proxy[1]=https-dns-proxy
https-dns-proxy.@https-dns-proxy[1].bootstrap_dns='1.1.1.1,1.0.0.1'
https-dns-proxy.@https-dns-proxy[1].resolver_url='https://cloudflare-dns.com/dns-query'
https-dns-proxy.@https-dns-proxy[1].listen_addr='127.0.0.1'
https-dns-proxy.@https-dns-proxy[1].listen_port='5054'
https-dns-proxy.@https-dns-proxy[1].user='nobody'
https-dns-proxy.@https-dns-proxy[1].group='nogroup'

Any suggestions anyone has to troubleshoot further and ultimately fix this would be greatly appreciated!

Just tried increasing the verbosity level to 4, here's that output, including before/after the crashes:

Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.797859 main.c:113 Received request for id: E179, len: 50
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.797915 https_client.c:251 E179: Requesting HTTP/2
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798647 https_client.c:205 E179: > POST /dns-query HTTP/2
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798673 https_client.c:205 E179: > Host: cloudflare-dns.com
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798719 https_client.c:205 E179: > User-Agent: https_dns_proxy/0.3
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798732 https_client.c:205 E179: > Accept: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798744 https_client.c:205 E179: > Content-Type: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798755 https_client.c:205 E179: > Content-Length: 50
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.798818 https_client.c:597 Reserved new io event: 0x7ffbd88d48
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.803583 main.c:113 Received request for id: 2165, len: 50
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.803634 https_client.c:251 2165: Requesting HTTP/2
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804318 https_client.c:205 2165: > POST /dns-query HTTP/2
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804343 https_client.c:205 2165: > Host: cloudflare-dns.com
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804355 https_client.c:205 2165: > User-Agent: https_dns_proxy/0.3
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804404 https_client.c:205 2165: > Accept: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804417 https_client.c:205 2165: > Content-Type: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804428 https_client.c:205 2165: > Content-Length: 50
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804491 https_client.c:587 Released used io event: 0x7ffbd88d48
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.804502 https_client.c:597 Reserved new io event: 0x7ffbd88d48
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817199 https_client.c:205 E179: < HTTP/2 200
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817225 https_client.c:205 E179: < server: cloudflare
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817241 https_client.c:205 E179: < date: Fri, 20 Oct 2023 21:45:41 GMT
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817255 https_client.c:205 E179: < content-type: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817267 https_client.c:205 E179: < access-control-allow-origin: *
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817281 https_client.c:205 E179: < content-length: 306
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817292 https_client.c:205 E179: < cf-ray: 819473045b4c8cae-EWR
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817340 https_client.c:344 E179: curl request succeeded
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817353 https_client.c:435 E179: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817393 https_client.c:447 E179: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817403 https_client.c:453 E179: CURLINFO_HTTP_VERSION: 2
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817412 https_client.c:482 E179: Times: 0.000155, 0.000000, 0.000000, 0.000704, 0.019106, 0.019293
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.817458 main.c:84 Received response for id: E179, len: 306
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822526 https_client.c:205 2165: < HTTP/2 200
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822550 https_client.c:205 2165: < server: cloudflare
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822567 https_client.c:205 2165: < date: Fri, 20 Oct 2023 21:45:41 GMT
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822582 https_client.c:205 2165: < content-type: application/dns-message
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822594 https_client.c:205 2165: < access-control-allow-origin: *
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822611 https_client.c:205 2165: < content-length: 162
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822622 https_client.c:205 2165: < cf-ray: 819473046b5a8cae-EWR
Fri Oct 20 17:45:41 2023 daemon.info https-dns-proxy[3558]: [D] 1697838341.822743 https_client.c:587 Released used io event: 0x7ffbd88d48
Fri Oct 20 17:45:42 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 116 crashes, 26 seconds since last crash
Fri Oct 20 17:45:42 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 116 crashes, 26 seconds since last crash
Fri Oct 20 17:45:42 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.933599 main.c:219 Version: 2023-05-25-1
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.933609 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.933615 main.c:221 System c-ares: 1.19.1
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.933679 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [W] 1697838342.933714 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.934983 dns_server.c:50 Listening on ::1:5054
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [D] 1697838342.935393 logging.c:40 starting periodic log flush timer
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [D] 1697838342.935924 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3753]: [I] 1697838342.935944 main.c:309 DNS polling initialized for 'cloudflare-dns.com'
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.937856 main.c:219 Version: 2023-05-25-1
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.937868 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.937874 main.c:221 System c-ares: 1.19.1
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.937927 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [W] 1697838342.937949 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.938172 dns_server.c:50 Listening on 127.0.0.1:5054
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [D] 1697838342.939125 logging.c:40 starting periodic log flush timer
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [D] 1697838342.939478 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 17:45:42 2023 daemon.info https-dns-proxy[3754]: [I] 1697838342.939496 main.c:309 DNS polling initialized for 'cloudflare-dns.com'
Fri Oct 20 17:45:46 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 117 crashes, 3 seconds since last crash
Fri Oct 20 17:45:46 2023 daemon.info procd: Instance https-dns-proxy::instance2 s in a crash loop 117 crashes, 3 seconds since last crash
Fri Oct 20 17:45:46 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓✓
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.538329 main.c:219 Version: 2023-05-25-1
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.538343 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.538349 main.c:221 System c-ares: 1.19.1
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.538414 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [W] 1697838346.538437 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.540406 main.c:219 Version: 2023-05-25-1
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.540419 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.540424 main.c:221 System c-ares: 1.19.1
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.540486 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [W] 1697838346.540508 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.540737 dns_server.c:50 Listening on 127.0.0.1:5054
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [D] 1697838346.540787 logging.c:40 starting periodic log flush timer
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [D] 1697838346.541084 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3915]: [I] 1697838346.541096 main.c:309 DNS polling initialized for 'cloudflare-dns.com'
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.539990 dns_server.c:50 Listening on ::1:5054
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [D] 1697838346.540112 logging.c:40 starting periodic log flush timer
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [D] 1697838346.543167 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 17:45:46 2023 daemon.info https-dns-proxy[3914]: [I] 1697838346.543184 main.c:309 DNS polling initialized for 'cloudflare-dns.com'

You seem to have two instances with the same listen_port, my guess is that the second instance crashes because it can't bind to the port 5054.

If I were you, I'd remove listen_addr, listen_port, user and group options from both instances and let the service use default values.

PS. Based on below:

Either something malfunction or you've tried to manually edit backup settings while the service was running. I'd suggest you either disable service from auto-updating dnsmasq (if you want to control dnsmasq settings yourself) or fix the backup server so that when you stop https-dns-proxy you still have a working DNS.

Thanks for the suggestions!

This was working fine on my previous router - should be fine, as one is IPv4 and one is IPv6. That said, I've made the changes you suggested, and it continues crashing:

service log restart; service dnsmasq restart; service https-dns-proxy restart

udhcpc: started, v1.36.1
udhcpc: broadcasting discover
udhcpc: no lease, failing
Starting https-dns-proxy 2023-05-25-7 instances ✓

logread -e dnsmasq; netstat -l -n -p | grep -e dnsmasq

Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: started, version 2.89 cachesize 150
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: DNS service limited to local subnets
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-nftset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: UBus support enabled: connected to system bus
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq-dhcp[1]: DHCP, IP range 192.168.5.100 -- 192.168.5.249, lease time 12h
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using nameserver 127.0.0.1#5053
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for test
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for onion
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for localhost
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for local
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for invalid
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for bind
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: using only locally-known addresses for lan
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: read /etc/hosts - 12 names
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 9 names
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq[1]: read /tmp/hosts/odhcpd - 20 names
Fri Oct 20 18:00:19 2023 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
tcp        0      0 10.14.0.1:53            0.0.0.0:*               LISTEN      14338/dnsmasq
tcp        0      0 69.253.xxx.xxx:53       0.0.0.0:*               LISTEN      14338/dnsmasq
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      14338/dnsmasq
tcp        0      0 192.168.5.1:53          0.0.0.0:*               LISTEN      14338/dnsmasq
tcp        0      0 2001:558:6027:86:c11b:d34c:xxx:xxx:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fd9f:5b3c:8f49::1:53    :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599b:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 2601:41:xxx:xxx::1:53  :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599e:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 fe80::ea9f:80ff:feed:599d:53 :::*                    LISTEN      14338/dnsmasq
tcp        0      0 ::1:53                  :::*                    LISTEN      14338/dnsmasq
udp        0      0 127.0.0.1:53            0.0.0.0:*                           14338/dnsmasq
udp        0      0 69.253.xxx.xxx:53       0.0.0.0:*                           14338/dnsmasq
udp        0      0 192.168.5.1:53          0.0.0.0:*                           14338/dnsmasq
udp        0      0 10.14.0.1:53            0.0.0.0:*                           14338/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           14338/dnsmasq
udp        0      0 ::1:53                  :::*                                14338/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                                14338/dnsmasq
udp        0      0 2001:558:6027:86:c11b:d34c:xxx:xxx:53 :::*                                14338/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599b:53 :::*                                14338/dnsmasq
udp        0      0 2601:41:xxx:xxx::1:53  :::*                                14338/dnsmasq
udp        0      0 fd9f:5b3c:8f49::1:53    :::*                                14338/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599c:53 :::*                                14338/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599e:53 :::*                                14338/dnsmasq
udp        0      0 fe80::ea9f:80ff:feed:599d:53 :::*                                14338/dnsmasq

logread -e https-dns; netstat -l -n -p | grep -e https-dns

Fri Oct 20 18:00:19 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 146 crashes, 81 seconds since last crash
Fri Oct 20 18:00:20 2023 user.notice https-dns-proxy: Starting service instances ✓
Fri Oct 20 18:00:41 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 147 crashes, 22 seconds since last crash
Fri Oct 20 18:00:41 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 18:00:48 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 148 crashes, 7 seconds since last crash
Fri Oct 20 18:00:48 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 18:02:16 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 149 crashes, 88 seconds since last crash
Fri Oct 20 18:02:16 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 18:02:16 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 150 crashes, 0 seconds since last crash
Fri Oct 20 18:02:17 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
udp        0      0 127.0.0.1:5053          0.0.0.0:*                           15181/https-dns-pro

pgrep -f -a dnsmasq; pgrep -f -a https-dns

14334 /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq.d -r /tmp/hosts -r /usr/bin/jshn -r /usr/lib/dnsmasq/dhcp-script.sh -r /usr/share/dnsmasq/dhcpbogushostname.conf -r /usr/share/dnsmasq/rfc6761.conf -r /usr/share/dnsmasq/trust-anchors.conf -r /usr/share/libubox/jshn.sh -r /var/etc/dnsmasq.conf.cfg01411c -w /var/run/dnsmasq/ -- /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
14338 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
15181 /usr/sbin/https-dns-proxy -r https://cloudflare-dns.com/dns-query -a 127.0.0.1 -p 5053 -b 1.1.1.1,1.0.0.1 -4 -u nobody -g nogroup

head -v -n -0 /etc/resolv.* /tmp/resolv.* /tmp/resolv./

==> /etc/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf <==
search lan
nameserver 127.0.0.1
nameserver ::1

==> /tmp/resolv.conf.d <==
head: /tmp/resolv.conf.d: I/O error

==> /tmp/resolv.conf.d/resolv.conf.auto <==
# Interface wan
nameserver 75.75.75.75
nameserver 75.75.76.76
search hsd1.pa.comcast.net.
# Interface wan6
nameserver 2001:558:feed::1
nameserver 2001:558:feed::2

uci show dhcp; uci show https-dns-proxy

dhcp.@dnsmasq[0]=dnsmasq
dhcp.@dnsmasq[0].domainneeded='1'
dhcp.@dnsmasq[0].localise_queries='1'
dhcp.@dnsmasq[0].rebind_protection='1'
dhcp.@dnsmasq[0].local='/lan/'
dhcp.@dnsmasq[0].domain='lan'
dhcp.@dnsmasq[0].expandhosts='1'
dhcp.@dnsmasq[0].authoritative='1'
dhcp.@dnsmasq[0].readethers='1'
dhcp.@dnsmasq[0].leasefile='/tmp/dhcp.leases'
dhcp.@dnsmasq[0].localservice='1'
dhcp.@dnsmasq[0].ednspacket_max='1232'
dhcp.@dnsmasq[0].quietdhcp='1'
dhcp.@dnsmasq[0].filterwin2k='1'
dhcp.@dnsmasq[0].doh_backup_noresolv='-1'
dhcp.@dnsmasq[0].noresolv='1'
dhcp.@dnsmasq[0].doh_backup_server='::1#5054' '127.0.0.1#5054'
dhcp.@dnsmasq[0].doh_server='::1#5054' '127.0.0.1#5054' '::1#5053' '127.0.0.1#5053'
dhcp.@dnsmasq[0].server='127.0.0.1#5053'
dhcp.lan=dhcp
dhcp.lan.interface='lan'
dhcp.lan.start='100'
dhcp.lan.limit='150'
dhcp.lan.leasetime='12h'
dhcp.lan.dhcpv4='server'
dhcp.lan.dhcpv6='server'
dhcp.lan.ra='server'
dhcp.lan.ra_flags='managed-config' 'other-config'
dhcp.wan=dhcp
dhcp.wan.interface='wan'
dhcp.wan.ignore='1'
dhcp.odhcpd=odhcpd
dhcp.odhcpd.maindhcp='0'
dhcp.odhcpd.leasefile='/tmp/hosts/odhcpd'
dhcp.odhcpd.leasetrigger='/usr/sbin/odhcpd-update'
dhcp.odhcpd.loglevel='4'
https-dns-proxy.config=main
https-dns-proxy.config.dnsmasq_config_update='-'
https-dns-proxy.config.force_dns='0'
https-dns-proxy.config.force_dns_port='53' '853'
https-dns-proxy.config.procd_trigger_wan6='0'
https-dns-proxy.@https-dns-proxy[0]=https-dns-proxy
https-dns-proxy.@https-dns-proxy[0].bootstrap_dns='1.1.1.1,1.0.0.1'
https-dns-proxy.@https-dns-proxy[0].resolver_url='https://cloudflare-dns.com/dns-query'

Additionally, I've enabled verbose logging, an example of the output, including content before and after the crashes, is below:

Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.495608 main.c:113 Received request for id: C76B, len: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.495660 https_client.c:251 C76B: Requesting HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496308 https_client.c:205 C76B: > POST /dns-query HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496333 https_client.c:205 C76B: > Host: cloudflare-dns.com
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496345 https_client.c:205 C76B: > User-Agent: https_dns_proxy/0.3
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496357 https_client.c:205 C76B: > Accept: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496368 https_client.c:205 C76B: > Content-Type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496425 https_client.c:205 C76B: > Content-Length: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496491 https_client.c:597 Reserved new io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496567 main.c:113 Received request for id: 150C, len: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.496604 https_client.c:251 150C: Requesting HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497197 https_client.c:205 150C: > POST /dns-query HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497222 https_client.c:205 150C: > Host: cloudflare-dns.com
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497234 https_client.c:205 150C: > User-Agent: https_dns_proxy/0.3
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497284 https_client.c:205 150C: > Accept: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497296 https_client.c:205 150C: > Content-Type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497308 https_client.c:205 150C: > Content-Length: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497371 https_client.c:587 Released used io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497383 https_client.c:597 Reserved new io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497463 main.c:113 Received request for id: C838, len: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.497554 https_client.c:251 C838: Requesting HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498157 https_client.c:205 C838: > POST /dns-query HTTP/2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498180 https_client.c:205 C838: > Host: cloudflare-dns.com
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498191 https_client.c:205 C838: > User-Agent: https_dns_proxy/0.3
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498203 https_client.c:205 C838: > Accept: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498214 https_client.c:205 C838: > Content-Type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498225 https_client.c:205 C838: > Content-Length: 62
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498286 https_client.c:587 Released used io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.498319 https_client.c:597 Reserved new io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515755 https_client.c:205 C76B: < HTTP/2 200
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515774 https_client.c:205 C76B: < server: cloudflare
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515788 https_client.c:205 C76B: < date: Fri, 20 Oct 2023 22:19:09 GMT
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515799 https_client.c:205 C76B: < content-type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515809 https_client.c:205 C76B: < access-control-allow-origin: *
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515821 https_client.c:205 C76B: < content-length: 144
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515830 https_client.c:205 C76B: < cf-ray: 8194a408794f1768-EWR
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515870 https_client.c:344 C76B: curl request succeeded
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515881 https_client.c:435 C76B: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515888 https_client.c:447 C76B: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515895 https_client.c:453 C76B: CURLINFO_HTTP_VERSION: 2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515902 https_client.c:482 C76B: Times: 0.000147, 0.000000, 0.000000, 0.000654, 0.019938, 0.020073
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.515971 main.c:84 Received response for id: C76B, len: 144
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517292 https_client.c:205 C838: < HTTP/2 200
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517309 https_client.c:205 C838: < server: cloudflare
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517324 https_client.c:205 C838: < date: Fri, 20 Oct 2023 22:19:09 GMT
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517336 https_client.c:205 C838: < content-type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517345 https_client.c:205 C838: < access-control-allow-origin: *
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517356 https_client.c:205 C838: < content-length: 94
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517365 https_client.c:205 C838: < cf-ray: 8194a40879551768-EWR
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517406 https_client.c:344 C838: curl request succeeded
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517418 https_client.c:435 C838: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517425 https_client.c:447 C838: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517432 https_client.c:453 C838: CURLINFO_HTTP_VERSION: 2
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517440 https_client.c:482 C838: Times: 0.000000, 0.000000, 0.000000, 0.000580, 0.019605, 0.019734
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.517492 main.c:84 Received response for id: C838, len: 94
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536643 https_client.c:205 150C: < HTTP/2 200
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536660 https_client.c:205 150C: < server: cloudflare
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536672 https_client.c:205 150C: < date: Fri, 20 Oct 2023 22:19:09 GMT
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536681 https_client.c:205 150C: < content-type: application/dns-message
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536688 https_client.c:205 150C: < access-control-allow-origin: *
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536697 https_client.c:205 150C: < content-length: 144
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536704 https_client.c:205 150C: < cf-ray: 8194a40879521768-EWR
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536758 https_client.c:587 Released used io event: 0x7fc9019018
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536781 https_client.c:344 150C: curl request succeeded
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536832 https_client.c:435 150C: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 18:19:09 2023 daemon.info https-dns-proxy[22191]: [D] 1697840349.536839 https_client.c:447 150C: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 18:19:12 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 163 crashes, 35 seconds since last crash
Fri Oct 20 18:19:12 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.847714 main.c:219 Version: 2023-05-25-1
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.847729 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.847733 main.c:221 System c-ares: 1.19.1
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.847781 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [W] 1697840352.847803 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.848345 dns_server.c:50 Listening on 127.0.0.1:5053
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [D] 1697840352.848411 logging.c:40 starting periodic log flush timer
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [D] 1697840352.848688 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 18:19:12 2023 daemon.info https-dns-proxy[22595]: [I] 1697840352.848699 main.c:309 DNS polling initialized for 'cloudflare-dns.com'

You're right, I should have paid more attention.

Weird, on my x86 system that part is:

15:45:25 https-dns-proxy[16535]: [D] 1697841925.387965 https_client.c:435 774C: CURLINFO_NUM_CONNECTS: 0
15:45:25 https-dns-proxy[16535]: [D] 1697841925.387968 https_client.c:447 774C: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
15:45:25 https-dns-proxy[16535]: [D] 1697841925.387971 https_client.c:453 774C: CURLINFO_HTTP_VERSION: 2
15:45:25 https-dns-proxy[16535]: [D] 1697841925.387974 https_client.c:482 774C: Times: 0.000040, 0.000000, 0.000000, 0.000182, 0.002356, 0.002560
15:45:25 https-dns-proxy[16535]: [D] 1697841925.387982 main.c:84 Received response for id: 774C, len: 149
15:45:25 https-dns-proxy[16535]: [D] 1697841925.439075 main.c:113 Received request for id: DF2F, len: 65

I've also noticed that nghttp2 on x86 is still at 1.51.0 whereas on mips_24kc it's on 1.57.0, maybe there's something there. I wonder if you can try to force HTTP/1 if that'd help.

PS. Maybe for testing you can stop the https-dns-proxy service and start an instance from CLI to keep dns resolution working.

Good thought - I did try that, but it's still crashing:

Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.517973 https_client.c:205 270F: < Connection: keep-alive
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.518004 https_client.c:205 270F: < Access-Control-Allow-Origin: *
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.518016 https_client.c:205 270F: < Content-Length: 100
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.518025 https_client.c:205 270F: < CF-RAY: 8194ed6bfdb9424d-EWR
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519366 https_client.c:112 curl closed socket: 12
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519420 https_client.c:587 Released used io event: 0x7fe7e334a8
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519454 https_client.c:344 270F: curl request succeeded
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519467 https_client.c:435 270F: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519474 https_client.c:447 270F: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519481 https_client.c:453 270F: CURLINFO_HTTP_VERSION: 1.1
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519489 https_client.c:482 270F: Times: 0.000234, 0.000000, 0.000000, 0.001235, 0.018431, 0.018599
Fri Oct 20 19:09:18 2023 daemon.info https-dns-proxy[5703]: [D] 1697843355.519528 main.c:84 Received response for id: 270F, len: 100
Fri Oct 20 19:09:20 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 221 crashes, 63 seconds since last crash
Fri Oct 20 19:09:20 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.832441 main.c:219 Version: 2023-05-25-1
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.832456 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.832461 main.c:221 System c-ares: 1.19.1
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.832669 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [W] 1697843360.832701 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.833237 dns_server.c:50 Listening on 127.0.0.1:5053
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [D] 1697843360.833305 logging.c:40 starting periodic log flush timer
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [D] 1697843360.833587 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 19:09:20 2023 daemon.info https-dns-proxy[5993]: [I] 1697843360.833598 main.c:309 DNS polling initialized for 'cloudflare-dns.com'

I haven't actually noticed any problems with DNS resolution (of course, DNS may be failing occasionally, but it's either something like an ad server that I don't notice, or it reattempts and is successful so quickly I don't notice). My only concern is the perpetual crashing of https-dns-proxy in syslog; that's the only 'symptom' I've experienced so far.

I re-enabled auto-negotiation, and here's another crash, which is not immediately after CURLINFO_EFFECTIVE_URL:

Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9050]: [D] 1697843877.837881 https_client.c:435 E221: CURLINFO_NUM_CONNECTS: 0
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9050]: [D] 1697843877.837911 https_client.c:447 E221: CURLINFO_EFFECTIVE_URL: https://cloudflare-dns.com/dns-query
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9050]: [D] 1697843877.837923 https_client.c:453 E221: CURLINFO_HTTP_VERSION: 2
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9050]: [D] 1697843877.837988 https_client.c:482 E221: Times: 0.000000, 0.000000, 0.000000, 0.000646, 0.045725, 0.045957
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9050]: [D] 1697843877.838033 main.c:84 Received response for id: E221, len: 174
Fri Oct 20 19:17:58 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 232 crashes, 11 seconds since last crash
Fri Oct 20 19:17:58 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.792655 main.c:219 Version: 2023-05-25-1
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.792669 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.792674 main.c:221 System c-ares: 1.19.1
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.792748 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [W] 1697843878.792771 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.792976 dns_server.c:50 Listening on 127.0.0.1:5053
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [D] 1697843878.793039 logging.c:40 starting periodic log flush timer
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [D] 1697843878.793537 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 19:17:58 2023 daemon.info https-dns-proxy[9221]: [I] 1697843878.793553 main.c:309 DNS polling initialized for 'cloudflare-dns.com'

The crashing seems to be unrelated to whatever debug logs are being printed at the time.

And another crash, a full seven seconds after the prior debug message:

Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.995282 main.c:219 Version: 2023-05-25-1
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.995296 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.995302 main.c:221 System c-ares: 1.19.1
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.995353 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [W] 1697844057.995377 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.997542 dns_server.c:50 Listening on 127.0.0.1:5053
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [D] 1697844057.997656 logging.c:40 starting periodic log flush timer
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [D] 1697844057.998122 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 19:20:57 2023 daemon.info https-dns-proxy[10734]: [I] 1697844057.998141 main.c:309 DNS polling initialized for 'cloudflare-dns.com'
Fri Oct 20 19:21:04 2023 daemon.info procd: Instance https-dns-proxy::instance1 s in a crash loop 242 crashes, 6 seconds since last crash
Fri Oct 20 19:21:04 2023 user.notice https-dns-proxy: Starting service instances on_interface_up ✓
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.711545 main.c:219 Version: 2023-05-25-1
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.711560 main.c:220 Built: Oct 15 2023 17:51:39
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.711564 main.c:221 System c-ares: 1.19.1
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.711952 main.c:222 System libcurl: libcurl/8.4.0 mbedTLS/2.28.5 nghttp2/1.57.0
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [W] 1697844064.711984 main.c:236 HTTP/3 is not supported by current libcurl
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.713624 dns_server.c:50 Listening on 127.0.0.1:5053
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [D] 1697844064.713695 logging.c:40 starting periodic log flush timer
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [D] 1697844064.714346 dns_poller.c:178 Nameservers count: 2
Fri Oct 20 19:21:04 2023 daemon.info https-dns-proxy[10896]: [I] 1697844064.714367 main.c:309 DNS polling initialized for 'cloudflare-dns.com'

I'm out of ideas then, maybe create an issue at openwrt/packages repo and I'll tag the upstream devs, maybe they would have some suggestions.

PS. Since you said it worked before -- there's a patch in the -7 revision which actually aims to prevent crashing, I wonder if you can try to compile the binary for your platform without the 030-src-logging.c-fix-crash.patch and see if it helps?

Thanks for the guidance; I've created an issue here: https://github.com/openwrt/packages/issues/22463

I'm not particularly familiar with that process, though I may end up Googling it and taking a crack at it over the weekend. I appreciate the suggestion!

Not sure how helpful this info is but I am getting the same crash with the stable release (and latest 23.05 snapshot) on my R7800. Also, I did not have this issue on my R7800 with RC3. I have not tried RC4 though.

Check for the updates in the linked github issue.

1 Like