Attended Sysupgrade Issue

I recently installed a fresh install of 24.10.0 on a Raspberry Pi CM4/DFRobot board. Got the network/VLANs all working fine and everything seems great except that I can't use the attended sysupgrade webUI in LuCI. It just sits there trying to check for upgrades. Yes, I also know that there are no newer releases yet but I thought I would at least check to see if this is working.

When I refresh the list of available app updates in the Software section, that works fine with opkg.

I thought maybe it's something with the web interface but when I ssh into it and run owut check, it also errors out eventually with:

root@OpenWrt:~# owut check
ERROR: Failed to connect
ERROR: Response status (null) while downloading
  https://sysupgrade.openwrt.org/json/v1/overview.json

For what it's worth, I can't seem to get to that URL whether I am behind this OpenWRT box or my regular router. So I thought maybe the sysupgrade server is down? Tried it the last 2 days and it's the same results.

Let me know if I just missed it somewhere that sysupgrade is down or if there is something else I need to try because owut replaced auc.

DNS issues on the main router? The URL is loading just fine.

That's usually indicative of the ASU server being down, but it looks fine to me.

Maybe like @Dante says, DNS issues? What does an nslookup show?

$ nslookup sysupgrade.openwrt.org
Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-02.infra.openwrt.org
Name:   asu-02.infra.openwrt.org
Address: 45.140.183.87

Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-02.infra.openwrt.org
Name:   asu-02.infra.openwrt.org
Address: 2001:678:6e1:1001:be24:11ff:fe23:4c6d

Looks like DNS from the router seems okay to me...

root@OpenWrt:~# nslookup sysupgrade.openwrt.org
Server:		127.0.0.1
Address:	127.0.0.1:53

Non-authoritative answer:
sysupgrade.openwrt.org	canonical name = asu-02.infra.openwrt.org
Name:	asu-02.infra.openwrt.org
Address: 45.140.183.87

Non-authoritative answer:
sysupgrade.openwrt.org	canonical name = asu-02.infra.openwrt.org
Name:	asu-02.infra.openwrt.org
Address: 2001:678:6e1:1001:be24:11ff:fe23:4c6d

Also traceroute from the openwrt router gets past the gateway into AT&T (my ISP) and then seems to go all over the place and dies after 30 hops. So maybe some kind of routing issue within the larger internet?

root@OpenWrt:~# traceroute sysupgrade.openwrt.org
traceroute to sysupgrade.openwrt.org (45.140.183.87), 30 hops max, 46 byte packets
 1  192.168.1.254 (192.168.1.254)  0.879 ms  0.414 ms  0.257 ms
 2  107-210-168-1.lightspeed.sndgca.sbcglobal.net (107.210.168.1)  1.549 ms  1.663 ms  1.324 ms
 3  71.157.16.114 (71.157.16.114)  1.376 ms  1.331 ms  0.824 ms
 4  *  *  *
 5  *  *  *
 6  *  *  *
 7  32.130.90.123 (32.130.90.123)  5.026 ms  *  *
 8  *  *  *
 9  rest-bb1-link.ip.twelve99.net (62.115.137.36)  74.391 ms  ash-bb2-link.ip.twelve99.net (62.115.137.38)  76.190 ms  72.912 ms
10  *  *  prs-bb1-link.ip.twelve99.net (62.115.140.104)  148.513 ms
11  ffm-bb1-link.ip.twelve99.net (62.115.123.12)  153.615 ms  156.067 ms  156.627 ms
12  ffm-b5-link.ip.twelve99.net (62.115.114.91)  162.496 ms  158.123 ms  ffm-b5-link.ip.twelve99.net (62.115.136.219)  162.319 ms
13  universitt-ic-349246.ip.twelve99-cust.net (213.248.88.26)  158.442 ms  897.064 ms  152.995 ms
14  stu-eti-a99-hu0-4-0-11.belwue.net (129.143.57.126)  158.369 ms  fra-tc-a99-hu0-1-0-7.belwue.net (129.143.57.124)  154.297 ms  stu-eti-a99-hu0-4-0-11.belwue.net (129.143.57.126)  162.724 ms
15  kar-rz-a99-hu0-1-0-0.belwue.net (129.143.60.77)  166.214 ms  166.106 ms  166.159 ms
16  *  *  *
17  *  *  *
18  gw0.scc.kae.bb.vzffnrmo.de (45.140.183.69)  162.949 ms  165.651 ms  163.780 ms
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

Hmm, how about trying both of these to see if it's something to do with the addresses? (I've seen issues with owut/uclient-fetch/LuCI ASU - which all use the same underlying library - when the IPv6 for the CDN is borked.)

uclient-fetch -4 https://sysupgrade.openwrt.org/json/v1/overview.json
uclient-fetch -6 https://sysupgrade.openwrt.org/json/v1/overview.json

No luck. Both error with:

Downloading 'https://sysupgrade.openwrt.org/json/v1/overview.json'
Failed to send request: Operation not permitted

Uh, wtf is going on here?

Let's grasp at some straws. First, check the system clock on the router, make sure it's correct. Assuming that's good, how about try those uclient-fetch runs again, but use http instead of https...

Firstly, I appreciate the help/support here. I knew this one seemed strange and it's sort of satisfying to also know I am not crazy :slight_smile:. Operation not permitted also makes me think I am missing something.

Date/Clock seem good. Accurate and an NTP client as well BTW.

uclient-fetch (http) results:

root@OpenWrt:~# uclient-fetch -4 http://sysupgrade.openwrt.org/json/v1/overview.json
Downloading 'http://sysupgrade.openwrt.org/json/v1/overview.json'
Failed to send request: Operation not permitted
root@OpenWrt:~# uclient-fetch -6 http://sysupgrade.openwrt.org/json/v1/overview.json
Downloading 'http://sysupgrade.openwrt.org/json/v1/overview.json'
Failed to send request: Operation not permitted

Just for tests sake, I also ssh'd into a Pi4 I had sitting in front of the OpenWRT router but obviously still behind my normal AT&T residential gateway and tried wget:

@pi4:~$ wget http://sysupgrade.openwrt.org/json/v1/overview.json
--2025-03-14 09:36:20--  http://sysupgrade.openwrt.org/json/v1/overview.json
Resolving sysupgrade.openwrt.org (sysupgrade.openwrt.org)... 45.140.183.87, 2001:678:6e1:1001:be24:11ff:fe23:4c6d
Connecting to sysupgrade.openwrt.org (sysupgrade.openwrt.org)|45.140.183.87|:80... failed: Connection timed out.
Connecting to sysupgrade.openwrt.org (sysupgrade.openwrt.org)|2001:678:6e1:1001:be24:11ff:fe23:4c6d|:80... failed: Network is unreachable.

Well, that looks like it's upstream from you, but since I see the same IP addresses and the owut and wget downloads are working for me, it must be downstream from the openwrt.org servers, so... Not sure how to proceed here.

Maybe bypass the CNAMEs and try asu-02.infra.openwrt.org directly??? Could it be a local CDN issue for your region???

Same result. Yeah, almost has to be something upstream from me. We know we can resolve it fine and it leaves my house fine. This is the larger universe telling me to mess with something else today :grinning:. Who knows. I'll let it all just burn for a few days and check again if I get anything different.

Since this has already been going on for a couple days, I'd have thought it would have resolved already if it were a CDN issue. And more to the point, geo location of the first couple addresses in your traceroute puts you somewhere in San Diego county, and since I'm in south Orange county, we're probably like 60 miles apart so its almost certainly not regional... You might have to talk to your ISP to figure this out.

A couple thoughts...

I did not test this but this error made me think that maybe the current directory is not writable but it looks like you were in ~ which I assume is /root/. Make sure wget (uclient-fetch) can get a file from another url like maybe a small file from github and write it to the local directory.
Try changing directory to /tmp or specifying an out file with -O.

To test rpi vs upstream issues, try getting the https://sysupgrade.openwrt.org/json/v1/overview.json file from a desktop pc, even a browser. It displays fine in my browser but I might have set something so it recognized json files. Alternatively yours might just save the file to your downloads directory.
EDIT: If your residential gateway is providing appropriate firewall protection, connect the desktop pc just inside or direct to a lan port on the residential gateway for the test.

You indicated testing from two rpi boards. I wonder if there might be an issue with ciphers or encryption method compatibility. If the desktop test works then this might be worth looking into.
EDIT: I know this doesn't apply with the non tls fetch.

Good luck!

Test from my mac using Chrome yields same results (no file displayed/downloaded). ERR_CONNECTION_TIMED_OUT is technically the browser error. Definitely feel like there is something strange going on with the ISP/routing situation.

it looks like you were in ~ which I assume is /root/

Yes, correct.

Also, yes, connected my mac directly to the AT&T gateway's wifi. Same result.

Thanks again for the help or at least validation that it's most likely something with my ISP or routing within the larger routing tables they use.

1 Like

For comparison, here are a couple traces from a different provider on the US East coast ...

tracepath -4 -b -p 443 sysupgrade.openwrt.org
 1?: [LOCALHOST]                      pmtu 1500
 1:  R4S-wrt.lan (192.168.21.1)                              2.226ms 
 1:  R4S-wrt.lan (192.168.21.1)                              2.928ms 
 2:  <REDACTED>                                              4.009ms 
 3:  no reply
 4:  <REDACTED>                                              8.193ms 
 5:  <REDACTED>                                             16.733ms asymm  7 
 6:  <REDACTED>                                              9.626ms asymm  7 
 7:  <REDACTED>                                              9.747ms 
 8:  be3628.ccr42.par01.atlas.cogentco.com (154.54.27.170)  85.868ms asymm  9 
 9:  be7946.ccr42.fra05.atlas.cogentco.com (154.54.72.118)  95.762ms asymm 10 
10:  be7946.ccr42.fra05.atlas.cogentco.com (154.54.72.118)  91.029ms 
11:  be3472.ccr32.bos01.atlas.cogentco.com (154.54.46.33)   99.673ms asymm 13 
12:  be5519.rcr61.str01.atlas.cogentco.com (154.54.62.134)  93.409ms 
13:  149.6.20.6 (149.6.20.6)                                96.794ms 
14:  be4975.ccr41.fra05.atlas.cogentco.com (154.54.63.69)  126.132ms asymm 10 
15:  no reply
16:  be5519.rcr61.str01.atlas.cogentco.com (154.54.62.134) 116.369ms asymm 12 
17:  149.6.20.6 (149.6.20.6)                               203.427ms asymm 13 
18:  sysupgrade.openwrt.org (45.140.183.87)                 99.695ms reached
     Resume: pmtu 1500 hops 18 back 21 

And a second run a couple minutes later:

...
 8:  be2262.ccr42.jfk02.atlas.cogentco.com (154.54.47.121)  11.375ms 
 9:  be4975.ccr41.fra05.atlas.cogentco.com (154.54.63.69)   95.960ms asymm 10 
10:  be7946.ccr42.fra05.atlas.cogentco.com (154.54.72.118)  94.351ms 
11:  be3471.ccr31.bos01.atlas.cogentco.com (154.54.40.153) 210.524ms asymm 13 
12:  149.6.20.6 (149.6.20.6)                               100.424ms asymm 13 
13:  149.6.20.6 (149.6.20.6)                               100.073ms 
14:  kar-rz-a99-hu0-1-0-0.belwue.net (129.143.60.77)       171.725ms 
15:  be3283.agr61.fra05.atlas.cogentco.com (154.54.76.122) 204.043ms asymm 11 
16:  gw0.scc.kae.bb.vzffnrmo.de (45.140.183.69)            101.208ms asymm 20 
17:  gw0.scc.kae.bb.vzffnrmo.de (45.140.183.69)             99.809ms asymm 20 
18:  sysupgrade.openwrt.org (45.140.183.87)                100.953ms reached
     Resume: pmtu 1500 hops 18 back 21 

There is more variability than I'm used to seeing but I have not been looking much the past few years. There is lots of DDOS activity these days and new tier 1 lines going online all the time. BGP issues pop up too. For my trace, it looks like cogent communications gets my traffic between continents.

Looking into the tier1 networks in use, I found a looking-glass page for twelve99 at https://lg.twelve99.net/ .

A trace from their Reston Core was slow but completed:

https://lg.twelve99.net/?type=traceroute&router=rest-b2&address=45.140.183.87

Router: rest-b2 / Reston (CoreSite)
Command: traceroute ipv4 45.140.183.87 timeout 1 source Loopback0

Tracing the route to 45.140.183.87

 1  rest-bb1-link.ip.twelve99.net (62.115.123.40) 1 msec 
    ash-bb2-link.ip.twelve99.net (62.115.121.217) 1 msec 
    rest-bb1-link.ip.twelve99.net (62.115.123.40) 1 msec 
 2  prs-bb1-link.ip.twelve99.net (62.115.140.104) 76 msec  *  * 
 3  ffm-bb1-link.ip.twelve99.net (62.115.123.12) 84 msec  84 msec 
    ffm-bb2-link.ip.twelve99.net (62.115.122.139) 96 msec 
 4  ffm-b5-link.ip.twelve99.net (62.115.136.213) 90 msec 
    ffm-b5-link.ip.twelve99.net (62.115.136.219) 90 msec 
    ffm-b5-link.ip.twelve99.net (62.115.114.91) 96 msec 
 5  universitt-ic-349246.ip.twelve99-cust.net (213.248.88.26) 84 msec  84 msec  91 msec 
 6  stu-eti-a99-hu0-4-0-7.belwue.net (129.143.60.112) 93 msec 
    fra-tc-a99-hu0-1-0-0.belwue.net (129.143.56.35) 85 msec 
    stu-eti-a99-hu0-4-0-7.belwue.net (129.143.60.112) 93 msec 
 7  kar-rz-a99-hu0-2-0-7.belwue.net (129.143.56.32) 88 msec 
    kar-rz-a99-hu0-1-0-0.belwue.net (129.143.60.77) 90 msec  96 msec 
 8   *  *  * 
 9   *  *  * 
 10 gw0.scc.kae.bb.vzffnrmo.de (45.140.183.69) 88 msec  94 msec  94 msec 
 11 sysupgrade.openwrt.org (45.140.183.87) 88 msec  91 msec  96 msec 

That source node was hop 9 on your trace.

If you happen to already have VPN account, try connecting to a different city through it and see if you get through.

Another thought, if you don't just want to wait it out, is to run tcpdump or wireshark while you try to connect if you are familiar with either tool. You might see something useful

Sadly, this problem still persists.
Was able to make it work though, upon configuring and starting Cloudflare's WARP as Wireguard interface.