Owut: OpenWrt Upgrade Tool

efahl · 2024-06-26T16:57:23.466Z

villahed94:

cp: cannot stat '/builder/d030386f7cd46115a6be623a9c182a6b/openwrt-6f67fa131d74-qualcommax-ipq807x-netgear_rax120v2-initramfs-uImage.itb': No such file or directory

I replicated your build (but without any extra packages) and got the same thing. Then tried with x86 and it worked fine, so...

Regrettably, I can't figure out how to make this one go away. The imagebuilder on the ASU server is broken somehow for that specific device, so the best I can do is that final message:

The above errors are often due to the upgrade server lagging behind the
build server, first suggestion is to wait a while and try again.

Once the buildbots create a new imagebuilder (you will see the Version-to SNAPSHOT r26792-646ebbd32c (kernel 6.6.35) "r-number" bump up beyond 26792), then it should (SHOULD!) work. Fingers crossed.

EDIT
I'm going to see if I can detect that specific class of error in the response logs and add my final paragraph-ish to the owut output... https://github.com/efahl/owut/issues/14

cdes · 2024-06-26T17:38:16.616Z

efahl:

uclient-fetch

uclient-fetch works.

root@wifiap1:~# uclient-fetch https://sysupgrade.openwrt.org/json/v1/overview.json
Downloading 'https://sysupgrade.openwrt.org/json/v1/overview.json'
Connecting to 206.81.21.114:443
Writing to 'overview.json'
overview.json        100% |*******************************| 17634   0:00:00 ETA
Download completed (17634 bytes)
root@wifiap-conroe1:~# owut check --verbose
owut - OpenWrt Upgrade Tool 2024.06.24~1b222bdc-r1 (/usr/bin/owut)
ERROR: uclient error code=1
Response status: 0
ERROR: Could not download 'https://sysupgrade.openwrt.org/json/v1/overview.json', terminating.
root@wifiap1:~#

efahl · 2024-06-26T19:09:34.943Z

Hmm. I'm pretty sure that error code 1 is UCLIENT_ERROR_CONNECT (https://github.com/openwrt/uclient/blob/master/uclient.h#L34), but it's being reported several calls past where we actually connect.

I should have thought of this earlier: could you run again with an extra verbose, see if we're getting back any headers?

$ owut check -v -v
...
Downloaded https://sysupgrade.openwrt.org/json/v1/overview.json to /tmp/owut-overview.json
Response headers = {
  "accept-ranges": "bytes",
  "access-control-allow-headers": "*",
  "access-control-allow-methods": "POST, GET, OPTIONS",
  "access-control-allow-origin": "*",
  "alt-svc": "h3=\":443\"; ma=2592000",
  "content-length": "17634",
  "content-type": "application/json",
  "etag": "\"d2a6it85fz7bdlu\"",
  "last-modified": "Wed, 26 Jun 2024 19:01:07 GMT",
  "server": "Caddy",
  "vary": "Accept-Encoding",
  "date": "Wed, 26 Jun 2024 19:04:41 GMT",
  "status": 200
}

cdes · 2024-06-26T20:18:23.190Z

efahl:

owut check -v -v

No change in output or result.

root@wifiap-conroe1:~# owut check -v -v
owut - OpenWrt Upgrade Tool 2024.06.24~1b222bdc-r1 (/usr/bin/owut)
ERROR: uclient error code=1
Response status: 0
ERROR: Could not download 'https://sysupgrade.openwrt.org/json/v1/overview.json', terminating.

EDIT: ran a tcpdump. curl produces traffic to asu-01.infra.openwrt.org.443, owut unfortunately does not put anything on the line.

efahl · 2024-06-26T21:08:06.432Z

cdes:

curl produces traffic to asu-01.infra.openwrt.org.443

Oho, good catch! I'm guessing something with a redirect is failing in owut. I'll see if I can chase anything down that rabbit hole... (Strange that it only happens on one of your devices, seems like you'd get the same redirect to a regional CDN from the same network, doesn't it?)

efahl · 2024-06-26T21:34:18.039Z

@aparcar
Ok, I found it. The IPv6 address for sysupgrade.openwrt.org site is bad.

Using GNU wget -4 ... works fine, wget -6 ... hangs forever (well, I killed it after three minutes, so close enough to forever).

Also

$ nslookup sysupgrade.openwrt.org
Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-01.infra.openwrt.org
Name:   asu-01.infra.openwrt.org
Address: 206.81.21.114

Non-authoritative answer:
sysupgrade.openwrt.org  canonical name = asu-01.infra.openwrt.org
Name:   asu-01.infra.openwrt.org
Address: 2a03:b0c0:3:d0::1574:1

$ ping 206.81.21.114
PING 206.81.21.114 (206.81.21.114): 56 data bytes
64 bytes from 206.81.21.114: seq=0 ttl=47 time=175.577 ms
64 bytes from 206.81.21.114: seq=1 ttl=47 time=172.185 ms

$ ping 2a03:b0c0:3:d0::1574:1
PING 2a03:b0c0:3:d0::1574:1 (2a03:b0c0:3:d0::1574:1): 56 data bytes
< hangs forever >

spence · 2024-06-26T22:03:56.437Z

efahl:

$ ping 2a03:b0c0:3:d0::1574:1
PING 2a03:b0c0:3:d0::1574:1 (2a03:b0c0:3:d0::1574:1): 56 data bytes
< hangs forever >

try a traceroute to the v6 address and see if it breaks early or close to the destination LAN. You might need a working trace from someone to compare to though.

efahl · 2024-06-26T22:48:09.235Z

Thanks, Daniel is looking at it, too, over on the devel list. He did a traceroute on his end and it works fine for him, so someone has a bad entry in BGP or something...

efahl · 2024-06-26T23:08:18.003Z

@cdes, what's your gross geography. Same as me in North America?

Could you (or anyone else who has the time) run a traceroute or tracepath to that address and report back? (tracepath output is prettier, but traceroute is just fine, too. Host doesn't matter either, any computer or router where you've got a trace* command.)

MAKE SURE TO SANITIZE those first few addresses, as they'll be your local devices.

tracepath -6 2a03:b0c0:3:d0::1574:1

EDIT
As of 2024-06-26 17:33 PDT the routes all seem to be working again...

 6:  las-b23-link.ip.twelve99.net                         16.024ms
 7:  rest-bb1-v6.ip.twelve99.net                          72.928ms
 8:  nyk-bb1-v6.ip.twelve99.net                           89.371ms
 9:  ldn-bb2-v6.ip.twelve99.net                          150.534ms asymm 11
10:  slou-b2-v6.ip.twelve99.net                          149.089ms
11:  digitalocean-ic-378006.ip.twelve99-cust.net         151.966ms
12:  2a03:b0c0:fffe::92                                  148.894ms
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  asu-01.infra.openwrt.org                            170.037ms reached
     Resume: pmtu 1500 hops 17 back 17

Fuzzy2000 · 2024-06-27T10:29:47.720Z

traceroute to 2a03:b0c0:3:d0::1574:1 (2a03:b0c0:3:d0::1574:1) , 5 relative hops max, 52 byte packets
1 asu-01.infra.openwrt.org (2a03:b0c0:3:d0::1574:1) 21.277 ms 23.473 ms 24.971 ms

I think this will Not help

bam80 · 2024-06-29T01:19:59.642Z

efahl:

The --init-script option allows you to specify a uci-defaults script, just like Firmware Selector does with its 'Script to run on first boot (uci-defaults)' input field. LuCI Attended Sysupgrade sort of does this by looking for /rom/etc/uci-defaults/99-asu-defaults and relaying that implicitly, but it doesn't allow you to delete or change what's already there. owut makes this explicit, if you want the script included in your new image, then you must specify it when you request a build.

Can I still specify the script in /rom?

efahl · 2024-06-29T02:15:20.597Z

bam80:

Can I still specify the script in /rom?

Yup, that works fine:

owut upgrade --init-script /rom/etc/uci-defaults/99-asu-defaults

And when you decide that the /rom version needs modification:

cp /rom/etc/uci-defaults/99-asu-defaults /tmp
vi /tmp/99-asu-defaults
... make your changes ...
owut upgrade --init-script /tmp/99-asu-defaults

And if you want to reduce typing and memory load, just set it as default:

uci set attendedsysupgrade.owut.init_script=/rom/etc/uci-defaults/99-asu-defaults
uci commit

Then, when you use just owut upgrade, it defaults to /rom, but if you say owut --init-script /some/where/else then that overrides the config and uses your CLI one.

efahl · 2024-06-29T12:53:31.596Z

Just to be clear, it will only default to /rom after you've set the value in the config. I've edited my post above, as I thought it was sort of ambiguous...

bam80 · 2024-06-29T12:57:53.542Z

Yep, it was indeed.

efahl:

And if you want to reduce typing and memory load, just set it as default:

So it differs from LuCI Attended Sysupgrade's behavior in that you need to explicitly specify the file now.

It looks to me there might be a problem here, as one might not know about /rom/etc/uci-defaults/99-asu-defaults special meaning, even if he has one in his firmware..

bam80 · 2024-06-29T13:11:18.413Z

@efahl if you insist the uci-defaults feature should be "opt-in", as a possible solution I see the following:
owut help listing for the --init-script option should specifically mention Firmware Selector's built firmwares use /rom/etc/uci-defaults/99-asu-defaults file, so user would check this file in the first place.

I personally do not see a reason to complicate things and prefer the uci-defaults feature would be just "opt-out" instead, if needed.

efahl · 2024-06-29T13:14:01.561Z

bam80:

as one might not know about /rom/etc/uci-defaults/99-asu-defaults special meaning

Yes, that's the specific problem I addressed. Not knowing that this specific file exists and is implicitly being propagated by LuCI ASU is a bug that should be addressed in the LuCI code by making it explicit, as it is in FS and owut.

bam80 · 2024-06-29T13:15:32.770Z

OK, but please pay attention to what I said above. Thanks.

efahl · 2024-06-29T13:44:06.794Z

bam80:

Firmware Selector's built firmwares use /rom/etc/uci-defaults/99-asu-defaults

But, Firmware Selector doesn't know anything about that file. That specific file is the ASU server's convention, so is shared by all of auc, owut, FS, LuCI ASU and anything else that uses ASU builds by setting defaults value in the build json.

I will add a warning if /rom.../99-asu-defaults exists and init-script is not define, but the beyond that, it will be up to the user to manage inclusion of the file.

bam80 · 2024-06-29T14:06:16.273Z

efahl:

I will add a warning if /rom.../99-asu-defaults exists and init-script is not define, but the beyond that, it will be up to the user to manage inclusion of the file.

I would still additionally mention the file in the help listing of your tool, as otherwise user might try to override the uci-defaults in the first place without knowing about /rom.../99-asu-defaults and therefor won't get the warning.

bam80 · 2024-06-29T17:19:37.700Z

efahl:

Yes, that's the specific problem I addressed. Not knowing that this specific file exists and is implicitly being propagated by LuCI ASU is a bug that should be addressed in the LuCI code by making it explicit, as it is in FS and owut.

Frankly, I don't see the exact problem with implicitly propagating the file, and why you do think it is a bug. It looks proper behavior for me, and we could always opt-out.

Whereas with the changed behavior we introduced a new, real and severe problem.
Take this scenario:

user generates fw with uci-defaults essential for accessing the device
user creates the /etc/config/attendedsysupgrade configuration for owut needed to include the uci-defaults in the fw on upgrade by default
user resets the firmware and forgets (or doesn't want) to restore the configuration saved
user upgrades with owut next time and looses access to the device

So we made maintaining access to the device in somewhat fragile, whereas it should be robust and don't depend on the user's actions.

uci-defaults is just a part of the firmware which should be preserved, until we don't explicitly set to not to.