Can't find package lists after upgrading to 19.07

BusyBox v1.30.1 () built-in shell (ash)

OpenWrt 19.07.0, r10860-a3ffeb413b

root@fb4040:~# opkg update
Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/packages/Packages.gz
Failed to establish connection
*** Failed to download the package list from http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/packages/Packages.gz

Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/kmods/4.14.162-1-fa00c1231ac7d7840ec6ffe62dcad926/Packages.gz
Failed to establish connection
*** Failed to download the package list from http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/kmods/4.14.162-1-fa00c1231ac7d7840ec6ffe62dcad926/Packages.gz
...
etc...

I just re-flashed 18.06.5 again, and everything's working again as it is supposed to. I think I'll rather stay with release 18 until the first maintenance release 19.07.1 is out.

Are you running IPv4, IPv6, or dual stack? You might have an if you have a wan IPv6 in the interfaces list, try removing it. It is possible that the system is trying to use IPv6 but not actually getting a true dual-stack, so it fails.

That sounds reasonable, but if that is true, shouldn't version 18 also fail after flashing?

I flashed the image, not keeping settings. Then I ssh into the router and first thing I do is

opkg update

That's the point where I get the error in version 19. If what you say is true, the error should also occur in version 18. Unless, there have been some important changes.

I've had some time to spend today, so I tried again. And I failed again. :frowning:

This is the result after flashing, rebooting and restoring my network settings:

root@fb4040:~# opkg update
Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/packages/Packages.gz
Failed to establish connection
*** Failed to download the package list from http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/packages/Packages.gz

Downloading http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/kmods/4.14.162-1-fa00c1231ac7d7840ec6ffe62dcad926/Packages.gz
Failed to establish connection
*** Failed to download the package list from http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/kmods/4.14.162-1-fa00c1231ac7d7840ec6ffe62dcad926/Packages.gz

Downloading http://downloads.openwrt.org/releases/19.07.0/packages/arm_cortex-a7_neon-vfpv4/base/Packages.gz
Failed to establish connection
*** Failed to download the package list from http://downloads.openwrt.org/releases/19.07.0/packages/arm_cortex-a7_neon-vfpv4/base/Packages.gz
...
...
Collected errors:
 * opkg_download: Failed to download http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.openwrt.org/releases/19.07.0/targets/ipq40xx/generic/kmods/4.14.162-1-fa00c1231ac7d7840ec6ffe62dcad926/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.
...
...

But I also get these results:

root@fb4040:~# nslookup downloads.openwrt.org
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:      downloads.openwrt.org
downloads.openwrt.org   canonical name = mirror-02.infra.openwrt.org
Name:      mirror-02.infra.openwrt.org
Address 1: 176.9.48.73
downloads.openwrt.org   canonical name = mirror-02.infra.openwrt.org
Address 2: 2a01:4f8:150:6449::2

root@fb4040:~# traceroute downloads.openwrt.org
traceroute to downloads.openwrt.org (176.9.48.73), 30 hops max, 38 byte packets
 1  gateway.localnet  (1.1.0.1)  0.637 ms  0.560 ms  0.768 ms
 2  *  *  *
 3  xxx.zzz.yyy  (a.b.c.d)  15.516 ms  17.971 ms  14.324 ms
 4  ...   21.217 ms  15.529 ms  20.792 ms
 5  ...  48.470 ms  17.503 ms  18.248 ms
 6  ...  22.115 ms  26.973 ms  19.068 ms
 7  ix-ae-27-0.tcore1.fr0-frankfurt.as6453.net (195.219.50.106)  17.373 ms  15.977 ms  20.826 ms
 8  195.219.219.10 (195.219.219.10)  32.173 ms  19.312 ms  25.706 ms
 9  core22.fsn1.hetzner.com (213.239.245.17)  26.795 ms  core21.fsn1.hetzner.com (213.239.245.13)  20.854 ms  27.898 ms
10  ex9k1.dc6.fsn1.hetzner.com (213.239.229.90)  22.440 ms  27.093 ms  28.567 ms
11  mirror-02.infra.openwrt.org (176.9.48.73)  26.510 ms  25.901 ms  21.827 ms


root@fb4040:~# ping downloads.openwrt.org
PING downloads.openwrt.org (176.9.48.73): 56 data bytes
64 bytes from 176.9.48.73: seq=0 ttl=54 time=21.680 ms
64 bytes from 176.9.48.73: seq=1 ttl=54 time=25.885 ms
64 bytes from 176.9.48.73: seq=2 ttl=54 time=23.363 ms
^C
--- downloads.openwrt.org ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 21.680/23.642/25.885 ms

I deleted some server names, but everything else you see is what OpenWRT 19.07 throws at me, and I really can not make any sense of this. Every thing works: my VLAN settings, name resolving, connecting to the outside world.

Why can the system reach the download server via traceroute, while at the same time it can't download the package lists?

My guess is that you have a misconfigured IPv6. You have DNS for it, but maybe not routing and connectivity. IPv6 is preferred, so wget tries to download with it and fails.

Either disable IPv6 DNS, of fix the connectivity.

You might be right, because I also don't use ipv6, but every network has had this option set before the update and afterwards:

option ipv6 '0'

But to make sure I will give it another try and add

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

to /etc/sysctl.conf. That should disable ipv6 completely.

You are right: there is something fishy going on: wget wants to use ipv6, even after I disabled ipv6 in sysctl.conf. If I use option "-4" to wget, it starts downloading.

So, two questions remain:

  1. How do I turn off ipv6 completely, when those entries in /etc/sysctl.conf.d/10-default_conf don't work, and if every network is added the open option ipv6 '0' in /etc/config/network?

  2. What has changed since version 18.06 concerning wget? Like I told before: version 18 did not have this problem!

You should probably disable the IPv6 related items in DHCP config /etc/config/dhcp . Also network config has ipv6 assignment stuff.

Those sysctl things are about actual low-level IPv6 routing and traffic, while your problems are with much higher level tools and config.

Which options are to be disabled in /etc/config/dhcp? I thoroughly looked into that file, but I can not find any option connected to ipv6.

Here are some for you from the standar network config. (But naturally we do not know, what you have in your own config)

config dhcp 'lan'
option interface 'lan'
option start '100'
option limit '150'
option leasetime '12h'
option force '1'
option dhcpv6 'server'
option ra 'server'

config dhcp 'wan'
option interface 'wan'
option ignore '1'

config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/hosts/odhcpd'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'

You could also disable the odhcpd service that takes care of the IPv6 assignments on LAN.

Ps. I have never really tried to disable IPv6, as I have been eagerly using the IPv6 since etc. since 2010, already well before I had proper IPv6 connectivity from ISP. So, my practical knowledge about really disabling it is negligible.

No offence meant, but that example doesn't make sense to me: there's the option

option dhcpv6 'server'

which means, ipv6 is active. Isn't it?

This gets more and more bizarre. Even if I put

176.9.48.73 downloads.openwrt.org mirror-02.infra.openwrt.org

into /etc/hosts, wget is not capable of downloading anything, unless I use the option "-4" on command line.

Also putting

prefer-family = IPv4

into a file /etc/wgetrc doesn't do the job, although it should make wget use ip4, according to the manual.

For today I am fed up with trying. :frowning: I will re-flash version 18 again and I will open an official bug report for this. I do not believe that this malfunction is all my fault.

1 Like

I meant that remove those, as they are IPv6 related.

Those option have never been activated, and every network was set to disable anything that is related to ipv6.

While I am typing theses lines, opkg on 18.06.6 is downloading the package lists and reinstalling additional packages without any error.

It is jinxed. :frowning:

As a test - why not try changing domain to IPv4 address in /etc/opkg/distfeeds.conf?

sed -i 's/downloads.openwrt.org/176.9.48.73/g' /etc/opkg/distfeeds.conf && opkg update

If this works - another argument for an IPv6 issue.

Please don't feel insulted, but I don't think another test is necessary. There is a problem with ipv6, I see that now. But the big question is: what's the reason for this problem? There has to be a small, but very important difference between version 18 and 19, that causes wget not being able to download, unless the command line option "-4" is used. And that option really has to be the command line option, it is even disregarded as part of a config file.

Maybe wget has been compiled incorrect, and the use of ipv6 has been hard-coded in, without any alternative. I don't know, but it would explain why this error even occurs right after flashing, without restoring and previously backed up options. You see: that is the part that puzzles me the most. If it happened after I restored my network settings, the case would be obvious. But the error even happens on the "naked" system, without any option altered or restored.

In other words: after flashing 18 and resetting, "opkg update" works, flashing 19 and resetting "opkg update" does not work. Same situation, just two different OS versions. I would say that limits the number of suspects very much.

Does ip -family inet6 route list show any IPv6 routes?

Definitely remove the option ipv6 server and option ra server from the DHCP configuration for an IPv4 only network. But I don't think this will affect Internet access from OpenWrt itself.

Eureka!!! :slight_smile:

I found the error. Or at least I found a way to circumvent it:

Because the trouble only happened when wget was involved (see above: traceroute, ping and dns was working), I came to the idea that it might be worth it to try another version of wget, So I downloaded the package wget-nossl from the repository, then I flashed version 19, without keeping settings, and then I copied the downloaded package onto the router, installed it and -voila!- suddenly opkg could download the package lists and additional packages.

Once wget was working, I replaced it with the ssl version. I couldn't install the ssl version before, because it has too many dependencies.

Even now, after applying all the necessary networking settings and installing all the required additional packages, I can still reproduce this error: as soon as I uninstall the package wget-ssl, opkg can't download anything at all again. If I re-install the package again, opkg can download again.

I can only think of one explanation of for this: the busybox version of wget is faulty and does not work with ip4, unless given "-4" as a command line argument. :frowning:

It is probably actually the OpenWrt specific wget clone, uclient-fetch, that misbehaves. (It masquerades as the default wget.)

And it works nicely with plain IPv4, but somehow you have IPv6 addressing info but no proper connectivity, and that probably confuses it.

But why does this not happen when I flash the previous version? In OpenWRT 18 the wget clone works perfectly from the beginning.

Another point: it is just this wget clone, that misbehaves. See above: traceroute, ping and name resolving works.

And last, but not least: how can there be any ipv6 info, when I turn off ipv6 on any and each network device?