Opkg update fails: uclient-fetch cannot handle redirects when a proxy is used

batchman · November 24, 2018, 12:47pm

Hi,

I have a sub net in which attached devices do not have access to the internet.
In other words, the OpenWrt router rejects packets from the LAN zone to the WAN zone (= the outer zone).
This is to enforce the usage of application level gateways including squid as a web proxy.

This works fine except for opkg when run on another OpenWrt device in that sub net.
The http_proxy option is set correctly in /etc/opkg.conf, and the environment variable http_proxy is set correctly as well.

$ grep http_proxy /etc/opkg.conf
option http_proxy http://web-proxy:3128

$ echo $http_proxy
http://web-proxy:3128

$ opkg update
Downloading http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/base/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/base/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/packages/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/packages/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/routing/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/routing/Packages.gz

Downloading http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/telephony/Packages.gz
*** Failed to download the package list from http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/telephony/Packages.gz

Collected errors:
 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/base/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/luci/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/packages/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/routing/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

 * opkg_download: Failed to download http://downloads.lede-project.org/releases/17.01.4/packages/mips_24kc/telephony/Packages.gz, wget returned 4.
 * opkg_download: Check your network settings and connectivity.

I have tracked down the problem, and my conclusion so far is that uclient-fetch does not handle HTTP redirects correctly when a proxy is used.

So a direct connection is expected to fail:

$ unset http_proxy
$ uclient-fetch http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Downloading 'http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz'
Connecting to 148.251.78.235:80
Connection error: Connection failed

OTOH with a proxy, this should work, but it does fail:

$ http_proxy=http://web-proxy:3128 uclient-fetch http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packa
ges/Packages.gz
Downloading 'http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz'
Connecting to 192.168.178.25:3128
Redirected to /releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz on downloads.openwrt.org
Connection error: Connection failed

So it seems that the redirected request is not routed through the proxy.

For comparison, on a Raspberry Pi in the same sub net, a direct connection is expected to fail:

$ LANG=en wget http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
--2018-11-24 13:34:05--  http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Resolving downloads.lede-project.org (downloads.lede-project.org)... 148.251.78.235, 2a01:4f8:202:43ea::3
Connecting to downloads.lede-project.org (downloads.lede-project.org)|148.251.78.235|:80... failed: Connection refused.
Connecting to downloads.lede-project.org (downloads.lede-project.org)|2a01:4f8:202:43ea::3|:80... failed: Network is unreachable.

On a Raspberry Pi in the same sub net, with a proxy, this is expected to work, and it does work:

$ http_proxy=http://web-proxy:3128 LANG=en wget http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz 
--2018-11-24 13:38:14--  http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Resolving web-proxy (web-proxy)... 192.168.178.25
Connecting to web-proxy (web-proxy)|192.168.178.25|:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://downloads.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz [following]
--2018-11-24 13:38:14--  http://downloads.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Reusing existing connection to web-proxy:3128.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://archive.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz [following]
--2018-11-24 13:38:14--  http://archive.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz
Reusing existing connection to web-proxy:3128.
Proxy request sent, awaiting response... 200 OK
Length: 63217 (62K) [application/octet-stream]
Saving to: 'Packages.gz'

Packages.gz         100%[===================>]  61.74K  --.-KB/s    in 0.05s   

2018-11-24 13:38:14 (1.20 MB/s) - 'Packages.gz' saved [63217/63217]

Since the affected OpenWrt device only has 4 MB flash, I cannot use a (non busybox) wget.

I am using

$ opkg list-installed | grep uclient
libuclient - 2018-08-03-ae1c656f-1
uclient-fetch - 2018-08-03-ae1c656f-1

$ opkg list-installed | grep opkg
opkg - 2017-03-23-1d0263bb-1

$ cat /etc/openwrt_release 
DISTRIB_ID='LEDE'
DISTRIB_RELEASE='17.01.4'
DISTRIB_REVISION='r3560-79f57e422d'
DISTRIB_CODENAME='reboot'
DISTRIB_TARGET='ar71xx/generic'
DISTRIB_ARCH='mips_24kc'
DISTRIB_DESCRIPTION='LEDE Reboot 17.01.4 r3560-79f57e422d'
DISTRIB_TAINTS='no-all'

Any suggestions, hints? Is this really a bug?

Thanks in advance

hnyman · November 24, 2018, 1:07pm

Simply download the .ipk packages as files to your PC and then use scp to transfer them into router's /tmp and then use opkg to install them.

With a 4 MB flash device your best approach would be to compile a personal firmware with the needed packages included.

batchman · November 24, 2018, 5:42pm

Thanks for that idea. Yes, that would be possible. Looking at the many LuCI updates that's a large number of packages.

Sorry I forgot to mention that a have a workaround: A transparent proxy on the router and "no proxy settings at all" at the affected devices. This works, but it complicates the firewall setup.

Does anyone else have this problem? Can anyone confirm that it's really a bug in uclient-fetch?

jow · November 24, 2018, 5:44pm

I'll look into the uclient proxy issue. In the meanwhile you can replace lede-project.org with openwrt.org in the URLs to avoid the extra redirect.

Edit: I was able to reproduce the redirect-via-proxy error and pushed a fix to uclient in https://git.openwrt.org/?p=project/uclient.git;a=commitdiff;h=3ba74ebc9363993ea2a83fd73b4c1b1a96e73940

Updated binaries will be available within the next 24 hours. Use opkg update; opkg upgrade libuclient uclient-fetch to install them.

batchman · November 25, 2018, 2:58pm

Wow, thanks for that quick response!

It looks different now, yet uclient-fetch reports HTTP 400 (bad request)

$ uclient-fetch http://downloads.lede-project.org/releases/17.01.4/targets/ar71x
x/generic/packages/Packages.gz
Downloading 'http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz'
Connecting to 192.168.178.25:3128
Redirected to / on web-proxy
HTTP error 400

The corresponding squid access log:

1543157086.366     60 10.100.0.20 TCP_MISS/301 548 GET http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz - HIER_DIRECT/148.251.78.235 text/html
1543157086.371      1 10.100.0.20 TAG_NONE/400 2165 GET /releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz - HIER_NONE/- text/html

For comparison the squid log (using wget on Raspbian):

1543157265.719     63 10.100.0.76 TCP_MISS/301 548 GET http://downloads.lede-project.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz - HIER_DIRECT/148.251.78.235 text/html
1543157265.794     73 10.100.0.76 TCP_MISS/301 759 GET http://downloads.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz - HIER_DIRECT/148.251.78.235 text/html
1543157265.803      6 10.100.0.76 TCP_MEM_HIT/200 63668 GET http://archive.openwrt.org/releases/17.01.4/targets/ar71xx/generic/packages/Packages.gz - HIER_NONE/- application/octet-stream

so it looks like that the 2nd URL is missing the target host

Thanks again

batchman · December 4, 2018, 3:37pm

Sorry, have you had the chance to inspect the follow-up issue?

jow · December 4, 2018, 6:58pm

@batchman - sorry, not yet but I didn't forget about it. When testing my "fix" I was using a simple setup using tinyproxy which might not deal with redirects the same way squid does. I was planning to take a deeper look at the issue soon to figure out how to best deal with relative urls.