Building my package: Errors with make download

For Developers
1

Hi folks

So I'm trying to have the build system download the tarball for my package from my github repo. The package is not yet included in the OpenWrt repository.

I managed to have it actually fetch the tarball, however I'm not sure it's doing this correctly because it spits out some errors.

Here is the relevant portion of the Makefile:

PKG_NAME:=geoip-shell
PKG_VERSION:=0.5
PKG_RELEASE:=r1
PKG_LICENSE:=GPL-3.0-or-later
PKG_MAINTAINER:=antonk <email_redacted>
PKG_SOURCE_PROTO:=git
PKG_SOURCE_VERSION:=7e3f8efc4197cf72d11102bb6ade9b9225595ded
PKG_SOURCE_URL:=https://github.com/friendly-bits/geoip-shell-openwrt.git
PKG_MIRROR_HASH:=46386d4d08a331799ffe873d1519c4fded664b93bf69f2d59f0876adf771af83

Here is the relevant portion of the output for the command make -j1 V=s "package/geoip-shell/download":

mkdir -p /home/antonk/openwrt/dl
SHELL= flock /home/antonk/openwrt/tmp/.geoip-shell-0.5.tar.xz.flock -c '        /home/antonk/openwrt/scripts/download.pl "/home/antonk/openwrt/dl" "geoip-shell-0.5.tar.xz" "46386d4d08a331799ffe873d1519c4fded664b93bf69f2d59f0876adf771af83" "" || (  /home/antonk/openwrt/scripts/dl_github_archive.py --dl-dir="/home/antonk/openwrt/dl" --url="https://github.com/friendly-bits/geoip-shell-openwrt.git" --version="7e3f8efc4197cf72d11102bb6ade9b9225595ded" --subdir="geoip-shell-0.5" --source="geoip-shell-0.5.tar.xz" --hash="46386d4d08a331799ffe873d1519c4fded664b93bf69f2d59f0876adf771af83" || (   echo "Checking out files from the git repository..."; mkdir -p /home/antonk/openwrt/tmp/dl && cd /home/antonk/openwrt/tmp/dl && rm -rf geoip-shell-0.5 && [ \! -d geoip-shell-0.5 ] && git clone  https://github.com/friendly-bits/geoip-shell-openwrt.git geoip-shell-0.5 && (cd geoip-shell-0.5 && git checkout 7e3f8efc4197cf72d11102bb6ade9b9225595ded && git submodule update --init --recursive) && echo "Packing checkout..." && export TAR_TIMESTAMP=`cd geoip-shell-0.5 && git log -1 --format='\''@%ct'\''` && rm -rf geoip-shell-0.5/.git &&  tar --numeric-owner --owner=0 --group=0 --mode=a-s --sort=name ${TAR_TIMESTAMP:+--mtime="$TAR_TIMESTAMP"} -c geoip-shell-0.5 |     xz -zc -7e > /home/antonk/openwrt/tmp/dl/geoip-shell-0.5.tar.xz && mv /home/antonk/openwrt/tmp/dl/geoip-shell-0.5.tar.xz /home/antonk/openwrt/dl/ && rm -rf geoip-shell-0.5; );  )    '
+ curl -f --connect-timeout 20 --retry 5 --location https://sources.cdn.openwrt.org/geoip-shell-0.5.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   146    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
Download failed.
+ curl -f --connect-timeout 20 --retry 5 --location https://sources.openwrt.org/geoip-shell-0.5.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   146    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
Download failed.
+ curl -f --connect-timeout 20 --retry 5 --location https://mirror2.openwrt.org/sources/geoip-shell-0.5.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   153    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
Download failed.
No more mirrors to try - giving up.
geoip-shell-0.5.tar.xz: Download from https://github.com/friendly-bits/geoip-shell-openwrt.git failed
geoip-shell-0.5.tar.xz: Cannot fetch commit ts:
  /repos/friendly-bits/geoip-shell-openwrt/git/commits/7e3f8efc4197cf72d11102bb6ade9b9225595ded: HTTP Error 404: Not Found
  /repos/friendly-bits/geoip-shell-openwrt/commits/7e3f8efc4197cf72d11102bb6ade9b9225595ded: HTTP Error 404: Not Found
Checking out files from the git repository...
Cloning into 'geoip-shell-0.5'...
.....

Then it proceeds to correctly clone the repo and eventually the download completes successfully.

So for some reason, it tries to fetch from OpenWrt servers (where my package is not yet stored), then says that download from my github failed, and only then clones the repo. Is this normal and expected or did I do something wrong?

2

Partial download fails mostly happen at night and weekends when the servers are busiest.
Compete download fails usually happen when there is a DNS error.

Try it when most people are at work.

1 Like
3

It returns Error 404, which means "file not found". Which is expected because my package is not in the URL it's trying to download from. This is not a network error.

4

Oh and I meant to publish this under "For Developers", not sure how this ended up in Community Builds. Moved to the correct forum.

5

404 means you got to the server and the server cannot find it.

Since you are not typing in the URL, we can presume it is not a typo on your part.

1 Like
6

I don't understand why it's trying to fetch from the OpenWrt servers at all, since the URL I specified in the Makefile is the URL of my repo...

PKG_SOURCE_URL:=https://github.com/friendly-bits/geoip-shell-openwrt.git

7

Oh wait, I just realized that the repo is still private. :slight_smile: Probably this explains it.

8

Nope, changing to public didn't change the behavior.

9

You can search but you will find 404 means the server cannot find it and it is looking for the file in a mirror stored on an OpenWRT server.

Could be an indexing problem but it is 2240 EDT and the servers could just be very busy.

I don't know how GitHub and OpenWRT work out storage on OpenWRT centric information but I'm sure someone can explain why it is forwarding your request to an OpenWRT mirror.

10 
https://sources.cdn.openwrt.org/
https://sources.openwrt.org/
https://mirror2.openwrt.org/sources/

Is used as final fallback (to fulfill the GPL source requirements), if the primary download location failed (if the original download location went away), this obviously only works if those versions were ever encountered (and thereby stored) on the buildbots. For your private builds and/or before these particular versions are pushed to the official OpenWrt repos, those links will obviously fail - but on the other hand the buildsystem shouldn't even try them in the first place, if the primary location ( PKG_SOURCE_PROTO:=git and PKG_SOURCE_URL:=https://github.com/friendly-bits/geoip-shell-openwrt.git) succeeded.

2 Likes
11

Looks ok to me.

I is a git clone type of download, so there are no upstream archives to download. First choice is OpenWrt source archive, if not found there or the mirror hash of found files does not match, then it is cloned from upstream.

2 Likes
12

For some reason it tries the buildbot urls first. And somehow since I changed the repo visibility to public, it won't even download from it at all. It just says download failed, but then the .xz file magically appears in the dl directory. I suppose it's cached somewhere.

13

With what download date?

14

Good question. I'll check that later when I'm near the computer.

15

Keep in mind that main has switched to *.tar.zstd tarballs recently, so you may have to fix your sha256sum to match that.

1 Like
16

To my understanding, the build system (as installed in my computer) defaults to xz. Is there a way to override this to make it use zstd?

17

Just upgrade your (main openwrt.git) git clone to current main/HEAD and it will try zstd.

1 Like
18

Once the package is in the OpenWrt feed repo, (after the PR is accepted), buildbot will place the .zst archive to the OpenWrt source archive mirror and the download will happen from there. Until that the download will always fail, and the archive is cloned from Github. (unless there have been no changes and your own local copy in dl/ is ok. )

1 Like