Very slow git clone

I am in the Philippines, on the fiber ISP named "Globe". I tried to use OpenWrt SDK today, and it failed due to network timeouts:

[aep@aep-haswell openwrt-sdk-24.10.1-mediatek-filogic_gcc-13.3.0_musl.Linux-x86_64]$ ./scripts/feeds update
Updating feed 'base' from 'https://git.openwrt.org/openwrt/openwrt.git;openwrt-24.10' ...
Cloning into './feeds/base'...
remote: Enumerating objects: 27561, done.
remote: Counting objects: 100% (27561/27561), done.
remote: Compressing objects: 100% (12838/12838), done.
error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
failed.
Updating feed 'packages' from 'https://git.openwrt.org/feed/packages.git^d8cd30f4e281d6853b3de134c4f147a807583e43' ...
Cloning into './feeds/packages'...
remote: Enumerating objects: 7349, done.
remote: Counting objects: 100% (7349/7349), done.
remote: Compressing objects: 100% (6028/6028), done.
error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 0
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
failed.
...

While git was doing its download job, the speed was oscillating between 10 KB/s and 1 MB/s, mostly staying on the low side.

I know that the server is in Germany, so quite far, and the networks between the two countries are quite lossy.

Dear infra admins: The situation can possibly be improved by switching the TCP congestion control algorithm to BBR on the servers. Could you please do it, unless it's already set up this way or there are reasons not to?

$ cat /etc/sysctl.d/99-bbr.conf 
net.ipv4.tcp_congestion_control=bbr

Here is my old blog post regarding this optimization: https://patrakov.blogspot.com/2018/02/a-case-of-network-throughput.html

Heh, but that would not close open connection...

Current non-event-based HTTP servers enforce a minimum transfer speed in order to mitigate the Slow Loris attack. So "a few seconds with zero bytes transferred" would be a good reason to close the connection server-side abruptly.

Besides, the complaint here is that, before the close, the connection struggled to keep even 10 KB/s transferred.

Although, on a second thought, the observed behavior is also consistent with a memory overuse by something else running on the same server + OOM.

Try git config core.compression 1 might jump over timeouts or server load (not involved with project's servers, just a blind guess from symptoms)

Probably won't help...

BBR can be single ended, so enabling on your end also works...

According to Google presentations of that time, it needs to be enabled on the sender size.

BBR won't solve the problem...

This is more of a CDN issue - there's options here I suppose...

If this git service has no defined service goals, whoever runs that server should start with doing that. If the person running that server doesn't want to do it, someone else with an interest should become the new administrator.

Onze that is resolved, one could include access from such countries as a goal oe a non-goal.

If it's considered a goal, then it could be resolved. Given the lack of response of anyone identifying as the administrator, it suggests nobody is there.

It seems like a reasonable request to me.
Having said that, if there is a huge demand, you could always find other users willing to setup e.g. a cached server in China at a cloud company. That way you can solve the problem without depending on anyone from OpenWrt.

1 Like

Sounds cool, will you foot the bill?

2 Likes

For what should I foot the bill? I identified various steps. The first parts are entirely political in nature. If nobody in control of the OpenWrt.org domain wants anything, then it's obvious that OP needs to find other solutions.

A bill only comes in play once there is an implemented solution, unless you meant that even designing a proper server configuration comes with a bill.

I don't live in Unconnectia.

AI nonsense? I can assure you no AI was involved. Please stop the insults.

Also, I don't have any problem. I am just trying to see whether there's an administrator anywhere to help OP.

Couple of things...

  1. use the github mirror to clone against...

git clone https://github.com/openwrt/openwrt.git

  1. this is for the maintainers - feeds still pull against git.openwrt.org, so the feeds are not mirrored...

Not sure who to reach out to about the feeds thing, but if this can be addressed, this should sort out most of the concerns regarding slow updates/clones...

Regarding AI - one of the scourges of the Web Space at the moment is all of the AI companies grabbing anything and everything they can to train their models, and they really don't honor the robots.txt files - I had a small family site to host private conversations, and I've had to take it offline due to this kind of activity...

1 Like

I believe this issue will disappear soon:

Then https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem (and anywhere else git.openwrt.org is specified) should be updated to instead specify Github.

This has been a problem for over half a decade: Clone git.openwrt.org slow

Fortunately it looks like one can now log into the docs wiki with Github, we will see if I can fix it: :crossed_fingers:

Edit: fixed. Whoever made it possible to edit the docs with a Github account has truly benefited humanity. I will now fix problems wherever I notice them.

This is a bad solution, because there's nothing stopping GitHub.com from hosting something entirely different from the original domain, AFAIK.

So, whoever is responsible for this, clearly doesn't know what they are doing, which in turn raises the question as to why they take on the responsibility for something they can't.

I will also note that no actual problem solving has happened.

2 Likes