Git repository timeout issues (git.openwrt.org)

Site Feedback and Other Questions
1

For the past several day https://git.openwrt.org has been pretty consistently responding with 504 errors. I have been using this to look for newly added devices to update the TOH. Is there an alternative with similar search features?

2

It is responding as I'm testing right now :blush:. Try again.

3

How are you searching? I've switched my buildroot to use github.com for the feeds, as they were the worst offenders for 503s and 504s, and I'm wondering if you could do your work using the github logs...

(Or maybe use something like https://downloads.openwrt.org/snapshots/.overview.json - I have one of my daily tests diff that file as it shows all the new targets/subtargets/profiles.)

4

I’m using it to search for the commits that added new devices that show up in the snapshot builds in order to add those to the TOH. Prior to early January I never had issues with this. Now I get a 504 Gateway Time-out multiple times for each search before it eventually responds. Just wondering if something has changed recently that is causing the time-outs.

5

A massive, continuous DDoS attack by an increasing number of competing 'ai' companies. Which, contrary to their mission, are not acting smart or intelligent (which would imply just cloning the git repos and doing their 'learning' locally), but which are web-scraping the gitweb interface in the most stupid way, following each and every link, for every single git commit, from thousands of IP addresses, every second of the day and evading rate limiting or blocking.

1 Like
6

Right, but what's your process. I suspect I could automate some of that for you, but need some details on what you look at and what data you need. My testing (mostly focused on owut functionality, but that overlaps with looking for new targets, subtargets and profiles) shows me a diff every time a new profile is added or changed.

Here's an example of what I saw when the HPE 1920 switch was added:

 7571         {
 7572             "id": "hpe_1920-16g",
 7573             "target": "realtek/rtl838x",
 7574             "titles": [
 7575                 {
 7576                     "model": "1920-16G (JG923A)",
 7577                     "vendor": "HPE"
 7578                 }
 7579             ]
 7580         },

Above stuff is run on my development OpenWrt device, and uses some tiny ucode scripts, but should be able to rewrite those bits to maybe python, so it would run on a Linux host device quite easily (or even windows, I suppose).

7

What I’m looking for is the actual git.openwrt.org link to the commit that added the device as that is what the Create New Dataentry Page is asking for. I can find most of the other information needed using the search features on GitHub but I haven’t figured out how that corresponds to the git.openwrt.org commit.

8

Aha, that should be pretty straightforward as the hashes are identical in all clones:

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=c1a9e35a77c1a4892d1fd89c11415ab579759f50
https://github.com/openwrt/openwrt/commit/c1a9e35a77c1a4892d1fd89c11415ab579759f50

So a script like this, run on desktop linux, would pop up the same commit on both repos:

hash=c1a9e35
xdg-open "https://github.com/openwrt/openwrt/commit/${hash}"
xdg-open "https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=${hash}"
9 
root@iZk1a00chdq8niw6ja82v4Z:~# git clone https://git.openwrt.org/openwrt/openwrt.git
Cloning into 'openwrt'...
remote: Enumerating objects: 5441, done.
remote: Counting objects: 100% (5441/5441), done.
remote: Compressing objects: 100% (5022/5022), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fetch-pack: unexpected disconnect while reading sideband packetfatal: early EOF
fatal: fetch-pack: invalid index-pack output
root@iZk1a00chdq8niw6ja82v4Z:~#

Updating feed 'base' from 'https://git.openwrt.org/openwrt/openwrt.git;v24.10.5' ...
Cloning into './feeds/base'...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
failed.
10

If you can’t (or won’t) use the GitHub mirrors, you can also use the git protocol instead of https.

git clone git://git.openwrt.org/openwrt/openwrt.git
11

If your using a script try the below code

sed -i 's|https://git.openwrt.org|git://git.openwrt.org|g' feeds.conf.default
sed -i '/video/s|git://|https://|g' feeds.conf.default
12

And there are also the new codeberg mirrors:

https://codeberg.org/openwrt/

2 Likes