Scraper for X-posting newly added devices on the forum?

This one I can't do myself, not skilled enough ...

Would it be possible to create a scraper, posting all new device commits in a thread here at the forum ?

This is (IMHO) "needed" since there's (AFAIK) no connection between newly added devices, and the ToH.
If there's at least a post about it, it'd be possible to monitor the thread, and manually create the hwdata entry...

1 Like

Not a bad idea, I've noticed this problem as well. If the contributor does not create the datatable entry right away, it's not even visible that this device is already supported (The HP 1920G switch series is such an example, or the ZTE MF282 which was contributed by me and I simply didn't have the time/patience to create the dataentry).

A git commit hook would probably be another option.

1 Like

I would appreciate standardised commit messages for new devices.

To keep up to date I receive the OpenWrt rss feed from https://git.openwrt.org/?p=openwrt/openwrt.git;a=atom. Using Betterbird (A Thunderbird soft-fork), I created a filter that automatically sorts feed-messages that are likely newly added wifi 6 devices into a separate folder.

Here (most of) the wifi 6 filter:

Here my wifi 6 devices folder:

As you can see I use "add support for" as basis, but it's too generic, as it is also used to denote the introduction of a new feature.

Edit: Just to clarify. These filters will in the long run likely miss one or the other device, as it depends on committers to draft good commit-messages. E.g. my filters would not have captured following device, as it lacks the "add support for", while clearly being a new device:


But it's still better than nothing...

I would prefer commit messages to include something of the like:

  • add support for device
  • add new device
  • add device (too generic. Filters some messages that are not new devices).
  • ...

We also have the "new device" label at GitHub. But since the repo there is just a mirror and new check-ins will be merged via staging-trees, these pull-requests will all simply be closed, even when merged. At first glance, it is not clear if the pr was rejected or merged.

Maybe new automatic labels "merged" or "rejected" would help there.

I sat down today and tried to find a better pattern for keeping up with "new devices", but I just got crushed by reality. Here the adventure:

My first attempt was to filter for "hardware specifications:" in the commit message body, which soon became "specifications:", because not everybody bothers with hardware, which soon became "specifications", because not everybody bothers with the :, which soon became "specification", because not everybody bothers with the s.

Unfortunately, i found another commit that was not a new device, but with the word "specification" in the body. So i thought, why not add an additional condition to it? - Also filter for the word "RAM". Surely if its a new device, hardware details will be added to the commit message and every device will have a RAM, right? The end of that story was following javascript regex:

, which was pretty neat and yielded 25 non ax capable devices since January 2023. Could this have been the end of the story? Of course not! Noooo. Obviously there are those that refrain from posting the word "specification" at all in the commit message.

Rockchip for examle only mentions "hardware" in the body and has following subject header: "add Firefly ROC-RK3328-CC support": https://git.openwrt.org/?p=openwrt/openwrt.git;a=commitdiff;h=a7d6cf0bb3504cb0e81b2a4c484fed4dc3a5b4c5. So i was thinking of creating another complicated complex filter or filter with RegEx, maybe something along

but while on that path, i found other commits that are so hard to get my head around, i find it impossible to derive a working keyword filter without ending up marking completely arbitrary commits as well, mainly because the refrain from adding any hardware details.

Examples would be:

Currently, e-mail filters are a mess.
I think one of the following has to be the way forward:

  1. Enforce proper syntax in commit messages for "add new device" or "add support for device"
  2. (Somehow) create link between table of hardware (toh) and forum (i would not know how).
  3. newly added devices could be labeled / marked / added to a "new device" or "last added" database based on the particular files and specific content that was changed, similar to what labeler does, the github actions bot.

Be aware though, not even the "new devices" label on github captures all new devices. See here: mediatek: filogic: add Qihoo 360T7 support. Edit: That is because the "new devices" label has to be added manually and it is not triggered by a bot.

So, not only e-mail filters are a mess :frowning:

I had looked at this same issue about 2 years ago. I came to the conclusion that their was not enough consistency enforced in pull requests to accomplish the task. What might possibly work is to monitor for changes in https://downloads.openwrt.org/snapshots/targets and then do a git search for any new target file names.

1 Like

Looking at this again, I tried to heavily use the git search function, but I failed. Here are my (failed) attempts that come closest:

https://git.openwrt.org/?p=openwrt%2Fopenwrt.git&a=search&h=HEAD&st=grep&s=.dts
https://git.openwrt.org/?p=openwrt%2Fopenwrt.git&a=search&h=refs%2Fheads%2Fmain&st=grep&s=BOARDNAME%3A%3D

It must be possible somehow to get the shortlog of changes within a filepath, since there is git documentation for it: https://git-scm.com/docs/git-shortlog#Documentation/git-shortlog.txt---ltpathgt82308203. So following that idea, I searched for changes to .dts files under openwrt/openwrt.git/target/linux/. I tried multiple times with various syntax, but it does not find anything:
image. This gives doubt about the notion that this option is accessible on the website, but maybe my syntax was just wrong.

1 Like

Finally, i got inspired by https://stackoverflow.com/questions/11511894/monitoring-website-updates-in-realtime

Either you poll the site (and polling every page on Wikipedia seems like a big challenge) or you get them to tell you via some API

So, I would simply propose to add a rss feed to https://downloads.openwrt.org/snapshots/targets/ since this seems like the easiest solution. We could then link to the rss feed in the forum and maybe even make it a pinned post.

My last attempt was to approach the topic from the "forum" point of view:

I am not sure if the basic functionality of https://www.discourse.org/ covers periodic automated updates from external sources, but I found two promising plugins that might be able to do the job: