Help navigating the git repos to chase down a bug

My apologies for this likely silly question... I'm not a developer and my knowledge of this area is not great (but I'd love to learn).

I am trying to chase down a bug (described here) that has appeared in 21.02.0, but was not present on any 19.07.x builds. It would stand to reason that the bug is caused by one or more commits to Master after the 19.07 branch was created and likely before the 21.02 branch started (although it could be anywhere up to 21.02.0). My plan would be to do a binary search on the snapshots between those dates to see if I can find the offending changes. However...

I'm not really that adept at dealing with the git revision history. How can I

  • figure out the exact dates of the branches for 19.07 and 21.02?
  • How can I then translate from the dates to the snapshot revision numbers in the master branch (or can I pull by date)?
  • and how can I then pull the desired daily snapshots?
  • Finally, I know that git works well grabbing deltas going forward, but can it also deal with deltas moving backwards in my binary search?

I have built from source before (although it has been a while), but I was building from a specific release version, not master snapshots

Pointers to some documentation and/or a quick tutorial would be handy... I found some documentation, but it didn't really answer the specific questions I have about digging back through the historical snapshots.

Finally, if there is a better way to approach this, I'm all ears. I have looked through the git commit history for changes that might be relevant, but I was unable to find anything obvious, so I figure just building snapshots and getting empirical data might be the best way to find the date and corresponding commit(s) at issue.

Thanks!

use-buildsystem#selecting_code_revision

2 Likes

Thanks @anon50098793 :slight_smile: . This helps me get started.

But what about figuring out the dates of the branches and then the revision hashes from the dates I'm interested in pulling? I see how to select the desired revision, but not how to calculate the dates of the branches and then the revision hash from there. I'm not seeing it in that doc (although I could be just missing it)

well... i'm sure someone more skilled than myself will have some better practical ideas for you...

personally, with my limited skillset... i'd be looking at major kernel/compiler bumps, switchchip patches, dts/probing changes etc...

a simple git log TARGETDIR | more on your targets DIR can do alot for seeing the overall dates that major changes took place and converting those into hashes... ( or nearby hashes to lookup nearby via the graphical git web interface )...

if something odd happened, you'll often see fixup patches nearby... keep in mind tho' that major kernel/compiler bumps may have fallout that was only noticed / patched on alternate targets/chipsets so may not show up in the above git log...

i suggest you pick a date and hash 1/4 between after 19 branch ( branch can be up to 3 months before first rc )... and 21 branch... and start there... after this 1/2 way...

2 Likes

Thanks. I'll chew on this a bit.

I haven't found anything with the searches I've done so far, but maybe some of the searches you are recommending might turn something up.

Thanks for the help so far. I may need a bit of hand-holding here and there, but my goal is to learn (not to be spoon-fed), so any and all guidance is welcome and appreciated.

1 Like

So I've been looking through the commits -- I haven't found any immediately obvious ones, but some that I might want to check out in more detail.

Now, the part I'm still missing is how to figure out the source revision and revision hash from a date. So if I want to select, just as an example, Feb 21, 2020, how do I figure out what revision/hash to use? The selecting hash section describes synchronizing everything to the date of the hash, but doesn't explain how to find the hash (or revision) by date in the first place. Similarly, this page talks about finding the has from the revision code, but still doesn't explain how to find the revision from the date. Sorry if I'm being dense here... and thanks in advance!


git log --since "FEB 2 2020" --until "FEB 23 2020" --pretty=format:"%h%x09%an%x09%ad%x09%s"
2 Likes

Thanks. I'll give that a shot!

google git bisect, fairly automated, but still a wee bit tedious. And given what you are chasing check the PRs RE qca8k

3 Likes

@anomeome - this is much better than anything I had in mind -- a semi-automated binary search will make life so much easier.

Meanwhile, I'm not sure if the qca8k is the actual search term I should be using (although I could be totally wrong)... the issue is present on the standalone port (usually used as the WAN/uplink) in these devices -- it is not connected to the switch chip. So on my RS Pro, eth0 is the wan port and it is the one that is connecting at 100Mbps instead of gigabit. eth1 on this device goes to the built-in 3 port switch and that connects at gigabit (both CPU > switch and switch > computer or other ethernet device).

I think that the search terms of interest are the 7161 (SoC) and/or 7100/71xx, excluding the many commits that are there to add support for new devices on the ath79 target (migrations from ar71xx targets previously).

Does that sound right?

Na, a case of TL;DR, qca8k would be about a throughput WIP, not port speed negotiation, not apropos.

Not quite sure. With WNDR3700, WNDR3700v2, WNDR3800, a widely-used device with the same chip (ath79 / 7161), there has been intermittent reports with devices being unable to use 1000 Mbit on WAN also on 19.07. I think that there are reports also from the ar71xx era, but I didn't look for them (also in the old forum).

E.g.

See also related discussion on the original ath79 transition:

(EDIT: I found one ancient discussion: https://forum.archive.openwrt.org/viewtopic.php?id=53261 )

19.07 was branched in June 2019:

21.02 in February 2021.

By looking manually at the commit history web pages is the quickest approach... :wink:

You need to compile them from scratch with the full toolchain. And each time first compile also the respective tools & toolchain, so clearing the cbuildroot with "make dirclean" is required after each git checkout, especially if you move backward to history.

Easiest might be if you first move backward to the 19.07 branching point, compile there and test, then move e.g. 3 months forward. That might enable you to not clear the toolchain always, as the build would update that automatically. Note that some musl & gcc change may go undetected, so rebuilding everything is important.

Note that you should also backdate all feed repos to the same date. Otherwise you will get strange results in LuCI etc. But if you only compile the main openwrt without GUI and any etxtra packages, just the minimal bootable firmware, that will help.

git checkout b344ca9ca68322 would move you to the 19.07 initial point.

Instead of blind git bisect approach, look into dates of crucial changes:

1 Like

Hi.
I have two 3700v2 : one is able to link WAN at 1Gbit/s (not 1000 MHz !), but one is still stuck to 100Mbit/s.
Both with 19.07 and 21.02. I have no clue why. Every wires works well, and the pins seem clean.

Thanks for all this really helpful info, @hnyman! Seriously awesome!

I bricked my RouterStation pro with my build + cherrypicked git commit (the tip on the interesting commit was provided by @dreamlayers who has been doing quite a bit of troubleshooting and adding some great info to the bug report itself). So now I'm going back to square one to figure out if the commit broke things or if I had something wrong with my self-compiled build. A fresh self-built image of 21.02.0 will be my first test, then I'll add that commit. If those function but the commit doesn't solve the problem, I'll probably follow the bisect suggestions above.

The sad prediction is that you will likely end up with a kernel bump, which means that you should then into all relevant changes in the upstream Linux kernel (which will be really difficult, if you end up e.g. thinking about all changes between major versions 4.14 --> 5.4 , taken the later upstream backportis into account... :frowning: )

(like suggested in the path mentioned in Possible 21.02.0 bug with RouterStation Pro ethernet speed - #9 by dreamlayers . That upstream
commit 92ed2eb7f4b7 means a Linux commit in early 2019. )

@hnyman - you're such a buzz-kill :rofl:.

I wonder if I should leave this to the people who actually know what they're doing. I figure if I can identify a commit that broke things, at least that helps reduce the effort for someone else. But it is possible this is going to be more involved (for me, a novice) than it is worth.

In all seriousness, I'm enjoying this opportunity to learn more of the nuts and bolts here, and I really appreciate the help so far!

I'll keep this thread updated with my toils!

I am not sure if you already read that link in my message above about the original ath79 transition? The section (~20 messages before, ~20 messages after the link location) contains relevant discussion about WNDR3800 link detection.

1 Like

I didn't read the link in detail, but I did see it. My particular device 'survived' the ar71xx to ath79 transition, and gigabit functionality was working perfectly on 19.07.x (ath79). It was the update to 21.02 that surfaced the issue. It is possible there was some latent bug in that target platform transition that actually became relevant with kernel bumps or some other minor changes -- if that's the case, I am absolutely not qualified to troubleshoot it. I was hoping for some low hanging fruit. Aside from the learning opportunity here, selfishly, I want this bug (that is probably not on the radar of most of the dev team) fixed... so if I can help, that's a benefit. It remains to be seen if I can actually make a dent on this :sweat_smile:

I'm happy to report success on both recompiling 21.02.0 for a functioning image and for the cherrypicked commit to solve the actual bug.

I sincerely appreciate the help from @anon50098793 , @anomeome , and @hnyman in helping me work with the build system and git. Luckily, I didn't actually need to go down the route of the git bisect method, but it is super useful to have learned a bit that tool, too!

Thanks again everyone!

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.