You know reading that I wonder if it's some temperature related problem since it's only after a large amount of data has been transferred. Would be interesting to check soc temp when the problem happens.
Thank you for clarifying this. Can anyone who has used the workaround confirm that this resolves the packet corruption?
Unfortunately, I am 200% swamped with work this week. I will try to find the instructions and PM them to you, or just post them here.
2 Likes
Hi, any updates for corruption?
I think the process of collecting the data dump with "packet corruption" log (as described by @patrakov) might take some time. After that, identifying the root cause of the issue could also take some time...
Thank you! Please let me know if I can assist you with your initiative. I have ~500 GB of free SSD space and a Windows 11 machine capable of running Ubuntu 24.10 via WSL (Windows Subsystem for Linux).
Yes, these are the almost-correct instructions:
The incorrect pieces are:
- The successful reproducer used rsync over ssh, not scp. The transfer was from a cloud server to a PC in the LAN.
- I forgot to disable NIC offloads on the client and on the server, thus making the resulting PCAP files hard to interpret; on the other hand, this also made it easy to say "I definitely have not made any error reassembling the TCP stream".
Also, while reading the communication with @whitedd, I noticed another failing test, which might be more straightforward but which I forgot to post publicly:
- Get a large-capacity USB flash drive with some large files and connect it to the router, then mount it
- Install
rsyncon the router - Use the following command on a Linux PC in LAN to transfer the files from the flash drive:
rsync --av --progress root@192.168.1.1:/mnt/sda1/large-dir /home/user/some-dir
We did not collect PCAP files in this case.
I can perform this simple test this weekend and share the rsync output with -v flag while transferring files from the USB flash drive plugged into the Asus TUF-AX6000 to the Linux machine.
Q: Should I use multiple large files (~5-10 Gb each) or a single large file (~150-200 Gb)?
The rsync output is not needed for determining if the bug is fixed. Just a simple fact whether it fails or not, and whether it fails on the version from the time when the bug was originally reported (i.e., whether this test is valid in your environment).
PCAP files are what would be needed for root-causing the issue, but in this test, there is no place on the router that would be able to keep them and not be the bottleneck.
A single large file was used in the original cloud-based test. I don't know the details of the payload in the flash drive based test.
- fact: Transferred 205 GB within the local LAN from Asus TUF-AX6000.
- The "packet corruption" issue did not reproduce. Not failed.
Details:
Initial parameters: 512 GB M.2 NVMe SSD inserted into a USB-C 3.1 External Adapter.
3 files were created on it: 10 GB, 15 GB, and 180 GB (please ignore the 50 GB file as it was not tested). Laptop with Windows 11 where WLS with Ubuntu 24.04 LTS is installed.
1. Calculated SHA-256 checksums for the files on USB SSD drive:
> Files in USB SSD drive
root@HP-Zeus:/# ls /mnt/usb
10GB 15GB 180GB 50GB lost+found
root@HP-Zeus:/# ls -lh /mnt/usb
total 256G
-rwxrwxrwx 1 root root 10G Dec 6 12:57 10GB
-rwxrwxrwx 1 root root 15G Dec 6 12:57 15GB
-rwxrwxrwx 1 root root 180G Dec 6 12:58 180GB
-rwxrwxrwx 1 root root 50G Dec 6 12:57 50GB
drwx------ 2 root root 4.0K Dec 6 12:08 lost+found
root@HP-Zeus:/# sha256sum /mnt/usb/10GB
732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d /mnt/usb/10GB
root@HP-Zeus:/# sha256sum /mnt/usb/15GB
55a20aa6ffa6c4c931daa2fcf823213783ab13a8bd9e15e05fc93143b038c5c4 /mnt/usb/15GB
root@HP-Zeus:/# sha256sum /mnt/usb/50GB
ab743e145f643a1f6237b7390baf2e6edc71d83997f5bf4ed40d975fb50ba423 /mnt/usb/50GB
root@HP-Zeus:/# sha256sum /mnt/usb/180GB
4d2e78b39cc45da6458bfe7827429b831b4e72f1184d35d0350f584f1cd56dd6 /mnt/usb/180GB
root@HP-Zeus:/#
2. Inserted the USB SSD drive with files into the USB port on the Asus TUF-AX6000.
3. Transferred files from Asus TUF-AX6000 router to local Linux machine ( WSL with Ubuntu 24.04 LTS on Win11 machine,):
> Transferred files from the router to the Linux machine via rsync.
root@HP-Zeus:/#
root@HP-Zeus:/# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/10GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
10GB
10,737,418,240 100% 57.10MB/s 0:02:59 (xfr#1, to-chk=0/1)
sent 43 bytes received 10,740,039,774 bytes 58,849,533.24 bytes/sec
total size is 10,737,418,240 speedup is 1.00
root@HP-Zeus:/# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/180GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
180GB
193,273,528,320 100% 57.60MB/s 0:53:20 (xfr#1, to-chk=0/1)
sent 43 bytes received 193,320,714,337 bytes 60,309,067.03 bytes/sec
total size is 193,273,528,320 speedup is 1.00
root@HP-Zeus:/# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/15GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
15GB
16,106,127,360 100% 56.88MB/s 0:04:30 (xfr#1, to-chk=0/1)
sent 43 bytes received 16,110,059,615 bytes 58,054,269.04 bytes/sec
total size is 16,106,127,360 speedup is 1.00
root@HP-Zeus:/#
Verifing checksum for the copied files:
> SHA-256 checksum for the copied files
root@HP-Zeus:/# ls /mnt/d/folder
10GB 15GB 180GB
root@HP-Zeus:/# ls -lh /mnt/d/folder
total 205G
-rwxrwxrwx 1 root root 10G Dec 6 12:57 10GB
-rwxrwxrwx 1 root root 15G Dec 6 12:57 15GB
-rwxrwxrwx 1 root root 180G Dec 6 12:58 180GB
root@HP-Zeus:/# sha256sum /mnt/d/folder/10GB
732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d /mnt/d/folder/10GB
root@HP-Zeus:/# sha256sum /mnt/d/folder/15GB
55a20aa6ffa6c4c931daa2fcf823213783ab13a8bd9e15e05fc93143b038c5c4 /mnt/d/folder/15GB
root@HP-Zeus:/# sha256sum /mnt/d/folder/180GB
4d2e78b39cc45da6458bfe7827429b831b4e72f1184d35d0350f584f1cd56dd6 /mnt/d/folder/180GB
root@HP-Zeus:/#
So, the checksums of the files on the USB SSD drive and the files copied to the Linux PC are identical.
The "packet corruption" issue was NOT reproduced.
The test is scientifically invalid, as there is no proof that it works at all in your environment. Please downgrade to a known-buggy version from one year ago and retest. If it catches the bug there, then your current results will magically become valid.
Oh, man, believe me, I would like to make every effort to detect the issue and finally get it fixed. I have no intention of hiding anything. Please just help me provide as many details as needed for bug fixing.
Should I repeat my local test on the 23.05.2 OpenWrt version.
Or maybe it would be more beneficial to retry using these steps while keeping my current 23.05.5 OpenWrt version?
Please provide direction and assistance to help reproduce the issue together with @Llethur
I am not sure what would be more valuable.
I think that downgrading to 23.05.2 (assuming that this is where the other reports come from) would be the most beneficial first step, and then trying both methods to reproduce the issue there. This is because, without this anchor that validates the methodology, all testing done on 23.05.5 would be pointless.
1 Like
I just discovered that official support for the ASUS TUF-AX6000 started with version 23.05.3, which was released on March 25, 2024. However, the "packet corruption" issue was reported ~3 months earlier in December 2023. So, should the version 23.05.3 be still used to reproduce this issue, or it has not sense?
Also, are there any prerequisites to assume that the issue could occur only when the 2.5Gb port is used? I don’t have any 2.5Gb devices, so I’m using only a regular 1Gb lan1 port (the 2.5Gb lan5 port is not used in my home setup).
Thank you.
Yes, the issue only affects 2.5 GbE ports. If you don't have any other 2.5 GbE device, I suggest buying a USB adapter from WavLink.
This is the exact one which I recommend:
However, Lazada only works in the Philippines, so, if you need it, please find the equivalent one on a different shopping platform (e.g., Amazon). EDIT: added the link below.
Trying on 23.05.3 makes sense.
1 Like
I’m looking with hope at the Support for RTL838x-based managed switches topic, hoping that someday I can replace my 1Gb 5 ports managed Zyxel GS1200-5. Unfortunately, support has only been added for 2.5Gb routers with 10+ ports, which is excessive for my home VLAN setup.
I will downgrade my ASUS TUF-AX6000 to 23.05.3 and repeat the tests with the 2.5Gb LAN5 port (but only with a 1Gb connection. Again, I hope this makes sense, @patrakov ?).
On the other hand, the "packet corruption" issue has become less relevant to me, as it potentially affects the 2.5Gb lan5 port only, which I am not currently using in my home setup.
I don't know whether testing on a 2.5 GbE port but with only 1 Gbps link makes sense. However, it's a quick test, so it doesn't hurt to try, and if the test produces corruption, then there is no need to find a 2.5 GbE device.
Again, I don't (and won't) have this device, so I have to rely on reports from others. It might make sense to ask @deeddy @whitedd how they tested. Maybe they also have old snapshot builds, i.e., exactly what they tested.
Ok, we have clarified all the necessary requirements for reproducing the "packet corruption" issue:
- Utilize the 2.5Gb lan5 port with a 2.5Gb link.
- OpenWrt version 23.05.3 (or even Snapshot from Jan. 2024).
At this time, I will not proceed with further testing as my home network config lacks the necessary env. conditions (missing 2.5 Gb link), and the issue is not reproduced on the 1Gb port of router (potentially, only the 2.5Gb port of the ASUS TUF-AX6000 might be affected.).
- for those who interested in attempting to reproduce this issue with the 2.5Gb port, a clear starting point is available. You can refer to Method 1 and Method 2 for guidance.
note: there is also a possibility that the "packet corruption" issue has already been resolved, as the "packet corruption" issue has been initially reported on the Snapshot version of the ASUS TUF-AX6000. It may no longer exist in the Stable OpenWrt versions.
Anyway, feel free to investigate or report any new incidents (if any).
Thank you.
1 Like
ok, here are the test results for the 2.5GbE lan5 port with a 1Gbps link on the ASUS TUF-AX6000 running stable OpenWrt firmware ver. 23.05.5 (I didn't test it on the older 23.05.3 version or the Jan. 2024 Snapshot, where the issue was initially reported.).
- fact: Transferred 205 GB within the local LAN from Asus TUF-AX6000.
- The "packet corruption" issue did not reproduce. Not failed.
Test Results with Details
- Initial parameters: 512 GB M.2 NVMe SSD inserted into a USB-C 3.1 External Adapter.
3 files were created on it: 10 GB, 15 GB, and 180 GB (please ignore the 50 GB file as it was not tested). Laptop with Windows 11 where WSL with Ubuntu 24.04 LTS is installed.
1. Calculated SHA-256 checksums for the files on USB SSD drive:
> Files in USB SSD drive
root@HP-Zeus:/# ls /mnt/usb
10GB 15GB 180GB 50GB lost+found
root@HP-Zeus:/# ls -lh /mnt/usb
total 256G
-rwxrwxrwx 1 root root 10G Dec 6 12:57 10GB
-rwxrwxrwx 1 root root 15G Dec 6 12:57 15GB
-rwxrwxrwx 1 root root 180G Dec 6 12:58 180GB
-rwxrwxrwx 1 root root 50G Dec 6 12:57 50GB
drwx------ 2 root root 4.0K Dec 6 12:08 lost+found
root@HP-Zeus:/# sha256sum /mnt/usb/10GB
732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d /mnt/usb/10GB
root@HP-Zeus:/# sha256sum /mnt/usb/15GB
55a20aa6ffa6c4c931daa2fcf823213783ab13a8bd9e15e05fc93143b038c5c4 /mnt/usb/15GB
root@HP-Zeus:/# sha256sum /mnt/usb/50GB
ab743e145f643a1f6237b7390baf2e6edc71d83997f5bf4ed40d975fb50ba423 /mnt/usb/50GB
root@HP-Zeus:/# sha256sum /mnt/usb/180GB
4d2e78b39cc45da6458bfe7827429b831b4e72f1184d35d0350f584f1cd56dd6 /mnt/usb/180GB
root@HP-Zeus:/#
2. Inserted the USB SSD drive with files into the USB port on the Asus TUF-AX6000.
3. Transferred files from the ASUS TUF-AX6000 router to a local Linux machine (WSL running Ubuntu 24.04 LTS on a Win11 laptop):
> Transferred files from the router to the Linux machine via rsync.
root@HP-Zeus:~# date
Sat Dec 7 09:42:19 CET 2024
root@HP-Zeus:~# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/10GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
10GB
10,737,418,240 100% 57.56MB/s 0:02:57 (xfr#1, to-chk=0/1)
sent 43 bytes received 10,740,039,774 bytes 58,849,533.24 bytes/sec
total size is 10,737,418,240 speedup is 1.00
root@HP-Zeus:~# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/15GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
15GB
16,106,127,360 100% 57.59MB/s 0:04:26 (xfr#1, to-chk=0/1)
sent 43 bytes received 16,110,059,615 bytes 58,903,325.99 bytes/sec
total size is 16,106,127,360 speedup is 1.00
root@HP-Zeus:~# rsync -av --progress -e ssh root@192.168.1.1:/mnt/usb/180GB /mnt/d/folder
root@192.168.1.1's password:
receiving incremental file list
180GB
193,273,528,320 100% 56.74MB/s 0:54:08 (xfr#1, to-chk=0/1)
sent 43 bytes received 193,320,714,337 bytes 59,364,567.60 bytes/sec
total size is 193,273,528,320 speedup is 1.00
root@HP-Zeus:~#
4. Verifying the checksums of the copied files:
> SHA-256 checksum for the copied files
root@HP-Zeus:~# ls /mnt/d/folder
10GB 15GB 180GB
root@HP-Zeus:~# ls -lh /mnt/d/folder
total 205G
-rwxrwxrwx 1 root root 10G Dec 6 12:57 10GB
-rwxrwxrwx 1 root root 15G Dec 6 12:57 15GB
-rwxrwxrwx 1 root root 180G Dec 6 12:58 180GB
root@HP-Zeus:~# sha256sum /mnt/d/folder/10GB
732377e7f4a2abdc13ddfa1eb4c9c497fd2a2b294674d056cf51581b47dd586d /mnt/d/folder/10GB
root@HP-Zeus:~# sha256sum /mnt/d/folder/15GB
55a20aa6ffa6c4c931daa2fcf823213783ab13a8bd9e15e05fc93143b038c5c4 /mnt/d/folder/15GB
root@HP-Zeus:~# sha256sum /mnt/d/folder/180GB
4d2e78b39cc45da6458bfe7827429b831b4e72f1184d35d0350f584f1cd56dd6 /mnt/d/folder/180GB
root@HP-Zeus:~#
So, the checksums of the files on the USB SSD drive and the files copied to the Linux PC are identical.
- The "packet corruption" issue was NOT reproduced on the latest stable OpenWrt 23.05.5 for the 2.5GbE lan5 port with a 1Gbps link.
Anyone is welcome to attempt replicating the test results for the 2.5GbE lan5 port using a 2.5Gbps link:
1 Like
Has anyone tried sysupgrade from 23.05.5 to the newest OpenWrt 24.10.0-rc2 on the Asus TUF-AX6000?
Do we need to be concerned about specifics mentioned for the Asus TUF-AX4200?
Does this confirm that if using 1Gbps link devices only on all LAN ports then packet corruption won't be an issue? I've received a delivery of the router so deciding whether I keep or return it!