KVM DOES NOT WORK AT ALL due to 6.x kernel bug, affects OpenWRT x86/64 snapshots

For Developers
1

Just a heads up: I've installed latest OpenWRT snapshot (x86/64) on some KVM-based virtual machines to act as a router.

I've noticed ABYSMAL network performance from OpenWRT, should have been gigabit, it was like 20Mbps.

The dmesg spewed out those messages constantly:

[50290.335674] eth0: bad gso: type: 1, size: 1240
[50290.336562] eth0: bad gso: type: 1, size: 1240

After some googling, I've discovered this is a serious kernel bug that is still around:

https://bugzilla.kernel.org/show_bug.cgi?id=219129

This happens every time KVM/virtio-net is involved.

I've solved it by running latest stable OpenWRT 23.05, that uses kernel 5.15 and it's NOT AFFECTED by the bug.

2

Do you mean kernel 6.1 or 6.6 because we doesn’t have a 6.4.
And 6.4 is right in the middle so it is hard to guess which one you mean.

1 Like
3

Let's assume 6.6 because x86/64 snapshots turned to 6.6 several months ago.
Probably a typo error from OP.

1 Like
4

All kernels 6.x are affected as far as I know. And yes, it was a typo, corrected.

A patch is in the works, but for some reason has not reached upstream yet.

1 Like
5

A serious bug indeed to any virtualization.
Bug reports states MTU 9000. Same result with a more classical MTU?

6

Pretty much any interface virtio-net based. Even some virtual machines with Alpine Linux were affected, kernel 6.10.3 (edge).

Rather painful, but I think will be fixed any time now.

7

so it's not really an openwrt issue ?

1 Like
9

No, it's not an OpenWRT bug, but a kernel/virtio-net issue.

I can confirm that kernel 5.15.x is NOT affected i.e. OpenWRT 23.05.x in a KVM environment works beautifully.

10

It is a little interesting that this hasn’t been noticed on kernel 6.1 for over a year on OpenWrt snapshot.

11

If the discussion in the Bugzilla thread is to be believed, the issue is caused by e269d79 which got backported to 6.1 as 5b19974 and 6.6 as 90d41eb on the 3rd of August, so not even 2 weeks. The fix in 89add40 hasn't been backported tho, which would've fixed the issue.

3 Likes
12

Thanks, that clarifies the problem: I have several other Alpine Linux machines with kernel 6.x on KVM and never had this issue.

It appeared last week, testing wireguard links (that need to have a smaller MTU, 1420 by default) on OpenWRT snapshot virtualized via QEMU/KVM.

Gso starts complaining in dmesg and packets get dropped.

1 Like
13
14

workaround:

echo options virtio_net gso=0 >> /etc/modules.conf

Then rmmod/modprobe or reboot.

3 Likes
15

Also worth mentioning outer virtio and kernel versions.

16

The fix should be in 6.6.46

Confirmed that the error is gone once https://github.com/openwrt/openwrt/pull/16173 is merged.

4 Likes