Hi, I’m using OpenWRT in a Proxmox VM and I’m facing some issue
I’ve tried to pinpoint the issue and it seems to be caused by the use of VirtIO NICs.
The issue is only happening to traffic to the WAN (egress) incoming traffic from WAN is working fine.
To keep this post easy to read I have provided some description of the problem over here:
doc_udp_loss_egress.md
# UDP Packet Loss / Malformed Packet Investigation
## 1. Scope and Context
This document describes the investigation of a **UDP packet loss / corruption issue** in a virtualized networking environment.
While `iperf` is used as a **controlled and repeatable test case**, the issue is **not limited to iperf**.
Similar behavior has been observed in **other UDP-based applications**. `iperf` is used solely to demonstrate and quantify the issue.
show original
If anyone have any hint to fix/troubleshoot this further.
brada4
January 19, 2026, 8:33pm
2
(x86 pc or VM is expected to forward gigabit without breaking the sweat)using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button (red circle; this works best in the 'Markdown' composer view in the blue oval):
Remember to redact passwords, VPN keys, MAC addresses and any public IP addresses you may have:
ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
cat /proc/net/softnet_stat
ethtool -S eth1
ethtool -S eth0
Here you go
root@OpenWrt:~# ubus call system board
{
"kernel": "6.12.63",
"hostname": "OpenWrt",
"system": "Common KVM processor",
"model": "QEMU Standard PC (Q35 + ICH9, 2009)",
"board_name": "qemu-standard-pc-q35-ich9-2009",
"rootfs_type": "ext4",
"release": {
"distribution": "OpenWrt",
"version": "25.12.0-rc2",
"firmware_url": "https://downloads.openwrt.org/",
"revision": "r32429-d76c64ad00",
"target": "x86/64",
"description": "OpenWrt 25.12.0-rc2 r32429-d76c64ad00",
"builddate": "1767653330"
}
}
Network config:
root@OpenWrt:~# cat /etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdf9:826f:0b55::/48'
option packet_steering '2'
option dhcp_default_duid '<redacted>'
config interface 'lan'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '64'
option ip6hint '1'
option ipaddr '192.168.1.254'
option device 'br-lan'
config interface 'public'
option proto 'static'
option netmask '255.255.255.0'
option ipaddr '192.168.100.254'
option ip6assign '64'
option ip6hint '100'
option device 'eth1'
config interface 'wan'
option proto 'dhcp'
option device 'eth2.101'
option auto '0'
config interface 'wan6'
option proto 'dhcpv6'
option device 'eth2.101'
option reqaddress 'try'
option reqprefix 'auto'
option peerdns '0'
list dns '2606:4700:4700::1111'
list dns '2606:4700:4700::1001'
list dns '1.1.1.1'
list dns '8.8.8.8'
list dns '4.4.4.4'
list dns '2001:4860:4860::8888'
list dns '2001:4860:4860::8844'
config device
option name 'eth2.101'
option type '8021q'
option ifname 'eth2'
option vid '101'
config device
option type 'bridge'
option name 'br-lan'
list ports 'bond-lacp'
list ports 'eth0'
config device
option type 'bonding'
option name 'bond-lacp'
option policy '802.3ad'
option min_links '1'
option ad_select 'bandwidth'
list ports 'eth3'
list ports 'eth4'
list ports 'eth5'
DHCP:
root@OpenWrt:~# cat /etc/config/dhcp
config dnsmasq
option domainneeded '1'
option localise_queries '1'
option rebind_protection '1'
option rebind_localhost '1'
option local '/lan/'
option domain 'lan'
option expandhosts '1'
option authoritative '1'
option readethers '1'
option leasefile '/tmp/dhcp.leases'
option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
option localservice '1'
option ednspacket_max '1232'
option confdir '/tmp/dnsmasq.d'
config odhcpd 'odhcpd'
option maindhcp '0'
option leasefile '/tmp/odhcpd.leases'
option leasetrigger '/usr/sbin/odhcpd-update'
option loglevel '4'
option piodir '/tmp/odhcpd-piodir'
option hostsdir '/tmp/hosts'
config dhcp 'lan'
option interface 'lan'
option start '100'
option leasetime '12h'
option dhcpv4 'server'
option dhcpv6 'server'
option ra 'server'
list ra_flags 'managed-config'
list ra_flags 'other-config'
option limit '149'
config dhcp 'wan'
option interface 'wan'
option ignore '1'
config dhcp 'public'
option interface 'public'
option start '100'
option limit '150'
option leasetime '12h'
option ra 'server'
root@OpenWrt:~#
Firewall:
root@OpenWrt:~# cat /etc/config/firewall
config defaults
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
option flow_offloading '1'
config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
list network 'VPN'
list network 'lan'
config zone
option name 'public'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
list network 'public'
config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wan'
list network 'wan6'
config forwarding
option src 'lan'
option dest 'wan'
config forwarding
option src 'public'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option src_ip 'fc00::/6'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config rule
option name 'Support-UDP-Traceroute'
option src 'wan'
option dest_port '33434:33689'
option proto 'udp'
option family 'ipv4'
option target 'REJECT'
option enabled '0'
config include
option path '/etc/firewall.user'
config redirect
option dest 'public'
option target 'DNAT'
option name 'IPERF-WAN'
option src 'wan'
option src_dport '8777'
option dest_ip '192.168.100.236'
option dest_port '8777'
list proto 'tcp'
list proto 'udp'
config redirect
option dest 'public'
option target 'DNAT'
option name 'IPERF-LAN'
option src 'lan'
option src_dport '8777'
option dest_ip '192.168.100.236'
option dest_port '8777'
list proto 'tcp'
list proto 'udp'
root@OpenWrt:~#
NICs data after a run of failing IPERF.
root@OpenWrt:~# cat /proc/net/softnet_stat
009db270 00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000 002d32d7 00000000 00000000 00000000 00000000 00000000
05e74436 00000000 00000088 00000000 00000000 00000000 00000000 00000000 00000000 01f0d861 00000000 00000000 00000001 00000000 00000000
0055c189 000003b8 00000003 00000000 00000000 00000000 00000000 00000000 00000000 00161a93 00000000 00000000 00000002 00000000 00000000
00980e60 00000000 00000411 00000000 00000000 00000000 00000000 00000000 00000000 00164539 00000000 00000000 00000003 00000000 00000000
root@OpenWrt:~# ethtool -S eth1
NIC statistics:
rx_drops: 0
rx_xdp_packets: 0
rx_xdp_tx: 0
rx_xdp_redirects: 0
rx_xdp_drops: 0
rx_kicks: 1
tx_xdp_tx: 0
tx_xdp_tx_drops: 0
tx_kicks: 31834
tx_tx_timeouts: 0
rx0_drops: 0
rx0_xdp_packets: 0
rx0_xdp_tx: 0
rx0_xdp_redirects: 0
rx0_xdp_drops: 0
rx0_kicks: 1
tx0_xdp_tx: 0
tx0_xdp_tx_drops: 0
tx0_kicks: 31834
tx0_tx_timeouts: 0
root@OpenWrt:~# ethtool -S eth0
NIC statistics:
rx_drops: 0
rx_xdp_packets: 0
rx_xdp_tx: 0
rx_xdp_redirects: 0
rx_xdp_drops: 0
rx_kicks: 1
tx_xdp_tx: 0
tx_xdp_tx_drops: 0
tx_kicks: 1
tx_tx_timeouts: 0
rx0_drops: 0
rx0_xdp_packets: 0
rx0_xdp_tx: 0
rx0_xdp_redirects: 0
rx0_xdp_drops: 0
rx0_kicks: 1
tx0_xdp_tx: 0
tx0_xdp_tx_drops: 0
tx0_kicks: 1
tx0_tx_timeouts: 0
root@OpenWrt:~#
brada4
January 19, 2026, 11:10pm
4
Install and enable irqbalance and disable packet steering. Also set CPU to Host-passthrough or host-model.
echo "options virtio_net gso=0" | tee -a /etc/modules.conf
will make virtio network more stable. Also setting them to same thread count as VM is of great help.
I'll try this tomorrow but this does not explain a more than 50% packet drop on a 15Mbps ratio.
What do you mean HW wise or SW wise? Its PVE 9.1.4.
Update:
No difference:
nicolo@modulus:~$ iperf3 -c <redacted> -p 8777 --udp -R --bitrate 15M
Connecting to host <redacted>, port 8777
Reverse mode, remote host <redacted> is sending
[ 5] local 192.168.43.95 port 37223 connected to <redacted> port 8777
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 1.08 MBytes 9.02 Mbits/sec 1.974 ms 530/1343 (39%)
[ 5] 1.00-2.00 sec 901 KBytes 7.39 Mbits/sec 1.673 ms 709/1374 (52%)
[ 5] 2.00-3.00 sec 843 KBytes 6.90 Mbits/sec 3.921 ms 672/1294 (52%)
[ 5] 3.00-4.00 sec 923 KBytes 7.56 Mbits/sec 2.018 ms 724/1405 (52%)
[ 5] 4.00-5.00 sec 885 KBytes 7.25 Mbits/sec 1.880 ms 699/1352 (52%)
[ 5] 5.00-6.00 sec 882 KBytes 7.23 Mbits/sec 1.953 ms 692/1343 (52%)
[ 5] 6.00-7.00 sec 882 KBytes 7.23 Mbits/sec 1.990 ms 701/1352 (52%)
[ 5] 7.00-8.00 sec 892 KBytes 7.31 Mbits/sec 1.844 ms 701/1359 (52%)
[ 5] 8.00-9.00 sec 888 KBytes 7.27 Mbits/sec 2.109 ms 690/1345 (51%)
[ 5] 9.00-10.00 sec 869 KBytes 7.12 Mbits/sec 2.135 ms 690/1331 (52%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.06 sec 18.0 MBytes 15.0 Mbits/sec 0.000 ms 0/0 (0%) sender
[ 5] 0.00-10.00 sec 8.86 MBytes 7.43 Mbits/sec 2.135 ms 6808/13498 (50%) receiver
iperf Done.
nicolo@modulus:~$
brada4
January 19, 2026, 11:38pm
6
virtualbox virtio totally glitches with default gso, unlikely PVC is any better.
1 Like
I performed the test before going to bed,
brada4
January 19, 2026, 11:53pm
8
Ok, hope it does not reboot in the meantime - any 50% drops in softnet_stat 2nd 3rd column? If not all the happines is outside OpenWrt (ethtool and softnet still applies)
Not sure what you need to see:
root@OpenWrt:~# cat /proc/net/softnet_stat
0000dc2e 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00006982 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000 00000000
001b4ce5 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000 00000000
0000670b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000003 00000000 00000000
root@OpenWrt:~#
brada4
January 20, 2026, 8:14am
10
All eyes on virtualisation.
_bernd
January 20, 2026, 9:32am
11
Besides gso deactivate all other offloads on the virt host.
But with local only traffic from:
Ubuntu VM > vmbr0 (virtio) > firewall(forwarding zones with dnat lan>public) > vmbr1 > LXC with iperf server
There are no dropped packets. What I dont get is why only WAN traffic have the issue.
Debian Machine > INTERNET (4G, GPON) > eth2 (Pcie passthrou) > firewall(forwarding zones with dnat wan>public) > vmbr1 > LXC with iperf server
I'm expecting that if virtualization virtio is broken then it is always broken.
Ubuntu VM > vmbr1 > LXC with iperf server
I don't get what you mean.
Uhm, you are talking about the host...
@brada4 So you wanted me to put this in the host not the VM?echo "options virtio_net gso=0" | tee -a /etc/modules.conf
_bernd
January 20, 2026, 9:49am
13
It's safe to disable these offloads on both sides.
Ps I'm lazy to dig and get you a proper source but this summary is not wrong: https://www.google.com/search?q=why+to+deactivate+gso+and+other+offloads
Maybe the host kernel just needs an update.
To test if it’s some incompatibility between host kernel and vm kernel you can try using an older version of Openwrt. A totally random suggestion to try: if the nics are Realtek don’t use the vendor driver if you are.
Are vlans involved in this ? Usually if the problem was with vlan filtering it wouldn’t work at all, so not much to suggest there.
Got it!
I thing I had this issue for log time, but now I'm starting to have mode UDP dependant software.
PVE Host is running:
root@pve:~# uname -r
6.17.2-1-pve
While Openwrt is on:
root@OpenWrt:~# uname -r
6.12.63
We are talking virtio NICs for the vmbrs no real NICs attached to those.
Uhm, they should not be, only WAN in on a vlan.
Below the PVE configuration for the Openwrt VM:
The PVE network birdges configuration:
Only the vmbr(0/1) are used to connect the VMs/LXCs to internet.LACP to main switch)
bond0 is a active-backup as I'm lazy and I dont remember what eth to plug for the host, an I hate doing 3 flight of stairs when I connect the wrong eth port.
Openwrt enables “gro fraglist” by default, you can try disabling it.
ethtool -K [interface name] rx-gro-list off
You can also perhaps experiment with ..
ethtool -K <interface_name> rx-udp-gro-forwarding on
1 Like
wilsonyan:
Openwrt by default enables “gro fraglist” by default, you can try disabling it.
ethtool -K [interface name] rx-gro-list off
This seems to be the trick, now the question is, if I disabled GRO on the VM via:echo "options virtio_net gso=0 gro=0" | tee -a /etc/modules.conf
Why it is still enabled?
Edit:rx-gro-list should improve QUIC performance and this is excatly my use case that is broken.
1 Like
brada4
January 20, 2026, 1:23pm
18
There is no virtio_net parameter called "gro", you have to take them off by ethtool.
parm: napi_weight:int
parm: csum:bool
parm: gso:bool
parm: napi_tx:bool
Oh dumb me...
I did a small script for hotplug:
#!/bin/sh
[ "$ACTION" = "add" ] || exit 0
[ -n "$INTERFACE" ] || exit 0
# Check that the interface exists
[ -d "/sys/class/net/$INTERFACE" ] || exit 0
# Check that the driver is virtio_net
driver_link="/sys/class/net/$INTERFACE/device/driver"
[ -L "$driver_link" ] || exit 0
driver=$(basename "$(readlink "$driver_link")")
[ "$driver" = "virtio_net" ] || exit 0
# Disable rx-gro-list (ignore if unsupported)
ethtool -K "$INTERFACE" rx-gro-list off 2>/dev/null
So at this point is the offload in the driver that is somehow broken.
1 Like
brada4
January 20, 2026, 2:29pm
20
rx-gro-list is often broken, nothing against you or virtio_net. You can disable it everywhere.
