So, I upgraded to gigabit Ethernet, and I have tested with a laptop. I get about 900 Mbps/900 Mbps.
I'm using a VM running OpenWrt x86_64...I move my configs...and do some speed tests...
I'm getting about 4 up/4 down
...but it get weirder
Thru a Wireguard connection, I'm getting 200/300
...despite I'm still missing some bandwidth...how in the world am I'm only getting 4/4 with normal LAN-to-WAN; but the same client sent thru the the same interfaces to and passes Wiregusrd traffic at 200/300???
Basically, I want to increase the speed of my WAN traffic (obviously).
Any ideas?
(I did search and only found this: [Solved] Internet slowness x86_64 - #5 by thompdre84 - but it suggests changing to another PHY, that is not possible...and the PHY is obviously capable of more than 4/4 - because the Wireguard traffic seems to move faster.)
What hypervisor and host OS?
Which (emulated) network cards did you configure?
Just for a comparison, I can achieve 934 MBit/s via iperf3 through a test-VM (qemu-kvm, virtio-net-pci, with 2 cores and 1 GB RAM assigned on a sandy-bridge i7-2600k/ linux v5.16 host); the throughput is limited by the rest of my wired network. Getting my wired (1000BASE-T) network out of the way, testing the throughput between host and VM, I can achieve 26 GBit/s.
It's been roughly 15 years since I last used virtualbox, on non-kvm capable hardware…
I know the networking setup with qemu isn't quite as convenient (no GUI, manual bridge setup and tap interfaces), but it would be interesting to test that on the same hardware as well. The choice of the emulated network card (virtio vs fully emulated hardware) also has a major performance impact (block i/o as well, but that doesn't matter much for OpenWrt).
First, I tried another PHY (it was possible; but I had to use tagged ports and a managed switch) - I had the same issues...so then I changed the adapter settings.
I selected the Intel PRO/1000 MT Desktop adapter...and...I'll just make a list of the results:
Intel WAN and Intel LAN - 800dn/6up
virtio-net WAN and Intel LAN - 6dn/800up (but mostly in this config the LAN ARP fails and I cannot ping router)
Intel WAN and virtio-net LAN - 6dn/800up
...wow.
EDIT: I forgot to note...that with Intel selected, Wireguard downloads remain fast (faster than unencapsulated LAN-to-WAN traffic...odd), but now upload is slow.
I'm now getting about 300dn/540up (and about the same thru a Wireguard interface) with virtio-net/virtio-net - and running some ethtool command...so I tried the:
And ran ethtool on LAN...and different combinations of VM NIC PHYs...the fastest were virtio-net/virtio-net with the following conifg:
ethtool -K eth1 tso off
ethtool -K eth1 sg off
ethtool -K eth1 generic-receive-offload off
#
ethtool -K eth0 tso off
ethtool -K eth0 sg off
ethtool -K eth0 generic-receive-offload off
Without these settings in place, upload it nearly 0 and times out.
@jow@hnyman ...is this normal...or given offloading helped...any ideas?
at the time was intel(G5258) mostly tested on 5.x(vbox) ...
got an amd a few weeks ago and while installing vbox did note it was a little different (different virtualization bios setting) etc... but have not tested used it yet...
a setting dump (see near hardware virtualiztion etc.)
Name: 64tftpclient
Groups: /
Guest OS: Other Linux (64-bit)
UUID: 7e4d82dd-785c-4d2a-9c11-28418dc36237
Config file: /home/vert/VirtualBox VMs/64tftpclient/64tftpclient.vbox
Snapshot folder: /home/vert/VirtualBox VMs/64tftpclient/Snapshots
Log folder: /home/vert/VirtualBox VMs/64tftpclient/Logs
Hardware UUID: 7e4d82dd-785c-4d2a-9c11-28418dc36237
Memory size: 1024MB
Page Fusion: disabled
VRAM size: 16MB
CPU exec cap: 100%
HPET: disabled
CPUProfile: host
Chipset: piix3
Firmware: BIOS
Number of CPUs: 1
PAE: enabled
Long Mode: enabled
Triple Fault Reset: disabled
APIC: enabled
X2APIC: enabled
Nested VT-x/AMD-V: disabled
CPUID Portability Level: 0
CPUID overrides: None
Boot menu mode: message and menu
Boot Device 1: Network
Boot Device 2: HardDisk
Boot Device 3: Not Assigned
Boot Device 4: Not Assigned
ACPI: enabled
IOAPIC: enabled
BIOS APIC mode: APIC
Time offset: 0ms
RTC: UTC
Hardware Virtualization: enabled
Nested Paging: enabled
Large Pages: disabled
VT-x VPID: enabled
VT-x Unrestricted Exec.: enabled
Paravirt. Provider: Default
Effective Paravirt. Prov.: KVM
State: powered off (since 2021-11-10T23:16:42.000000000)
Graphics Controller: VMSVGA
Monitor count: 1
3D Acceleration: disabled
2D Video Acceleration: disabled
Teleporter Enabled: disabled
Teleporter Port: 0
Teleporter Address:
Teleporter Password:
Tracing Enabled: disabled
Allow Tracing to Access VM: disabled
Tracing Configuration:
Autostart Enabled: disabled
Autostart Delay: 0
Default Frontend:
VM process priority: default
Storage Controller Name (0): IDE
Storage Controller Type (0): PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 2
Storage Controller Port Count (0): 2
Storage Controller Bootable (0): on
IDE (0, 0): /home/vert/VirtualBox VMs/64tftpclient/64tftpclient.vdi (UUID: c039f8b5-3eff-4a5e-be31-e3d312bf1b3c)
NIC 1: MAC: 0800278B7810, Attachment: Bridged Interface 'enp0s25', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2: disabled
NIC 3: disabled
NIC 4: disabled
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled
Pointing Device: USB Tablet
Keyboard Device: PS/2 Keyboard
UART 1: disabled
UART 2: disabled
UART 3: disabled
UART 4: disabled
LPT 1: disabled
LPT 2: disabled
Audio: enabled (Driver: PulseAudio, Controller: AC97, Codec: STAC9700)
Audio playback: enabled
Audio capture: disabled
Clipboard Mode: disabled
Drag and drop Mode: disabled
VRDE: disabled
OHCI USB: enabled
EHCI USB: disabled
xHCI USB: disabled
USB Device Filters:
Name: pihole1
Groups: /
Guest OS: Other Linux (32-bit)
UUID: 7ea00ff0-ba93-4c74-9baa-4c4366cd2c56
Config file: /home/vert/VirtualBox VMs/pihole1/pihole1.vbox
Snapshot folder: /home/vert/VirtualBox VMs/pihole1/Snapshots
Log folder: /home/vert/VirtualBox VMs/pihole1/Logs
Hardware UUID: 7ea00ff0-ba93-4c74-9baa-4c4366cd2c56
Memory size: 1024MB
Page Fusion: disabled
VRAM size: 16MB
CPU exec cap: 100%
HPET: disabled
CPUProfile: host
Chipset: piix3
Firmware: BIOS
Number of CPUs: 1
PAE: disabled
Long Mode: disabled
Triple Fault Reset: disabled
APIC: enabled
X2APIC: enabled
Nested VT-x/AMD-V: disabled
CPUID Portability Level: 0
CPUID overrides: None
Boot menu mode: message and menu
Boot Device 1: Network
Boot Device 2: Not Assigned
Boot Device 3: Not Assigned
Boot Device 4: Not Assigned
ACPI: enabled
IOAPIC: disabled
BIOS APIC mode: APIC
Time offset: 0ms
RTC: UTC
Hardware Virtualization: enabled
Nested Paging: enabled
Large Pages: disabled
VT-x VPID: enabled
VT-x Unrestricted Exec.: enabled
Paravirt. Provider: Default
Effective Paravirt. Prov.: KVM
State: powered off (since 2021-11-08T10:21:28.000000000)
Graphics Controller: VMSVGA
Monitor count: 1
3D Acceleration: disabled
2D Video Acceleration: disabled
Teleporter Enabled: disabled
Teleporter Port: 0
Teleporter Address:
Teleporter Password:
Tracing Enabled: disabled
Allow Tracing to Access VM: disabled
Tracing Configuration:
Autostart Enabled: disabled
Autostart Delay: 0
Default Frontend:
VM process priority: default
Storage Controller Name (0): IDE
Storage Controller Type (0): PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 2
Storage Controller Port Count (0): 2
Storage Controller Bootable (0): on
IDE (0, 0): /home/vert/VirtualBox VMs/pihole1/pihole1.vdi (UUID: 5b16fd98-bb7d-44da-9ac2-dae9309caa20)
IDE (1, 0): Empty
NIC 1: MAC: 080027494788, Attachment: Bridged Interface 'enp0s25', Cable connected: on, Trace: off (file: none), Type: Am79C973, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2: disabled
NIC 3: disabled
NIC 4: disabled
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled
Pointing Device: USB Tablet
Keyboard Device: PS/2 Keyboard
UART 1: disabled
UART 2: disabled
UART 3: disabled
UART 4: disabled
LPT 1: disabled
LPT 2: disabled
Audio: enabled (Driver: PulseAudio, Controller: AC97, Codec: AD1980)
Audio playback: enabled
Audio capture: disabled
Clipboard Mode: disabled
Drag and drop Mode: disabled
VRDE: disabled
OHCI USB: enabled
EHCI USB: disabled
xHCI USB: disabled
USB Device Filters:
<none>
On my proxmox 7 (KVM-based) setup I have a OpenWrt VM that is connected to a MPTCP VM (Openwrt-based appliance that does multipath bonding to aggregate two WAN connections), and I routinely get 100-200MBit/s down and 50-100Mbit/s up internet speeds so I'm already well beyond your 4/4.
I'll have to test if my setup is actually capable of gigabit for the local network but I would be surprised if it isn't, especially after slh also said his kvm-based VMs can reach gigabit.
My understanding is that TSO and GSO in the virtio driver work by offloading the job to the actual hardware, and if the hardware or the host driver does not support it properly (Realtek is a common offender, Intel is usually good) then the VM will have issues and you will need to disable it.
virt-manager is a GUI application that handles QEMU/KVM virtualization, there are also script-based managers like quickemu that take a text-based config https://github.com/quickemu-project/quickemu
I have used virt-manager in the past, it can also connect to a headless server to remote-control its virtualization capabilities (using SSH tunnels). I have used it for a while before proxmox and it's fine for a single headless hypervisor system.
I do use OpenWrt/ x86_64 under qemu-kvm (no orchestration, other than some custom start scripts to save me the typing) regularly, to temporarily bring internet access to a walled-off VLAN (usually no internet access) for updating. When I originally set it up, I didn't care about the speed that much, as my WAN was the limiting factor anyways, but now I can reliably achieve 1 GBit/s wire speed (~934 MBit/s) from that qemu-kvm instance (running on a 14 year old sandy-bridge i7-2600k host). This instance also doubles up for testing various OpenWrt related things, as the virtualization makes that easy to accomplish (and to reset afterwards).
About a months ago, I switched my main router to x86_64 (ivy-bridge celeron 1037u), with OpenWrt/ master running on the bare iron. While testing the setup, I could confirm that this ~9 year old ULV CPU originally targeted at the mobile market can easily achieve 1 GBit/s wire speed between its two Intel 82574L (e1000e) onboard network cards (I would have been happy with r8168 as well, I didn't know what cards to expect when I bought it), with SQM/ cake being active (~53% CPU load on one core, without even clocking all the way up). I've been very happy with this setup so far, it's totally bored handling my 400/200 MBit/s ftth WAN uplink.
Personally I wouldn't want to rely on virtualization for the main router in a common household, as that's the single device that really must remain functional at all times - including the times when the hypervisor is off, updating, or broken. Additionally the router is the only device in my network that knows static DHCP leases and local DNS overrides and resolution, allowing my to quickly change the complete network topology as needed, and without having to care about syncing the configuration with other systems (or a hypervisor underneath the VM…) - maybe aside from the managed switch(es) (but VLAN assignments tend to remain more stable for the majority of devices than IP assignments). It's fine to manage 'optional' devices, e.g. other VM clusters or the update-once-a-month needs of an otherwise offline subnet, but imho not really as the sole bastion host in your network, that needs to remain functional for VoIP/ SIP, mail (-client usage), DHCP/ DNS and is needed for updating/ bootstrapping or fixing any system within the network. Obviously this perception may be different in an enterprise network (or sufficiently complex enthusiast networks managed as such), but I personally don't want to treat any individual system in my home-network as mission critical to that extent (down to the hypervisor); obviously I do have preconfigured spares for router/ SIP pbx, etc. as cold-standby (they won't give me 100% performance, but they'll work until I've sorted the problem).
This is a known issue.