X86_64: questions on transition from bare metal OpenWrt install to hypervisor + OpenWrt in VM

Installing and Using OpenWrt
1

I'm considering moving from the bare metal OpenWrt install to running OpenWrt in a VM on esxi (or some other hypervisor). The hardware is Sophos SG-135w rev 2.0 with the Atom C2558 CPU. There's an ath10k wireless radio, but it's not being used much, so I don't care if my wifi speeds are severely cut after transition, but overall performance (other than wireless radio) matters.

I have a little experience with esxi, zero experience with any other hypervisor/virtualization platform. What might be my best option for this, considering the CPU/HDD/traffic bandwidth overhead costs and ease of installation/use? If picking esxi what's the best version to go with right now, given my hardware?

If anyone already has experience running OpenWrt in an esxi VM on the Atom C2500 CPUs, what are the actual overhead costs?

I build my own images, would there be anything I may have to change when switching from building for bare metal to building for the VM?

Finally, is there a step-by-step guide for transition? I recon I would have to do the following:

  1. Download hypervisor, my own image and a vanilla 22.03 x86_64 image as a backup option to get networking
  2. Use serial connection to SG-135 box to install esxi and give it a static IP: 192.168.1.2
  3. Use esxi WebUI to create a VM for OpenWrt
  4. Configure said VM.
    4.1. how do I assign ethernet ports to the VM?
    4.2. Is pass-through an option or would the ports have to be virtualized?
    4.3. How do I share the ethernet ports so that both esxi and OpenWrt VM can be accessed thru it?
    4.4. Is it possible to set up the ath10k wireless radio in pass-through mode so only OpenWrt has access to it?
    4.4. Do I need to assign a static IP 192.168.1.1 to the VM or would OpenWrt take care of it once booted?
  5. Use a bootable linux ISO to start the VM and use dd to write the OpenWrt image
  6. Profit!!!

Anything I'm missing here? Anything else I need to consider?

The hardware information is below:

Network config 
SG-135 in ~ # uci export network
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	list ports 'eth2'
	list ports 'eth3'
	list ports 'eth4'
	list ports 'eth5'
	list ports 'eth6'
	list ports 'eth7'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.21.1'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	option hostname 'SG-135'
`/proc/cmdline` and `/proc/cpuinfo` 
SG-135 in ~ # cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz root=PARTUUID=c120d9f5-02 rootwait console=tty0 console=ttyS0,115200n8 noinitrd
SG-135 in ~ # cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 77
model name	: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz
stepping	: 8
microcode	: 0x11e
cpu MHz		: 2400.062
cache size	: 1024 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm arat
vmx flags	: vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs		: cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips	: 4800.19
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 77
model name	: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz
stepping	: 8
microcode	: 0x11e
cpu MHz		: 2400.151
cache size	: 1024 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm arat
vmx flags	: vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs		: cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips	: 4800.19
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 77
model name	: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz
stepping	: 8
microcode	: 0x11e
cpu MHz		: 2400.097
cache size	: 1024 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm arat
vmx flags	: vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs		: cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips	: 4800.19
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 77
model name	: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz
stepping	: 8
microcode	: 0x11e
cpu MHz		: 2400.272
cache size	: 1024 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch cpuid_fault epb pti tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm arat
vmx flags	: vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs		: cpu_meltdown spectre_v1 spectre_v2 mds msbds_only mmio_unknown
bogomips	: 4800.19
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:
`
dmidecode 
SG-135 in ~ # dmidecode --type 17
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.8 present.

Handle 0x003D, DMI type 17, 34 bytes
Memory Device
	Array Handle: 0x003B
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: 2048 MB
	Form Factor: DIMM
	Set: None
	Locator: DIMM0
	Bank Locator: BANK 0
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: 1600 MT/s
	Manufacturer: <BAD INDEX>
	Serial Number: <BAD INDEX>
	Asset Tag: <BAD INDEX>
	Part Number: <BAD INDEX>
	Rank: 1
	Configured Memory Speed: 1600 MT/s

Handle 0x003F, DMI type 17, 34 bytes
Memory Device
	Array Handle: 0x003B
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: 4096 MB
	Form Factor: DIMM
	Set: None
	Locator: DIMM0
	Bank Locator: BANK 1
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: 1600 MT/s
	Manufacturer: <BAD INDEX>
	Serial Number: <BAD INDEX>
	Asset Tag: <BAD INDEX>
	Part Number: <BAD INDEX>
	Rank: 2
	Configured Memory Speed: 1600 MT/s

Handle 0x0041, DMI type 17, 34 bytes
Memory Device
	Array Handle: 0x003B
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: DIMM1
	Bank Locator: BANK 0
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: Unknown
	Manufacturer: NO DIMM
	Serial Number: NO DIMM
	Asset Tag: NO DIMM
	Part Number: NO DIMM
	Rank: Unknown
	Configured Memory Speed: Unknown

Handle 0x0042, DMI type 17, 34 bytes
Memory Device
	Array Handle: 0x003B
	Error Information Handle: Not Provided
	Total Width: Unknown
	Data Width: Unknown
	Size: No Module Installed
	Form Factor: DIMM
	Set: None
	Locator: DIMM1
	Bank Locator: BANK 1
	Type: DDR3
	Type Detail: Synchronous Unbuffered (Unregistered)
	Speed: Unknown
	Manufacturer: NO DIMM
	Serial Number: NO DIMM
	Asset Tag: NO DIMM
	Part Number: NO DIMM
	Rank: Unknown
	Configured Memory Speed: Unknown
lsblk 
SG-135 in ~ # lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 59.6G  0 disk
├─sda1   8:1    0  128M  0 part /boot
│                               /boot
└─sda2   8:2    0    2G  0 part /
2

you might want to check Xen or VMware or similar virtualization forums on that.

I would say there are far more randoms popping up one in a while in this forum that ran into weird issues due to virtualized network issues of their virtualization layer than there are actual regular users using OpenWRT virtualized.

according to the issues that pop up one in a while, I would say the overhead price to pay is that you can typically run into weird network issues caused by the software defined network layer or passthrough or similar of the virtualization software. And then you are in a rather small group of pals worldwide to discuss such things. And finding only few blogs with helpful debugging content about the topic.

3

The Atom hardware isn't particularly fast to begin with, while hardware accelerated virtualization isn't that heavy weight, this wouldn't be my first, or second choice (if pressed, I'd rather run qemu(-kvm) on OpenWrt, with VMs offering additional services, just to retain 'simple' access to the network interfaces; not really a fan of this either).