Strange problem with ESXI

[NPeca75]

Hi to all

i wanted to use OpenWRT on ESXI
installed it succesfuly with

openwrt-22.03.2-x86-64-generic-ext4-combined.img.gz

my main goal was to make routing/filtering between vlans so i tried the "easy" way, access ports on esxi switch instead of tagging vlans on OWRT
so far, so good
4 pcs of ETH adapter on OpenWRT, working as expected

but...

problem arise when i tried to put 5-th ETH adapter
very strange problem
only the first (eth0) is working, other eth's are still present in OWRT but something is wrong. There is no communication on eth1-4
some rx/tx counters are increasing, but ARP is missing, pings does not work, etc
only on first eth0 everything is OK

if i remove last added (eth4) adapter from OWRT, thing get back in normal

what i tried ?
adapter type VMXNET3 and E1000e
tried to assign different portgroup to eth4
tried to rotate portgroups on eth1-4
whatever i do, OWRT break (half) with 5th adapter

does anyone have similar experience with esxi/owrt ?

edit: esxi is 6.7.0

[NPeca75]

ok, the problem seems to be with ESXI ? or OpenWRT ? could not decide

with 4 eth adapters:

[    5.748157] vmxnet3 0000:03:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.748665] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
[    5.748781] vmxnet3 0000:0b:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.748953] vmxnet3 0000:0b:00.0 eth1: NIC Link is Up 10000 Mbps
[    5.749056] vmxnet3 0000:13:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.749235] vmxnet3 0000:13:00.0 eth2: NIC Link is Up 10000 Mbps
[    5.749338] vmxnet3 0000:1b:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.749498] vmxnet3 0000:1b:00.0 eth3: NIC Link is Up 10000 Mbps

and when i insert 5th eth

[    5.755999] vmxnet3 0000:03:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.756192] vmxnet3 0000:03:00.0 eth0: NIC Link is Up 10000 Mbps
[    5.756307] vmxnet3 0000:04:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.756475] vmxnet3 0000:04:00.0 eth1: NIC Link is Up 10000 Mbps
[    5.756577] vmxnet3 0000:0b:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.756756] vmxnet3 0000:0b:00.0 eth2: NIC Link is Up 10000 Mbps
[    5.756858] vmxnet3 0000:13:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.757017] vmxnet3 0000:13:00.0 eth3: NIC Link is Up 10000 Mbps
[    5.757119] vmxnet3 0000:1b:00.0: # of Tx queues : 1, # of Rx queues : 1
[    5.757277] vmxnet3 0000:1b:00.0 eth4: NIC Link is Up 10000 Mbps

so, clearly, new (last) adapter is inserted as "0000:04:00.0 eth1" which will shift "down" existing eth1,2,3 to eth 2,3,4

is there any solution to tie ETH names to adapter MAC address ?? or anything else to keep existing order of ETH adapters ?

[grrr2]

hi, it is not owrt related bug but esxi. vmxnet3 does not keep order for some strange reason according to my experience. try with e1000(e).

[NPeca75]

hi @grrr2
tnx for reply
i tried with

1. pure vmxnet3
2. pure E1000e
3. mixed vmx/e1000e environment
   i have no option for plain E1000, only E1000e
   result is same, 5-th adapter is inserted on 04:00.0

so, basicaly, only hotplug/coldplug scripting could solve this problem ?

[grrr2]

not sure if VMWare have ever fixed it: https://communities.vmware.com/t5/Virtual-Machine-Guest-OS-and-VM/Rhel-6-nic-order-with-4-network-adapter-or-more/m-p/1793932
maybe this can help https://vnetwise.wordpress.com/2013/03/22/howto-force-vmware-esxi-to-re-order-nic-ports-to-maintain-standards/

I am using VMWare Workstation mostly and it is definitely randomize order in case of vmxnet3. so i keep using e1000 when order matters.

[NPeca75]

Since plain E1000 (without e) is not supported anymore in v6.7 and up
i decided to make a trunk port group (4095) and make vlan tagging on OpenWRT
as only viable solution