Hey friends, how are you doing.
Currently i am experiencing some issue.
I have got my x86 Supermicro server running OpenWrt for 2 months now.
My router is connected to my coreswitch using a dual link LACP bonding.
During the first month, after 3 days the LAG went down, so i was suspecting a bad switch, so i have replaced it by a fairly new one. Unfortunately the issue persists.
One link "dies" after some random amount of days. I can't determine any specific issue in the logs.
That's what the switch log says:
The router is connected to ge 1/0/47 and ge 1/0/48
Even when moving to a different Port-Channel interface the "dead" link doesn't come up again.
No LACPDUs are detected.
What might be the issue here?
ge 1/0/4 is the IPMI interface of my router/server.
Hardware is Gigabit Ethernet, address is ecc8.8236.c4b0 (bia ecc8.8236.c4b0)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:10, output 00:00:05, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
891510239 packets input, 1311325674536 bytes, 0 no buffer
Received 121567 broadcasts (117111 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 117111 multicast, 0 pause input
0 input packets with dribble condition detected
2005132 packets output, 366675955 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
2022-01-17T21:32:17.495124+01:00 172.20.32.100 2451: Jan 17 20:32:16.497: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:33:17.878510+01:00 172.20.32.100 2452: Jan 17 20:33:16.881: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:35:18.223341+01:00 172.20.32.100 2453: Jan 17 20:35:17.223: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:36:18.322035+01:00 172.20.32.100 2454: Jan 17 20:36:17.323: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:38:18.555996+01:00 172.20.32.100 2455: Jan 17 20:38:17.555: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:39:18.958971+01:00 172.20.32.100 2456: Jan 17 20:39:17.960: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:41:49.265175+01:00 172.20.32.100 2457: Jan 17 20:41:48.266: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:42:19.323236+01:00 172.20.32.100 2458: Jan 17 20:42:18.324: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:42:49.438336+01:00 172.20.32.100 2459: Jan 17 20:42:48.429: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:45:19.609046+01:00 172.20.32.100 2460: Jan 17 20:45:18.609: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:47:19.747142+01:00 172.20.32.100 2461: Jan 17 20:47:18.747: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:48:49.989247+01:00 172.20.32.100 2462: Jan 17 20:48:48.989: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:49:50.058184+01:00 172.20.32.100 2463: Jan 17 20:49:49.058: %SW_MATM-4-MACFLAP_NOTIF: Host a8a1.5946.4b85 in vlan 3 is flapping between port Gi1/0/34 and port Po7
2022-01-17T21:51:33.235048+01:00 172.20.32.100 2464: Jan 17 20:51:32.235: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/48, changed state to down
2022-01-17T21:51:33.235318+01:00 172.20.32.100 2465: Jan 17 20:51:32.256: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel7, changed state to down
2022-01-17T21:51:34.257419+01:00 172.20.32.100 2466: Jan 17 20:51:33.257: %LINK-3-UPDOWN: Interface Port-channel7, changed state to down
2022-01-17T21:51:40.271198+01:00 172.20.32.100 2467: Jan 17 20:51:39.271: %EC-5-L3DONTBNDL2: Gi1/0/48 suspended: LACP currently not enabled on the remote port.
2022-01-17T21:51:42.798251+01:00 172.20.32.100 2468: Jan 17 20:51:41.798: %EC-5-L3DONTBNDL2: Gi1/0/47 suspended: LACP currently not enabled on the remote port.
2022-01-17T21:55:49.387056+01:00 172.20.32.100 2469: Jan 17 20:55:48.386: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/34, changed state to down
2022-01-17T21:55:50.409379+01:00 172.20.32.100 2470: Jan 17 20:55:49.409: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/34, changed state to down
2022-01-17T21:55:50.409651+01:00 172.20.32.100 2471: Jan 17 20:55:49.409: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/48, changed state to down
2022-01-17T21:55:54.512013+01:00 172.20.32.100 2472: Jan 17 20:55:53.451: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/34, changed state to up
2022-01-17T21:55:54.512297+01:00 172.20.32.100 2473: Jan 17 20:55:53.451: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/48, changed state to up
2022-01-17T21:55:54.512719+01:00 172.20.32.100 2474: Jan 17 20:55:54.489: %EC-5-CANNOT_BUNDLE2: Gi1/0/48 is not compatible with Gi1/0/47 and will be suspended (speed of Gi1/0/48 is 10M, Gi1/0/47 is 1000M)
2022-01-17T21:55:56.486018+01:00 172.20.32.100 2475: Jan 17 20:55:55.485: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/34, changed state to up
2022-01-17T21:55:56.486286+01:00 172.20.32.100 2476: Jan 17 20:55:55.648: %EC-5-L3DONTBNDL2: Gi1/0/47 suspended: LACP currently not enabled on the remote port.
2022-01-17T21:55:57.985846+01:00 172.20.32.100 2477: Jan 17 20:55:56.985: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/34, changed state to down
2022-01-17T21:55:59.009662+01:00 172.20.32.100 2478: Jan 17 20:55:58.002: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/34, changed state to down
2022-01-17T21:55:59.009970+01:00 172.20.32.100 2479: Jan 17 20:55:58.007: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/48, changed state to down
Wrt Log:
Thu Jan 27 17:22:16 2022 kern.info kernel: [523585.831322] e1000e: eth3 NIC Link is Down
Thu Jan 27 17:22:16 2022 kern.err kernel: [523585.835759] bonding-i0: (slave eth3): speed changed to 0 on port 2
Thu Jan 27 17:22:16 2022 kern.info kernel: [523585.847179] bonding-i0: (slave eth3): link status definitely down, disabling slave
Thu Jan 27 17:22:19 2022 kern.info kernel: [523588.388203] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Thu Jan 27 17:22:19 2022 kern.info kernel: [523588.455228] bonding-i0: (slave eth3): link status definitely up, 1000 Mbps full duplex
Thu Jan 27 17:22:19 2022 kern.info kernel: [523588.783259] e1000e: eth3 NIC Link is Down
Thu Jan 27 17:22:19 2022 kern.err kernel: [523588.787684] bonding-i0: (slave eth3): speed changed to 0 on port 2
Thu Jan 27 17:22:19 2022 kern.info kernel: [523588.879112] bonding-i0: (slave eth3): link status definitely down, disabling slave
Thu Jan 27 17:22:22 2022 kern.info kernel: [523592.036117] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
Thu Jan 27 17:22:22 2022 kern.info kernel: [523592.107137] bonding-i0: (slave eth3): link status definitely up, 1000 Mbps full duplex
After switching from ge 1/0/47 and ge 1/0/48 on the switch to ge 1/0/43 and 44
Feb 7 12:19:48.723: %EC-5-L3DONTBNDL2: Gi1/0/43 suspended: LACP currently not enabled on the remote port.
Feb 7 12:19:50.537: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/44, changed state to up
Feb 7 12:19:51.518: %LINK-3-UPDOWN: Interface Port-channel5, changed state to up
Feb 7 12:19:52.519: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel5, changed state to up
