Huge ping spikes on wifi (FritzBox 4040 / ipq40xx)

I have trouble with my wifi. Different devices disconnect often and sometimes only stay connected for a few seconds.
While trying to find the problem I found out, that just pinging the router on wifi sometimes shows very high delay:

[1610057309.590818] 64 Bytes von 192.168.3.1: icmp_seq=126 ttl=64 Zeit=3.72 ms
[1610057310.592437] 64 Bytes von 192.168.3.1: icmp_seq=127 ttl=64 Zeit=3.45 ms
[1610057316.546251] 64 Bytes von 192.168.3.1: icmp_seq=129 ttl=64 Zeit=3947 ms
[1610057316.578107] 64 Bytes von 192.168.3.1: icmp_seq=130 ttl=64 Zeit=2955 ms
[1610057316.629660] 64 Bytes von 192.168.3.1: icmp_seq=131 ttl=64 Zeit=1983 ms
[1610057316.679028] 64 Bytes von 192.168.3.1: icmp_seq=133 ttl=64 Zeit=8.19 ms

The strange thing is, when looking at this from the other side by running tcpdump on the router everything looks normal:

1610057309.587529 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 126, length 64
1610057309.587782 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 126, length 64
1610057310.589527 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 127, length 64
1610057310.589777 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 127, length 64
1610057311.594186 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 128, length 64
1610057311.596094 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 128, length 64
1610057312.633526 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 129, length 64
1610057312.634994 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 129, length 64
1610057313.624050 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 130, length 64
1610057313.626164 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 130, length 64
1610057314.726934 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 131, length 64
1610057314.728997 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 131, length 64
1610057315.671744 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 132, length 64
1610057315.671977 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 132, length 64
1610057316.671661 IP 192.168.3.209 > OpenWrt.lan: ICMP echo request, id 6, seq 133, length 64
1610057316.673594 IP OpenWrt.lan > 192.168.3.209: ICMP echo reply, id 6, seq 133, length 64

It seems as if the packages are stuck somewhere in the firmware for a second or two, before being transmitted.

I have 3 SSIDs on the 2.4GHz Wifi with maybe 15 devices but not much traffic.
The Wifi is on channel 11 and the only other Wifis I can see are on channel 1 and 6.