"Drop invalid packets" causes Proxmox server to be unreachable

[Arno]

Yesterday Firewall blocking traffic within lan zone - #36 by Arno was solved.
Today I found the cause of part of the problems I had.

After enabling "Drop invalid packets" my Proxmox server is unreachable from another computer.
Other server are reachable.

Before posting this I read "Drop invalid packets" prevents ssh over lan - #4 by keith24
As I understand that post it had something to do with duplicate IP addresses.

Different IP addresses in my case.
Why is "Drop invalid packets" causes Proxmox to be unreachable?
Both computers are in same zone: lan.

Proxmox has some specific network configuration with the /etc/hosts file but I can't believe that causes this behavior.

The solution is simple: Uncheck the "Drop invalid packets" option but does anybody know why the server is unreachable when the option is enabled?

[lleachii]

@lleachii

Also, I though of something:

@Arno

option log '1'

Is there a log of the block you say occurred?

• Maybe there's a log?
• tcpdump?

(i.e., attempt to get a record of the "invalid packet")

Edit:

Also, did you make any edits to br-lan, routing, other bridges, gateways, etc?

[Arno]

@lleachii

Also, did you make any edits to br-lan, routing, other bridges, gateways, etc?

Only changed reject to drop. When I first saw this today I restored a working backup from yesterday.
Then tested. All fine.
Checked "Drop invalid packages". Unreachable.
Unchecked. All fine.

IIRC dropped packets are not logged so I changed General Settings - Forward to reject.
Added option log '1' to the firewall configuration file at the flan zone.
Restarted the firewall.
No log of packets.

Edit: Intra zone forward of lan = accept
Used tcpdump on the router but don't know what to look for in the output of tcpdump.

[lleachii]

To be clear, this is still a test between a wireless client and wired Proxmox, both on LAN (br-lan), same subnet?

@Arno

Only changed reject to drop.

Changed reject to drop on which option (and for what zone, global configuration, etc)?

@Arno

When I first saw this today I restored a working backup from yesterday.
Then tested. All fine.
Checked "Drop invalid packages". Unreachable.
Unchecked. All fine.

So this was actually done both with a previous backup, and with defaults?

(To be clear, the purpose of advice in the previous thread to reset to defaults is to eliminate any issue with the former configuration.)

@Arno

IIRC dropped packets are not logged so I changed General Settings - Forward to reject.
Added option log '1' to the firewall configuration file at the flan zone.
Restarted the firewall.
No log of packets.

Edit: Intra zone forward of lan = accept

Yea, because traffic within the LAN (bridge) wouldn't traverse the firewall.

I'm curious, can you run from (with "Drop Invalid" filter enabled):

1. The client - ip route get <IP_of_Proxmox>
2. The Proxmox - ip route get <IP_of_client>
3. Client - tcpdump -i br-lan -nn host <IP_of_Proxmox> (or Wireshark, run on WiFi NIC and filter fir Proxmox IP)
4. Proxmox - tcpdump -i vmbr0 -nn host <IP_of_client>
5. While tcpdump is running, proceed with testing again

If you need more advanced help with troubleshooting, let us know.

[Arno]

@lleachii

Changed reject to drop on which option (and for what zone, global configuration, etc)?

At General Settings

@lleachii

So this was actually done both with a previous backup, and with defaults?

The backup I restored was a backup made right after the previous thread.
In that backup "Drop invalid packets is off".
Proxmox is reachable.
Enabled the option. Proxmox is unreachable.

I will run your tests now. Will post in a bit.

[Arno]

On client
ip route get <IP_of_Proxmox>

192.168.3.3 dev wlan0 src 192.168.3.104 uid 1000 
    cache

On server
ip route get <IP_of_client>
Can't do this using ssh because I can't login to the sever with "drop invalid packets" enabled
But using ipmi this is the result

192.168.3.104 via 192.168.3.1 dev vmbr0 src 192.168.3.3 uid 0
    cache

(192.168.3.1 is br-lan on OpenWRT router)

Before your last post I already did a tcpdump on the router
While tcpdump was running I did ssh to the Proxmox server:

drop invalid packets ENABLED

12:06:16.019361 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752146013 ecr 0,nop,wscale 9], length 0
12:06:16.019664 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984009166 ecr 2752146013,nop,wscale 7], length 0
12:06:17.020208 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752147015 ecr 0,nop,wscale 9], length 0
12:06:17.020456 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984010167 ecr 2752146013,nop,wscale 7], length 0
12:06:18.041105 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752148039 ecr 0,nop,wscale 9], length 0
12:06:18.041373 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984011188 ecr 2752146013,nop,wscale 7], length 0
12:06:19.068134 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752149063 ecr 0,nop,wscale 9], length 0
12:06:19.068382 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984012215 ecr 2752146013,nop,wscale 7], length 0
12:06:20.092122 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752150087 ecr 0,nop,wscale 9], length 0
12:06:20.092417 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984013239 ecr 2752146013,nop,wscale 7], length 0
12:06:21.116106 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752151111 ecr 0,nop,wscale 9], length 0
12:06:21.116410 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984014263 ecr 2752146013,nop,wscale 7], length 0
12:06:22.152249 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984015299 ecr 2752146013,nop,wscale 7], length 0
12:06:23.136455 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752153127 ecr 0,nop,wscale 9], length 0
12:06:23.136743 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984016283 ecr 2752146013,nop,wscale 7], length 0
12:06:25.160258 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984018307 ecr 2752146013,nop,wscale 7], length 0
12:06:27.388650 IP 192.168.3.104.48424 > 192.168.3.3.22: Flags [S], seq 2965312652, win 64240, options [mss 1460,sackOK,TS val 2752157383 ecr 0,nop,wscale 9], length 0
12:06:27.388929 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984020535 ecr 2752146013,nop,wscale 7], length 0
12:06:31.432244 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984024579 ecr 2752146013,nop,wscale 7], length 0
12:06:34.303492 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [F.], seq 3999533906, ack 4008499537, win 141, options [nop,nop,TS val 3041038324 ecr 3983885787], length 0
12:06:34.303809 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [R], seq 4008499537, win 0, length 0
12:06:39.497249 IP 192.168.3.3.22 > 192.168.3.104.48424: Flags [S.], seq 1773916871, ack 2965312653, win 65160, options [mss 1460,sackOK,TS val 3984032644 ecr 2752146013,nop,wscale 7], length 0

drop invalid packets DISABLED

12:02:09.531201 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [S], seq 3999530876, win 64240, options [mss 1460,sackOK,TS val 3040773553 ecr 0,nop,wscale 9], length 0
12:02:09.531507 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [S.], seq 4008496566, ack 3999530877, win 65160, options [mss 1460,sackOK,TS val 3983762678 ecr 3040773553,nop,wscale 7], length 0
12:02:09.531531 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [S.], seq 4008496566, ack 3999530877, win 65160, options [mss 1460,sackOK,TS val 3983762678 ecr 3040773553,nop,wscale 7], length 0
12:02:09.534345 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [.], ack 1, win 126, options [nop,nop,TS val 3040773560 ecr 3983762678], length 0
12:02:09.534408 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 1:42, ack 1, win 126, options [nop,nop,TS val 3040773560 ecr 3983762678], length 41: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
12:02:09.534635 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 42, win 509, options [nop,nop,TS val 3983762681 ecr 3040773560], length 0
12:02:09.534669 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 42, win 509, options [nop,nop,TS val 3983762681 ecr 3040773560], length 0
12:02:09.545114 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 1:42, ack 42, win 509, options [nop,nop,TS val 3983762692 ecr 3040773560], length 41: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
12:02:09.545146 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 1:42, ack 42, win 509, options [nop,nop,TS val 3983762692 ecr 3040773560], length 41: SSH: SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u4
12:02:09.546163 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [.], ack 42, win 126, options [nop,nop,TS val 3040773572 ecr 3983762692], length 0
12:02:09.547744 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [.], seq 42:1490, ack 42, win 126, options [nop,nop,TS val 3040773574 ecr 3983762692], length 1448
12:02:09.547872 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 1490:1610, ack 42, win 126, options [nop,nop,TS val 3040773574 ecr 3983762692], length 120
12:02:09.548109 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 1610, win 532, options [nop,nop,TS val 3983762695 ecr 3040773574], length 0
12:02:09.548134 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 1610, win 532, options [nop,nop,TS val 3983762695 ecr 3040773574], length 0
12:02:09.552867 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 42:1082, ack 1610, win 532, options [nop,nop,TS val 3983762699 ecr 3040773574], length 1040
12:02:09.552894 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 42:1082, ack 1610, win 532, options [nop,nop,TS val 3983762699 ecr 3040773574], length 1040
12:02:09.558659 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 1610:2842, ack 1082, win 124, options [nop,nop,TS val 3040773585 ecr 3983762699], length 1232
12:02:09.564901 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], seq 1082:2530, ack 2842, win 555, options [nop,nop,TS val 3983762711 ecr 3040773585], length 1448
12:02:09.564925 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], seq 1082:2530, ack 2842, win 555, options [nop,nop,TS val 3983762711 ecr 3040773585], length 1448
12:02:09.564954 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2530:2662, ack 2842, win 555, options [nop,nop,TS val 3983762711 ecr 3040773585], length 132
12:02:09.564976 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2530:2662, ack 2842, win 555, options [nop,nop,TS val 3983762711 ecr 3040773585], length 132
12:02:09.569312 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [.], ack 2662, win 136, options [nop,nop,TS val 3040773595 ecr 3983762711], length 0
12:02:09.574514 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 2842:2926, ack 2662, win 136, options [nop,nop,TS val 3040773601 ecr 3983762711], length 84
12:02:09.615029 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 2926, win 555, options [nop,nop,TS val 3983762762 ecr 3040773601], length 0
12:02:09.615057 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 2926, win 555, options [nop,nop,TS val 3983762762 ecr 3040773601], length 0
12:02:09.619485 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 2926:2970, ack 2662, win 136, options [nop,nop,TS val 3040773646 ecr 3983762762], length 44
12:02:09.619765 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 2970, win 555, options [nop,nop,TS val 3983762766 ecr 3040773646], length 0
12:02:09.619790 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [.], ack 2970, win 555, options [nop,nop,TS val 3983762766 ecr 3040773646], length 0
12:02:09.619816 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2662:2706, ack 2970, win 555, options [nop,nop,TS val 3983762766 ecr 3040773646], length 44
12:02:09.619836 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2662:2706, ack 2970, win 555, options [nop,nop,TS val 3983762766 ecr 3040773646], length 44
12:02:09.620758 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [P.], seq 2970:3030, ack 2706, win 136, options [nop,nop,TS val 3040773647 ecr 3983762766], length 60
12:02:09.621966 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2706:2970, ack 3030, win 555, options [nop,nop,TS val 3983762768 ecr 3040773647], length 264
12:02:09.621991 IP 192.168.3.3.22 > 192.168.3.104.55968: Flags [P.], seq 2706:2970, ack 3030, win 555, options [nop,nop,TS val 3983762768 ecr 3040773647], length 264
12:02:09.665504 IP 192.168.3.104.55968 > 192.168.3.3.22: Flags [.], ack 2970, win 141, options [nop,nop,TS val 3040773692 ecr 3983762768], length 0

[wilsonyan]

impressive to see the guys help you out so far

I don't really see how you can get any further without describing how you have set this proxmox server up but i'm a newbie. Are you using vlans ? Using some other kind of virtual interfaces ? macvlan ?

Is it just a host with 1 network port and you've made a bridge ?

[lleachii]

@Arno

192.168.3.104 via 192.168.3.1 dev vmbr0 src 192.168.3.3 uid 0
    cache

(192.168.3.1 is br-lan on OpenWRT router)

@wilsonyan

I don't really see how you can get any further without describing how you have set this proxmox server up but i'm a newbie.

I agree.

I would check the subnet setting of the Proxmox's network configuration. For some reason, the Proxmox is trying to route a packet in the same subnet.

[Arno]

@wilsonyan

impressive to see the guys help you out so far

Yes. That's why I wrote in my first post of this topic: Solution is simple, just uncheck the box.
The reason for this topic is the question: Why does this happen?

The only difference between fail and success is the "Drop invalid packets" option.
Proxmox can be the cause, it's the only server that is unreachable.
On the other side: Proxmox can't be the problem because the issue is solved by unchecking the option.
Which one it is and even if the cause can be determined, I don't know.

@wilsonyan

Is it just a host with 1 network port and you've made a bridge ?

The default configuration of Proxmox is to create vmbr0 and the /etc/hosts file has to be edited.

[lleachii]

@Arno

The reason for this topic is the question: Why does this happen?

Because of this:

@lleachii

For some reason, the Proxmox is trying to route a packet in the same subnet.

@Arno

But using ipmi this is the result

192.168.3.104 via 192.168.3.1 dev vmbr0 src 192.168.3.3 uid 0
    cache

(192.168.3.1 is br-lan on OpenWRT router)

@Arno

On the other side: Proxmox can't be the problem because the issue is solved by unchecking the option.

This is untrue. A packet on same LAN that wants to be routed to a machine on the same LAN - is invalid.

[Arno]

@lleachii

Proxmox is trying to route a packet in the same subnet.

Client and server are on the same subnet. Why is this incorrect?
If routing is wrong then Proxmox should be unreachable all the time.

The Proxmox server is only unreachable when "drop invalid packets" is enabled.

[lleachii]

@Arno

Client and server are on the same subnet. Why is this incorrect?
If routing is wrong then Proxmox should be unreachable all the time.

It's incorrect because they should be Ethernet/Wireless Ethernet Switching (Layer 2)- not IP Routing (Layer 3).

@Arno

On client
ip route get <IP_of_Proxmox>

192.168.3.3 dev wlan0 src 192.168.3.104 uid 1000 
    cache

Client: No router.

@Arno

192.168.3.104 via 192.168.3.1 dev vmbr0 src 192.168.3.3 uid 0
    cache

(192.168.3.1 is br-lan on OpenWRT router)

Proxmox: Router

This indicates an issue with the Proxmox's network configuration.

[Arno]

@lleachii

This indicates an issue with the Proxmox's network configuration.

Then I go to the Proxmox forum. IF I want to change its configuration. If I mess that up I don't know if I can fix it.
From an OpenWRT perspective I can just disable "drop invalid packets". Problem 'solved'.
Thanks for answering why this happens.

[lleachii]

@Arno

From an OpenWRT perspective I can just disable "drop invalid packets". Problem 'solved'.
Thanks for answering why this happens.

Not exactly, and I would advise going to a Proxmox forum. If your Proxmox is routing when it shouldn't be, there could be other issues (e.g., security-related, as evidence by your firewall interacting with a connection that it shouldn't see).

As I noted previously, the OpenWrt should not be involved in this connection whatsoever - from a correct perspective.

Edit:

to clarify - "the OpenWrt firewall shouldn't be involved whatsoever"

[Arno]

@lleachii

OpenWrt should not be involved in this connection whatsoever

Thanks for the edit. I was confused about that part.

[MisterW]

For information, I have a Proxmox server running and dont have any problem with accessing it via an Openwrt router.

A route get for a local device on the Proxmox console gives

root@pve-1:~# ip route get 192.168.5.175
192.168.5.175 dev vmbr0 src 192.168.5.63 uid 0
cache

so no route showing

[lleachii]

Yea, I surmise the user needs to ensure the Proxmox IP is configured as 192.168.3.3/24 (subnet mask: 255.255.255.0). The Proxmox seems to think 192.168.3.104 is not within the local subnet's range (hence, it sends the packets to the OpenWrt for routing and the firewall interaction occurs).

I applaud the OP for first consulting a Proxmox forum, although I think this might be a simple network misconfiguration.

[wilsonyan]

@lleachii

The Proxmox seems to think 192.168.3.104 is not within the local subnet's range (hence, it sends the packets to the OpenWrt for routing and the firewall interaction occurs).

I applaud the OP for first consulting a Proxmox forum, although I think this might be a simple network misconfiguration.

Probably, unless it was already a workaround that happened to paper over that host networking had a problem.

[Arno]

@MisterW came up with the solution on the Proxmox forum: add "/24" to the IP address of vmbr0.

[MisterW]

To expand on that for others. The bridge in proxmox was defined with a static ip but no subnet mask. Despite the ip address being a class c private one it would seem that proxmox/debian did not assume a /24 subnet. So proxmox decided it needed to route the packet via the gateway even though the destination was actually in the same subnet. With the drop invalid packet setting on, Openwrt did not like this