Xiaomi Mini weird firewall issue

Hello folks.

For some weird reason I am unable to figure out, the Xiaomi Mini running latest trunk 31/01/2017 is not allowing the HOST to ssh into the VMs. Let me try to quickly explain the setup.

Essentially I have a Windows 10 Pro Workstation with VMware Workstation Pro where I host several guests, primarily FreeBSD 10.3 / 11 and CentOS 7. Every VM is bridged with the HOST as they all are on the same subnet which is 192.168.1.0/24, nothing fancy really.

When the HOST is connected to the Mini, the HOST is unable to ssh into any VMs, but every VMs as well as any physical devices on the same subnet can ssh just fine into eachother. If I take the firewall down on the Mini, the HOST is able to ssh into the VMs again.

Here is the log error on the HOST:

Connecting to 192.168.1.100:8895...
Could not connect to '192.168.1.100' (port 8895): Connection failed.

Here is ssd -d on the FreeBSD 10.3 Guest:

debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.1s-freebsd 1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:emNFXiEpr6a7Z0AIvveSfJ/HILf0+JpP8SRDrsTgM24
debug1: private host key #1: ssh-dss SHA256:36cqF7PzM3gedktTE67QcvSUdjFueTIgRlo5p9RBvGE
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:J9ARmbU39El2zJyw6h6y5BotU/EEhrWjYJ/RkDEkHhw
debug1: private host key #3: ssh-ed25519 SHA256:9rwy0lZ72L1+XTwP+UfmmrhraTKYUeSJ0FhLXYIg1xQ
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='8895'
debug1: rexec_argv[4]='-E'
debug1: rexec_argv[5]='/usr/home/psammarco/sshd_debug'
debug1: Bind to port 8895 on 192.168.1.100.
debug1: Server TCP RWIN socket size: 65536
Server listening on 192.168.1.100 port 8895.
debug1: fd 5 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.1s-freebsd 1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:emNFXiEpr6a7Z0AIvveSfJ/HILf0+JpP8SRDrsTgM24
debug1: private host key #1: ssh-dss SHA256:36cqF7PzM3gedktTE67QcvSUdjFueTIgRlo5p9RBvGE
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:J9ARmbU39El2zJyw6h6y5BotU/EEhrWjYJ/RkDEkHhw
debug1: private host key #3: ssh-ed25519 SHA256:9rwy0lZ72L1+XTwP+UfmmrhraTKYUeSJ0FhLXYIg1xQ
debug1: inetd sockets after dupping: 5, 5
debug1: res_init()
Connection from 192.168.1.195 port 53441 on 192.168.1.100 port 8895
debug1: Client protocol version 2.0; client software version nsssh2_5.0.0037 NetSarang Computer, Inc.
debug1: no match: nsssh2_5.0.0037 NetSarang Computer, Inc.
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2 FreeBSD-20160310
debug1: permanently_set_uid: 22/22 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: ecdh-sha2-nistp256 [preauth]
debug1: kex: host key algorithm: ssh-rsa [preauth]
debug1: kex: client->server cipher: aes128-cbc MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: kex: server->client cipher: aes128-cbc MAC: hmac-sha2-256-etm@openssh.com compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]

I tried few trunk builds but it did not make any difference. Ultimately I hocked up my carambola 2 running lede trunk as well, default firewall configuration and same network configuration and boom, the HOST is able to ssh into any VMs.

Windows 10 Pro HOST; 192.168.1.195
FreeBSD 10.3 Guest; 192.168.1.100
etc etc. It's pointless to list out all of the other VMs since the result is pretty much the same.

Network configuration here:

config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'

config globals 'globals'
option ula_prefix 'fd84:ad16:156c::/48'

config interface 'lan'
option ifname 'eth0.1'
option type 'bridge'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option dns '80.80.80.80 80.80.81.81'
option ip6assign '60'

config device 'lan_dev'
option name 'eth0.1'
option macaddr 'f0:b4:29:ec:b8:70'

config interface 'wan'
option ifname 'eth0.2'
option proto 'static'
option ipaddr '192.168.0.2'
option netmask '255.255.255.0'
option gateway '192.168.0.1'
option dns '80.80.80.80 80.80.81.81'

config device 'wan_dev'
option name 'eth0.2'
option macaddr 'f0:b4:29:ec:b8:71'

config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'

config switch_vlan
option device 'switch0'
option vlan '1'
option ports '0 1 2 3 6t'

config switch_vlan
option device 'switch0'
option vlan '2'
option ports '4 6t'

config route
option interface lan
option target 10.1.0.0/24
option gateway 192.168.1.100

config route
option interface lan
option target 10.2.0.0/24
option gateway 192.168.1.100

Any help would be gladly appreciated.

Regards,
Pietro

Do you have any port forward configured involving the host?

Hey Jow. No I do not have any port forward at all. I run default firewall configuration. How could I open the ssh port for each VMs so that I can ssh from the HOST ?

Thanks

I have just added the following rule to the firewall config file but it didn't do much of a difference. Apparently every once and awhile the HOST does manage to get connected to the VMs but soon after it gets kicked out with the message connection closed by foreign host.

config rule
option src lan
option src_ip '192.168.1.0/24'
option dest_port 8895
option target ACCEPT
option proto tcp

I may be wrong but I have reason to believe that this is something weird with the Xiaomi Mini firmware as like I said I don't have such a issue with the Carambola 2 that has been serving me well for over two years. Default firewall configuration on both and never had issues with LAN traffic.

I am leaving a shell connected overnight pinging google and I am debugging the connection using sshd -d again. Additionally, right now having a shell connected it doesn't let me open another remote shell as I get the very same message.

Connecting to 192.168.1.100:8895...
Could not connect to '192.168.1.100' (port 8895): Connection failed.

I am desperate!

Edit: I have realized I am getting connection refused when trying to open another remote shell from another device in LAN, this has never happened before, the issue was only faced by the HOST. Could that be because sshd -d does only allow a remote session? No idea...

Thanks

I wonder if some kind of masquerade is involved. Maybe some traffic is accidentially masqueraded which isn't supposed to.

Hey @jow, I seriously don't have any idea, and that rule I mentioned above didn't do any difference.

Btw, as requested network config here:

root@XiaomiMini:~# cat /etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdde:0262:61fd::/48'

config interface 'lan'
        option ifname 'eth0.1'
	option type 'bridge'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
        option dns '80.80.80.80 80.80.81.81'

config device 'lan_dev'
	option name 'eth0.1'
	option macaddr 'f2:b4:29:ec:b8:6f'

config interface 'wan'
	option ifname 'eth0.2'
        option proto 'static'
        option ipaddr '192.168.0.2'
        option netmask '255.255.255.0'
        option gateway '192.168.0.1'
        option dns '80.80.80.80 80.80.81.81'    

config device 'wan_dev'
	option name 'eth0.2'
	option macaddr 'f0:b4:29:ec:b8:6f'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '0 1 2 3 6t'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '4 6t'

config route
option interface lan
option target 10.1.0.0/24
option gateway 192.168.1.100

config route
option interface lan
option target 10.2.0.0/24
option gateway 192.168.1.100

firewall configuration here:

root@XiaomiMini:~# cat /etc/config/firewall
config defaults
	option syn_flood	1
	option input		ACCEPT
	option output		ACCEPT
	option forward		REJECT
# Uncomment this line to disable ipv6 rules
#	option disable_ipv6	1

config zone
	option name		lan
	list   network		'lan'
	option input		ACCEPT
	option output		ACCEPT
	option forward		ACCEPT

config zone
	option name		wan
	list   network		'wan'
	list   network		'wan6'
	option input		REJECT
	option output		ACCEPT
	option forward		REJECT
	option masq		1
	option mtu_fix		1

config forwarding
	option src		lan
	option dest		wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
	option name		Allow-DHCP-Renew
	option src		wan
	option proto		udp
	option dest_port	68
	option target		ACCEPT
	option family		ipv4

# Allow IPv4 ping
config rule
	option name		Allow-Ping
	option src		wan
	option proto		icmp
	option icmp_type	echo-request
	option family		ipv4
	option target		ACCEPT

config rule
	option name		Allow-IGMP
	option src		wan
	option proto		igmp
	option family		ipv4
	option target		ACCEPT

# Allow DHCPv6 replies
# see https://dev.openwrt.org/ticket/10381
config rule
	option name		Allow-DHCPv6
	option src		wan
	option proto		udp
	option src_ip		fc00::/6
	option dest_ip		fc00::/6
	option dest_port	546
	option family		ipv6
	option target		ACCEPT

config rule
	option name		Allow-MLD
	option src		wan
	option proto		icmp
	option src_ip		fe80::/10
	list icmp_type		'130/0'
	list icmp_type		'131/0'
	list icmp_type		'132/0'
	list icmp_type		'143/0'
	option family		ipv6
	option target		ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Input
	option src		wan
	option proto	icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	list icmp_type		router-solicitation
	list icmp_type		neighbour-solicitation
	list icmp_type		router-advertisement
	list icmp_type		neighbour-advertisement
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Forward
	option src		wan
	option dest		*
	option proto		icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT

# include a file with users custom iptables rules
config include
	option path /etc/firewall.user


### EXAMPLE CONFIG SECTIONS
# do not allow a specific ip to access wan
#config rule
#	option src		lan
#	option src_ip	192.168.45.2
#	option dest		wan
#	option proto	tcp
#	option target	REJECT

# block a specific mac on wan
#config rule
#	option dest		wan
#	option src_mac	00:11:22:33:44:66
#	option target	REJECT

# block incoming ICMP traffic on a zone
#config rule
#	option src		lan
#	option proto	ICMP
#	option target	DROP

# port redirect port coming in on wan to lan
#config redirect
#	option src			wan
#	option src_dport	80
#	option dest			lan
#	option dest_ip		192.168.16.235
#	option dest_port	80
#	option proto		tcp

# port redirect of remapped ssh port (22001) on wan
#config redirect
#	option src		wan
#	option src_dport	22001
#	option dest		lan
#	option dest_port	22
#	option proto		tcp

# allow IPsec/ESP and ISAKMP passthrough
config rule
	option src		wan
	option dest		lan
	option proto		esp
	option target		ACCEPT

config rule
	option src		wan
	option dest		lan
	option dest_port	500
	option proto		udp
	option target		ACCEPT

### FULL CONFIG SECTIONS
#config rule
#	option src		lan
#	option src_ip	192.168.45.2
#	option src_mac	00:11:22:33:44:55
#	option src_port	80
#	option dest		wan
#	option dest_ip	194.25.2.129
#	option dest_port	120
#	option proto	tcp
#	option target	REJECT

#config redirect
#	option src		lan
#	option src_ip	192.168.45.2
#	option src_mac	00:11:22:33:44:55
#	option src_port		1024
#	option src_dport	80
#	option dest_ip	194.25.2.129
#	option dest_port	120
#	option proto	tcp

config rule
        option src              lan
        option src_ip           '192.168.1.0/24'
        option dest_port        8895
        option target           ACCEPT
        option proto            tcp
root@XiaomiMini:~# 

Lastly firewall.user here

iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -d 10.1.0.0/24 -j MASQUERADE

Also, I am posting sshd_config of the FreeBSD 10.3 VM http://pastebin.com/vFQQh9a0

Thank you for your help. I truly appreciate it.

Anyone ? I just can't get to figure this out by my own. :sweat:

Thanks

Whats the purpose of the masquerade rule in firewall.user?

Hey Jow. I have a couple of VMs on 10.1.0.0/24. Although that's not the issue since it has been there for years, I removed it and made no difference.

I'm still having a hard time wrapping my head around your network layout. Can you make some small ascii art diagram or similar?

Hello Jow.

My network is very simple. I have a proprietary modem called Arris something. Unfortunately my ISP does not offer a way to connect using a standard modem. The modem also provides TV channels as well as VOIP.

The modem is set as bridged NAT with the Xiaomi Mini which does NAT under a different subnet.

Modem ip; 192.168.0.1
Xiaomi Mini wan ip; 192.168.0.2
Xiaomi mini LAN ip; 192.168.1.0/24

So modem bridged NAT > Xiaomi mini > LAN 192.168.1.0/24

Anyways. It turned out that I sort of worked around the issue. Since the 2.4Ghz wireless became hell of unstable lately, I decided to turn off the 2.4Ghz wireless interface and hock up a Carambola 2 to serve as gw as well as 2.4Ghz wireless.

So my current setup now is; modem bridged NAT > Carabola 2 gw > LAN > Xiaomi mini that acts like a switch + 5Ghz wireless with dnsmasq and firewall disabled.

Still with this setup the Windows 10 HOST would get disconnected by any VMs through SSH. Powering off the Xiaomi mini and the HOST would be able to ssh to any VMs again.

What I did in the end is I migrated a FreeBSD VM from VMware Workstation 12 to Virtualbox and hell it works. I had the HOST connected to the VM trough ssh the whole night and I didn't get disconnected.

I am sorry for the confusion but I cannot explain what happened. I had this setup for years and VMware never complained. Apparently for some reason VMware is not liking the Xiaomi mini regardless of the fw installed.

Thanks