Possible Static Routing Bug with 172.16.0.0/16 CIDR?

wsy · July 17, 2019, 6:34pm

Sorry about the text formatting. I'm not good at writing MarkDown text.

Version: OpenWrt 18.06.4 r7808-ef686b7292, installed from official combined-ext4 image.
Hardware: Hyper-V Gen 1 VM, i3 8100, 512MiB RAM, 4GiB vhdx, running on top of Windows Server 2019
Network Adapter: 1 for WAN and 1 for LAN
Software: SoftEther VPN Server v4.22 Build 9634, Compiled 2016/11/27 14:33:59(ipk file downloaded from official OpenWrt opkg repository)

Step1. I successfully configured NAT between LAN(192.168.200.0/24) and WAN.
Step2. I created a virtual hub inside SoftEther VPN Server called HubA. HubA has a cascade connection to a remote server called RemoteA. RemoteA is running in NAT mode, with client DHCP range configured to 192.168.222.0/24.
Step3. In SoftEther VPN Server, I created a bridge from HubA to tap_hub_a.
Step4. In LuCI, I configured an interface for tun_hub_a named InterfaceA. InterfaceA successfully got IP address 192.168.222.11/24 with gateway 192.168.222.1.
Step5. I configured some static routes, routing 192.168.222.0/24 and 10.0.0.0/9(reachable by RemoteA) to InterfaceA and 0.0.0.0/0 to WAN.

So far, everything works fine. From LAN side, if I ping 8.8.8.8(I just randomly picked a famous public IP address for testing Internet connectivity), my traffic goes through WAN and I got 0% packet loss. If I ping 192.168.222.1, my traffic goes through tap_hub_a and soft ether forwarded my data to RemoteA, and I also got 0% packet loss. BELOW IS WHERE THE BUG BEGINS.

Step6. I configured another static route entry, routing 172.16.0.0/16(Reachable by RemoteA) to InterfaceA. (No other configuration was changed) Then I ping 192.168.222.1, my traffic still goes through tap_hub_a to RemoteA, but I got regularly packet loss: Exactly ten successful response from RemoteA and then I got about 30 to 40 lost packets. Then another cycle: Exactly 10 successful response and it's again followed by 30 to 40 lost packets.
In each cycle, I got exactly 10 continuously successful responses and then some continuous packet loss (About 30 ~ 40 packets). However, whether I configured 172.16.0.0/16 static route entry or not, I got no problem ping 8.8.8.8 through WAN from LAN side.

Analysis:
Possibility1: Too many static route rules? Without 172.16.0.0/16 CIDR in the rules, I configured more than 10 static route rules and no bug occurred. I tried to configure 172.16.0.0/16 as the only static route rule, and still, I got regularly packet loss.

Possibility2: Is this a bug with Linux TAP device? I tried to install another virtual machine running Ubuntu 19.04 and installed SoftEther VPN Server there, and I used a Hyper-V virtual switch to connect InterfaceA to HubA, and still, I got packet loss with 192.168.222.1 and no packet loss with 8.8.8.8.

Another Attempt: I tear apart the 172.16.0.0/16 subnet into 172.16.128.0/17 + 172.16.64.0/18 + 172.16.32.0/19 + ...... + 172.16.2.0/23 + 172.16.1.0/24 + 172.16.0.0/24.
If I configure 172.16.128.0/17 into static route rules, I got no packet loss to remoteA.
If I configure 172.16.128.0/17 and 172.16.64.0/18 and all the way to 172.16.8.0/21 (5 entries), I got no packet loss to remoteA.
If I configure from 172.16.128.0/17 to 172.16.1.0/24 (8 entries), I got packet loss to remoteA.

I tried to google this problem, and I found one thread saying someone named Lintel has already fixed a regularly-packet-loss bug in hardware driver about three years ago.(That thread is in a Chinese forum)
Now, I think the bug still exists and I believe it's related to 172.16.0.0/16 CIDR.

lleachii · July 17, 2019, 8:41pm

Can you explain why you tested your 172.16.0.0/16 route by pinging 8.8.8.8 and 192.168.80.1?
Can you do a traceroute?
Can you show us the route you added for it in /etc/config/network ?
Why didn't you test by pinging a host in 172.16.0.0/16?
Is 172.16.0.0/16 directly connected to an interface on your router (InterfaceA is 172.16.0.0/16), or downstream of it (on router Remote A on InerfaceA) - the difference is important?

Can you show us the link to this?

wsy · July 18, 2019, 12:04am

Thanks for your reply.
192.168.80.1 is a remote VPN server (let's call it RemoteA) for me to contact my development servers in the cloud. These development servers have address 172.16.0.0/16 and 10.0.0.0/9.
RemoteA is working in NAT mode and it's assigning 192.168.80.0/24 addresses to VPN Clients and it has IP address 192.168.80.1.
So, if I can ping 192.168.80.1, then I can reach 172.16.0.0/16 and 10.0.0.0/9 servers behind 192.168.80.1.

The purpose of setting up this OpenWrt virtual machine is to split my data in my office.

Sorry I didn't put a picture in my thread.

Now, this would better explain my scenario.

wsy · July 18, 2019, 1:31am

This is an OpenWrt setup in my test environment and this is the list of the interfaces in my OpenWrt.

This is my static route configuration. Under this configuration, I got no packet loss to QCloud.

If I add the last route, 172.16.4.0/22, I got packet loss to QCloud.
Or if I use 172.16.0.0/16 directly instead of those 172.16.x.0 above, I still got packet loss to QCloud.

trendy · July 18, 2019, 7:50am

First it is not convenient to paste images with configuration. Better copy the text and paste it here in Preformatted text (Ctrl-Shift-c)

Second can you post the following?

uci show firewall; \
ip -4 addr ; ip -4 ro ; ip -4 ru

wsy · July 19, 2019, 3:58am

Thanks for your reply and sorry for the screenshot. I will see if I can replace those screenshots to text later today.

root@BBB-Router:~# uci show firewall; \
> ip -4 addr ; ip -4 ro ; ip -4 ru
firewall.@defaults[0]=defaults
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='ACCEPT'
firewall.@zone[0]=zone
firewall.@zone[0].name='lan'
firewall.@zone[0].input='ACCEPT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='ACCEPT'
firewall.@zone[0].network='lan'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].masq='1'
firewall.@zone[1].mtu_fix='1'
firewall.@zone[1].network='wan wan6'
firewall.@zone[1].forward='ACCEPT'
firewall.@zone[1].input='ACCEPT'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[2]=rule
firewall.@rule[2].name='Allow-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='ACCEPT'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-DHCPv6'
firewall.@rule[3].src='wan'
firewall.@rule[3].proto='udp'
firewall.@rule[3].src_ip='fc00::/6'
firewall.@rule[3].dest_ip='fc00::/6'
firewall.@rule[3].dest_port='546'
firewall.@rule[3].family='ipv6'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-MLD'
firewall.@rule[4].src='wan'
firewall.@rule[4].proto='icmp'
firewall.@rule[4].src_ip='fe80::/10'
firewall.@rule[4].icmp_type='130/0' '131/0' '132/0' '143/0'
firewall.@rule[4].family='ipv6'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[5]=rule
firewall.@rule[5].name='Allow-ICMPv6-Input'
firewall.@rule[5].src='wan'
firewall.@rule[5].proto='icmp'
firewall.@rule[5].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type' 'router-solicitation' 'neighbour-solicitation' 'router-advertisement' 'neighbour-advertisement'
firewall.@rule[5].limit='1000/sec'
firewall.@rule[5].family='ipv6'
firewall.@rule[5].target='ACCEPT'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-ICMPv6-Forward'
firewall.@rule[6].src='wan'
firewall.@rule[6].dest='*'
firewall.@rule[6].proto='icmp'
firewall.@rule[6].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type'
firewall.@rule[6].limit='1000/sec'
firewall.@rule[6].family='ipv6'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-IPSec-ESP'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='lan'
firewall.@rule[7].proto='esp'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[8]=rule
firewall.@rule[8].name='Allow-ISAKMP'
firewall.@rule[8].src='wan'
firewall.@rule[8].dest='lan'
firewall.@rule[8].dest_port='500'
firewall.@rule[8].proto='udp'
firewall.@rule[8].target='ACCEPT'
firewall.@include[0]=include
firewall.@include[0].path='/etc/firewall.user'
firewall.@rule[9]=rule
firewall.@rule[9].target='ACCEPT'
firewall.@rule[9].src='wan'
firewall.@rule[9].proto='tcp udp'
firewall.@rule[9].dest_port='5555'
firewall.@rule[9].name='SoftEther5555'
firewall.@redirect[0]=redirect
firewall.@redirect[0].target='DNAT'
firewall.@redirect[0].src='wan'
firewall.@redirect[0].dest='lan'
firewall.@redirect[0].proto='tcp'
firewall.@redirect[0].src_dport='443'
firewall.@redirect[0].dest_port='443'
firewall.@redirect[0].name='HTTPS'
firewall.@redirect[0].dest_ip='192.168.124.2'
firewall.@rule[10]=rule
firewall.@rule[10].target='ACCEPT'
firewall.@rule[10].src='wan'
firewall.@rule[10].proto='tcp udp'
firewall.@rule[10].dest_port='500'
firewall.@rule[10].name='IKE'
firewall.@rule[11]=rule
firewall.@rule[11].target='ACCEPT'
firewall.@rule[11].src='wan'
firewall.@rule[11].proto='tcp udp'
firewall.@rule[11].dest_port='4500'
firewall.@rule[11].name='IKE_NAT'
firewall.@rule[12]=rule
firewall.@rule[12].target='ACCEPT'
firewall.@rule[12].src='wan'
firewall.@rule[12].proto='tcp udp'
firewall.@rule[12].dest_port='1701'
firewall.@rule[12].name='L2TP'
firewall.@zone[2]=zone
firewall.@zone[2].output='ACCEPT'
firewall.@zone[2].masq='1'
firewall.@zone[2].forward='ACCEPT'
firewall.@zone[2].name='SoftEther'
firewall.@zone[2].input='REJECT'
firewall.@zone[2].mtu_fix='1'
firewall.@zone[2].network='QCloud WSY_HK'
firewall.@redirect[1]=redirect
firewall.@redirect[1].target='DNAT'
firewall.@redirect[1].src='wan'
firewall.@redirect[1].dest='lan'
firewall.@redirect[1].proto='tcp'
firewall.@redirect[1].src_dport='3389'
firewall.@redirect[1].dest_port='3389'
firewall.@redirect[1].name='RDP'
firewall.@redirect[1].dest_ip='192.168.124.2'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].dest='wan'
firewall.@forwarding[0].src='lan'
firewall.@rule[13]=rule
firewall.@rule[13].target='ACCEPT'
firewall.@rule[13].src='wan'
firewall.@rule[13].dest='lan'
firewall.@rule[13].name='Default'
firewall.@rule[14]=rule
firewall.@rule[14].target='ACCEPT'
firewall.@rule[14].src='wan'
firewall.@rule[14].proto='tcp'
firewall.@rule[14].dest_port='80'
firewall.@rule[14].name='HTTP'
firewall.@redirect[2]=redirect
firewall.@redirect[2].target='DNAT'
firewall.@redirect[2].src='wan'
firewall.@redirect[2].dest='lan'
firewall.@redirect[2].proto='tcp udp'
firewall.@redirect[2].src_dport='33889'
firewall.@redirect[2].dest_port='33889'
firewall.@redirect[2].name='RDP2'
firewall.@redirect[2].dest_ip='192.168.124.2'
firewall.@redirect[3]=redirect
firewall.@redirect[3].target='DNAT'
firewall.@redirect[3].src='wan'
firewall.@redirect[3].dest='lan'
firewall.@redirect[3].proto='tcp udp'
firewall.@redirect[3].src_dport='33389'
firewall.@redirect[3].dest_port='33389'
firewall.@redirect[3].name='RDP3'
firewall.@redirect[3].dest_ip='192.168.124.2'
firewall.@forwarding[1]=forwarding
firewall.@forwarding[1].dest='SoftEther'
firewall.@forwarding[1].src='lan'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 192.168.122.4/24 brd 192.168.122.255 scope global eth1
       valid_lft forever preferred_lft forever
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP100> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.124.1/24 brd 192.168.124.255 scope global br-lan
       valid_lft forever preferred_lft forever
7: tap_qcloud: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN qlen 1000
    inet 192.168.222.12/24 brd 192.168.222.255 scope global tap_qcloud
       valid_lft forever preferred_lft forever
default via 192.168.122.1 dev eth1  src 192.168.122.4  metric 3
10.0.0.0/9 via 192.168.222.1 dev tap_qcloud  metric 1
172.16.8.0/21 via 192.168.222.1 dev tap_qcloud  metric 1
172.16.16.0/20 via 192.168.222.1 dev tap_qcloud  metric 1
172.16.32.0/19 via 192.168.222.1 dev tap_qcloud  metric 1
172.16.64.0/18 via 192.168.222.1 dev tap_qcloud  metric 1
172.16.128.0/17 via 192.168.222.1 dev tap_qcloud  metric 1
192.168.122.0/24 dev eth1 scope link  metric 3
192.168.124.0/24 dev br-lan scope link  src 192.168.124.1
192.168.222.0/24 via 192.168.222.1 dev tap_qcloud  src 192.168.222.12  metric 1
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

eduperez · July 19, 2019, 7:14am

I would use "traceroute" instead of "ping", to see where are the packets being lost.

trendy · July 19, 2019, 10:41am

Are you sure you are using tun and not tap? In your configuration I can see a lot of tap interfaces. In your case you can use the tun interface.
Also I cannot spot the addresses you mention in your first two posts, like 192.168.80.0/24 or 192.168.200.0/24. They are not in the routing table.
Other than that, verify that there is a return route for all those networks that you try to reach and that there is no routing loop on the other side.

wsy · July 22, 2019, 6:39am

With traceroute command, you can only do one-time trace. What I encountered was periodically packet-loss.
By the way, my servers behind 192.168.222.1 does not answer to ICMP requests.

wsy · July 22, 2019, 6:40am

Above configuration is from a test environment which is built to repro and debug this issue.
My local LAN has IPSeg 192.168.200.0/24
WAN Side has IPSeg 192.168.122.0/24 which connects to Internet through another router.
QCloud is the interface to my 'server environment' which has 192.168.222.0/24. And its gateway, 192.168.222.1, connects to another 'server environment' which has IPSeg 172.16.0.0/16 and 10.0.0.0/9.

I'm sorry I don't quite know the difference between tap and tun devices. Here's my screenshot of SoftEther and TAP is the only option.

About this, I'm sure there's no routing loop.
And the periodical packet-loss only occures when I configured 172.16.0.0/16 or 172.16.4.0/22 and occurs only on that specific interface.

mikma · July 22, 2019, 7:04am

Tap is layer 2 (Ethernet) which is bridged and tun is layer 3 (IP) which is routed.

wsy · July 22, 2019, 7:13am

Thanks for your knowledge. Now I understand their differences.

wsy · July 22, 2019, 7:15am

Thank you all for your advice. I have edited my original post to better explain my problem.

moeller0 · July 22, 2019, 8:27am

Try mtr then, which will allow runner ng for long time intervals and which reports percent packet loss, mean, maximum and minimum RTT, as well standard deviation, the max should catch your periodic loss events....

trendy · July 22, 2019, 12:34pm

I presume this SoftEther is the software running on the Server residing in the Cloud segment.
For sure you should not mix tun and tap interfaces.
And if you have a lot of tap interfaces you need to be careful not to have loops in layer 2.
Unfortunately I have no experience with SoftEther, therefore I cannot help further, however I don't see anything wrong in the OpenWrt configuration. The amount of routes cannot affect the packet loss, unless there is some routing issue.

vgaetera · July 22, 2019, 2:28pm

Did you try to use other VPN solutions?
SoftEther VPN may be a good choice if you just want to work and not think how it works.
But if you look inside, there are some things that may cause your concern.

beh0ld · December 16, 2019, 8:12am

Hi! Did you resolve this issue? I have the same problem.

wsy · December 19, 2019, 5:31am

No, it's not solved. I gave up on routing 172.16.0.0/16 and used 172.16.255.0/24 instead.
Although I got some services running on 172.16.3.0/24 and 172.16.4.0/24, I was able to find some work arounds for them. So I just routed 172.16.255.0/24.