[Tutorial] Build OpenWrt with Multipath TCP

Hey Andy. I plan to write a full-fledged tutorial about MPTCP aggregation and encryption as a whole, (including v2ray, wireguard, configuration on the client & server side).

In the meantime, since you already set up your VPS using OMR script, I can just refer you to an example v2ray configuration.

For OpenWrt, you need to compile iptables-mod-tproxy with the image to use Tproxy method of transparent proxy, so you can forward all the traffic to v2ray and aggregate links.

Server:

{
  "inbounds": [
    {
      "port": 31351,
      "protocol": "socks",
      "settings": {
        "auth": "password",
        "accounts": [
          {
            "user": "change this",
            "pass": "change this"
          }
        ]
      }
    }
  ],
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {}
    }
  ]
}

Client:

Put this config to /etc/v2ray/config.json

{
  "inbounds": [
    {
      "port": 12345,
      "listen": "127.0.0.1",
      "protocol": "dokodemo-door",
      "settings": {
        "network": "tcp",
        "followRedirect": true
      },
      "streamSettings": {
        "sockopt": {
          "tproxy": "tproxy"
        }
      }
    }
  ],
  "outbounds": [
    {
      "protocol":"socks",
      "settings": {
        "servers": [
          {
            "address": "ServerIP",
            "port": 31351,
            "users": [
              {
                "user": "change this",
                "pass": "change this"
              }
            ]
          }
        ]
      }
    }
  ]
}

iptables OpenWrt

Change your Server IP on the iptables rule then issue the commands.

# Identify packets with destination address matching a local socket, set the packet mark to 1
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT

# Match on packet mark 1 using policy routing to have those packets delivered locally
ip rule add fwmark 1 table 100
ip route add local default dev lo table 100

# Create new chain
iptables -t mangle -N V2RAY

# Ignore v2ray server address, routed devices won't be able to reach the server with this
# iptables -t mangle -A V2RAY -d "ServerIP" -j RETURN

# Ignore local IPv4 addresses to bypass the proxy
iptables -t mangle -A V2RAY -d 0.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A V2RAY -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A V2RAY -d 192.168.0.0/16 -j RETURN
iptables -t mangle -A V2RAY -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A V2RAY -d 240.0.0.0/4 -j RETURN

# Anything else should be redirected to Dokodemo-door's local port
iptables -t mangle -A V2RAY -p tcp -j TPROXY --on-ip 127.0.0.1 --on-port 12345 --tproxy-mark 1

# Transparent proxy for routed devices
iptables -t mangle -A PREROUTING -j V2RAY

# ---

# Create new chain for proxying the router
iptables -t mangle -N V2RAY_LOCAL

# Ignore v2ray server address
iptables -t mangle -A V2RAY_LOCAL -d "ServerIP" -j RETURN

# Ignore local IPv4 addresses to bypass the proxy
iptables -t mangle -A V2RAY_LOCAL -d 0.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 10.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 127.0.0.0/8 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 169.254.0.0/16 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 172.16.0.0/12 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 192.168.0.0/16 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 224.0.0.0/4 -j RETURN
iptables -t mangle -A V2RAY_LOCAL -d 240.0.0.0/4 -j RETURN

# Mark anything else as 1
iptables -t mangle -A V2RAY_LOCAL -p tcp -j MARK --set-mark 1

# Transparent proxy for the router
iptables -t mangle -A OUTPUT -j V2RAY_LOCAL

You can put the iptables rules on firewall settings and ip rule & ip route commands to startup section on LuCI for the configuration to survive reboots.

v2ray OpenWrt startup script

I also wrote a simple script to automatically initialize v2ray after reboot. Put it to /etc/init.d/v2ray
Run service v2ray enable && service v2ray start

#!/bin/sh /etc/rc.common

USE_PROCD=1

START=99

CONF=/etc/v2ray/config.json
EXEC=/usr/bin/v2ray

start_service() {
  procd_open_instance
  procd_set_param command $EXEC -c $CONF
  procd_set_param file $CONF
  procd_set_param pidfile /var/run/v2ray.pid
  procd_set_param respawn
  procd_close_instance
}

start() {
        service_start $EXEC
}

stop() {
        service_stop $EXEC
}

reload() {
        service_reload $EXEC
}

Your OpenWrt router should survive reboots and automatically aggregate links now.

Hi Arinc9,
Thanks for the reply. Because the OMR VPS doesn't support the socks protocol but the shadowsocks. So I try to change client outbound protocol into shadowsocks.
And I see your tproxy is for the client inbound streaming method.
Is the tproxy necessary for the inboud ?
Thanks for the reply again.

Best,
Andy

It’s not OMR that supports SOCKS. It’s v2ray. If you have v2ray (you do) then you can use it.
Tproxy is required on the client side. I recommend you not to change anything but the server IP on the v2ray configurations.
It will work fine (enable sniffing if it won’t).

Hi Arinc9,
It's OMR VPS v2ray server config.

{
	"log": {
		"loglevel": "warning",
		"error": "/tmp/v2rayError.log"
	},
	"transport": {
		"tcpSettings": {},
		"wsSettings": {},
		"kcpSettings": {
			"mtu": 1460,
			"tti": 10,
			"uplinkCapacity": 100,
			"downlinkCapacity": 100,
			"congestion": false,
			"readBufferSize": 8,
			"writeBufferSize": 8
		}
	},
	"inbounds": [
		{
			"tag": "omrin-tunnel",
			"port": 65228,
			"protocol": "vless",
			"settings": {
				"decryption": "none",
				"clients": [
					{
						"id": "5b6d20d7-1495-4028-ade4-41727f8d47d3",
						"level": 0,
						"alterId": 0,
						"email": "openmptcprouter"
					}
				]
			},
			"streamSettings": {
				"sockopt": {
					"mark": 0
				},
				"network": "tcp",
				"security": "tls",
				"tlsSettings": {
					"certificates": [
						{
							"certificateFile": "/etc/openvpn/ca/pki/issued/server.crt",
							"keyFile": "/etc/openvpn/ca/pki/private/server.key"
						}
					]
				}
			}
		},
		{
			"listen": "127.0.0.1",
			"port": 10085,
			"protocol": "dokodemo-door",
			"settings": {
				"address": "127.0.0.1"
			},
			"tag": "api"
		}
	],
	"outbounds": [
		{
			"protocol": "freedom",
			"settings": {
				"userLevel": 0
			},
			"tag": "direct"
		}
	],
	"routing": {
		"rules": [
			{
				"type": "field",
				"inboundTag": [
					"omrin-tunnel"
				],
				"outboundTag": "OMRLan",
				"domain": [
					"full:omr.lan"
				]
			},
			{
				"inboundTag": [
					"api"
				],
				"outboundTag": "api",
				"type": "field"
			}
		]
	},
	"reverse": {
		"portals": [
			{
				"tag": "OMRLan",
				"domain": "omr.lan"
			}
		]
	},
	"stats": {},
	"api": {
		"tag": "api",
		"services": [
			"HandlerService",
			"LoggerService",
			"StatsService"
		]
	},
	"policy": {
		"levels": {
			"0": {
				"uplinkOnly": 0,
				"downlinkOnly": 0,
				"bufferSize": 512,
				"connIdle": 2400,
				"statsUserUplink": true,
				"statsUserDownlink": true
			}
		},
		"system": {
			"statsInboundUplink": true,
			"statsInboundDownlink": true
		}
	}
}

So ,

  1. If I don't want change my OMR VPS v2ray server config
    I need to modify client outbound to match OMR VPS v2ray inbound setting
    But I can't find the protocol "vless" setting which OMR VPS v2ray used.
    ---> what should I config to match OMR VPS inbound setting ?

  2. Another question is how does OMR VPS know using the "dokodemo-door" with 12345 port to match my client inbound rule ?
    --> I found the answer , it's because this iptables rules

# Anything else should be redirected to Dokodemo-door's local port
iptables -t nat -A V2RAY -p tcp -j REDIRECT --to-ports 12345

Sorry for the stupid questions and thank your again.

Best,
Andy

You will want to, change it with what I provided for server above. If not, I can't help you.

Server side doesn't know that, you're forwarding traffic from dokodemo-door to socks and vice versa on the client. dokodemo-door is used to achieve transparent proxy on the client.
If you want to understand how v2ray works, read the v2ray documentation. I can't help you.

Just do what I advised and you'll be fine.

Hi Arinc9,
I follow your comments.

It's my openwrt side v2ray config ( /etc/v2ray/v2ray.main.json)

{
	"log": {
		"access": "/dev/null",
		"loglevel": "warning",
		"error": "/var/log/v2ray-error.log"
	},
	"inbounds": [
		{
			"listen": "127.0.0.1",
			"port": 12345,
			"protocol": "dokodemo-door",
			"settings": {
				"followRedirect": true,
				"network": "tcp"
			},
			"streamSettings": {
				"sockopt": {
					"tproxy": "tproxy"
				}
			},
			"tag": "transparent",
			"sniffing": {
				"enabled": true,
				"destOverride": [
					"http",
					"tls"
				]
			}
		}
	],
	"outbounds": [
		{
			"protocol": "socks",
			"settings": {
				"servers": [
					{
						"address": "x.x.x.x",
						"port": 31351,
						"users": [
							{
								"user": "andy",
								"pass": "andy"
							}
						]
					}
				]
			},
			"streamSettings": {
				"sockopt": {
					"tcpFastOpen": true
				}
			},
			"tag": "proxy"
		}
	]
}

It's my VPS side v2ray config

{
        "inbounds": [
                {
                        "port": 31351,
                        "protocol": "socks",
                        "settings": {
                                "auth": "password",
                                "accounts": [
                                        {
                                                "user": "andy",
                                                "pass": "andy"
                                        }
                                ]
                        }
                }
        ],
        "outbounds": [
                {
                        "protocol": "freedom",
                        "settings": {}
                }
        ]
}

And fill the firewall rules in custom rules field.
But I found I can't open the webpages.
And there is no connections entry in /proc/net/mptcp_net/mptcp .

Do you know what else I miss ?

Thanks,
Andy

DM me your server IP. I’ll see if the server is working properly.

Hi Arinc9,
I found I can't sent message through this forum.
I sent you by the gmail.
Hope you don't mind.

Best,
Andy

Hi Arinc9,
I reinstall a clean VPS which only install v2ray.
And I can bypass my openwrt traffic to VPS with the same configurations which I posted.
But the mptcp feature doesn't work, the traffic only go though the first route in my routing table.
WAN and LAN2 are my openwrt wan interface name.

I still can't see any connections in mptcp node.
Do you have any ideas ?

Thanks,
Andy

root@OpenWrt:/proc/sys/net/mptcp# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 lan2
0.0.0.0         192.168.8.1     0.0.0.0         UG    0      0        0 wan
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 lan1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 lan2
192.168.8.0     0.0.0.0         255.255.255.0   U     0      0        0 wan
root@OpenWrt:/proc/sys/net/mptcp# cat mptcp_enabled 
1
root@OpenWrt:/proc/sys/net/mptcp# cat mptcp_path_manager 
fullmesh
root@OpenWrt:/proc/sys/net/mptcp# cat mptcp_scheduler 
blest
root@OpenWrt:/proc/sys/net/mptcp# cat mptcp_syn_retries 
3
root@OpenWrt:/proc/sys/net/mptcp# cat /proc/net/mptcp_fullmesh 
Index, Address-ID, Backup, IP-address, if-idx
IPv4, next v4-index: 5
2, 3, 0, 10.10.10.1, 5
3, 4, 0, 192.168.1.115, 4
4, 5, 0, 192.168.8.100, 3
IPv6, next v6-index: 1
0, 8, 0, fd90:0551:e3a2:0000:0000:0000:0000:0001, 5
root@OpenWrt:/proc/sys/net/mptcp# cat /proc/net/mptcp_net/
mptcp  snmp
root@OpenWrt:/proc/sys/net/mptcp# cat /proc/net/mptcp_net/mptcp 
  sl  loc_tok  rem_tok  v6 local_address                         remote_address                        st ns tx_queue rx_queue inode

You also need linux kernel on the VPS with mptcpv0 support.

You can find my precompiled releases here:

Download the defconfig, linux-image-5.4.132

Hi Arinc9,
I install the 5.4.100-mptcp kernel from OMR repo.

andy@v2ray:/proc/sys/net/mptcp$ uname -a
Linux v2ray 5.4.100-mptcp #1 SMP Wed Feb 24 09:39:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
andy@v2ray:/proc/sys/net/mptcp$ lsmod | grep mptcp
mptcp_blest            16384  0
mptcp_ecf              16384  0
mptcp_rr               16384  0
mptcp_redundant        16384  0
mptcp_ndiffports       20480  0
mptcp_balia            20480  0
mptcp_wvegas           16384  0
mptcp_olia             20480  0

And right now I can see connections entry from /proc/net/mptcp_net/mptcp in both sides.
But the traffic still only goes through only one interface, no aggregation. And it seems very slow. (1xx Mbps without mptcp vs 3Mbps with mptcp kernel )
Is there any parameter need to check ?

Thanks,
Andy

Sorry, I can’t help any further.

Thanks you for the help these days.

Hi Arinc9,

As per your comment, you are planning to use wireguard as well.. May I know how will wireguard work along with MPTCP. Is it for UDP traffic ?

Does it do multipath as well..

If yes, how to set it up ?

Thanks

I never got to writing the tutorial. I still plan to do it, but it's not a high priority right now.

I have a setup that I use for weeks now. I'm very satisfied with it.
I use OpenWrt 21.02, Linux kernel 5.4.x with MPTCPv0. I got v2ray, wireguard & TPROXY feature of iptables.
I forward TCP traffic on anything but the router itself (PREROUTING) to v2ray which will go over multiple links thanks to MPTCP. This goes cleartext, because my router can only perform max of 200mbps with ChaCha & 100mbps with AES. I need 400mbps.
For non-TCP traffic & to reach my devices from outside, I use WireGuard. This goes encrypted.

I wrote an initscript to launch v2ray at boot. WireGuard already does this. Then, I made iptables & ip commands for TPROXY to run at boot.
Everything will initialize at boot, completely autonomous.

Until I write the tutorial, this page I wrote has most of this covered:

In short, TCP traffic is aggregated. Non-TCP traffic goes over WireGuard on a single link.

1 Like

Thanks for the reply..

I wanted to discuss a usecase. Lets say I have multiple branch offices each having 2-3 wan connections. I place a high-end x86 box running v2ray with vless along with tls encryption.

Would the server be able to aggregate the connections of say, 50 branches. If yes, what kind of configuration should I be looking at for the server.

To optimize, I will not pass the internet traffic to the server, just the enterprise traffic like internal applications.

Any suggestions ?

I don't think it's the amount of connections (branches) to the server but rather how much throughput the server will have to handle. And for that, it really depends on the specifs of your server. For example, I got a VPS with 2.5GHz 1-core CPU. Using v2ray's SOCKS implementation (cleartext) creates ~75% load on 411mbps throughput.

If you plan to aggregate TCP traffic just to reach corporate network:
instead of the iptables rules like -A PROXY -d 0.0.0.0/8 -j RETURN
You could change:
-A PROXY -p tcp -j TPROXY --on-port 12346 --on-ip 127.0.0.1 --tproxy-mark 0x1/0xffffffff
to
-A PROXY -d 10.0.0.0/24 -p tcp -j TPROXY --on-port 12346 --on-ip 127.0.0.1 --tproxy-mark 0x1/0xffffffff

Meaning that only destination to 10.0.0.0/24 will go over v2ray, therefore aggregated. You can change the IPv4 subnet with whatever the hosts on your corporate network operate at.

I can't give you specific configuration for your usecase, as I don't have time or interest to do so. My Notion pages will help you figure it out yourself.

Understood !!

Sorry to bother you but last 2 questions.

A. I was going through this document :

There are 2 ways to direct the traffic - tproxy and redirect. Can you suggest which one is preferable.

B. Have you found any good way to multipath udp traffic. Openmptcprouter gives a lot of ways like glorytun, mlpvpn , dsvpn etc. But are they any good ? What's your take on that. OR should we just loadbalance udp traffic and ignore aggregation ?

I suggest TPROXY, linux documentation explains clearly why this is better.

I did mess with aggregating links over UDP. I tried glorytun, MLVPN and ubond. The biggest problem I had was automation. Every time my router reboots or a link goes down and up, I had to reconfigure the program with the newest IP addresses the ISPs assigned to my device. That's why I like MPTCPv0. It's got a full-mesh path manager which sets up all links automatically.

Encryption is a problem too. I don't want my TCP traffic to be encrypted because I require high throughput and my router can't handle it either with ChaCha or AES. However I still want my DNS traffic to go over encrypted.

So I landed with the solution I use now.

1 Like

Thanks a lot for the info. Could you please recommend VM configuration: how much gb of ram needed? is it ok to run with 1gb on vm for 2-4 openvpn connections?