A guide for Japan NTT users on IPoE/IPv6 setting (MAP-E only)

Installing and Using OpenWrt Network and Wireless Configuration
1

For the fiber internet services in Japan, unless you are using 10G one (which is very new), most 1G fiber connections come with 2 ways of connection: PPPoE & IPoE.

The PPPoE is IPv4 only (some providers have IPv6 PPPoE before but I think should be deprecated now) and it can be slow in busy hours or areas, also the PPPoE dialup speed really depends on router CPU single core speed so even my older x86 router cannot get better PPPoE speed. ISP has provided IPv6 which doesn't need any dialup, and to retain IPv4 connectivity they use a few different types of techniques: MAP-E/transix/DS-LITE.

ISP has a supplied router (or rental) to achieve this, however if we want custom router to be used it's very difficult due to lack of some important information. I have written a guide and tested on my NTT 1G internet (provider: plala), to address the 2 issues: Only /64 prefix assignment without router advertisement (RA); MAP-E tunnel which is important if you need to use IPv4.

There were many Japanese trying to work on this, information spreading around but mostly incomplete (I was never able to work it out with regarding to a single guideline), looked at OpenWrt forum here there is also no actual solution can be found for such a special environment in Japan, so I've spent a few months to test and make configuration changes to see how I can make it work with my own OpenWrt router.

My guide was a bit long so I wrote it in GitHub: https://github.com/fakemanhk/openwrt-jp-ipoe (Since I don't have transix/DS-LITE connection so I can only workout the MAP-E solution).

After implementing the solution, I am able to kick out the ISP router, with my OpenWrt router connecting ISP ONU directly, all clients are able to get IPv6 addresses with same prefix as WAN6 interface, and pure IPv4 connection is also possible.

5 Likes
2

Got confirmation from another user with a different provider (same NTT brand) that using my approach and it also worked!

So as long as you are NTT Docomo based network, MAP-E implementations among all providers are the same.

3

I have a line working with IPoE+DS-Lite on NTT with Asahi-Net. There's nothing particular about it, DS-Lite installation and configuration is completely straightforward.

The only challenge is finding the DS-Lite AFTR address for Transix v6connect. For some reason, people are hesitant to write out that address in public, it's always just spoken of in a hushed voice and I'm not entirely sure why that is. Do people think that 2001:c28:5:301::11 is some sort of trade secret, or mentioning it opens one up to DDOS or something?

Speed-wise I don't see any difference between PPPoE and DS-Lite here, but I hear that differs between areas (we are in Kansai) or even between lines. I have PPPoE running on the same router, mainly for incoming connections.

1 Like
4

Yeah I tried to look around and found more people talking about DS-Lite, seems to be easier.
MAP-E config is not really a secret, because when I log in the rental router some important parameters are already there, and then the rules are pretty much based on standard IETF calculations, i.e. everyone would be able to calculate.
However all those Japanese websites I found, they started to test MAP-E on 19.07 or even earlier but I couldn't get success by following instructions, while most of their "successful screenshots" are gone (maybe too old, server removed), it was a pain for me to setup the whole thing.

In some not so busy areas, like me living in Kanagawa, it's also not too much difference. While I heard from some people living in central Tokyo, especially the "closed down" dates during COVID, working from home with or without using IPoE can show significant difference. I didn't feel that it was probably far away from Tokyo, however I noticed that PPPoE is using single core so even my Celeron N2930 based x86 cannot achieve > 750M, but with IPoE, the thread limiting factor is gone, I tried to use my Linksys WRT3200ACM (7 years old!) and I can get ~900M easily.

5

I think maybe it's incomplete. There are some iptables (or nftables with 22.03 and firewall4) rules and a modification to the /lib/netifd/proto/map.sh file that you may need to put in in order to get all the ports to be used properly. These modifications mark the packets and distribute them (but cause things like mwan3 to break in the process)

There is the "nichiban benchmark" where you try to open https://www.nichiban.co.jp/ and see how it does. Usually I've needed to add the rules for it to pass the "nichiban benchmark" or else the number of usable ports are all used up and things don't go well.

I also noticed on versions that use fw4, you can't ping any external ipv4 addresses, but with Openwrt 21.X you can. Are you able to ping out?

The rules are listed on this link
https://blog.mamemaki.com/entry/2019/12/15/171104

Working nftables versions of these rules would be something like this:
https://pastebin.com/raw/NHe0x5ub

On top of what you have said to do, I can add the following as steps I've made:

  1. Add the following changes to /lib/netifd/proto/map.sh
    https://pastebin.com/raw/ydLaT2MA

  2. (Assuming you are on 22.03 or a version with fw4) Make a file called /etc/firewall.user and put the following in it:
    https://pastebin.com/raw/NHe0x5ub

  3. Add the following lines to /etc/config/firewall
    https://pastebin.com/raw/JHUetmUs

This will get it using all the ports, at the least, and it will pass the nichiban benchmark.

There is also the issue of only getting a /64 prefix. You can work around this as well, if you want to, by setting a static address with a /65 prefix on the WAN interface, and another with a /65 prefix on a different subnet on the LAN interface, to split the /64 prefix. Then disable SLAAC and use dhcpv6 to give clients v6 addresses. I find that in order to manage ipv6, use things like masq6 if needed, and just to generally have IPV6 reliably work I need to do this, but it's a different discussion.

1 Like
6

But if you split subnet then you are doing NAT on IPv6?
In my original write up since we relay everything so LAN side with the same /64 also getting DHCPv6 (not SLAAC)

BTW did you linked to wrong benchmark website? I didn't find any benchmark thing there.

And yeah I do believe something is wrong on firewall rule that IPv4 pings not passing (I saw that from firewall logs), that's why I am throwing out this so to bring up discussions, myself is not too familiar with tunneling thing.

7

I need NAT on IPV6 for mwan3 and VPNs to work. But it is a separate discussion.

The site is a site showing products for sale made by nichiban. It is not a site designed as a benchmark. That site uses a lot of sessions, so whether the contents on that site loads up properly is considered a benchmark in the Japanese community for testing implementation of MAP-E correct or not, because it's so easy to run out of ports. If you read Japanese, you can see a lot of talk about it on twitter, like below link.
https://twitter.com/search?q=ニチバンベンチ&src=typed_query

You can also notice some issues like very slow to connect, or speed test not working well at times, because the used ports are overloaded. I re-confirmed with latest OpenWRT and the modifications are indeed still necessary unfortunately.

Pings work OK with fw3 versions of openwrt, but not fw4. I don't know the reason, though. I haven't really looked into it much.

8

OK I get what you mean, the website does load, but as I clicking deeper inside it's getting much slower.

In my GitHub page there is another discussion about map.sh change, is that you as well? (However the map.sh content change looks difference from yours after running a diff check) Seems we are all trying to find solution to optimize.

BTW when I just finished the setup with default map.sh, I already saw that under firewall rules there is thousand of ports listed to map the ports (of course I didn't check one by one yet), maybe I should spend some more time to read them all.

9

I found out that the map.sh script is trying to map ICMP to a port, but fails due to some check in OpenWRT that does not allow ICMP to be mapped to a port. Doing so manually via nft (position is # handle at the end of the rule - you can get it by "nft -a list ruleset") and place it in between one of the rules in chain snat.
Here's the command I ran (replace position, oifname, IP and port range to match your setup):

nft add rule inet fw4 srcnat position 999 meta nfproto ipv4 meta l4proto icmp oifname "map-replaceme" counter packets 0 bytes 0 snat ip to 1.1.1.1:1234-1234

I needed only one rule for ping to work, but if we can make it generate the rules, then all can be used.

10

Why do we need port range for ICMP?

And I've checked the firewall already had the following srcnat rule which should indicate that ping is allowed?

ICMP egress device [MAP interface] rewrite source to [MAP IPv4 address]

11

Great work! You are right, map.sh has these troubles in fw4. In fw3, it worked.
This effectively fixed ping for me. Now just need to automate it. I will look into it.

12

So we can fix ping with a (very hacky) edit to /lib/netifd/proto/map.sh

I just added a new table in nftables called inet nathack and put the necessary icmp rules there, and put it in map.sh to automate it.

I can confirm that this effectively lets pings work.

If you are not using fwmarks to determine ports:

+nft add table inet nathack
+nft add chain inet nathack srcnat {type nat hook postrouting priority 0\; policy accept\; }
	    for portset in $(eval "echo \$RULE_${k}_PORTSETS"); do
              for proto in icmp tcp udp; do
	        json_add_object ""
...

                  json_add_string snat_port "$portset"
	        json_close_object
              done
+	      nft add rule inet nathack srcnat meta nfproto ipv4 meta l4proto icmp oifname "map-$cfg" counter packets 0 bytes 0 snat ip to $(eval "echo \$RULE_${k}_IPV4ADDR"):$portset

If you are using firewall.user to set the fwmark (recommended if it is JPNE/v6plus)

	json_add_array firewall
	  if [ -z "$(eval "echo \$RULE_${k}_PORTSETS")" ]; then
	    json_add_object ""
	      json_add_string type nat
	      json_add_string target SNAT
	      json_add_string family inet
	      json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
	    json_close_object
	  else
+local mark=10
+nft add table inet nathack
+nft add chain inet nathack srcnat {type nat hook postrouting priority 0\; policy accept\; }
	    for portset in $(eval "echo \$RULE_${k}_PORTSETS"); do
              for proto in icmp tcp udp; do
	        json_add_object ""
	          json_add_string type nat
	          json_add_string target SNAT
	          json_add_string family inet
	          json_add_string proto "$proto"
		 + json_add_string mark "$mark"
                  json_add_boolean connlimit_ports 1
                  json_add_string snat_ip $(eval "echo \$RULE_${k}_IPV4ADDR")
                  json_add_string snat_port "$portset"
	        json_close_object
              done
+	      nft add rule inet nathack srcnat meta nfproto ipv4 meta l4proto icmp oifname "map-$cfg" meta mark $mark counter packets 0 bytes 0 snat ip to $(eval "echo \$RULE_${k}_IPV4ADDR"):$portset

+	      mark=`expr $mark + 1`

	    done
	  fi
	  if [ "$maptype" = "map-t" ]; then
1 Like
13

Just tested this and it worked perfectly. Seems like a complete solution now with fwmark. All sites load, passes nichibanbench and the connection is just as fast as IPv6 (350/350).

14

Almost complete solution....
There is the issue of fwmarks interfering with other things that use fwmarks like mwan3 or policy based routing.
But as long as those functions are not needed it's good enough I suppose.

In my case I need these functions, so I just run two OpenWRT VMs.

15

What was affected? Since I am also planning to put VPN on it and do policy routing

16

The basic thing that causes affects is that everything is fwmarked, and the fwmark that is set by other things like policy based routing get overwritten.

So far example, if you use mwan3, and tell it to route some traffic through another interface, it will all still go through the map-e interface.

I think by using masks, it can be resolved maybe, by editing the map.sh and the firewall.user files correctly, but I haven't looked into it hard enough. Maybe I will dedicate some time to do it.

17

I just managed to set up Wireguard via IPv4, but the performance is terrible at only 2mbps. Doing the same via IPv6 gives me around 152mbps. CPU usage is low, so not really sure what's causing it.

18

router as WG client? Did you try OpenVPN?

19

Understand, I have never done this with OpenWrt before (I use pfSense before and it's not working in this way), this packet marking way is more or less like the RouterOS from Mikrotik, so if we implement solution with mwan3 we have to think about it. Just don't know the precedence of fwmark, if we also use mwan3, will the fwmark by map-e go first (i.e. will be overwritten by mwan3 rules) or the other way?

I suggest to follow up this in my GitHub page and open new issue there to discuss (as this thread here going longer would be difficult to track everything)

20

I am adjusting the fwmark rules to use masks instead. It's still a hacky solution, but it's likely to work. Will do some tests and ensure mwan and pbr working then post the working files on your github.

EDIT: I have now fixed policy based routing and mwan. Now it is a complete solution. I will post everything at your github in the discussion as well.

1 Like