For the fiber internet services in Japan, unless you are using 10G one (which is very new), most 1G fiber connections come with 2 ways of connection: PPPoE & IPoE.
The PPPoE is IPv4 only (some providers have IPv6 PPPoE before but I think should be deprecated now) and it can be slow in busy hours or areas, also the PPPoE dialup speed really depends on router CPU single core speed so even my older x86 router cannot get better PPPoE speed. ISP has provided IPv6 which doesn't need any dialup, and to retain IPv4 connectivity they use a few different types of techniques: MAP-E/transix/DS-LITE.
ISP has a supplied router (or rental) to achieve this, however if we want custom router to be used it's very difficult due to lack of some important information. I have written a guide and tested on my NTT 1G internet (provider: plala), to address the 2 issues: Only /64 prefix assignment without router advertisement (RA); MAP-E tunnel which is important if you need to use IPv4.
There were many Japanese trying to work on this, information spreading around but mostly incomplete (I was never able to work it out with regarding to a single guideline), looked at OpenWrt forum here there is also no actual solution can be found for such a special environment in Japan, so I've spent a few months to test and make configuration changes to see how I can make it work with my own OpenWrt router.
After implementing the solution, I am able to kick out the ISP router, with my OpenWrt router connecting ISP ONU directly, all clients are able to get IPv6 addresses with same prefix as WAN6 interface, and pure IPv4 connection is also possible.
I have a line working with IPoE+DS-Lite on NTT with Asahi-Net. There's nothing particular about it, DS-Lite installation and configuration is completely straightforward.
The only challenge is finding the DS-Lite AFTR address for Transix v6connect. For some reason, people are hesitant to write out that address in public, it's always just spoken of in a hushed voice and I'm not entirely sure why that is. Do people think that 2001:c28:5:301::11 is some sort of trade secret, or mentioning it opens one up to DDOS or something?
Speed-wise I don't see any difference between PPPoE and DS-Lite here, but I hear that differs between areas (we are in Kansai) or even between lines. I have PPPoE running on the same router, mainly for incoming connections.
Yeah I tried to look around and found more people talking about DS-Lite, seems to be easier.
MAP-E config is not really a secret, because when I log in the rental router some important parameters are already there, and then the rules are pretty much based on standard IETF calculations, i.e. everyone would be able to calculate.
However all those Japanese websites I found, they started to test MAP-E on 19.07 or even earlier but I couldn't get success by following instructions, while most of their "successful screenshots" are gone (maybe too old, server removed), it was a pain for me to setup the whole thing.
In some not so busy areas, like me living in Kanagawa, it's also not too much difference. While I heard from some people living in central Tokyo, especially the "closed down" dates during COVID, working from home with or without using IPoE can show significant difference. I didn't feel that it was probably far away from Tokyo, however I noticed that PPPoE is using single core so even my Celeron N2930 based x86 cannot achieve > 750M, but with IPoE, the thread limiting factor is gone, I tried to use my Linksys WRT3200ACM (7 years old!) and I can get ~900M easily.
I think maybe it's incomplete. There are some iptables (or nftables with 22.03 and firewall4) rules and a modification to the /lib/netifd/proto/map.sh file that you may need to put in in order to get all the ports to be used properly. These modifications mark the packets and distribute them (but cause things like mwan3 to break in the process)
There is the "nichiban benchmark" where you try to open https://www.nichiban.co.jp/ and see how it does. Usually I've needed to add the rules for it to pass the "nichiban benchmark" or else the number of usable ports are all used up and things don't go well.
I also noticed on versions that use fw4, you can't ping any external ipv4 addresses, but with Openwrt 21.X you can. Are you able to ping out?
This will get it using all the ports, at the least, and it will pass the nichiban benchmark.
There is also the issue of only getting a /64 prefix. You can work around this as well, if you want to, by setting a static address with a /65 prefix on the WAN interface, and another with a /65 prefix on a different subnet on the LAN interface, to split the /64 prefix. Then disable SLAAC and use dhcpv6 to give clients v6 addresses. I find that in order to manage ipv6, use things like masq6 if needed, and just to generally have IPV6 reliably work I need to do this, but it's a different discussion.
But if you split subnet then you are doing NAT on IPv6?
In my original write up since we relay everything so LAN side with the same /64 also getting DHCPv6 (not SLAAC)
BTW did you linked to wrong benchmark website? I didn't find any benchmark thing there.
And yeah I do believe something is wrong on firewall rule that IPv4 pings not passing (I saw that from firewall logs), that's why I am throwing out this so to bring up discussions, myself is not too familiar with tunneling thing.
I need NAT on IPV6 for mwan3 and VPNs to work. But it is a separate discussion.
The site is a site showing products for sale made by nichiban. It is not a site designed as a benchmark. That site uses a lot of sessions, so whether the contents on that site loads up properly is considered a benchmark in the Japanese community for testing implementation of MAP-E correct or not, because it's so easy to run out of ports. If you read Japanese, you can see a lot of talk about it on twitter, like below link. https://twitter.com/search?q=ニチバンベンチ&src=typed_query
You can also notice some issues like very slow to connect, or speed test not working well at times, because the used ports are overloaded. I re-confirmed with latest OpenWRT and the modifications are indeed still necessary unfortunately.
Pings work OK with fw3 versions of openwrt, but not fw4. I don't know the reason, though. I haven't really looked into it much.
OK I get what you mean, the website does load, but as I clicking deeper inside it's getting much slower.
In my GitHub page there is another discussion about map.sh change, is that you as well? (However the map.sh content change looks difference from yours after running a diff check) Seems we are all trying to find solution to optimize.
BTW when I just finished the setup with default map.sh, I already saw that under firewall rules there is thousand of ports listed to map the ports (of course I didn't check one by one yet), maybe I should spend some more time to read them all.
I found out that the map.sh script is trying to map ICMP to a port, but fails due to some check in OpenWRT that does not allow ICMP to be mapped to a port. Doing so manually via nft (position is # handle at the end of the rule - you can get it by "nft -a list ruleset") and place it in between one of the rules in chain snat.
Here's the command I ran (replace position, oifname, IP and port range to match your setup):
nft add rule inet fw4 srcnat position 999 meta nfproto ipv4 meta l4proto icmp oifname "map-replaceme" counter packets 0 bytes 0 snat ip to 22.214.171.124:1234-1234
I needed only one rule for ping to work, but if we can make it generate the rules, then all can be used.
Almost complete solution....
There is the issue of fwmarks interfering with other things that use fwmarks like mwan3 or policy based routing.
But as long as those functions are not needed it's good enough I suppose.
In my case I need these functions, so I just run two OpenWRT VMs.
Understand, I have never done this with OpenWrt before (I use pfSense before and it's not working in this way), this packet marking way is more or less like the RouterOS from Mikrotik, so if we implement solution with mwan3 we have to think about it. Just don't know the precedence of fwmark, if we also use mwan3, will the fwmark by map-e go first (i.e. will be overwritten by mwan3 rules) or the other way?
I suggest to follow up this in my GitHub page and open new issue there to discuss (as this thread here going longer would be difficult to track everything)
I am adjusting the fwmark rules to use masks instead. It's still a hacky solution, but it's likely to work. Will do some tests and ensure mwan and pbr working then post the working files on your github.
EDIT: I have now fixed policy based routing and mwan. Now it is a complete solution. I will post everything at your github in the discussion as well.