# This doesn't work
$ ip -6 route get 1:: dev ${NET_IF}
RTNETLINK answers: Permission denied
# This works
$ ip -6 route get 1:: dev ${NET_IF} from ${NET_ADDR}
1 Like
@yousong I don't really know what you did, but it worked!
Any other IPv6 host I try to ping I get permission denied error, but 2800:3f0:4001:800::200e works!
root@EdgeOpenWrt:/# ping -c 2 -I eth0.3 ipv6.google.com
PING ipv6.google.com (2800:3f0:4004:800::200e): 56 data bytes
ping: sendto: Permission denied
root@EdgeOpenWrt:/# ping -c 2 -I eth0.3 2800:3f0:4001:800::200e
PING 2800:3f0:4001:800::200e (2800:3f0:4001:800::200e): 56 data bytes
64 bytes from 2800:3f0:4001:800::200e: seq=0 ttl=49 time=40.800 ms
64 bytes from 2800:3f0:4001:800::200e: seq=1 ttl=49 time=44.981 ms
--- 2800:3f0:4001:800::200e ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 40.800/42.890/44.981 ms
Sadly I don't understand ip command and ip6tables. I tested and pppoe-wan remains working.
@vgaetera thanks! I also wasn't able to understand your test. I'll read that link.
What might be wrong on my router? I didn't manually change any routes.
1 Like
It's likely a source routing issue.
Post your persistent and runtime network configuration:
uci show network
ip -6 a; ip -6 r s t all; ip -6 ru
1 Like
I'm over an hour trying to post and it's failing, IDK why.
I placed the output on https://pastebin.com/RLSnjR16
2 Likes
Try this way:
ip -6 route get 1::
ip -6 route get 1:: dev pppoe-wan
ip -6 route get 1:: dev pppoe-wan from 2804:1b2:206:520:a9f2:ca03:4dec:7c71
ip -6 route get 1:: dev eth0.3
ip -6 route get 1:: dev eth0.3 from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd
ping6 -c 3 example.org
ping6 -c 3 -I 2804:1b2:206:520:a9f2:ca03:4dec:7c71 example.org
ping6 -c 3 -I 2804:14c:658b:1000:45d7:3d14:cfdc:66cd example.org
It seems that in some scenarios OpenWrt can't correctly select the source IPv6 address for routing.
I have a similar bug report here:
https://bugs.openwrt.org/index.php?do=details&task_id=2167
The only workaround is to add another IPv6 default route without source address filter:
https://openwrt.org/docs/guide-user/network/ipv6/ipv6_henet#default_route
2 Likes
This didn't seem promissing...
root@EdgeOpenWrt:/# ip -6 route get 1::
RTNETLINK answers: Permission denied
root@EdgeOpenWrt:/# ip -6 route get 1:: dev pppoe-wan
RTNETLINK answers: Permission denied
root@EdgeOpenWrt:/# ip -6 route get 1:: dev pppoe-wan from 2804:1b2:206:520:a9f2
:ca03:4dec:7c71
1:: from 2804:1b2:206:520:a9f2:ca03:4dec:7c71 via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static src 2804:1b2:206:520:a9f2:ca03:4dec:7c71 metric 512 pref medium
root@EdgeOpenWrt:/# ip -6 route get 1:: dev eth0.3
RTNETLINK answers: Permission denied
root@EdgeOpenWrt:/# ip -6 route get 1:: dev eth0.3 from 2804:14c:658b:1000:45d7:
3d14:cfdc:66cd
1:: from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd via fe80::201:5cff:fe66:7646 dev eth0.3 proto static src 2804:14c:658b:1000:45d7:3d14:cfdc:66cd metric 512 pref medium
root@EdgeOpenWrt:/# ping6 -c 3 openwrt.org
PING openwrt.org (2a03:b0c0:3:d0::1af1:1): 56 data bytes
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=0 ttl=52 time=238.547 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=1 ttl=52 time=227.981 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=2 ttl=52 time=228.015 ms
--- openwrt.org ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 227.981/231.514/238.547 ms
root@EdgeOpenWrt:/# ping6 -c 3 -I 2804:1b2:206:520:a9f2:ca03:4dec:7c71 openwrt.o
rg
PING openwrt.org (2a03:b0c0:3:d0::1af1:1) from 2804:1b2:206:520:a9f2:ca03:4dec:7c71: 56 data bytes
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=0 ttl=52 time=231.257 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=1 ttl=52 time=230.968 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=2 ttl=52 time=231.466 ms
--- openwrt.org ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 230.968/231.230/231.466 ms
root@EdgeOpenWrt:/# ping6 -c 3 -I 2804:14c:658b:1000:45d7:3d14:cfdc:66cd openwrt
.org
PING openwrt.org (2a03:b0c0:3:d0::1af1:1) from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd: 56 data bytes
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=0 ttl=46 time=232.976 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=1 ttl=46 time=235.716 ms
64 bytes from 2a03:b0c0:3:d0::1af1:1: seq=2 ttl=46 time=232.910 ms
--- openwrt.org ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 232.910/233.867/235.716 ms
Won't this route you suggested mess with mwan3? Now I remembered, it's reporting warnings with this interface too. Could you take a look before I add this route? Maybe it's the cause of this permission issue.
If this is caused by some problem on routing, I think permission denied is a misleading error message. It implies that root isn't permitted to run this command, not that it's a routing problem. Should I add a bug report suggesting a better message?
Actually, is works exactly as I expected.
So, if you want to use ping6 via specific interface, you need to specify the source address.
The reason is here:
About mwan3 I'm not sure as I don't use it, however it works the same way as the one you added earlier:
2 Likes
Thanks a lot to taking the time to help me understand this.
IDK how it handles routes, but for this matter it uses ping to verify if a WAN/interface is reaching internet and can be used for load balancing and fail over. It relies on interface name for that, so it's not working because passing the interface name as parameter isn't working.
What's odd is that it was all working before. I'm not sure, but I believe it started when I moved this modem to bridge and changed network settings to PPPoE connect.
Anyway, so if I don't specify an interface to where the ping must use, as in ping -c 2 ipv6.google.com, then OpenWRT just chooses the route randomly or by mwan3 rules.
But if I need to use a specific interface as in ping -c 2 -I eth0.3 ipv6.google.com, then it doesn't work with permission denied message.
Indeed, if I use ping -c 2 -I 2804:14c:658b:5fed::1 ipv6.google.com (ISP1 prefix + "suffix" [config interface lan option ip6ifaceid]) or ping -c 2 -I 2804:1b2:180:afc1::1 ipv6.google.com (ISP2 prefix), both works. The same is valid for traceroute -i.
What I don't understand is why ping -c 2 -I eth0.3 ipv6.google.com doesn't work but ping -c 2 -I pppoe-wan ipv6.google.com does.
The problem is that these global prefixes are dynamic. And my main use for this is precisely mwan3 pinging to verify if each WAN/interface is up and can be used.
Other than ping, it offers arping and httping to do the job. I'm gonna read about them before testing on my router.
But, isn't there a stable way to test if a WAN/interface is reaching internet, that can be used in scripts?
ddns is modular and allows us to use custom script to find out whatever IP address we wanna attribute for a domain name. mwan3 doesn't seem to have such feature. If needed be, I could add a feature request for that. Sadly I'm not good on sh/bash/python to help develop it 
1 Like
Try to compare these:
ip -6 r s t all dev eth0.3
ip -6 r s t all dev pppoe-wan
The output should be symmetrical.
1 Like
root@EdgeOpenWrt:/# ip -6 r s t all dev pppoe-wan
default via fe80::6b0:e7ff:fec9:4557 table 2 metric 1024 pref medium
default from 2804:1b2:180:afc1::/64 via fe80::6b0:e7ff:fec9:4557 proto static metric 512 pref medium
default from 2804:1b2:206:2174::/64 via fe80::6b0:e7ff:fec9:4557 proto static metric 512 pref medium
2804:1b2:206:2174::/64 proto static metric 256 pref medium
fe80::/10 metric 1 pref medium
fe80::/10 proto kernel metric 256 pref medium
anycast 2804:1b2:206:2174:: table local proto kernel metric 0 pref medium
local 2804:1b2:206:2174:6d97:138e:9bf8:fc30 table local proto kernel metric 0 pref medium
anycast fe80:: table local proto kernel metric 0 pref medium
local fe80::6d97:138e:9bf8:fc30 table local proto kernel metric 0 pref medium
ff00::/8 table local metric 256 pref medium
root@EdgeOpenWrt:/# ip -6 r s t all dev eth0.3
default via fe80::201:5cff:fe66:7646 table 4 metric 1024 pref medium
default from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd via fe80::201:5cff:fe66:7646 proto static metric 512 pref medium
default from 2804:14c:658b:5fed::/64 via fe80::201:5cff:fe66:7646 proto static metric 512 pref medium
2800:3f0:4001:800::200e metric 1024 pref medium
fe80::/64 proto kernel metric 256 pref medium
local 2804:14c:658b:1000:45d7:3d14:cfdc:66cd table local proto kernel metric 0 pref medium
anycast fe80:: table local proto kernel metric 0 pref medium
local fe80::822a:a8ff:fe5d:79d7 table local proto kernel metric 0 pref medium
ff00::/8 table local metric 256 pref medium
Yeah they aren't exactly symmetrical 
But then one is PPPoE and the other is DHCPv6.
2 Likes
I checked on my OpenWrt with the routing configuration as close to yours as possible and it still works, so the problem might not be related to routing directly.
Try to temporary stop the firewall service:
/etc/init.d/firewall stop
ping6 -c 3 -I pppoe-wan example.org
ping6 -c 3 -I eth0.3 example.org
/etc/init.d/firewall start
1 Like
It took me a while to find out the root cause of "Permission defined" (EACCES). They should be caused by iif lo failed_policy rule installed by OpenWrt patches
- target/linux/generic/pending-4.19/670-ipv6-allow-rejecting-with-source-address-failed-policy.patch
- package/network/utils/iproute2/patches/180-drop_FAILED_POLICY.patch
My current understanding of the situation is that when doing source address selection
- Kernel does route lookup to find output device. This will fail because no route matches. Those default routes in main table are all constrained by "from" directive. Those default routes in table 2 and 4 are guarded by "from" directive in ip-rules
- With output device unspecified, "Rule 5: Prefer outgoing interface." of rfc3484 will not apply. "Rule 8: Use longest matching prefix. " results in tie for destination 2800:3f0:4001:800::200e. The source address selection algorithm is free to use scope global addresses from other interfaces, e.g. pppoe-wan, etc.
If the above assumption holds, after source address selection, we end up with fl6.saddr from "pppoe-wan" and fl6.flowi6_oif being "eth0.3" (SO_BINDTODEVICE).
Then kernel does route lookup again to find output route, it fails again hitting iif lo failed_policy. And this time the EACCES will be returned to userspace.
My conclusion is that source address constrained routes in main table should be the cause. However I am not sure why they are there and preferred over the traditional destination based routes.
There are a few ways to check and confirm the level of correctness the above words are.
# ping another address so that "Rule 8: Use longest matching prefix. "
# prefers "2804:14c:658b:1000:45d7:3d14:cfdc:66cd/128" from eth0.3 .
#
# Timeout is not a concern we just want to make sense the
# "Permission denied" error.
ping -I eth0.3 2804:14...
ping -I pppoe-wan 2804:14...
# remove those "default from xxx" routes from main table
# and replace them with traditional ones.
ip -6 route del default from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 pref medium
ip -6 route del default from 2804:14c:658b:5fed::/64 via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 pref medium
ip -6 route del default from 2804:1b2:182:d768::/64 via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 512 pref medium
ip -6 route del default from 2804:1b2:206:520::/64 via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 512 pref medium
ip -6 route add default via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 pref medium
ip -6 route add default via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 pref medium
ip -6 route add default via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 512 pref medium
ip -6 route add default via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 512 pref medium
ping -c 2 -I pppoe-wan ipv6.google.com
ping -c 2 -I eth0.3 ipv6.google.com
3 Likes
Did you ever get to the bottom of this?
It's worth updating this that @brianjmurrell recently committed a fix to mwan3 which was merged which resolves ping issues with IPv6 interfaces. The issue appears to be OpenWrt using the wrong source address in some cases. This leads to mwan3 stating the interface as down when it's not.
The change in mwan3 2.8.3-2 and newer makes track_ip ping the interface with the source address obtained from the interface which is more reliable, than the interface name because of the mentioned source address issue.
There are also differences when using the busybox ping6 vs iputils-ping6 packages, which I think is all down to the source address problem.
1 Like
My system is:
Model: TP-Link Archer C7 v5
Architecture: Qualcomm Atheros QCA956X ver 1 rev 0
Firmware Version: OpenWrt 19.07.2 r10947-65030d81f3 / LuCI openwrt-19.07 branch git-20.064.69776-e8c638c
Kernel Version: 4.14.171
I am just using PPPoE and thats all. I seem to have same problem.
root@OpenWrt:~# ping 2800:3f0:4001:800::200e
PING 2800:3f0:4001:800::200e (2800:3f0:4001:800::200e): 56 data bytes
ping: sendto: Permission denied
root@OpenWrt:~#
I did not install any mwan3 package. Though I see mwan3 2.8.4-2 is available in LUCI.
So, how one should be solving that problem? Simply by installing that package?
Thanks.
1 Like
Not the same issue, this issue is related to multiple WANs and IPv6 in relation to mwan3.
Possibly you are hitting the same source address scenario but it's not the same, confirm you have working IPv6 connectivity and open a new topic if you'd like help with IPv6 and PPPoE.
1 Like
Hello. At that time I just got tired of trying and gave up, sorry for no reply.
I'm trying to get back and get this working. I just used opkg to upgrade everything possible and hope that new versions would fix this error.
Unfortunately, it didn't. I redid original tests and got same results.
I checked on my OpenWrt with the routing configuration as close to yours as possible and it still works, so the problem might not be related to routing directly.
@vgaetera I tried to stop firewall, and whole internet went down. Starting it didn't work, I had to reboot router :-x
@yousong tnx a lot for your work, I'm unable to understand what you said. I'm gonna test your code and update the post with results.
root@EdgeOpenWrt:/home/hikari# ping -c 2 -I eth0.3 2804:14c:658b:1000:45d7:3d14:cfdc:66cd
PING 2804:14c:658b:1000:45d7:3d14:cfdc:66cd (2804:14c:658b:1000:45d7:3d14:cfdc:66cd): 56 data bytes
--- 2804:14c:658b:1000:45d7:3d14:cfdc:66cd ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
root@EdgeOpenWrt:/home/hikari# ping -c 2 -I pppoe-wan 2804:14c:658b:1000:45d7:3d14:cfdc:66cd
PING 2804:14c:658b:1000:45d7:3d14:cfdc:66cd (2804:14c:658b:1000:45d7:3d14:cfdc:66cd): 56 data bytes
ping: sendto: Permission denied
root@EdgeOpenWrt:/home/hikari# ip -6 route del default from 2804:14c:658b:1000:45d7:3d14:cfdc:66cd via fe80::201:5cff:fe
66:7646 dev eth0.3 proto static metric 512 pref medium
RTNETLINK answers: No such process
root@EdgeOpenWrt:/home/hikari# ip -6 route del default from 2804:14c:658b:5fed::/64 via fe80::201:5cff:fe66:7646 dev eth
0.3 proto static metric 512 pref medium
RTNETLINK answers: No such process
root@EdgeOpenWrt:/home/hikari# ip -6 route del default from 2804:1b2:182:d768::/64 via fe80::6b0:e7ff:fec9:4557 dev pppo
e-wan proto static metric 512 pref medium
RTNETLINK answers: No such process
root@EdgeOpenWrt:/home/hikari# ip -6 route del default from 2804:1b2:206:520::/64 via fe80::6b0:e7ff:fec9:4557 dev pppoe
-wan proto static metric 512 pref medium
RTNETLINK answers: No such process
root@EdgeOpenWrt:/home/hikari# ip -6 route add default via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 p
ref medium
root@EdgeOpenWrt:/home/hikari# ip -6 route add default via fe80::201:5cff:fe66:7646 dev eth0.3 proto static metric 512 p
ref medium
RTNETLINK answers: File exists
root@EdgeOpenWrt:/home/hikari# ip -6 route add default via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 51
2 pref medium
RTNETLINK answers: File exists
root@EdgeOpenWrt:/home/hikari# ip -6 route add default via fe80::6b0:e7ff:fec9:4557 dev pppoe-wan proto static metric 51
2 pref medium
RTNETLINK answers: File exists
root@EdgeOpenWrt:/home/hikari# ping -c 2 -I pppoe-wan ipv6.google.com
PING ipv6.google.com (2800:3f0:4001:81b::200e): 56 data bytes
64 bytes from 2800:3f0:4001:81b::200e: seq=0 ttl=117 time=21.786 ms
64 bytes from 2800:3f0:4001:81b::200e: seq=1 ttl=117 time=21.961 ms
--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 21.786/21.873/21.961 ms
root@EdgeOpenWrt:/home/hikari# ping -c 2 -I eth0.3 ipv6.google.com
PING ipv6.google.com (2800:3f0:4001:81b::200e): 56 data bytes
64 bytes from 2800:3f0:4001:81b::200e: seq=0 ttl=112 time=41.481 ms
64 bytes from 2800:3f0:4001:81b::200e: seq=1 ttl=112 time=35.011 ms
--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 35.011/38.246/41.481 ms
Looks like it worked 
I'm gonna give it some time and see if mwan3 will have it working.
Well good and bad news.
It kept working for a few hours. Then I decided to reboot router just to be sure. And it stopped working, and again ping failed.
I executed the code again, and again it's working.
I wonder if I'd need to put it in a script under /etc/init.d or the sorts, so that it's fixed every sad time router reboots.
ubus call system board
What is the output?
1 Like
{
"kernel": "4.14.131",
"hostname": "EdgeOpenWrt",
"system": "MediaTek MT7621 ver:1 eco:3",
"model": "UBNT-ERX-SFP",
"board_name": "ubnt-erx-sfp",
"release": {
"distribution": "OpenWrt",
"version": "18.06.4",
"revision": "r7808-ef686b7292",
"target": "ramips\/mt7621",
"description": "OpenWrt 18.06.4 r7808-ef686b7292"
}
}
1 Like