haha, yes that will happen because it creates new veth each time you run fireqos 
probably you want a line like this instead of just unconditionally add new veth
ip link show | grep veth0 || ip link add type veth
1 Like
this is my last config :
## set up a pair of veth devices to handle inbound and outbound traffic
ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip link set veth1 master br-lan ## place the veth1 into br-lan bridge
ip route add 192.168.1.0/24 via veth0 table 100 ## traffic to LAN goes by default to veth0 where it comes out veth1 into the br-lan bridge
ip rule add not from 192.168.1.0/24 table 100 ## traffic from the internet uses routing table 100
## set up some mangle tables... this is an example,
iptables -t mangle -A PREROUTING -i pppoe-wan -j DSCP --set-dscp 0
iptables -t mangle -A PREROUTING -p udp --sport 5000:5500 -j DSCP --set-dscp 48
iptables -t mangle -A PREROUTING -p udp --dport 5000:5500 -j DSCP --set-dscp 48
iptables -t mangle -A PREROUTING -m set --match-set gvideostream4 src -j DSCP --set-dscp-class AF41
## etc... you need to mark similarly for google or facebook non-video traffic etc
## we send inbound packets from the internet through veth0 towards the br-lan bridge so this
## handles our inbound bandwidth, as output on a virtual ethernet device
interface veth0 lanin output rate 15500kbit qdisc fq_codel overhead 8
class highprio rate 300kbit ceil 500kbit
match dscp CS6
class group prio rate 14000kbit ceil 14000kbit
match dscp AF41
match dscp AF21
class video rate 69%
match dscp AF41
class other rate 29%
match dscp AF21
class group end
class default ceil 1000kbit
class torrent
match dports 6881:6889 ### note this is a default range, please insert the range used by your actual torrent client
## traffic from the LAN gets sent via pppoe-wan so this handles our outbound traffic
interface pppoe-wan wanout output rate 5500kbit qdisc fq_codel overhead 8
class highprio rate 300kbit ceil 500kbit
match dscp CS6change to
ip link show | grep veth0 || ip link add type veth
change to
ip route add 192.168.1.0/24 dev veth0 table 100 ## traffic to LAN goes by default to veth0 where it comes out veth1 into the br-lan bridge
check that it works, then reboot the router to clear out the old extra veth devices etc.
1 Like
after reboot,still no internet.
i think there's something missing or problem with "ip route add 192.168.1.0/24 dev veth0 table 100".
can you show output of
ip route show
and
ip route show table 100
that will help debug the routing.
EDIT: I suspect this is a firewall issue. The firewall doesn't know anything about the veth devices yet. So we should somehow let it know about it being ok to forward traffic from the internet to the veth0 in the same way that it used to forward to original LEDE LAN
1 Like
output of "ip route show" is :
default via 10.16.172.1 dev pppoe-wan proto static
10.16.172.1 dev pppoe-wan proto kernel scope link src 10.16.172.241
20.20.10.0/24 dev eth0.4 proto kernel scope link src 20.20.10.11
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
output of "ip route show table 100" is :
192.168.1.0/24 dev veth0 scope link
i forget to said that i don't need to use recovery mode at all.
ok, I think we actually don't need the special routing table, instead we just tell the regular routing table to send 192.168.1.0/24 to dev veth0 instead of dev br-lan in the regular table.
change:
## set up a pair of veth devices to handle inbound and outbound traffic
ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip link set veth1 master br-lan ## place the veth1 into br-lan bridge
ip route add 192.168.1.0/24 via veth0 table 100 ## traffic to LAN goes by default to veth0 where it comes out veth1 into the br-lan bridge
ip rule add not from 192.168.1.0/24 table 100 ## traffic from the internet uses routing table 100
to
## set up a pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip route del 192.168.1.0/24 dev br-lan
ip route add 192.168.1.0/24 dev veth0
reboot to eliminate the old veth and route tables
This still may not work because firewall issues.
1 Like
it's not working, i can't access my router.so i used recovery mode it's was some hard because i use extroot and firehol is installed in ext storage.
maybe this link can give us some idea about config qos.
Yes, this veth component is tricky. It is necessary for handling DSCP tagging but it is non-standard in LEDE so it probably interacts strangely with firewall. I suspect the firewall blocks us from sending traffic to veth0 and this then makes it so you have no connection to the router.
This will be fixed if we add the appropriate forwarding rule, but I'm not quite sure where to put it. You can examine firewall under LuCI status->firewall and then examine firewall settings under network -> firewall and see if you can figure it out. I will think about it a bit and try to give you some suggestions.
we are very close to a comprehensive solution here, so please don't give up 
Some things you could try:
Create new LEDE network called "QOS" set protocol to "unmanaged" and under physical settings set veth0 as the only device (do not create bridge over interfaces), then add this to the LAN firewall zone. This might be enough!
1 Like
thank you, i can't give up easily,but we need a way to fix this problem.
i had created that interface and added it to lan firewall zone.
what next?
Ok, I thought about this a while, and here is what I think are the issues:
-
We need firewall to accept packets from pppoe-wan to be routed to veth0... so by creating new LEDE network QOS containing veth0 and putting it in firewall zone LAN I think the firewall will be ok. I hope.
-
We need packets coming to 192.168.1.0/24 via wifi or ethernet or received on veth1 from veth0 to be bridged to appropriate place. This will be done already by br-lan
-
We need packets coming vie wifi or ethernet for input to the router at 192.168.1.1/32 to be delivered to br-lan so we can access the router itself... this requires a special routing rule
change our config to add an additional route rule for case (3)
## set up a pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip route del 192.168.1.0/24 dev br-lan
ip route add 192.168.1.0/24 dev veth0
ip route add 192.168.1.1/32 dev br-lan
/etc/init.d/firewall restart ## just in case because veth devices didn't exist before now
##.... the rest is unchanged.
With this in place, It seems like we've made an effort to take care of all the cases, and I think it should work.
1 Like
ok, but can i run fireqos conf file from /tmp so if there's and problem a reboot will fix the problem.
because the /tmp is in ram.
same thing i can't connect to router.
i copied the conf file to /tmp then ran fireqos /tmp/fireqos.conf start.so if there's something isn't correct i can reboot and back normally.
awesome good debugging skills!
I see we had a bug where we weren't adding veth1 into the bridge!!! that line got dropped somehow, so putting that back should help. But also thinking about it some more the policy routing was working better for us, and might have other advantages. It lets us keep the effect of the QoS more tightly controlled. let's go back to that, then you can hopefully connect to the router and see how well it's working... Continue to the use /tmp trick to avoid being locked out after reboot.
ip link show | grep veth0 || ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip link set veth1 master br-lan ## place the veth1 into br-lan bridge (We forgot this last time!)
ip route add 192.168.1.0/24 dev veth0 table 100
ip route add 192.168.1.1/32 dev br-lan table 100
ip rule add not from 192.168.1.0/24 to 192.168.1.0/24 table 100 ## traffic from the internet to the lan uses routing table 100, other traffic uses standard routing table.
/etc/init.d/firewall restart ## just in case because veth devices didn't exist before now
Hopefully, now that we're remembering to put the veth1 into the bridge, and we have policy routing so that routing only is affected for packets not coming from the LAN, we can get somewhere useful. First reboot router, then copy file to /tmp and run, see if you stay connected.
EDIT: thinking about this, I'd expect we may recover ability to talk to router from LAN side, but the routing table for packets from the internet is not complete enough, for example it won't let the router receive DHCP packets we should be more selective of the traffic we route through veth0. so I changed the rule above.
EDIT AGAIN: another debugging trick. You could try putting your fireqos config into /tmp/testqos.conf and then executing the following command:
fireqos /tmp/testqos.conf; sleep 60; /etc/init.d/network restart
If the router becomes unreachable there's a good chance that after 60 seconds it will reset its network back to normal and become reachable again.
1 Like
"fireqos /tmp/testqos.conf; sleep 60; /etc/init.d/network restart" should be "fireqos /tmp/testqos.conf start; sleep 60; /etc/init.d/network restart"
EDIT:
still not working i can't access my router, but "sleep 60; /etc/init.d/network restart" let me access router after 1 min.
you can see there's some data transferred on qos interface:
this is what i get when i run "fireqos /tmp/testqos.conf start; sleep 60; /etc/init.d/network restart" :
FireQOS 3.1.5
(C) 2013-2014 Costa Tsaousis, GPL
Using file '/tmp/testqos.conf' for FireQOS configuration...
Warning: Section @zone[1] (wan) cannot resolve device of network 'wan6'
Warning: Section @rule[27] (maher-iphone) does not specify a protocol, assuming TCP+UDP
* Flushing IPv4 filter table
* Flushing IPv4 nat table
* Flushing IPv4 mangle table
* Flushing IPv4 raw table
* Flushing IPv6 filter table
* Flushing IPv6 mangle table
* Flushing conntrack table ...
* Populating IPv4 filter table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Rule 'ubus:igmpproxy[instance1] rule 0'
* Rule 'ubus:igmpproxy[instance1] rule 1'
* Rule 'ubus:igmpproxy[instance1] rule 2'
* Rule #3
* Rule #4
* Rule 'Allow-DHCP-Renew'
* Rule 'Allow-Ping'
* Rule 'Allow-IGMP'
* Rule 'Allow-IPSec-ESP'
* Rule 'Allow-ISAKMP'
* Rule 'Allow-Tor-DHCP'
* Rule 'tor_allow_DNS'
* Rule 'tor_allow_TCP'
* Rule 'Allow-Tor-DNS'
* Rule 'Allow-Tor-Transparent'
* Rule 'Allow-Tor-SOCKS'
* Rule 'https'
* Rule 'pptp'
* Rule 'gre'
* Rule 'esp'
* Rule 'port500'
* Rule 'port4500'
* Rule 'ah-protocol'
* Rule 'l2tp'
* Rule 'pvp-net'
* Rule 'Allow-Encapsulated-IPv6'
* Rule 'maher-iphone'
* Redirect 'tor_redirect_DNS'
* Redirect 'tor_redirect_TCP'
* Redirect 'ftp'
* Redirect 'League_of_Legends_Game_Client'
* Redirect 'League_of_Legends_Game_Client_1'
* Redirect 'Patcher_and_Maestro'
* Redirect 'Patcher_and_Maestro'
* Redirect 'PVP_Net'
* Redirect 'PVP_Net'
* Redirect 'PVP_Net'
* Redirect 'http'
* Redirect 'HTTPS'
* Redirect 'Spectator_Mode'
* Forward 'wan' -> 'lan'
* Forward 'lan' -> 'wan'
* Populating IPv4 nat table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Redirect 'tor_redirect_DNS'
* Redirect 'tor_redirect_TCP'
* Redirect 'ftp'
* Redirect 'League_of_Legends_Game_Client'
* Redirect 'League_of_Legends_Game_Client_1'
* Redirect 'Patcher_and_Maestro'
* Redirect 'Patcher_and_Maestro'
* Redirect 'PVP_Net'
* Redirect 'PVP_Net'
* Redirect 'PVP_Net'
* Redirect 'http'
* Redirect 'HTTPS'
* Redirect 'Spectator_Mode'
* Populating IPv4 mangle table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Populating IPv4 raw table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Populating IPv6 filter table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Rule 'ubus:igmpproxy[instance1] rule 0'
* Rule 'ubus:igmpproxy[instance1] rule 2'
* Rule #3
* Rule 'Allow-DHCPv6'
* Rule 'Allow-MLD'
* Rule 'Allow-ICMPv6-Input'
* Rule 'Allow-ICMPv6-Forward'
* Rule 'Allow-IPSec-ESP'
* Rule 'Allow-ISAKMP'
* Rule 'https'
* Rule 'pptp'
* Rule 'gre'
* Rule 'esp'
* Rule 'port500'
* Rule 'port4500'
* Rule 'ah-protocol'
* Rule 'l2tp'
* Rule 'pvp-net'
* Rule 'Allow-Encapsulated-IPv6'
* Rule 'maher-iphone'
* Forward 'wan' -> 'lan'
* Forward 'lan' -> 'wan'
* Populating IPv6 mangle table
* Zone 'lan'
* Zone 'wan'
* Zone 'tor'
* Set tcp_ecn to off
* Set tcp_syncookies to on
* Set tcp_window_scaling to on
* Running script '/etc/firewall.user'
* Running script '/usr/share/miniupnpd/firewall.include'
* Running script '/usr/lib/bcp38/run.sh'
: interface veth0 lanin output rate 15500kbit qdisc fq_codel overhead 8 (veth0, 15500kbit, mtu 1500, quantum 1500, minrate 155kbit)
: class highprio rate 300kbit ceil 500kbit (1:11, 300/500kbit, prio 0)
: class prio rate 14000kbit ceil 14000kbit (1:12, 14000/14000kbit, prio 1)
: class video rate 69% (1:13, 9660/14000kbit, prio 0)
: class other rate 29% (1:14, 4060/14000kbit, prio 1)
: class default (1:8001, 155/14000kbit, prio 2)
: committed rate 13875kbit (99%), the remaining 125kbit will be spare bandwidth.
: class default ceil 1000kbit (1:8000, 155/1000kbit, prio 2)
: class torrent (1:17, 155/15500kbit, prio 3)
: committed rate 14610kbit (94%), the remaining 890kbit will be spare bandwidth.
: interface pppoe-wan wanout output rate 5500kbit qdisc fq_codel overhead 8 (pppoe-wan, 5500kbit, mtu 1480, quantum 1480, minrate 55kbit)
: class highprio rate 300kbit ceil 500kbit (1:11, 300/500kbit, prio 0)
: class default (1:8000, 55/5500kbit, prio 1)
: committed rate 355kbit (6%), the remaining 5145kbit will be spare bandwidth.
Traffic is classified:
- on 2 interfaces
- to 10 classes
- by 8 FireQOS matches
34 TC commands executed
All Done! Enjoy...
bye...
i think i can only receive data from router but i can't send 'upload'.
So if I understand correctly, when you run our latests script it lets you access the router itself, but you never get information to/from the internet?
This is getting closer. First, please paste script here so I know exactly what it is fireqos is doing, can look for typos/bugs etc. If you are able to access the router, then I think we should try to see whether after fireqos script how much tx/rx is there over pppoe-wan and over qos network. Run the fireqos script, then try to do google search, try to read non-google new site, try to do various things... while in separate tab watching the statistics on the interfaces on your router. Are the packet counts changing? Which interface? Then after doing that, check kernel logs to see if you see anything useful, and check system logs too. anything useful?
EDIT: specifically, try to see if maybe you are able to transmit the request out to the internet, and it comes back through the pppoe-wan but somehow doesn't get back to your computer over the veth0.
hi
i can't access router but when i run “fireqos /tmp/testqos.conf start; sleep 60; /etc/init.d/network restart”
i get everything printed on the screen then i can't access router,but i feel that i can't transmit data to router but i can't receive data,also no internet access,also i see when 1 min is passed i can access router but no internet access even if i try to reconnect wan or restart firewall so i should reboot.
this is the script:
## set up a pair of veth devices to handle inbound and outbound traffic
ip link show | grep veth0 || ip link add type veth
ip link set veth0 up
ip link set veth1 up
ip link set veth1 master br-lan ## place the veth1 into br-lan bridge (We forgot this last time!)
ip route add 192.168.1.0/24 dev veth0 table 100
ip route add 192.168.1.1/32 dev br-lan table 100
ip rule add not from 192.168.1.0/24 to 192.168.1.0/24 table 100 ## traffic from the internet to the lan uses routing table 100, other traffic uses standard routing table.
/etc/init.d/firewall restart ## just in case because veth devices didn't exist before now
##ipset for googlevideo,etc.they are beening filled by dnsmasq
ipset create gvideostream4 hash:ip
## set up some mangle tables... this is an example,
#mark packets for googlevideo and others
iptables -t mangle -A PREROUTING -i pppoe-wan -j DSCP --set-dscp 0
iptables -t mangle -A PREROUTING -p udp --sport 5000:5500 -j DSCP --set-dscp 48
iptables -t mangle -A PREROUTING -p udp --dport 5000:5500 -j DSCP --set-dscp 48
iptables -t mangle -A PREROUTING -m set --match-set gvideostream4 src -j DSCP --set-dscp-class AF41
## etc... you need to mark similarly for google or facebook non-video traffic etc
## we send inbound packets from the internet through veth0 towards the br-lan bridge so this
## handles our inbound bandwidth, as output on a virtual ethernet device
interface veth0 lanin output rate 15500kbit qdisc fq_codel overhead 8
class highprio rate 300kbit ceil 500kbit
match dscp CS6
class group prio rate 14000kbit ceil 14000kbit
match dscp AF41
match dscp AF21
class video rate 69%
match dscp AF41
class other rate 29%
match dscp AF21
class group end
class default ceil 1000kbit
class torrent
match dports 6881:6889 ### note this is a default range, please insert the range used by your actual torrent client
## traffic from the LAN gets sent via pppoe-wan so this handles our outbound traffic
interface pppoe-wan wanout output rate 5500kbit qdisc fq_codel overhead 8
class highprio rate 300kbit ceil 500kbit
match dscp CS6
BTW, can we use netdata to have look at qos statics,netdata is already installed on my router.
Ok, good, that is helpful information.
If Netdata is already installed then yes I suggest you try to use it for finding out information about what is going on. One way might be to have fireqos run, then instead of 60 second timeout... give it say 120 second, and do some activity such as try to do google search, try to load your favorite website, etc, and then when 120 second timeout occurs and you can talk to the router again, see what netdata says happened. Which interfaces did traffic flow over during the 120 seconds? How much traffic? etc. If you find something useful please post it.
I think the question now is really whether our routing rule is wrong, or whether a firewall rule is blocking access, or both. I suspect the routing is confused but I don't know why, the new routing table only affects data that isn't from your LAN and is going to your LAN so I don't see why it would affect your ability to access the router for example, since that is data coming from your LAN headed to your router, and should go through the standard routing table.
Another option is that we're creating a bridge loop of some sort... for example data you send from your LAN hits br-lan and instead of delivering it to your router, it tries to send it over veth1 and then it pops out veth0 and gets routed back somewhere into the bridge.... looping forever. I don't see why that would happen, but it's concievable, and it might have that symptom of a few lines get printed and then everything locks up. Netdata output during the period when you don't have access but are trying to surf would be informative.
1 Like
but how can i integrate fireqos into netdata to see the qos output, cause now i can't see qos data netdata.