SQM on (bridged) WAN interface - No NAT

I have been trying to implement this ever since I learn the issues about Bufferbloat, after tried more than 20 different ways to set it up over the period of 7 months, finally I was able to set it up the way I wanted. This is mainly for me to remember how-to do it, and if others find it helpful, I am glad I can be of help.

In this how-to, I am going to move the LAN Port 3 of my QOS router to the WAN, and let it get a DHCP address from my ISP's cable modem (even though my ISP said it's not possible), this way, the traffic through it won't be NAT'ed.

The story started about a year ago, my son started to play graphic intensive online video games, and started to complain about the network is "lagging". Being a wireless router guru dad, I showed him a couple of speedtest websites / apps, with acceptable results. Until one day, he was angry and said he would go somewhere else to paly games.

For the simplicity, my network set up can be viewed as this: ISP -> Cable Modem -> Wireless Router (running TomatoUSB), so my first try was to use QOS feature included in Tomato firmware. I find the Tomato's QOS is very complicated, I spent lots of time reading the forum posts, and finally enabled the QOS according to my understanding. The result was my son couldn't even log into the game servers anymore. I am not saying that QOS doesn't work, it's probably my understandings were incorrect.

I need something simpler that just work. One day I found this article while browsing, I then measured my speed with DSL Reports, my connection was so badly affected by Bufferbloat, most of the times, the test couldn't finish.

I bought 3 Linksys EA4500 wireless routers a couple years ago at a yard sale, it's time to put them into use as QOS devices (the bump referred in the article). I didn't want to replace my main wireless router yet, just in case the SQM implementation causes troubles like the QOS I did on Tomato.

Flashed EA4500 with 18.06, and used almost all the default settings, connected it's WAN to my cable modem and LAN port 1 to my main wireless router's WAN port, and enabled SQM / cake / piece_of_cake for it's WAN, and I made it work!!! Bufferbloat were significantly reduced.

I had 3 options to go further with the experiment:

  1. Just leave it the way it is, but I didn't like it, because my main wireless router (TomatoUSB) is doing NAT, now I have another EA4500 router (QOS device) also doing NAT, double NAT is bad, plus it makes the port forwarding (required by some games) hard and unreliable;
  2. I could replace my main router with EA4500, but I have a very complicated wireless network setup, my main router has been configured with lots of stuff, such as complex wifi schedule, access rules, plus I have 6 other routers running Wireless Ethernet Bridge to it. Replacing the main router is certainly doable but would have been painful;
  3. Removing NAT on EA4500, let it work just like a switch, no NAT/DHCP/DNS/Firewall etc.

At the moment, it seemed to me the 3rd option would be the easiest. Oh man, what I mistake I made.

1 Like

It looks like it's easier to do if you went through the first link I posted in my previous post, people tried it but had troubles, like this, and this. The reason could be the order in which you perform each step, I am guessing, because I have tried this so many times, it didn't work till now.

Finally I did ityesterday, the way I wanted and here go folks:

  1. Back up your configuration just in case, then flash your device with Openwrt 18.06; Or factory reset it if you are already on Openwrt 18.06;
  2. Make sure your computer is using DHCP and connect to Port 3 of your wireless router;
  3. Open a browser, and enter 192.168.1.1 in the address field, then ;
  4. Goto Network -> Switch, change LAN Ports 1, 2 and 4 to "off" in VLAN 1
  5. Click on Add to add VLAN 3, and change CPU (eth0) to "tagged", LAN Port 1 to "untagged" for VLAN 3, Save & Apply. Now it looks like this: ;

Network -> Interfaces, click on "Add new interface...", pick a name for the new interface, I call it "managmnt", because it's going to be used for manage the router only, if you follow this how-to. Keep "Static address" for the Protocol, leave the "Create a bridge" unticked, select "eth0.3" for "Cover the following interface", then "Submit"

Next page, use "192.168.2.1" as it's IPv4 address and "255.255.255.0" as netmark, Save & Apply, then you will be presented with this:

This management Interface will be used to access the device; In the case I mess up the configuration, I can just plug in a network cable and reset it to factory default. Remember it's address is 192.168.2.1/24.

Cool, please note if you do this it might be better to not use a single sqm instance on eth1.2, but rather one on eth1.2 and one on eth0.1? That way you can avoid the cost for the ifb rerouting, but keep in mind that now you need to configure download speedd for eth1.2 to 0 (to disable ingress shaping), and instead set the desired internet ingress rate as upload/egress rate on a second sqm instance on eth0.1 (the directions called download/upload in the GUI are in reality ingress/egree from the view of the respective interface; in the common recommended case of sqm on wan these are aligned with the internet directions given in the GUI).
The drawback from putting the bump in the wire between modem and tomato touter is that you lack information about the internal IP addresses and hence will not be able to configure per-internal-IP-fairness. For this you would need to put the bump between your computer(s) and the tomato router (and for this to work reliably you would also need to disable wlan on the tomato and put a secondary AP into your network, in that case you might as well set up the EA4500 as dumb AP with SQM).

Also, cool project and nice write-up!

1 Like

Thank you Moeller0 for the comments. Wow, there are so many things to learn. Can you please share some information on "ifb rerouting"? I tried to google it, but couldn't find any useful information.

Anyway, I am in the middle of wrting this up, my final goal is to bridge LAN Port3 with eth1.2, so no NAT there. Stay tuned.

Let me try :wink: Linux traditionally only allowed to attach qdiscs to egress interfaces, as the kernel really can only control the outgoing data completely. Now there are use cases where processing of incoming packets is quite helpful, like traffic shaping. The kernel accommodated these uses by introducing the Intermediate Functional Block device which among other things allows to effectively attaching qdiscs to the ingress side of a real interface. I believe it achieved that by acting as a pretend interface between the outside and the ingress side of the real interface and allows qdiscs to be attached the the egress side of the ifb. This is conceptually actually similar to your bump in the wire idea, only the ifb is not a full fledged device.
Any way sqm scripts uses ifb for the ingress shaper, but that carries a computational cost that can be saved if we can move the internet download shaping from the ingress side of the want interface to the egress side of a LAN interface. The reason why we do not do this as default is that typically in a wifi router, the wifi interfaces talk to the CPU directly, and hence our download shaper will not affect traffic from the internet to wifi hosts (making ingress shaping ineffective) BUT the shaper will affect traffic from the wifi hosts and the router's CPU to the lab hosts. But since your bump does not act as an AP and will not offer services to your network this will not affect your use-case.

I hope this helps....

It makes sense, but it's also a little bit beyond me now, I probably need lots of reading on the subject, hopefull I will be there one day, but thank you for spending the time to write it up.

Ok, Continuing with my how-to.

Go Network -> Interfaces, click on "Edit" for LAN, then "Common Configuration" -> "Physical settings" -> "Interface", untick 'Switch VLAN "eth0.1" (lan)', it looks like this:

Click "Save & Apply", the router will think for 30 seconds and tell you the configuration is no good and has been rolled back, don't worry, just click on the red "Apply unchecked".

Even better, another 30 seconds later, you will see this:

This is because the LAN Port 3 is removed from the LAN bridge, I will fix it in the next step.

To solve the problem, change your computer's network to 192.168.2.2/24, and connect it to router's LAN Port 1. Go "192.168.2.1" from your browser, you will have access to the router again.

From within the browser, Network -> Interfaces, "Edit" for "WAN", then "Common Configuration" -> "Physical settings", tick "Bridge interfaces" and add 'Switch VLAN "eth0.1"' from "Interface" drop down list.

Save & Apply.

You might have noticed I didn't enable DHCP anywhere, this is because early in my experiment, I had it happened twice to me that the LAN Port 1, now is part of WAN, would get IP address from the Managmnt IP Pool when DHCP is enabled there, even though they are totally separated.

Now the EA4500 should be working as a QOS device if I put it in between my Cable Modem and my wireless router, but just in case I need to make any changes to it, I would have to connect my laptop to it's LAN Port 1 using a Ethernet cable, that's no good. Let's enable one wifi and attached it to Managmnt Interface.

Go Network -> Wireless, click on "Edit" and under "Interface Configuration" -> Network, untick lan and tick managmnt, and change other settings, such as SSID, security, etc. to your preference.

Save & Apply. Test to see if you can manage the QOS device (192.168.2.1) when you connected to it's wifi. Remember, DHCP is not enabled, so you need to configure your wireless connection's IP settings manually.

Now it's time to put it into real life test. Connect the newly configured EA4500 between Cable Modem, your wireless router, connect your wireless router's WAN to EA4500's LAN Port3, and Connect EA4500's WAN to you Modem, power it on.

You should have internet connection just like before, if not, try to power cycle your main wireless router. The next step is the install and configure the SQM package, there are plenty of tutorials for that, one thing to pay attention is that under this set up, you want to enable SQM on "eth1.2 (wan, wan6)" interface.

Cheers!

Great work!

You might also use a second VLAN on the same port as the data VLAN and all you need to add on the main router is an additional interface, that way you do not need wifi. That said I use this method to reach my modem's (running openwrt) management GUI over the same cat 6 cable that also use to push data to the bridged vdsl modem; but I also have set up a separate wlan SSID on the modem to be able to reach it even in case of VLAN issues :wink:

Thank you meoller0,that's a brilliant idea! Based on my set up, how do I go about to do it?

Good question. So in my set-up I did the following to allow to pass data through my bridged modem and also access its management GUI:

  1. I decided that I do not want to deal with vlan trunking (trying to route untagged and tagged packets over the same interface), which might be a deal breaker for you as it requires to use VLAN tagging on the real router (but that seems unavoidable if the same cable/port are to be used for data and management traffic)

  2. I created a new "interface" on the openwrt modem, bridging the real WAN interface (dsl0.7) with the intended VLAN (eth0.7) in my case, and I opted to not assign these to any firewall zone and I selected protocol unmanaged to basically create a dumb ethernet bridge. This is the core of the data path. Now to access this I opted to assign both the CPU port and one switch port (say LAN 1) to VLAN ID 7 in tagged mode. This now will pass all packets with VLAN tag 7 from LAN 1 to WAN (and the other way around). This is analog to your solution except I need a VLAN in both directions as my ISP requires VLAN7, in your case the WAN interface does not need to be tagged and probably should not be.

  3. I also added tagged VLAN ID 2 to both the CPU and LAN 1 port (actually die to a LuCI bug I actually added the CPU port and LAN 1 to all VLAN IDs from 1 to 7, but that bug is hopefully fixed). Then I added eth0.2 to the lan bridge (br-lan) and set the the protocol for the LAN interface to static address (I selected 192.168.100.1, which will not work for you if you also want tp access the docsis modem's management interface as that modem also uses 192.168.100.1).

  4. on the actual router (openwrt master in my case) I selected eth1.7 as the physical wan interface (this allowed data to flow to the internet).

  5. I created a new interface on the router (WAN4BTHH5A) with protocol set to static address, and address set to 192.168.100.2/255.255.255.0 (must be the same subnet as the LAN address in the modem/bump), and selected as physical interface eth1.2 and assigned that interface to the wan firewall zone.

And that basically allows concurrent internet acceess and management of the modem over the same cable, all I need to do is browse to http://192.168.100.1 to get access to the modem's GUI from any host behind my router.

Whether this is fully applicable to your bump-in-the-wire idea really depends on how hard it is to create another interface on the main router to reach the bump via a dedicated tagged vlan...

I hope this helps.

Interesting concept, but I will try to avoid doing tagged and untagged on the same interface. I am pretty sure I read somewhere there is a bug in Openwrt with that, I am not sure if it's fixed in 18.06.

My main goal for this exercise is to reduce the Bufferbloat and not to lose too much throughput (no double NAT), for now the solution is pretty stable, I seldom have to touch the EA4500, so manage the device is not a big deal.

Yepp, I disabled untagged on the LAN1 port and therefore need to appropriately tag out going packets from the router to the modem explicitly, so I do not use eth0 on the router directly at all, but just eth0.7 and eth0.2.

Sure and having WLAN access for rare interventions is just fine...

Many times people struggle with QoS settings when in fact the root cause is a broken ISP modem / network that cannot be solved with QoS...

1 Like

Thanks, ISP / problematic network might have affected my set up, but I was mainly trying to combat Bufferbloat, feels like the solution worked.

Now I have been using the QOS device for a while, sometimes I have to connect it wirelessly to change something, only to realize that I forgot the syntax, but while on the "Managmnt" network, I have no Internet access, I have to reconnect to my main router, then I can google again. Kind of PIA. Would it be great if I have internet access even if I am connected to my QOS device?

It turns out not that different, all you need to do is to change the firewall rule for "MANAGMNT" interface, as outlined in this how-to article .

Here is what my /etc/config/firewall looks like after the changes:

Please note that you will need to enable Dropbear for "MANAGMNT" interface and utility "putty" to change the /etc/config/firewall from terminal window.

They have apparently fixed this so now you can effectively put the bump between the modem and router. Just implemented this today and A+ across the board on DSLReports tests, including bufferbloat. Thanks for the multiple SQM interface info, it does improve throughput on faster connections. For my lowly 30/4 connection an old DIR-810L fits the bill just fine. CPU utilization never gets above 20%.


To enable Per-Host Isolation Add the following to the “Advanced option strings” (in the Interfaces → SQM-QoS page; Queue Discipline tab, look for the Dangerous Configuration options):

For queueing disciplines handling incoming packets from the internet (internet-ingress): nat dual-dsthost ingress

For queueing disciplines handling outgoing packets to the internet (internet-egress): nat dual-srchost