Lede as a dedicated QoS /Bufferbloat appliance

You can't ! pppoe is layer 2.

LEDE has to be the edge and then Peplink handling link balance with DHCP server.

That's the thing that I've been running up against...thinking somehow to
make the LEDE transparent.

I am doing this on a network with 600 routers. I have a LEDE box with 3 gigabit interfaces, 2 bridged (unmanaged) and 1 management. The box is transparent to the network and it is sitting in between my customers and the main NAT router. I modified the cake code to use up to 65535 flows. It's running on a dual core pentium pushing 300mbits/sec during peak hours. CPU load 0-1%

And now the 64K question is, how low can you push the number of flows and still get decent performance and flow separation :wink: ? Or put differently with 600 users how many active flows do you typically see? (This is an honest question, BTW, I am really curious, just having home network experience myself)

I can manage to login PPOE with PEPLINK and the LEDE router between it and the transparent bridge modem. Problem is with DNS. I've yet to get fast DNS lookups with to the PEPLINK. I can set PEPLINK to look for DNS servers at the Lan interface of the LEDE router, at the WAN interface of the LEDE router and at DNS servers but no setting I've found lets things move along at normal speeds and sometime web pages just refuse to load. Guess, for now, I'll go back to a single LEDE router between PEPLINK and my unmanaged switch...so PEPLINK is edge.

if you login in the modem, your LEDE router is acting as switch and you won't do qos at all.

See also the "Transparent Cake Box" topic that seems to be running along the same lines. Transparent Cake Box

docpecos, the only real solution here is to have a transparent LEDE box in between each wan connection and your peplink box. Its overkill i know and cake in this scenario will only work based on flows. Get rid of the multi wan setup.

Orangetek this is exactly what I have every wan connection goes into a transparent bridge modem, then goes into a R6300v2 "cake box" and then goes into a PEPLINK 380 for aggregation

Felipee07 the modems are in transparent bridge formation and the Cake Boxes do the login via PPOE. They work great.

The problem the current system has is that each additional cake box when added in makes connections to a given web page etc more and more difficult, like maybe there is a loop or something in the system or just too many wan to lan connections. Currently the cake boxes are connected to the modem on their wan connections and to the PEPLINK via a single lan line.

One way I have thought of possibly decreasing the dns problems and complications is to try and put the cake boxes in a configuration where they act simply as as a switch so the PEPLINK handles the PPOE and the cake boxes do zero in terms of dns and PPOE. As noted by Felipee07 if it is a single switch then no bufferbloat control is possible.

I was thinking since with the R6300v2 there really is only a single switch for wan and lan and the function of the 5 ports (4 lan, 1 wan is controlled by VLAN switches) that maybe there is a way to set up those switches so that there are two lans (lan1, lan2) and hence two interfaces so that luci-app-sqm can be applied. Basically this would turn the linkage between the 4 lan ports from an unmanaged switch into a managed switch between lan1 and lan2. Problem is, I've not been able to set this up and would like some help on how to do this.

This is what the Luci Lua looks like in a cake box:

Here I added in VLAN 3 and turned on LAN4 on VLAN3 and turned off all other lans and wan. In VLAN1, I turned off LAN4. Now if LAN4 could be made to communicate with LAN1-3 and be on a different physical setting and label, eth2, say, then maybe luci-app-sqm could be applied usefully to eth2. I think LAN4 would also have to be not bridged for this to work.

I've tried to add an interface but haven't managed to make it so, say three of the lan ports are on lan1 and one on lan2 with a connection between them (like when you pug into a wan and all 4 lan ports are then connected with the wan).

It seems ip conflict.

just to be sure.
Are you using different ip address for each cake router right ? like

192.168.0.1/24
192.168.1.1/24
192.168.2.1/24
192.168.3.1/24

and Peplink gives(DHCP) another totally different like 10.0.0.1/24.

For the DNS, just use google on Peplink and all your client will use Peplink DNS(google).

Yes Felipee07, I'm toying with forwarding IPs from the PEPLINK to the cake routers, that might make for a less complex routing;

I've not done IP forwarding before but moving the IPs forward to the cake routers might be handy.

I still don't get where you're going.

Why PPPOE on PepLink, the should be DHCP(Connection Method) for the 4 wan

Connection Method (DHCP)
Routing Mode (NAT)

Felipee07:

Probably you already know this, but I would like you to understand where I am headed.

Most dsl+ modems are either just modems and don't do much very well other than connect. Modem/router combinations provided by your ISP are ok as modems and generally mediocre at best as routers. That's why folks get pricey fast routers or less expensive fast LEDE routers to connect with their modems. Generally the modems are left in "transparent bridge" configuration so the better routers can do other things routers do better. One thing good routers do, like LEDE routers and PEPLINK routers, is to handle the PPOE connection to the modem in transparent bridge. Generally you know your modem doesn't know much so you try to take as much of the burden off it as is possible. Perhaps you have a point here...I've not had a modem do PPOE for at least 5 years but, hmmm, maybe pure static IPs at the routers is worth a try. That way the LEDE router would have a static IP and the PEPLINK would also have a static IP.

I have read that the device that does the PPOE does NAT so one might get double NAT and the associated slow down if the modem does the PPOE.

Just the size of the CPU on a modem and on a modern router are so much different, its hard to imagine the modem doing better. It's sorta like thinking an earth worm can out smart a dog, but hey, might be worth a try. Maybe an earth worm is better at burrowing than a dog...

Currently my LEDE routers are handling the PPOE, which is fine, but PPOE is generally set up to be handed off easy to the router doing the connection. Things are streamlined within the router, I believe, to hand off the connection to DHCP DNS and all that. So what I would like to do, where I'd like to head, but haven't been able to configure, is to let the PEPLINK handle the PPOE, DHCP, DNS link aggregation etc. The LEDE router would be simply a "cake box".

In order to do this, the cake box has to be very transparent. Perhaps this would be best accomplished by having Lan1 in and Lan2 out. I believe I have a double NAT going on with the cake boxes: I think each LEDE router does NAT and there is no way to turn that off and the PEPLINK also does NAT. By letting the PEPLINK do PPOE through both the LEDE router and the modem double and maybe triple NAT can be avoided.

Parenthetically your comments and JMJones comments made me further examine my static IPs and I did find one set of possible ambiguities. I fixed those and....the connections are better, some web pages actually completely load etc with 4 modems feeding to 4 cake boxes feeding to the PEPLINK. But, still very annoying and slow. Currently I am running 2 cake boxes, which helps with the blufferbloat, and 2 lines with no cake boxes.

Another idea I plan to explore is not allowing the LEDE router and the PEPLINK to automatically handle MTU. Undoubtedly the algorithms could be different so packaging and repackaging information handed off from one router to the other seems like it could induce lag. Thinking I'll set them both at 1438 and see if I get any increase in speed.

Thanks again for all those who have been watching me struggle with this. I've been building and using computers since MSDOS 2.0 and earlier but never did get much into networks, whole new bowl of spaghetti.

Currently...

I've currently taken apart my system so that my gaming is done on a single R6300v2 with bufferbloat control and the rest of my system is on the PEPLINK aggregating 3 DSL WANs. I have successfully managed to get 4 bridged modems -> 4 LEDE R6300v2 routers -> PEPLINK but, as mentioned above in several places this leads to very slow connects to web pages etc.

My current reading and experimentation is focused on Double Nats as mentioned by JMJones. I think that it may be that this problem may be exacerbated by 4 WANs on the PEPLINK.

Good references I have found on the internet include:

http://www.practicallynetworked.com/networking/fixing_double_nat.htm

This reference provides this advice:


The Remedy

To check for double NAT on your network, log into your router and look up the IP address of its WAN port. If you see an address in the 10.x.x.x or 192.168.x.x range (both of which are private) it means that the device your router's WAN port connects to is doing NAT, and hence, you're dealing with double NAT.

There are a several options available to correct -- or circumvent -- a double NAT situation. If the culprit is your ISP-supplied equipment, you may be able to access the device's configuration interface via a browser and set it up to work in "bridge" mode. This will disable NAT on the device and essentially make it transparent on the network so your router will receive the public IP address and perform the NAT function on its own. Instructions on how to activate bridge mode for your specific device can usually be found on the ISP's or device manufacturer's support site, but if you can't find the information or aren't comfortable making the change, an ISP's phone tech support will often do it for you on request (or at least walk you through it).

If, on the other hand, your double NAT is being caused by a third-party piece of equipment that needs to be connected in front of your router (the aforementioned VoIP adapters usually require/recommend this for quality-of-service reasons), eliminating double NAT really isn't an option-- but you can still get around it.

One way to compensate for double NAT is to set up separate port forwarding rules on each device so that incoming traffic is shepherded through both layers of NAT. So for example, on the first NAT device (the one closest to your Internet connection) forward the port(s) you need to the IP address of your router's WAN port. Then on your router, forward the same port(s) to the address of the device you need to reach.

If you have a lot of ports to forward, doing them individually can get a bit cumbersome, so a simpler method is to configure the first NAT device to make your router's IP address the DMZ. This will hustle all incoming traffic through the first layer of NAT no questions asked, but when it hits your router it will be filtered or forwarded as appropriate.


https://portforward.com/help/doublerouterportforwarding.htm

This reference provides similar ideas and a handy diagram:

The idea of the first reference seems practical and easy:

"If you have a lot of ports to forward, doing them individually can get a bit cumbersome, so a simpler method is to configure the first NAT device to make your router's IP address the DMZ. This will hustle all incoming traffic through the first layer of NAT no questions asked, but when it hits your router it will be filtered or forwarded as appropriate."

This makes particular sense with a LEDE router that functions only as a bufferbloat appliance.

So I am currently looking at this advice from LEDE

https://lede-project.org/docs/user-guide/firewall_configuration#simple_dmz_rule

Our Cake box is in bridge mode only, no nat here. BUT, its in between our customers and our main nat gateway router so it sees our customer router ip's. In your situation, you can also use cake in bridge mode but you'll loose per internal ip sharing. Not so bad because you'll still get per stream sharing and the other stuff which is better than nothing. If you need cake to use more than 1024 flows, give me a shout, i can help.

Hi orangetek,

had you time to figure out how many concurrent flows you actually need for your 600 subscribers? Does the maximum of 2^16? works noticeably better than 2^15? Basically, how low can you push this and still get good isolation? I seem to recall that the number of flows active in a core router is actually much smaller than one would naively expect... It would be especially great if you could report these results also on the cake mailing list, where Jonathan Morton cake's principal developer will see it and have a chance to chime in? See https://lists.bufferbloat.net/listinfo/cake for getting on the cake mailing list...

Best Regards

Orangetek,

I have tried the Cake Box in your position. The problem for me is the use of the Peplink router to aggregate 4x10mbps connections. As a result a cakebox between the Peplink and clients is there the internet connection is about 38 mbps download and about 2,5 mbps upload. As a result it has virtually no bufferbloat control on the uploads unless it is set at about .7 mbps upload and that wrecks the fat pipe made in aggregation.

Both Peplink and IQ routers think to be effective I really need 4 cakeboxes, one per WAN. Problem there is a strange interference with connections with every additional cakebox added. I thought it could be something in the Peplink that makes it look at every possible connection like between cakeboxes that aren't attached to a given WAN but I made firewall rules prohibiting any but linear connections between a given modem, its cakebox and the Peplink but that doesn't help.

The advice of turning the Peplink into a DMZ behind each cakebox seemed to actually help, but not enough to make the connections viable. I'll have Peplink techs look over my router configuration and see if they can come up with anything but right now it seems like I've tried everything and nothing makes the system acceptably functional.

The one thing I have not managed to configure is to try and do away with the WAN to LAN connection in the cakeboxes and instead only use their lan ports. This would require making two VLans and then connecting them somehow, at least one would have to be in bridge formation. Maybe if I manage that I'd have one Vlan connection that might work as a point for bufferbloat control.

Most routers I know of have two physical ports, one for WAN and one to connect to the LAN switch, in that case you would be better of to turn the LEDE box into a transparent bridge between WAN and LAN, assuming it is possible to still instantiate cake on members of bridged interfaces (which I believe it is). That way cake shaping should be transparent to the PEP link, no?

I've made them as transparent as is possible with Wan to Lan links and bufferbloat control works great on the PPOE wan connection. DHCP is turned off, DMZ to Peplink configured, no firewall etc.

But...I still have this very aggravating issue on connectivity. At this point I don't think it is double NAT but somehow a difficulty the Peplink has with a static connection to the cakebox and associated once removed DNS connections (that would not be necessary if I could avoid the WAN connection on the cakebox).

Are your cake units running in bridge mode? No nat, no firewall, no dnsmasq?