Lede as a dedicated QoS /Bufferbloat appliance

It seems ip conflict.

just to be sure.
Are you using different ip address for each cake router right ? like

192.168.0.1/24
192.168.1.1/24
192.168.2.1/24
192.168.3.1/24

and Peplink gives(DHCP) another totally different like 10.0.0.1/24.

For the DNS, just use google on Peplink and all your client will use Peplink DNS(google).

Yes Felipee07, I'm toying with forwarding IPs from the PEPLINK to the cake routers, that might make for a less complex routing;

I've not done IP forwarding before but moving the IPs forward to the cake routers might be handy.

I still don't get where you're going.

Why PPPOE on PepLink, the should be DHCP(Connection Method) for the 4 wan

Connection Method (DHCP)
Routing Mode (NAT)

Felipee07:

Probably you already know this, but I would like you to understand where I am headed.

Most dsl+ modems are either just modems and don't do much very well other than connect. Modem/router combinations provided by your ISP are ok as modems and generally mediocre at best as routers. That's why folks get pricey fast routers or less expensive fast LEDE routers to connect with their modems. Generally the modems are left in "transparent bridge" configuration so the better routers can do other things routers do better. One thing good routers do, like LEDE routers and PEPLINK routers, is to handle the PPOE connection to the modem in transparent bridge. Generally you know your modem doesn't know much so you try to take as much of the burden off it as is possible. Perhaps you have a point here...I've not had a modem do PPOE for at least 5 years but, hmmm, maybe pure static IPs at the routers is worth a try. That way the LEDE router would have a static IP and the PEPLINK would also have a static IP.

I have read that the device that does the PPOE does NAT so one might get double NAT and the associated slow down if the modem does the PPOE.

Just the size of the CPU on a modem and on a modern router are so much different, its hard to imagine the modem doing better. It's sorta like thinking an earth worm can out smart a dog, but hey, might be worth a try. Maybe an earth worm is better at burrowing than a dog...

Currently my LEDE routers are handling the PPOE, which is fine, but PPOE is generally set up to be handed off easy to the router doing the connection. Things are streamlined within the router, I believe, to hand off the connection to DHCP DNS and all that. So what I would like to do, where I'd like to head, but haven't been able to configure, is to let the PEPLINK handle the PPOE, DHCP, DNS link aggregation etc. The LEDE router would be simply a "cake box".

In order to do this, the cake box has to be very transparent. Perhaps this would be best accomplished by having Lan1 in and Lan2 out. I believe I have a double NAT going on with the cake boxes: I think each LEDE router does NAT and there is no way to turn that off and the PEPLINK also does NAT. By letting the PEPLINK do PPOE through both the LEDE router and the modem double and maybe triple NAT can be avoided.

Parenthetically your comments and JMJones comments made me further examine my static IPs and I did find one set of possible ambiguities. I fixed those and....the connections are better, some web pages actually completely load etc with 4 modems feeding to 4 cake boxes feeding to the PEPLINK. But, still very annoying and slow. Currently I am running 2 cake boxes, which helps with the blufferbloat, and 2 lines with no cake boxes.

Another idea I plan to explore is not allowing the LEDE router and the PEPLINK to automatically handle MTU. Undoubtedly the algorithms could be different so packaging and repackaging information handed off from one router to the other seems like it could induce lag. Thinking I'll set them both at 1438 and see if I get any increase in speed.

Thanks again for all those who have been watching me struggle with this. I've been building and using computers since MSDOS 2.0 and earlier but never did get much into networks, whole new bowl of spaghetti.

Currently...

I've currently taken apart my system so that my gaming is done on a single R6300v2 with bufferbloat control and the rest of my system is on the PEPLINK aggregating 3 DSL WANs. I have successfully managed to get 4 bridged modems -> 4 LEDE R6300v2 routers -> PEPLINK but, as mentioned above in several places this leads to very slow connects to web pages etc.

My current reading and experimentation is focused on Double Nats as mentioned by JMJones. I think that it may be that this problem may be exacerbated by 4 WANs on the PEPLINK.

Good references I have found on the internet include:

http://www.practicallynetworked.com/networking/fixing_double_nat.htm

This reference provides this advice:


The Remedy

To check for double NAT on your network, log into your router and look up the IP address of its WAN port. If you see an address in the 10.x.x.x or 192.168.x.x range (both of which are private) it means that the device your router's WAN port connects to is doing NAT, and hence, you're dealing with double NAT.

There are a several options available to correct -- or circumvent -- a double NAT situation. If the culprit is your ISP-supplied equipment, you may be able to access the device's configuration interface via a browser and set it up to work in "bridge" mode. This will disable NAT on the device and essentially make it transparent on the network so your router will receive the public IP address and perform the NAT function on its own. Instructions on how to activate bridge mode for your specific device can usually be found on the ISP's or device manufacturer's support site, but if you can't find the information or aren't comfortable making the change, an ISP's phone tech support will often do it for you on request (or at least walk you through it).

If, on the other hand, your double NAT is being caused by a third-party piece of equipment that needs to be connected in front of your router (the aforementioned VoIP adapters usually require/recommend this for quality-of-service reasons), eliminating double NAT really isn't an option-- but you can still get around it.

One way to compensate for double NAT is to set up separate port forwarding rules on each device so that incoming traffic is shepherded through both layers of NAT. So for example, on the first NAT device (the one closest to your Internet connection) forward the port(s) you need to the IP address of your router's WAN port. Then on your router, forward the same port(s) to the address of the device you need to reach.

If you have a lot of ports to forward, doing them individually can get a bit cumbersome, so a simpler method is to configure the first NAT device to make your router's IP address the DMZ. This will hustle all incoming traffic through the first layer of NAT no questions asked, but when it hits your router it will be filtered or forwarded as appropriate.


https://portforward.com/help/doublerouterportforwarding.htm

This reference provides similar ideas and a handy diagram:

The idea of the first reference seems practical and easy:

"If you have a lot of ports to forward, doing them individually can get a bit cumbersome, so a simpler method is to configure the first NAT device to make your router's IP address the DMZ. This will hustle all incoming traffic through the first layer of NAT no questions asked, but when it hits your router it will be filtered or forwarded as appropriate."

This makes particular sense with a LEDE router that functions only as a bufferbloat appliance.

So I am currently looking at this advice from LEDE

https://lede-project.org/docs/user-guide/firewall_configuration#simple_dmz_rule

Our Cake box is in bridge mode only, no nat here. BUT, its in between our customers and our main nat gateway router so it sees our customer router ip's. In your situation, you can also use cake in bridge mode but you'll loose per internal ip sharing. Not so bad because you'll still get per stream sharing and the other stuff which is better than nothing. If you need cake to use more than 1024 flows, give me a shout, i can help.

Hi orangetek,

had you time to figure out how many concurrent flows you actually need for your 600 subscribers? Does the maximum of 2^16? works noticeably better than 2^15? Basically, how low can you push this and still get good isolation? I seem to recall that the number of flows active in a core router is actually much smaller than one would naively expect... It would be especially great if you could report these results also on the cake mailing list, where Jonathan Morton cake's principal developer will see it and have a chance to chime in? See https://lists.bufferbloat.net/listinfo/cake for getting on the cake mailing list...

Best Regards

Orangetek,

I have tried the Cake Box in your position. The problem for me is the use of the Peplink router to aggregate 4x10mbps connections. As a result a cakebox between the Peplink and clients is there the internet connection is about 38 mbps download and about 2,5 mbps upload. As a result it has virtually no bufferbloat control on the uploads unless it is set at about .7 mbps upload and that wrecks the fat pipe made in aggregation.

Both Peplink and IQ routers think to be effective I really need 4 cakeboxes, one per WAN. Problem there is a strange interference with connections with every additional cakebox added. I thought it could be something in the Peplink that makes it look at every possible connection like between cakeboxes that aren't attached to a given WAN but I made firewall rules prohibiting any but linear connections between a given modem, its cakebox and the Peplink but that doesn't help.

The advice of turning the Peplink into a DMZ behind each cakebox seemed to actually help, but not enough to make the connections viable. I'll have Peplink techs look over my router configuration and see if they can come up with anything but right now it seems like I've tried everything and nothing makes the system acceptably functional.

The one thing I have not managed to configure is to try and do away with the WAN to LAN connection in the cakeboxes and instead only use their lan ports. This would require making two VLans and then connecting them somehow, at least one would have to be in bridge formation. Maybe if I manage that I'd have one Vlan connection that might work as a point for bufferbloat control.

Most routers I know of have two physical ports, one for WAN and one to connect to the LAN switch, in that case you would be better of to turn the LEDE box into a transparent bridge between WAN and LAN, assuming it is possible to still instantiate cake on members of bridged interfaces (which I believe it is). That way cake shaping should be transparent to the PEP link, no?

I've made them as transparent as is possible with Wan to Lan links and bufferbloat control works great on the PPOE wan connection. DHCP is turned off, DMZ to Peplink configured, no firewall etc.

But...I still have this very aggravating issue on connectivity. At this point I don't think it is double NAT but somehow a difficulty the Peplink has with a static connection to the cakebox and associated once removed DNS connections (that would not be necessary if I could avoid the WAN connection on the cakebox).

Are your cake units running in bridge mode? No nat, no firewall, no dnsmasq?

As far as i can tell, around 25k-30k during busy hours.

This is nice, so using 32K queues should perform just as well as 64K, no?

Completely bridge mode.

I reiterate I do test each cakebox as a separate unit...one wan to one computer and assess bufferbloat and speed. They work great.

This would suggest you are running cake in router mode

I test each cakebox attached only to a computer and a transparent bridged modem. After the test I move it into my system. DHCP is not enabled during the testing as the single computer's IP is configured to be a part of the same network as the cakebox. If for example a cakebox has the address of 10.0. 5.1 then the computer is given the address of 10.0.5.10

Each of your cake boxes should have 1 bridge interface only, with 2 ethernet adapter inside the bridge. Thats it. Use 'dhcp-client' as the protocol for the bridge interface then set up cake on either ethernet interface, NOT the bridge interface. You do not want an ip assigned to either physical interface and your firewall and dnsmasq should not be loaded. This is a transparent bridge. What you are describing is a router. Your Computer should not be getting an ip from the cake box.

Try bridge mode. It has much lower overheads for your hardware and you dont have yet another NAT in the path to the internet.

I gave this a solid try yestday and re-configured all my routers. As I think of this, this turns the router into a switch and all connections are on the lan side of my R6300v2s.

To test the configuration I put the LEDE box between my unmanaged switch and one computer. There it worked perfectly. I configured the VLAN so 2 ethernet ports allowed one in and one line out lan applied luci-app-sqm to eth1.0. The two other ethernet ports were given a static address so I could communicate through those ports with the LEDE box.

Switch Configuration

In the test position, it was between a computer and a switch so it was, by force, in a per IP mode. When moved to between my transparent bridged modem and my PEPLINK it was no longer between a single computer and a switch, instead it is between the modem and a WAN connection on the PEPLINK. The PEPLINK made PPOE connections through the LEDE boxes with no problems. However, the per computer bufferbloat control no longer was in operation and bufferbloat control became more hit and miss. Sometimes I'd get an A or A+ from dslreports/speedtest, and sometimes an F. While not as consistent as in the previous position it did improve my scores as previously they were straight Fs.

Also there remained a few connectivity issues. Some websites, like Microsoft games (my wife likes these) would not connect very rapidly.

All in all your solution is much much better on connections but not as good at bufferbloat control where I positioned the LEDE cakeboxes.

Currently I'm a bit discouraged with the PEPLINK-Cakebox combo and have split off one line for gaming that is just a cakebox router and a transparent bridged modem. The other 3 lines are for business uses and just on the PEPLINK with no bufferbloat control.