BracketQos - Rust, EBPF, XDP, & cake, oh my!

For Developers
14

Actually, even Rust std uses libc for sockets it seems, so I've given up on getting rid of that dependency.
But for ICMP type 8 smoltcp could be nice, since it handles the whole chain, but it doesn't expose type 13, so I'll stick with what I've got for now.

I hope that something like https://crates.io/crates/syscalls will mature and be supported on all architectures in stable, then we could drop libc for the syscall related stuff.

For TCP_INFO, Nix has facilities for getting socket options, but currently doesn't implement TCP_INFO.
The major thing that makes it a bit painful to add (at least for me being a Rust newb) is that the tcp_info struct uses bitfields, which Rust doesn't have builtin support for. So that'd take some research on how to handle.

15

I feel the same way. I have always really liked the look and feel of C and am dubious about the way rust looks.

16

I checked this list, and the Marvell Amethyst 88E6393X switch inside doesn't seem to be supported, from what I can gather?

I'll check when I get home though.

17

Are you aiming to run something like libreqos.io or Bracket QOS on OpenWrt? I'm not sure I see the use case. Are you looking for something power efficient to cover a single WISP tower?

For optimizing CAKE in a home setting I'd look more in the direction of something like software flow offloading, not sure how the current state of that is.

1 Like
18

@Borromini Switch doesn't matter for this, MVPP2 is the ethernet controller.
I know it has some fancy stuff, but no idea if it has BQL

1 Like
19

@Zoxc There is a lot of "we shipped openwrt 22.03.0, now what?", sort of speculation going on.

In my case I'm very frustrated that vendors like mikrotik have only just added cake, and are 6-7 years behind openwrt in many respects. They make good hardware, but....

One of the biggest problems linux has had in the last few years is that it bottlenecks on reads, not writes. Some bleeding edge stuff like vpp and dpdk appeared that could use 1/16th the cores to blast more packets... but lack any method at all at doing good queue control. XDP and ebpf are responses to that while still retaining good egress characteristics.

"Something power efficient for a tower" & well queued & encrypted backhaul would be great! I am allergic to python, but rust seems fast and performant and easier to make run on openwrt/routerOS/vyatta/linux/etc

PS if you (or anyone) can figure out a way to poll tcp_info in rust in crusader, it would be GREAT. For those here that haven't played with it yet, it's here: https://github.com/Zoxc/crusader

20

@dtaht Does this help any?

root@RB5009UG+S+IN:/sys/class/net# ls
bonding_masters  br-lan.1         fiber            p1               p2.20            p4               p6               p8               sfp
br-lan           eth0             lo               p2               p3               p5               p7               pppoe-wan        wg0
root@RB5009UG+S+IN:/sys/class/net# cat */queues/tx-0/byte_queue_limits/limit
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
21 
root@turris:~# cat /sys/class/net/eth2/queues/tx-0/byte_queue_limits/*
1000
0
7590
1879048192
0

Turris Omnia, BQL seems to work... the 3rd number changes during load.

22

@Borromini doesn't look like BQL is there. Sigh. It's only 8 lines of code, and makes a huge difference, especially for bidirectional traffic. There's a mini-BQL tutorial over here: http://www.taht.net/~d/broadcom_aug9_2018.pdf

23

@moeller0 That's on OpenWrt or on Turris OS?

@dtaht I'm getting an 'SSL_ERROR_RX_RECORD_TOO_LONG' from your webserver here?

24

that is an http not https address.

25

Sorry. Overly zealous Firefox.

26

That is on OpenWrt19 based TurrisOS 5.4.3. I bought this thing as I wanted automatic updates (from a source I trust) and it has been delivering; however it currently straggles behind upstream OpenWrt*. But I assume that BQL if available in the old series 4.14 kernel, will also be available for more modern mvneta drivers.

*) This is not a complaint just a statement of current fact, upgrades to a more recent OpenWrt base are in the works and hopefully will be rolled-out soon. If I wanted/dared I could already test the upcoming TOS6.

1 Like
27

wow. that is a highly capable river, with xdp and tso support, but no bql. doesn't look hard to add though. https://elixir.bootlin.com/linux/v6.0-rc6/source/drivers/net/ethernet/marvell/mvpp2/mvpp2_main.c#L4417

28

I'd be happy to help (and I read the PDF you linked to) but my C skills are extremely limited. I'll gladly test any patches you'd have though, ideally for 5.10 (device is on 22.03).

29

Don't have the hardware, and doing BQL really requires having it, due to how hard it is to find all the sources of resets from code inspection. But I'll put it on the todo list. 1024 tx descriptors is a bit much! At a gbit,
30k (22 big packets), is all bql will put on the ring. Also the flow control to the switch was "interesting", I don't know how much buffering is in the switch itself.

1 Like
30

I tend to agree, but in C it's often too easy to do this:
image

While a bit alien to me as well, I really do like Rust's memory safety features a lot.
Although admittedly I curse at the borrow checker sometimes :grinning_face_with_smiling_eyes:

Half relatedly I came across this: https://github.com/carbon-language/carbon-lang
Another Google-backed language that aims to be an successor to C++, not ready for usage yet, and likely won't be for a while, but I'll be keeping an eye on it.

1 Like
31

What I wish for is a language that works well running inside an interpreter but that can also be compiled... for development and troubleshooting an interpreted autorate implementation is so much easier to work with. (I guess if the compiler and development environment is small enough to include on a router that would also work, but I somehow doubt that is a viable way forward for low storage all-in-one router)...

32

Seemed more appropriate to continue in the autorate thread, so I've posted my reply there instead

1 Like
33

I toyed around with getting tcp_info from a socket, seems to work, result here: https://github.com/Lochnair/tcp_info_test

I likely won't have time to add this into Crusader, so I'll leave that part for someone else :slight_smile: