IPFire vs. OpenWrt

How does the performance of OpenWrt compare to IPFire? Are there people here who have used both OSes on the same hardware? I am particularly interested in SQM with Cake.

Both should be about the same speed, since both are based on Linux (if the same features are used).

I am currently using IPFire 2.29 (x86_64) on a small x86_64 box (J3160) with Coreboot (similar to Protectli FW4B). When using Cake, my throughput is roughly halved (I have 600 down and 300 up). It would probably be no different with OpenWrt? The hardware is probably too weak.

But I'm also thinking of switching to OpenWrt for other reasons (e.g. VLANs).

The CPU cost of a qdisc like cake will not magically differ depending on the Linux flavor you run it under... my guess is simply that these cores are a bit on the weak side for 600/300... Maybe there is a bit tweaking that can help a bit but in the end, have a look at irqbalance and receive packet steering (which both should also work under non OpenWrt linux distributions...)

1 Like

You can very reasonably boot OpenWrt from a USB stick on x86_64, without touching/ harming your installed system. So doing a realistic back to back comparison should be relatively easy.

4 Likes

Thanks for the info. I might even do that if I find the time.
Regardless of that, I will probably switch to OpenWrt, as IPFire lacks some features and options, such as VLANs. I'm also keen to try something new. I have OpenWrt running on two access points but have never really used it.
I would also like to buy new hardware. And with OpenWrt you have more options (probably BPI-R4 or maybe Nanopi R6S).
I did a few quick tests with IPFire. I do have IPS running, but I haven't been using it recently. Interesting that Suricata seems to use so little processing power on IPFire?

IPFire 2.29 (x86_64) - Core-Update 190

Cake, Enable Intrusion Prevention System - Suricata with Emergingthreats.net Community Rules @ Red, Green, Blue
https://www.waveform.com/tools/bufferbloat?test-id=65f8f718-3990-4771-90f0-e00215cf3611
A+
Down: 255,5 Mbps
Up: 91,6 Mbps

Cake, No IPS
https://www.waveform.com/tools/bufferbloat?test-id=d5660195-5994-4e0d-be44-5d6460b0cade
A+
Down: 248,5 Mbps
Up: 90,0 Mbps

No Cake, No IPS
https://www.waveform.com/tools/bufferbloat?test-id=65f8f718-3990-4771-90f0-e00215cf3611
A
Down: 622,7 Mbps
Up: 232,9 Mbps <= ???

New Test
https://www.waveform.com/tools/bufferbloat?test-id=490820e6-91b3-40cc-adb9-9f7396c613f2
A
Down: 617.4 Mbps
Up: 317.6 Mbps

No Cake, Enabled IPS
https://www.waveform.com/tools/bufferbloat?test-id=ba64422d-9cef-49eb-9519-d2570849f56c
A
Down: 626,1 Mbps
Up: 311.7 Mbps

Edit:
I am not always getting an A without cake...

And OpenWRT?

Something clearly is odd here (besides cake being a tad expensive CPU wise, which alas is expected). Could you please post the output of tc -s qdisc so I can see the cake configuration?
I am well aware that this is IPFire and not OpenWrt, but I am quite interested in understanding why you end up with ~250/90 Mbps with cake. (Unless this reflects your shaper settings, I would have expected 250/250 giving that even 250 is well below your true upload limit of 300 Mbps).

That at least implies that cake is doing what it is intended to do, so hurray...

# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc htb 1: dev red0 root refcnt 9 r2q 10 default 0x110 direct_packets_stat 4 direct_qlen 1000
 Sent 39018351 bytes 316293 pkt (dropped 0, overlimits 3613 requeues 107)
 backlog 0b 0p requeues 107
qdisc cake 120: dev red0 parent 1:120 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 0b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:        65535 /       0
 min/max overhead-adjusted size:    65535 /       0
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us          0us          0us
  av_delay          0us          0us          0us
  sp_delay          0us          0us          0us
  backlog            0b           0b           0b
  pkts                0            0            0
  bytes               0            0            0
  way_inds            0            0            0
  way_miss            0            0            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len             0            0            0
  quantum          1514         1514         1514

qdisc cake 102: dev red0 parent 1:102 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 12582 bytes 128 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2304b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           63 /     118
 min/max overhead-adjusted size:      159 /     212
 average network hdr offset:            6

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         46us          0us
  av_delay          0us          7us          0us
  sp_delay          0us          6us          0us
  backlog            0b           0b           0b
  pkts                0          128            0
  bytes               0        12582            0
  way_inds            0            0            0
  way_miss            0          128            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len             0          132            0
  quantum          1514         1514         1514

qdisc cake 104: dev red0 parent 1:104 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 21205940 bytes 188913 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 48384b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           40 /    1492
 min/max overhead-adjusted size:      106 /    1749
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us        187us          0us
  av_delay          0us         24us          0us
  sp_delay          0us          7us          0us
  backlog            0b           0b           0b
  pkts                0       188913            0
  bytes               0     21205940            0
  way_inds            0          284            0
  way_miss            0         2272            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        15670            0
  quantum          1514         1514         1514

qdisc cake 110: dev red0 parent 1:110 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 17765323 bytes 127005 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 72319b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           28 /    1492
 min/max overhead-adjusted size:      106 /    1749
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us       1.64ms         15us
  av_delay          0us        199us          2us
  sp_delay          0us          9us          2us
  backlog            0b           0b           0b
  pkts                0       126937           68
  bytes               0     17762323         3000
  way_inds            0          400            0
  way_miss            0         2333            3
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        17176           90
  quantum          1514         1514         1514

qdisc cake 101: dev red0 parent 1:101 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 30761 bytes 187 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2304b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           56 /     576
 min/max overhead-adjusted size:      159 /     689
 average network hdr offset:            7

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         24us         32us
  av_delay          0us          3us          4us
  sp_delay          0us          3us          4us
  backlog            0b           0b           0b
  pkts                0           69          118
  bytes               0         7998        22763
  way_inds            0            0            0
  way_miss            0            4            6
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            1
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len             0          118          590
  quantum          1514         1514         1514

qdisc cake 103: dev red0 parent 1:103 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 2900 bytes 52 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2304b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           29 /     112
 min/max overhead-adjusted size:      106 /     212
 average network hdr offset:            3

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         31us          0us
  av_delay          0us          3us          0us
  sp_delay          0us          3us          0us
  backlog            0b           0b           0b
  pkts                0           52            0
  bytes               0         2900            0
  way_inds            0            0            0
  way_miss            0            1            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0          126            0
  quantum          1514         1514         1514

qdisc ingress ffff: dev red0 parent ffff:fff1 ----------------
 Sent 467539093 bytes 472548 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 8003: dev green0 root refcnt 9 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 22664716501 bytes 17763490 pkt (dropped 0, overlimits 0 requeues 11523)
 backlog 0b 0p requeues 11523
  maxpacket 52290 drop_overlimit 0 new_flow_count 180191 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 8004: dev blue0 root refcnt 9 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 8002: dev orange0 root refcnt 9 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
 Sent 445028805 bytes 478254 pkt (dropped 0, overlimits 0 requeues 608)
 backlog 0b 0p requeues 608
  maxpacket 25452 drop_overlimit 0 new_flow_count 1180 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc htb 2: dev imq0 root refcnt 2 r2q 10 default 0x210 direct_packets_stat 3 direct_qlen 32
 Sent 483107408 bytes 472533 pkt (dropped 13, overlimits 275526 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 203: dev imq0 parent 2:203 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 1260 bytes 10 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2304b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:          112 /     112
 min/max overhead-adjusted size:      212 /     212
 average network hdr offset:            1

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         17us          0us
  av_delay          0us          0us          0us
  sp_delay          0us          0us          0us
  backlog            0b           0b           0b
  pkts                0           10            0
  bytes               0         1260            0
  way_inds            0            0            0
  way_miss            0            1            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0          126            0
  quantum          1514         1514         1514

qdisc cake 220: dev imq0 parent 2:220 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 0b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:        65535 /       0
 min/max overhead-adjusted size:    65535 /       0
 average network hdr offset:            0

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us          0us          0us
  av_delay          0us          0us          0us
  sp_delay          0us          0us          0us
  backlog            0b           0b           0b
  pkts                0            0            0
  bytes               0            0            0
  way_inds            0            0            0
  way_miss            0            0            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len             0            0            0
  quantum          1514         1514         1514

qdisc cake 200: dev imq0 parent 2:200 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 7198 bytes 61 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2304b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:          104 /     104
 min/max overhead-adjusted size:      212 /     212
 average network hdr offset:            3

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         27us          0us
  av_delay          0us          3us          0us
  sp_delay          0us          3us          0us
  backlog            0b           0b           0b
  pkts                0           61            0
  bytes               0         7198            0
  way_inds            0            0            0
  way_miss            0            1            0
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            0            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0          118            0
  quantum          1514         1514         1514

qdisc cake 204: dev imq0 parent 2:204 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 442748120 bytes 326923 pkt (dropped 12, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 851424b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           46 /    1492
 min/max overhead-adjusted size:      106 /    1749
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us        107us          0us
  av_delay          0us         21us          0us
  sp_delay          0us          6us          0us
  backlog            0b           0b           0b
  pkts                0       326935            0
  bytes               0    442765708            0
  way_inds            0          251            0
  way_miss            0         2224            0
  way_cols            0            0            0
  drops               0           12            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0        67770            0
  quantum          1514         1514         1514

qdisc cake 210: dev imq0 parent 2:210 bandwidth unlimited diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms atm overhead 48
 Sent 40350154 bytes 145534 pkt (dropped 1, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 52992b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           46 /    1492
 min/max overhead-adjusted size:      106 /    1749
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh           0bit         0bit         0bit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay          0us         54us         23us
  av_delay          0us         20us          3us
  sp_delay          0us         12us          3us
  backlog            0b           0b           0b
  pkts                0       145470           65
  bytes               0     40347496         3900
  way_inds            0          415            0
  way_miss            0         2667            1
  way_cols            0            0            0
  drops               0            1            0
  marks               0            1            0
  ack_drop            0            0            0
  sp_flows            0            1            1
  bk_flows            0            2            0
  un_flows            0            0            0
  max_len             0        13446           60
  quantum          1514         1514         1514
1 Like

When I find the time.

Thanks, that looks complicated... so they likely create a HTB tree and attach one cake instance to each leaf. For that purpose the cake configuration looks sub-optimal, but I guess you inherited that from IPFire? And IPFire might have reasons for that configuration choices no matter how odd these look to me :wink:

1 Like

Yes, I took this over from IPFire. That is the default setting in the GUI. I only entered my Internet speed. I'm not familiar with Cake.

In any case, I would probably like to use OpenWrt in the future. The possibilities of IPFire 2.x are limited. But as long as you use IPFire as intended, it's pretty good and easy.

2 Likes

I was just about to do that. My IPFire box is in the basement without a monitor. I brought the box up to my office, connected the monitor and keyboard and booted an OpenWrt image via USB. Everything worked and I was able to connect directly from my Mac via LAN. But of course I had no internet connection.
So I took the box back to the basement and got confused with my cables. Then I brought the box back up to the office. Suddenly I could no longer boot from the USB stick.
In Coreboot you have to press F11 at the beginning to be able to boot via USB. The USB stick was no longer recognized. I then swapped the USB port. Then the USB stick was recognized but it did not boot.

I suspect that the USB stick has broken.

The whole process cost me a lot of time and my children were annoyed because they wanted to play online.

I may try this again later or in the next few days with another USB stick.

I have just started OpenWrt from the USB stick.
Test with the OOTB installation:

A
Down: 613.8 Mbps
Up: 285.1 Mbps <= Slightly less than under IPFire. However, I have only carried out this test.

https://www.waveform.com/tools/bufferbloat?test-id=0044af95-74e7-4c3a-9006-1f7db7112887

SQM/Cake (Smart Queue Management): https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm
A+
Down: 262.9 Mbps
Up: 240.7 Mbps <= Significantly better than with IPFire. However, it may not be possible to make a 1:1 comparison. To do this, it would have to be ensured that the same or comparable services are activated.

https://www.waveform.com/tools/bufferbloat?test-id=536ffcab-6037-4469-9247-f27a756900b3

What are your subscription parameters? J3160 needs memory interleaving enabled to comfortably forward gigabit each way.

What do you mean by that?
In the meantime, I have IPFire running again.
Out of curiosity I had installed SNORT and then the box stopped responding.

comfortably forward gigabit each way

But not with SQM/Cake?!

i am saying that each of 2 DDR3 ram modules are too slow to store and forward between 2 gigabit cards. You have to make them work both at once cf raid0

I was not aware of this function ("memory interleaving enabled").

FW4B:

I have a Yanling, but it is identical to the Protectli. I use CoreBoot and don't know if you can activate this feature in CoreBoot. The device also only has one memory slot.

I have also switched to a BPI-R4, but will perhaps look for the feature when I get the time.

Then you cannot forward at 2gbps. dual-channel memory setup supported by CPU could do it somewhat if little packet inspection/transform is done down the road.

1 Like