I think the queues used by (*)MII are not the actual queues used by the nic.
But the (8) queues are pretty much useless anyway.
Because it is not possible to configure them with ethtool.
ethtool -l eth0
Channel parameters for eth0:
Cannot get device channel parameters
: Not supported
This means that your driver has not implemented the ethtool get_channels operation. This could be because the NIC doesn’t support adjusting the number of queues, doesn’t support RSS / multiqueue, or your driver has not been updated to handle this feature.
ethtool -x eth0 (RX flow indirection table)
RX flow hash indirection table for eth0 with 4 RX ring(s):
0: 0
RSS hash key:
Operation not supported
RSS hash function:
toeplitz: on
xor: off
crc32: off
ethtool -n eth0 (RX network flow classification)
4 RX rings available
rxclass: Cannot get RX class rule count: Not supported
RX classification rule retrieval failed
ethtool -k eth0 | grep 'ntuple' ntuple-filters: off [fixed]
So configuring RX/RSS queues doesn't work?
I currently have a build running with only 2 queues (Well actually, 4 queues 2 RX / 2 TX). (one queue for each CPU.)
Then enable XPS, one queue assign to each CPU.
And don't use RPS (and don't see any significant change in CPU utilization anyway), I have read that it can introduce unwanted latency and RSS is the better choice.
But unfortunately, it isn't possible to configure RSS. (I think this is also the reason why one CPU core gets maxed out all the time when there is much traffic load)
//edit
2 queues also work fine.
One "strange" thing to add...
With the packet steering script disabled and 4 queues enabled, the XPS mapping is as follows:
Queue/CPU Mask:
1:1
2:2
3:0
4:1
Why is XPS automatically disabled for one Queue?
With the packet steering script disabled and 2 queues enabled, the XPS mapping is as follows:
Queue/CPU Mask:
1:1
2:2
Hmm...
The first queue shows: 15081 packets transmitted
maxpacket: 0 (max packet size?) how can this be 0?
The second queue shows: 1900689 packets transmitted
maxpacket: 1522, makes more sense...
new_flow_count: 4, seems a bit low to me....
And when I search the forum for some tc output, for example here:
Its the same, most queues show a maxpacket value of 0.
I will try a build with 1 queue, with swconfig that didn't work. maybe it works with DSA...
//edit
Nope.. doesn't work. Interfaces come up but no communication is possible.
The interface(s) packet counter shows a large amount of RX packets..
Hmm...
//edit 3
updated once more...
classify doesn't seem to work either...
Next bet, skbedit...
// edit 4
skbedit also not working...
Maybe it only works with mqprio qdisc but mqprio also doesn't work (also tried hw 0).
So back to the connmark match approach...
The hierarchy looks like this
root/parent 1:0
mq class 1:1 -> leaf 110: fq_codel
mq class 1:2 -> leaf 120: fq_codel
It is not possible to attach filters to root/parent 1: or class 1:1 / 1:2
It is only possible to attach filters to 110: / 120:
Matching for fw mark 1 in 110: makes no sense because it is already in there.
So match for fw mark 2 in 110: and redirect it to 1:2 and the other way around...
But actually I'm not sure if this works properly...
CPU0 still has twice as much interrupts as CPU1 but I think this has something to do with DSA...?
But this fixed the weird lag on my WiFi.
For example, when browsing on my phone the loading bar almost always got stuck at ~80% for a few seconds (but the website was fully loaded).
Master has a new backport to enable GRO for DSA.
I'm not sure if it actually does anything useful for mvebu because mvneta uses napi_gro_receive() instead of napi_gro_frags().
But re-enabled GRO to see if it makes any difference.
VID | lan0 | lan1 | lan2 | lan3 | wan |
----------------------------------------
201 | t | t | u | u | |
202 | t | t | | | |
204 | t | t | | | |
2 | | | | | u |
root@linksys0:~# cat /etc/config/network
config interface 'vlan201'
option type 'bridge'
option ifname 'lan1.201 lan2.201 lan3 lan4'
option proto 'static'
option ipaddr '10.254.201.1'
option netmask '255.255.255.0'
config interface 'vlan202'
option type 'bridge'
option ifname 'lan1.202 lan2.202'
option proto 'static'
option ipaddr '10.254.202.1'
config interface 'vlan204'
option type 'bridge'
option ifname 'lan1.204 lan2.204'
option proto 'static'
option ipaddr '10.254.204.1'
option netmask '255.255.255.0'
config interface 'wan'
option 'ifname' 'wan'
option 'proto' 'pppoe'
option 'username' 'user'
option 'password' 'password'
option 'timeout' '10'
root@linksys0:~# cat /etc/rc.local
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
#### enable vlan filtering
ip link set dev br-vlan201 type bridge vlan_filtering 1
ip link set dev br-vlan202 type bridge vlan_filtering 1
ip link set dev br-vlan204 type bridge vlan_filtering 1
#### set vlans
bridge vlan add dev lan3 vid 201 pvid untagged
bridge vlan add dev lan4 vid 201 pvid untagged
ip link set br-vlan201 type bridge vlan_default_pvid 201
ip link set br-vlan202 type bridge vlan_default_pvid 202
ip link set br-vlan204 type bridge vlan_default_pvid 204
#### clear out vlan 1
bridge vlan del dev lan3 vid 1
bridge vlan del dev lan4 vid 1
I have problem with communication between untagged and tagged interfaces ( lan1.201 and lan3 ) in bridge vlan201 .
This is tcpdump on openwrt when pinging something connected on lan1.201 from PC connected to lan3 .
This setup is based on setup provided by dengqf6 in PR2942.
The difference from DSA setup 1 is that there is only one bridge.
I tried configuration based on that setup, but the router is dead (bricked) after reboot.
I don't have serial cable to see where is the problem.
I saw that there is setup in /etc/hotplug.d/iface/21-lan file (not in /etc/rc.local ).
Ports lan1 and lan2 are trunk ports (with vlans 201 , 202 , and 203 ), and ports lan3 and lan4 are untagged ports with vlan 201.
I can boot the previous partition with old swconfig configuration.
# vi /etc/config/network
config interface 'lan'
option type 'bridge'
option ifname 'lan1 lan2 lan3 lan4'
option proto 'none'
config interface 'vlan201'
option ifname '@lan.201'
option proto 'static'
option ipaddr '10.254.201.1'
option netmask '255.255.255.0'
config interface 'vlan202'
option ifname '@lan.202'
option proto 'static'
option ipaddr '10.254.202.1'
option netmask '255.255.255.0'
config interface 'vlan204'
option ifname '@lan.204'
option proto 'static'
option ipaddr '10.254.204.1'
option netmask '255.255.255.0'
# vi /etc/hotplug.d/iface/21-lan
#!/bin/sh
[ $INTERFACE = lan -a $ACTION = ifup ] || exit 0
# enable VLAN filtering
ip link set dev br-lan type bridge vlan_filtering 1
# clear out vlan 1
bridge v del dev lan1 vid 1
bridge v del dev lan2 vid 1
bridge v del dev lan3 vid 1
bridge v del dev lan4 vid 1
bridge v del dev br-lan self vid 1
# set vlans lan1
bridge v add dev lan1 vid 201
bridge v add dev lan1 vid 202
bridge v add dev lan1 vid 204
# set vlans lan2
bridge v add dev lan2 vid 201
bridge v add dev lan2 vid 202
bridge v add dev lan2 vid 204
# set vlans lan1
bridge v add dev lan3 vid 201 pvid untagged
# set vlans lan2
bridge v add dev lan4 vid 201 pvid untagged
# set vlans cpu port
bridge v add dev br-lan self vid 201 pvid untagged
bridge v add dev br-lan self vid 202
bridge v add dev br-lan self vid 204
I hope somebody can find / locate the problem.
I am not sure where is is the problem, and is configuration in file /etc/hotplug.d/iface/21-lan correct, or it should go also in /etc/rc.local.
Hi,
Thanks!
I managed to solve my DSA configuration.
Setup with multiple bridges is not working ok, because there is a problem in communication between tagged and untagged interfaces in the same bridge.
So, it seems that setup with single bridge is working fine.
VID | lan0 | lan1 | lan2 | lan3 | wan |
----------------------------------------
201 | t | t | u | u | |
202 | t | t | | | |
204 | t | t | | | |
2 | | | | | u |
root@linksys0:~# cat /etc/hotplug.d/iface/21-lan
#!/bin/sh
[ $INTERFACE = lan -a $ACTION = ifup ] || exit 0
#### enable VLAN filtering
ip link set dev br-lan type bridge vlan_filtering 1
#### clear out vlan 1
bridge v del dev lan1 vid 1
bridge v del dev lan2 vid 1
bridge v del dev lan3 vid 1
bridge v del dev lan4 vid 1
bridge v del dev br-lan self vid 1
#### set vlans lan1
bridge v add dev lan1 vid 201
bridge v add dev lan1 vid 202
bridge v add dev lan1 vid 204
#### set vlans lan2
bridge v add dev lan2 vid 201
bridge v add dev lan2 vid 202
bridge v add dev lan2 vid 204
#### set vlans lan3
bridge v add dev lan3 vid 201 pvid untagged
#### set vlans lan4
bridge v add dev lan4 vid 201 pvid untagged
#### set vlans cpu port
# bridge v add dev br-lan self vid 201 pvid untagged
bridge v add dev br-lan self vid 201
bridge v add dev br-lan self vid 202
bridge v add dev br-lan self vid 204
root@linksys0:~#
Output:
root@linksys0:~# brctl show
bridge name bridge id STP enabled interfaces
br-lan 7fff.6038e0cd87c0 no lan1
lan2
lan3
lan4
root@linksys0:~#
root@linksys0:~# bridge vlan
port vlan-id
lan4 201 PVID Egress Untagged
lan3 201 PVID Egress Untagged
lan2 201
202
204
lan1 201
202
204
br-lan 201
202
204
root@linksys0:~#
root@linksys0:~# bridge link show
3: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 4
4: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 100
5: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 master br-lan state disabled priority 32 cost 100
6: lan1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-lan state forwarding priority 32 cost 4
root@linksys0:~#
This is only needed when the hardware doesn't support the dsa tagging protocol?
Why do they create the vlan interface on the dsa master interface ethX.0, ethX.1, ethX.2, etc?
Why doesn't the page show an example with tagging support + vlans?
Hi,
I have external switch TL-SG108E which supports vlans, and Mikrotik CAP ac as additional Access Point.
That is the reason why I have 2 trunk network interface ports on wrt3200acm: one for switch, one for Mikrotik Access Points.
All vlans are defined on trunk ports.
That's the issue I have been complaining about since the upgrade from kernel 4.14 to 4.19. I have no idea how to fix it cause I'm no good with patches and after extensive searching I've found nothing.
Also, this brings me to another point, why does OpenWRT use interface vlans instead of vlan filtering? From what I gathered (this is valid for DSA, swconfig has no benefit from this) Linux should have a bridge per switch and use vlan filtering to manage port vlan membership, correct?
Doesn't the driver initialize the rx/tx hardware queues?
But there is no logic to select a queue.
So there is only partial support for mq?
With the generic ndo select function (post above) in combination with xps/rps it seems like it is possible to get more interrupts on "the other" CPU.
This is with 2 Queues enabled (one for each CPU)
And settings 1 RX and TX Queue in the driver source code file, bricks my device.
A proper ndo select function to implement basic RSS would be nice...
But I have no clue how to do it :<
And it would be better to limit the queues to the number of CPUs, I think...
Because there is no performance gain when having more queues than CPUs.
Idk, I'm not sure.
Usually WiFi interfaces have 4 queues. (?)
Which CPU should process the interrupts is set by /proc/irq/< irq number>/smp_affinity
It it possible to set a wildcard there (higher value/mask than CPUs?) so any CPU can process interrupts.
So interrupts can be spread across the CPUs?
But I'm not sure when Linux starts to move interrupts from one CPU to another.
So for WiFi devices it is only possible to have all queues mapped to one CPU or to all CPUs?
It is not possible to change the affinity for the mvneta driver/mvneta devices.
By default the value is set to 3 (meaning CPU1+2)
But most interrupts are handled on one CPU.
Idk how interrupt handling is accomplish there.
If it is a ARM specific thing, some driver limitation or something else ...
I guess, the problem is, the driver has no function to select any queue.
Intel drivers for example have such functionality and is it possible to configure it via ethtool.
(So it is a driver limitation?)