I also noticed that the port LEDs do not work in U-Boot either. They are probably (that's an educated guess) controlled by the software: Using the stock firmware, a button push toggles the LED functionality between speed indication and PoE status indication.
The speed of all three fans is supervised by the stock firmware, as soon as I stop one of the fans, the fan LED changes its color.
Regarding tracing the pins: I don't want to take off any of the soldered heat sinks and they are, unfortunately, too big to have access to the pins.
What's the current status here? I was convinced that I had tested vlan_filtering, but of course I forgot that I didn't have the full ip-route2 suite. Turns out the command I had put in rc.local was failing:
ip link set br-lan type bridge vlan_filtering 1 ageing_time 5000
so vlan_filtering was never enabled.
But I now started writing a real network config, trying to use the recend netifd support for DSA. Which does enable vlan_filtering by default on any bridge with VLANs configured. Which unfortunately seems to be broken?
I am testing with a simple setup with a tagged uplink port and a local subinterface:
The setup on the other end of the link is similar, and I can see that the packets do arrive nicely tagged there, as expected. But as soon as I enable vlan_filtering, tre trafiic stops:
root@OpenWrt:/tmp# echo 1 >/sys/class/net/br-lan/bridge/vlan_filtering
[61526.831621] rtl838x_vlan_filtering: port 8
[61526.837085] rtl838x_vlan_filtering: port 9
[61526.843048] rtl838x_vlan_filtering: port 10
[61526.849091] rtl838x_vlan_filtering: port 11
[61526.853781] rtl838x_vlan_filtering: port 12
[61526.858469] rtl838x_vlan_filtering: port 13
[61526.863160] rtl838x_vlan_filtering: port 14
[61526.867852] rtl838x_vlan_filtering: port 15
[61526.872547] rtl838x_vlan_filtering: port 24
[61526.877233] rtl838x_vlan_filtering: port 26
root@OpenWrt:/tmp# ping 192.168.99.1 -c 3
PING 192.168.99.1 (192.168.99.1): 56 data bytes
--- 192.168.99.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
Nothing at all is received at the other side after this. Known issue? Something I broke? Or real bug?
I'm making some progress: If I play around with GPIOs 33 and 37 (which are not real GPIOs, as far as I've understood) I can get the port LEDs to flicker amber.
What I don't really understand: According to the RLT838x leaked datasheet (I'm on a RTL8393M, I don't have a datasheet for that), there are only very few GPIO lines available. However, if I toggle GPIO 15 - which is not one of the GPIOs defined in the datasheet - I can measure a changed pin on the PoE board. Which pin did I actually toggle!? If I try to toggle GPIO14 - which is defined as a GPIO in the datasheet - I get an operation not permitted error - why is that?
In the meantime, I've identified the buttons and fan status inputs.
I found that I can sort of make it work by adding VIDs to a port after enabling vlan_filtering. Which I believe is a well known issue with DSA drivers (and maybe not well defined in the spec). I believe there were similar issues on mvebu and ramips. Maybe there is a solution we can copy there?
But there are probably more snags. Toggling vlan_filtering multiple times tends to end up with completely broken network, with duplicate and lost packets
@bmork kindly run me by the exact sequence to make it work please. nbd added that new bridge-vlan feature to /e/c/network and it is not working out of the box for me. I have /etc/board.d patches ready to make the unit come up by default with a master switch0 bridge using that API. however the moment i add a vlan traffic dead locks. If I know the manual steps to fix it, I can ask Felix to augment that away inside netifd.
I was also thinking that it might make sense to have the default setup, that the switches run dhcp and have a static ip on vlan 100 on port1 for mgmt purposes ?!
"GPIO"s 32-63 map to the global LED control register, see a brief description here:
For the port leds you need to enable their control via the shift registers and then try pins:
* 64-95: PORT-LED 0 (green)
* 96-127: PORT-LED 1 /yellow
* 128-159: PORT-LED 2 (red)
This was never tested on an 9x, so maybe work on the driver is needed.
There is no datasheet for the RTL839x we know of, but the suspicion is that it has considerably more actual GPIO pins than the RTL838x, not just registers, which lead to no pins like the 8x.
If it says "operation not permitted" then it should be an issue with the .dts or a driver bug, the driver thinks that GPIO is not that type.
This state works in the meaning that there are VLAN 203 tagged packets coming out of lan8, and the VLAN 203 tagged packets received on lan8 are swtitched to the CPU port and end up on the subinterface there.
Enabling vlan_filtering in this state break this connectivity as expected. But then adding VLAN 203 to port lan8, makes it work again:
A ping from the other side og the link, showing the exact moment when I added the VLAN. Note the DUP. I typically see that at this point:
From 192.168.99.1 icmp_seq=24288 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24289 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24290 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24291 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24292 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24293 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24294 Destination Host Unreachable
From 192.168.99.1 icmp_seq=24295 Destination Host Unreachable
64 bytes from 192.168.99.51: icmp_seq=24297 ttl=64 time=24.1 ms
64 bytes from 192.168.99.51: icmp_seq=24297 ttl=64 time=24.3 ms (DUP!)
64 bytes from 192.168.99.51: icmp_seq=24298 ttl=64 time=0.506 ms
64 bytes from 192.168.99.51: icmp_seq=24299 ttl=64 time=0.493 ms
64 bytes from 192.168.99.51: icmp_seq=24300 ttl=64 time=0.510 ms
64 bytes from 192.168.99.51: icmp_seq=24301 ttl=64 time=0.502 ms
64 bytes from 192.168.99.51: icmp_seq=24302 ttl=64 time=0.537 ms
So this is what I meant by "working". There are lots of issues. I have not been able to do this after netifd has configured VLANs and therefore filtering. I don't know why. And filtering on the CPU port doesn't exist at all, for good and for bad. And I cannot make this work with any untagged port.
With some luck. all of these issues are related to the same minor bug
After this I can ping the br-lan.203 interface from the other end of the lan8 link!
Which is quite surprising. VLAN 1 is not configured on the other side of the link. Ssnooping shows neither untagged nor VLAN 1 tagged packets on the link. Removing VLAN 1 again blocks VLAN 203. Adding any other VLAN to lan8 makes no difference. Adding VLAN 1 as an untagged PVID does work. The only trigger I've found enabling the port is adding a tagged VLAN 1. Which should not really be configured at all on this bridge.
I am hoping this magic behaviour is a signature bug for anyone knowing the inner workings on this driver...
In the original firmware, there seems to always be a VLAN 1 set up by default. So maybe that is the reason for that magic. I had already considered to set it up in the driver by default. But was not sure it was necessary and anyway better left to userspace.
This is easy to work around in userspace, so let's leave it there for now and see if the assumption holds.
But I still have serious trouble with "access ports", i.e a port with a single VLAN configured as PVID and untagged. I read the excellent hw design description on https://biot.com/switches/description_of_the_rtl_socs , but had one question I couldn't find the answer to: Should we create and update entries in the VLAN table when adding an untagged VLAN?
So I went looking at the SDK for an answer. And to my suprise, this is actually quite readable once you look through the auto-generated files and figure out which parts are related to this chip.
Looking at src/dal/maple/dal_maple_vlan.c I see that dal_maple_vlan_port_add() always set member_portmask, and additionally set untag_portmask for untagged VLANs., And _dal_maple_setVlan(), which writes VLAN entries to hw, will always program both the VLAN table and the UNTAG table, regardless of adding tagged or untagged VLANs. The end result is that member_portmask has all ports with that VLAN enabled, and untag_portmask will always hold the (possibly empty) subset of ports where this VLAN is untagged.
What do you think? I can try to cook up a patch and test.
I am not sending it as a PR since there still is a major issue I can't figure out: Packets are duplicated when using PVID/untagged VLANs. I see the same packet transmitted both with and without a tag on both directions when forwarding between an untagged and a tagged port. So that works because the other end only picks up the correct one. But it's still a serious bug in my eyes.
And I am not sure I understand the outer/inner thing correctly. Is this supposed to deal with q-in-q? Or is the reference to outer and inner related to something else? I am not sure how we'd configure an inner PVID, but in my testing I could not get PVID VLANs working when programming the port for both inner and outer PVIDs. Might not be correct..
As for the duplicate packets - is there some egress filtering we are missing somewhere maybe?
Looking closer, I noticed that all ports in fact already are members of VLAN 1 after hw init. So you don't need to do anything to configure it.
The reason it doesn't work after netifd has setup vlan_filtering is that netifd explitly deletes VLAN 1. Probably because the bridge code add the default PVID, which also is 1 unless overridden, to all ports. So the netifd code works around that by deleting VLAN 1 (it probably wants to delete the current default_pvid of the bridge instead of the hard coded 1, but that's an entirely different discussion). This netifd workaround breaks the rtl838x since you honour the delete, removing the port from the hw default "allports" member mask.
Anyway, all that's needed to make it work and hide some of the ugliness is to drop the member mask update for VLAN 1 on delete.
Of course, it's still ugly in the way that you can't really remove VLAN 1 from any port. But this is a common enough problem that most people probably are aware of, and just stay away from ever using VLAN 1.
LuCI runs smoothly, also because all these devices have relatively large RAM of at least 128MB, often 256MB.
So far I am not aware of a NAT performance test done. But it is also probably not the most obvious use case for such a device to put it at the border of your network. More likely it sits behind a router, the developers even discussed to make it use DHCP for the initial setup instead of the usual OpenWRT static 192.168.1.1. In any case there are still a lot of possible performance improvements in that area, all unexplored. So it is probably a bit early to give a definitive answer on this.
This shows 3 packets which are all duplicated with and without tag (VID 7). The ARP request and the ICMP echo reply enter tagged on port 8, while the ICMP echo request entered untagged on port 3.
If I read the DSA trailer correctly, then we have consistently
0x800a1000 - untagged
0x800f1000 - tagged
regardless of actual source port. Which is weird, isn't it? Is the switch doing something wrong, or are we bouncing back packets here?
Could this trailer be the trailer the 83xx driver puts in, see function rtl838x_hw_receive()?
There also an 0x80pp1000 is appended, where pp is the port number, not sure this is removed by DSA.
FYI, I just test built the archine, and it seems both complete and functional. Just needs some of the usual fixups to build in a modern world: Splitting the two combined wildcard and non-wildcard rules in the kernel Makefile, changing "bin/sh is /bin/bash" assumptions in the (pre-generated) Makefiles for the snmp app, and fixing two mknod commands in the final romfs rule so they can be called without having to build the whole thing as root
Been thinking about this. Not sure running management on a VLAN by default is a good idea. Will probably confuse most users coming from OEM firmware, where managent is setup on VLAN 1. Which is untagged PVID on every port by default.
Assuming most new users are using the OEM web GUI to convert (once that is supported), then they will already have setup management access. How about simply generating OpenWrt defaults based on that? The OEM config is readily available in the "jffs" partition (better named "JFFS2_CFG" in OEM uboot/firmware - maybe we should use their partition names too?).
And parsing just the management settings should not be too difficult:
oot@OpenWrt:/# df -hT /mnt/
Filesystem Type Size Used Available Use% Mounted on
/dev/mtdblock3 jffs2 1.0M 248.0K 776.0K 24% /mnt
root@OpenWrt:/# egrep '^manage|^ip' /mnt/startup-config
ip address 192.168.99.51 mask 255.255.255.0
management-vlan vlan 203
management access-list default
You can "borrow" the ssh host keys there too if you want to make the transition even smoother:
root@OpenWrt:/# ls -la /mnt/ssh/
drwxr-xr-x 2 root root 0 Jan 1 2019 .
drwxr-xr-x 4 root root 0 Jan 1 1970 ..
-rw-r--r-- 1 root root 493 Jan 1 2019 http_rsa_key
-rw-r--r-- 1 root root 137 Jan 1 2019 http_rsa_modulus
-rw------- 1 root root 668 Jan 1 2019 ssh_host_dsa_v2_key
-rw-r--r-- 1 root root 601 Jan 1 2019 ssh_host_dsa_v2_key.pub
-rw------- 1 root root 1679 Jan 1 2019 ssh_host_rsa_v2_key
-rw-r--r-- 1 root root 393 Jan 1 2019 ssh_host_rsa_v2_key.pub
-rw-r--r-- 1 root root 1245 Jan 1 2019 ssl_cert.pem
-rw-r--r-- 1 root root 1704 Jan 1 2019 ssl_key.pem
