Context: I purchased a Fujitsu thin client, along with a PCIe 4-port gigabit NIC & a riser to make my own custom router. My reason for doing so is because i am interested in custom routers & my ISP is refusing to let me use port 80 & 443 on my own connection, unless i use a 3rd party router.
My connection is a 150mbps/150mbps Fiber Connection.
I am running into two main issues:
The router seems to randomly drop my NIC interfaces & in the local terminal, i can see "510.260596 ibg 0000:03:00.0: clearing RX timestamp hang". The interface seems to downed & does not come up, until the device is rebooted.
When I perform a speed test to test the router's throughput via fast.com, the test will run for a few seconds & then reboot. I have also tried pfsense. I get the same issue where the NICs drop & the speed test reboots the system.
I have tried the following so far:
Replaced NIC with a different 2-port NIC from intel, same dropout & reboot issues occur.
Replaced thermal paste on CPU/SOC. no difference.
Tried using pfsense & linux mint to see if the issue was isolated to Openwrt. I experience the reboot after performing a speed test. Sometimes, pfsense would drop the NICs.
Plugged the PCIe NIC directly into the motherboard, to eliminate the potential that the PCIe 4x riser is causing issues.
I haven't yet found a way to monitor the CPU temperature during the speed test, but using linux mint on a USB, i can see that the CPU shoots to 100% usage when doing the speed test. on some occasions where i have been testing & putting the system under load, I will sometimes experience random reboots, for example at the end of the bootup process.
Does anyone have any ideas on how to go about troubleshooting these issues? I don't mind trying to obtain logfiles, granted I will need the steps to do so, since I'm still learning the ropes of Openwrt.
I would suspect the raiser card but it could also very well be some kind of weird incompatbility between the hardware. Looking close I guess it could also be the NIC itself, going a quick search on Google suggests that it's using 82575 which appears to be well over 10y by now.
To provide some further context, i am basing my parts off of this review of the Fujitsu S920, along with whatever spare parts i have lying about (NICs) & some purchased parts i.e. PCIe riser. One difference to note is that the review has a quad-core CPU, whereas i found out i actually have a dual core CPU.
@frollic , The power brick looks to be original. The output on the brick is 19V 2.1A.
Replacing the card with a NIC sounds like a good idea I haven't tried yet. I have a spare USB NIC adapter i will try out at some point during the week. I tend to use it quite often, so i know it's a fairly reliable piece of equipment.
@diizzy , I too suspected the riser card, however taking the motherboard out the chassis & putting the NIC directly into the board yielded the same symptoms.
If the NIC is too old, I do have this D33682 NIC. I did already try it & had the same results.
Unsure if this would help, but i will link you to my cloud, where you can see in-depth images of the parts used. The D33682 NIC I have will be named as "2-Port NIC". Hopefully, that will help with any model numbers or part identification. Excuse my DIY PCIe bracket for the D33682 NIC, i didn't have the original & the bracket helps hold the NIC to the chassis with minimal movement.
@frollic The OP of the other post actually inspired me to make my current one, since i couldn't find any posts around the S920, until OP made their post. I did swap out the 4-port NIC mentioned above with another 2-port server card (D33682 NIC). One thing i don't think @geo999 experienced was the NIC randomly going down with the message "clearing RX timestamp hang". I'll try the USB NIC with the integrated NIC & let you know the result.
Indeed, on my side there was no "clearing RX timestamp hang" message, anyway any message that might have been there I could not see as the crashes where sudden with nothing observed while "logread -f".
there was no problem with the riser card/cable or while doing burn routing tests on a private network.
The crash occurred only when the ISP wire was connected.
Tested also any port combination (dual on intel + the one onboard)
Doing network speed tests will predictably determine a reboot.
In the end I replaced the intel x3859 with tp-link TG-3468 and it's all fine, no more heating, no more crashes with +900 mbit/sec up/down.
Hi Guys, I have a partial update. I hooked up a Rankie R1161 NIC to the S920 which didn't work at first. After the following commands:
opkg update
opkg install kmod-usb-net-rtl8152
to download a driver for the USB NIC, the NIC seems to work. My update is partial because I was able to do a quick speed test & obtain 95mbps down/90mbps up, however my laptop NIC is 100mbps so a limitation is being hit. I should be able to test some proper testing tomorrow. This speed test was done with the 4-port NIC still in the system, but with the WAN interface assigned to the built-in NIC & the LAN interface assigned to that USB adapter, essentially leaving the NIC with nothing to do.
The current plan will probably be to either:
Use inbuilt card + 2 or 4-port NIC that openwrt supports (suggestions accepted )
Only use PCIe NIC, as I believe some people reported issues with their built-in NICs, because they were Realtek & Realtek has some incompatibility or stability issue? I will need to check the built-in NIC in the S920 spec sheet to verify if it's using Realtek & if Realtek causes issues.
I will troubleshoot further tomorrow & advise on any interesting progress.
@geo999 thank you for your comment, I had no clue what caused the "Clearing RX timestamp" but such wasn't occurring via built-in + USB interfaces. It seems isolated to the PCIe cards, I suspect @diizzy may have a point regarding the hardware being too old which is causing compatibility issues. To be fair to @diizzy , I grabbed these NICs out of a parts box that had floppy disks with drivers for some of the parts.
It could very well be that I should purchase some recent NIC, ideally with 2 or 4 ports, I'd just need to know if there is some compatibility list with which NICs are compatible with OWRT or some NICs that the community recommends.
hey, i basically registered to answer you because i am in the same boat (even thoug i am running pfsense instead of openwrt on this machine).
I wanted to build up a cheap firewall for one of my customers and i eyed the very same review you linked in your opening post. In my homelab i have it virtualized in my mini server and generally customers who need a firewall either can virtualize it or can pay for the official hardware, in this case i had to make an exception.
I ran in the very same problem aka, after configuring everything and filling all ports (this customers has 4 WANs and 1 LAN) i set up a load balance and went to test it with a speed test. While it indeed worked after a few second the entire machine rebooted. I tried again and again it rebooted.
I banged my head for about six hours today and eventually after almost giving up ended up going back to the youtube video and scrolling through the comments:
if you scroll to Bjorn V comment (3rd or fourth from the top):
Also to mention that in the S920 there are 3 cpu versions : AMD G-Series GX415GA @1.5Ghz Quad Core cpu AMD G-Series GX222GC @2.2Ghz Dual Core cpu AMD G-Series GX424CC @2.4Ghz Quad Core cpu But the GX222GC model doesn't like a Quad port, i have tested this model with a few different Quad port cards, and with all, the ThinClient reboots when you pull traffic through it,even when browsing through the Web GUI of Pfsense and it reboots. With a Dual port card this model works fine, even at full speed. The GX415GA model like in this video works fine with Dual and Quad port cards, the GX424CC version i havn't tested yet, of this will except a Quad port card. The malfunction of a Quad port card in the GX222GC has (i guess) to do that the PCI-E slot doesn't provide enough power for the more energy hungry Quad port cards.
After reading it i indeed verified i had the 2.2 ghz version and tried searching again some more to see if someone found a workaround but i guess your topic, stating the same issue with a different firmware, kinda puts a nail in the coffin.
Either go dual nic instead of quad or get a 1.4 ghz version of the thin client.
Just wanted to give you a heads up before you end up buying extra quad nics since they're not cheap
Funny enough, my manager showed me that exact same comment on the YouTube video today.
It's a bit of an odd one, since I still get reboot issues with my 2-port NIC, then again it's more than a decade old.
I'm planning to test the USB NIC at gigabit speed. My ability to verify the comment made on the video rather limited, since I'm "borrowing" parts from my workplace, which mostly consist of EOL/obsolete parts.
I'll see if I can find any other 2-port or 1 port NICs.
Alternatively, I had one more idea, although I'm not very knowledgeable with this method & may end up damaging my system if I execute it. If you check my cloud link & navigate to the PCIe riser images, it has a 12V 4-pin connector. According to the seller, this provides extra power to the PCIe connected device, so I'm curious about if I can use the motherboard to give extra juice to the PCIe slot, let's say, that 2-port NIC that is likely not very efficient considering it's age.
I'll post any interesting updates when I get around to testing these ideas.
in my experience, if it happens once it's an unexpeceted event, twice a coincidence, the third time (the guy from youtube, me and you, that makes three of us) a known fact. In my case i need 5 gigabit lans.
My customer has a raspberry pi with openwrt at the moment with USB nics.. while the speed is consistent, since usb 3 is capable of delivering decent transfer speed, USB nics connection isn't. Sometimes they get disconnected, sometimes the order is swapped around after reboot and Wan 3 becomes Wan 1 for examble (i know there are scripts to lock to the usb ID but it's still very clunky).
On paper the thin client it's a very appealing piece of hardware because it allows you to get to almost an enterprise result for less than a consumer price (29 € for each on ebay without power supply). Power consuption is very low and it looks like an actual proper hardware too. I ordered a 1.5 ghz version as well as a 2 port intel nic to make some more tests. i don't mind throwing a few bucks at it for "science" especially if i end up with a couple of extra firewalls that i can use.
to the extent of my tests: heat can be an issue in an uncooled evironment, mine was getting pretty hot as the "rack" room of my customer wasn't cooled at all. Nothing a fan can't fix thoug, i slapped a regular CPU fan on top of it and monitored the temperature (Pfsense has inbuilt cpu temp monitor) and it never exceeded 55C. Considering we have 35-38C ambient temperature in my country this time of the year i think that's pretty good.
Regarding your "powered" riser: i am not sure, that extra power should be for GPUs and i don't know what will happen if you provide it to the intel nic. I know i tried to upscale the power supply to a 90W without any success (aka i had the same exact problem). If you end up trying let us know!
Looks like this particular S920 model can pull a gigabit through the integrated RealTek NIC & with this Rankie R1161 USB NIC. I ran this test for around 2 minutes with no crashes or reboots. I also am not getting any "Clearing RX timestamp" errors as mentioned before.
All things considered, the current theory is that my mix of my old NICs & potential power limitations, as mentioned in the you tube video could be causing the device to reboot under small loads like browsing the router GUI or starting a speed test due to potential power delivery issues.
I do not yet know how to monitor the temperature of this unit in OpenWrt, if any S920 owners have any ideas or know of any requirements e.g. drivers needed, i can try to benchmark the system & monitor the temps for the sake of having results to share.
I'll provide further updates regarding temperature & IPerf3 testing with NICs if i can find any extra 1-2 port NICs lying around & when i figure out how to monitor the temperature during the stress testing.
Hey, i have made more tests as well in the weekend.
I hooked the device with 2 USB Nics as well, but left the quad gigabit card plugged in too. I configured one of the USB as additional Lan and the other as additional WAN. While i was doing this i experienced a reboot because i was browsing the web configurator through the quad nic but no other ports were plugged so the amount of traffic through the nic must have been very minimal.
Then i connected my laptop to the USB LAN and my internet connection to the USB WAN and started doing your tests as well, including doing speedtest and a 10 hours 4K youtube video to give it persistend traffic load. Also i added a PWM 12v FAN because it was getting quite hot (80C) all while leaving the quad NIC plugged into the Pcie ports but without any Lan actually plugged.
The machine didn't crashed and performed well the whole time with a power load that must have been higher than when it was just with the quad port since it had the quad port in idle plus 2 USB nics + the PWM fan and was pulling heavy load of traffic for 10hours+
This basically confirm that the issue lies within the PCie port and it honestly start to feel like a bug more than anything else. NICs either works or don't, in my experience i have had NICs causing reboots because of some malfunction (most of the time fried) but this would happen right away, not when pulling traffic. This is easy verifiable by plugging the NIC into another board which i don't have lying around right now but i'm pretty sure it would work without issues.
I suppose the last thing left to try would be test a powered riser to see if it fixes anything but i'm not sure if this would fry the NIC honestly.
If that doesn't work, either swap for another revision like the youtube guy suggest or rely on USB NICs (bleargh!).
I don't have the tool to verify that, but if you give credit to the youtube video, without Screen and just the LAN cables connected we are talking about 12-16W power drawn. Even if a Quad port draws more power i highly doubt it would cap up the 40W output of the original power brick.
Plus i did try to use a 60W power supply and a 90W as well for what matters and nothing change.. as long as there is traffic going through the quad nic even if it is just a single client talking to the web browser gui of pfsense in my case, and OpenWRT for the OP's case, the machine will eventually reboot (more often and fast when there is more amount of traffic at the same time, while it could last several minutes or even one hour if you just browse the web gui). Again with USB NIC this doesn't happen at all and i went ahead and plugged 4 to make tests and everything has been rock solid for the entire day. I just don't like using them but it seems with this particular version it might be the only way.
also @ITXProtocol DO NOT attempt to connect a power source to the PCIe riser!! I had one like yours lying around and decided i wanted to make the test but before going through with it, my brain started to work again and i decided to test the pins on the riser with a multimeter: The power socket is meant to be an output not an input, meaning it actually gives current. So if you connect a power source there you are going to short-circuit and bllow the riser at the very minimum, possibly the NIC or motherboard as well.
There is not just the PSU to worry about, but also the voltage regulators on the mainboard itself. These devices are designed for thinclient usage, without extensibility in mind. The onboard PCIe slot is just supposed to power a WLAN card or something like that at most, but a quad-port network card may be another topic - that's why I would try a simple single-port PCIe card from another vendor (to rule out software incompatibilities) first, so e.g. a sub-10-buck Realtek one.
Seller could be completely wrong, but I'm not nearly knowledgeable enough to prove them wrong with my multimeter
I agree with your testing that when you send traffic through a quad-port, even small traffic like browsing GUI, it decides to reboot. I've decided to buy a cheap TP-Link 1-port NIC
TP-Link Gigabit PCI Express Network Adapter, 32-bit PCIe interface, Supports operating systems Windows 11/10/8.1/8/7/Vista/XP, Low-Profile Bracket(TG-3468) https://amzn.eu/d/eDiw1CT
, I believe with realtek chipset. I also bough a dual-port Intel server NIC
Gigabit PCI Express Network Card Intel E1G42ET - 82576 Chip, 1Gb NIC Ethernet LAN Card, Dual RJ45 Copper Ports for Windows Server, Linux, PC, VMware ESX https://amzn.eu/d/5HdNq5I
That has support until 2025 so this should eliminate the variable of my cards being aged.
I'll also test my aged NICs in my other PC to verify that the NICs don't have some kind of issue.
)
