Starlink vs 1G fiber

Installing and Using OpenWrt
1

TLDR;
Download through OpenWRT to Starlink: fast
Download through OpenWRT to Fiber: slow
Download through OpenWRT to Fiber before speedtest starts: slow
Download through OpenWRT to Fiber while running a speedtest: fast
Download through OpenWRT to Fiber after speedtest is finished: slow
Download direct to Fiber: fast

I've been using OpenWRT for a long time, on a lot of different devices - starting with a WRT54G. It's a great platform.

My most recent router is a generic Intel(R) Celeron(R) CPU J1900 @ 1.99GHz job you can get on Amazon for about $100. After maxing the ram at 8GB and installing an SSD, this little dual-nic computer does great with OpenWRT 23.05.0. Here's an LSPCI:

00:00.0 Host bridge: Intel Corporation Atom Processor Z36xxx/Z37xxx Series SoC Transaction Register (rev 0e)
00:02.0 VGA compatible controller: Intel Corporation Atom Processor Z36xxx/Z37xxx Series Graphics & Display (rev 0e)
00:13.0 SATA controller: Intel Corporation Atom Processor E3800 Series SATA AHCI Controller (rev 0e)
00:14.0 USB controller: Intel Corporation Atom Processor Z36xxx/Z37xxx, Celeron N2000 Series USB xHCI (rev 0e)
00:1a.0 Encryption controller: Intel Corporation Atom Processor Z36xxx/Z37xxx Series Trusted Execution Engine (rev 0e)
00:1b.0 Audio device: Intel Corporation Atom Processor Z36xxx/Z37xxx Series High Definition Audio Controller (rev 0e)
00:1c.0 PCI bridge: Intel Corporation Atom Processor E3800 Series PCI Express Root Port 1 (rev 0e)
00:1c.1 PCI bridge: Intel Corporation Atom Processor E3800 Series PCI Express Root Port 2 (rev 0e)
00:1c.2 PCI bridge: Intel Corporation Atom Processor E3800 Series PCI Express Root Port 3 (rev 0e)
00:1c.3 PCI bridge: Intel Corporation Atom Processor E3800 Series PCI Express Root Port 4 (rev 0e)
00:1d.0 USB controller: Intel Corporation Atom Processor Z36xxx/Z37xxx Series USB EHCI (rev 0e)
00:1f.0 ISA bridge: Intel Corporation Atom Processor Z36xxx/Z37xxx Series Power Control Unit (rev 0e)
00:1f.3 SMBus: Intel Corporation Atom Processor E3800/CE2700 Series SMBus Controller (rev 0e)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)

I've been using Starlink for about a year, and it's OK when it's OK. Storms and high winds knock it out for brief periods, but overall I'm happy to have something fast in my rural area (160Mbps/13Mbps). Recently, Spectrum has gained access to the power poles in my area, and 1G fiber became an option...so I ordered it.

Post install, tests against the fiber were fine. I don't like anybody messing with my cabinet - so I ran ethernet from my laptop directly to the fiber CPE and tests were like 930 Mbps up and down. It all seemed fine, and I ran ethernet to my OpenWRT router. This is where it gets weird.

Speedtest through the router to Starlink - 160Mbps/13Mbps
Speedtest through the router to Fiber - 930Mbps/930Mbps

Download speeds through OpenWRT/starlink (large AI models from huggingface.com) - 80Mbps
Download speeds through OpenWRT/fiber (same content) - 400Kbps <--- that's kilobits, not megabits
Download speeds through OpenWRT/fiber while running a speedtest - 500Mbps
Download speeds through OpenWRT/fiber AFTER running a speedtest - 400Kbps

The second the speedtest starts, my download speeds up, when the speedtest is finished, my download slows back to 400Kbps.

Starlink connection/downloads behave as you might expect - If I'm downloading and I start a speedtest - my download speed drops until the speedtest is finished.

If I connect my laptop directly to the fiber CPE, I download the large files at blazing speed, but when it goes through the router it gets cut down to nothing unless a speedtest is running.

I have a video showing this speedup-during-speedtest behavior, which took 2 hours for the fiber tech support mouthbreather to understand.

iperf3 test through starlink:

iperf3 -c ash.speedtest.clouvider.net -p 5200-5209 -u -w 64K -p 5201 -b 1000M
Connecting to host ash.speedtest.clouvider.net, port 5201
[ 5] local ... port 43107 connected to 45.250.25.133 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 108 MBytes 904 Mbits/sec 78075
[ 5] 1.00-2.00 sec 106 MBytes 888 Mbits/sec 76672
[ 5] 2.00-3.00 sec 105 MBytes 885 Mbits/sec 76368
[ 5] 3.00-4.00 sec 111 MBytes 928 Mbits/sec 80141
[ 5] 4.00-5.00 sec 110 MBytes 922 Mbits/sec 79603
[ 5] 5.00-6.00 sec 110 MBytes 920 Mbits/sec 79387
[ 5] 6.00-7.00 sec 108 MBytes 906 Mbits/sec 78241
[ 5] 7.00-8.00 sec 105 MBytes 878 Mbits/sec 75769
[ 5] 8.00-9.00 sec 107 MBytes 897 Mbits/sec 77457
[ 5] 9.00-10.00 sec 109 MBytes 918 Mbits/sec 79232

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.05 GBytes 905 Mbits/sec 0.000 ms 0/780945 (0%) sender
[ 5] 0.00-10.21 sec 26.4 MBytes 21.7 Mbits/sec 0.031 ms 0/779602 (0%) receiver

iperf Done.

iperf3 test through fiber:

iperf3 -c ash.speedtest.clouvider.net -p 5200-5209 -u -w 64K -p 5201 -b 1000M
Connecting to host ash.speedtest.clouvider.net, port 5201
[ 5] local ... port 46404 connected to 45.250.25.133 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 111 MBytes 928 Mbits/sec 80098
[ 5] 1.00-2.00 sec 111 MBytes 928 Mbits/sec 80128
[ 5] 2.00-3.00 sec 108 MBytes 907 Mbits/sec 78291
[ 5] 3.00-4.00 sec 111 MBytes 928 Mbits/sec 80099
[ 5] 4.00-5.00 sec 107 MBytes 897 Mbits/sec 77470
[ 5] 5.00-6.00 sec 111 MBytes 935 Mbits/sec 80743
[ 5] 6.00-7.00 sec 110 MBytes 926 Mbits/sec 79941
[ 5] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 81175
[ 5] 8.00-9.00 sec 110 MBytes 919 Mbits/sec 79317
[ 5] 9.00-10.00 sec 107 MBytes 900 Mbits/sec 77688

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.07 GBytes 921 Mbits/sec 0.000 ms 0/794950 (0%) sender
[ 5] 0.00-10.17 sec 593 MBytes 489 Mbits/sec 0.021 ms 0/429546 (0%) receiver

iperf Done.

What is it about a single file download that slows to a crawl on fiber while remaining fast on starlink? It boggles the mind!

2

Please also try different (to rule out specific peering issues) download sources (e.g. windows ISO from Microsoft's servers). If that shows the same behaviour, talk to your ISP, based on your description they're prioritizing known speedtest servers (quite a few do), while normal traffic starves out.

I am using an j1900 based router with OpenWrt myself, it has no problem at all routing 1 GBit/s (plain DHCP, can't test PPPoE, which would be more demanding), just sqm/cake would bring it to its knees (around 830 MBit/s max, but even then you wouldn't get a behaviour like that). But given that you have quite capable x86_64 hardware, you could also cross-check the behaviour with opnsense instead of OpenWrt on the router (it doesn't like running from a USB stick (very slow), but for a quick test that should be sufficient nevertheless). I do not expect any difference in behaviour, but the second test would bring some peace of mind.

Given that you have the ability (hardware), please also check what happens when you download the same file on the router (wget -O/dev/null URL), instead of through it from another system (ISP playing TTL games?).

Another test would be temporarily attaching router and notebook with short, known-good, patch cables to the cabinet just to rule out cabling issues.

(no need to do the full tests each time, just downloading a single large file from a source with known good connectivity).

Based on the description, I do blame the ISP, neither the hardware nor OpenWrt, these test suggestions just go the extra mile to rule out anything on your side.

EDIT: just to increase your own conscience (before talking to the ISP), I'd also just test the limits of your OpenWrt j1900 by putting its wan on the lan side of your starlink router (double-NAT and all, simple default config) and check how fast you can download through it in that scenario. Next check would be putting a computer next to the j1900 wan on the Starlink lan side and another on the j1900 lan side, to do some iperf3 and/ or file transfer tests through it (this will really check the physical limits of your j1900, I expect the full 1 GBit/s wirespeed here).

1 Like
3

What is the "router" in the above test? Also, is your fiber CPE a simple ONT (just bridge) or an HGU (router/NAT)? I mean, are you doing double-NAT by connecting your OpenWrt J1900 router to the ISP CPE? Or is your ISP fiber configured as bridge?

I'm asking this just to double-check if I have the right picture before I suggest anything...

4

Packet capture time... in both conditions then comparison of the traces, looking for DSCPs, ECN/CE and dupACKs and Co.....

2 Likes
5

router is my j1900 openwrt x86 router

to starlink is double nat
to fiber is to fiber bridge (public IP)

1 Like
6

I actually have a bunch of these J1900's that I use in my lab for testing software - so setting up an opnsense on identical hardware isn't a problem.