Slow upload speed with RPI 3B+ custom setup

RPI 3B+ running OpenWrt 18.06.8, r7989-82fbd85747

I am trying to get all the traffic of my home devices routed through a RPI.
I have done the following setup:

  1. connected the RPI to the LAN port of my ISP router.
  2. once I got an IP from the ISP router, I configured the RPI to use the same IP as a static IP.
  3. I disabled the DHCP server of my ISP router. the DHCP server of the RPI is still functioning.
  4. I rebooted my ISP router.

So, essentially for all the devices who used to connect to my ISP router, will now get an IP from the RPI.
the RPI during DHCP, will provide its own IP as the Default gateway and the Default DNS server to the devices.

Now, I can see all the OUTGOING traffic on the RPI.
I can't however see the INCOMING traffic.
is there a way to route the INCOMING traffic from the ISP router as well through the RPI?

I also did the speed tests with the above setup, which showed that the upload speeds are really poor when compared to the speeds when no such RPI setup exists.
the download speeds however are similar.

what could be wrong in my setup?

any help on this topic is appreciated.
thanks a lot!

What does this mean?

Can it go anywhere else?

Start by upgrading to a newer release.

So a lot more information is needed for us to help you.

First, can you draw a diagram of your network topology? This is necessary so that we can see how things are physically connected.

Next, how are you running the speed tests (wired or wireless, a website or iperf or something else)? What speeds are you actually getting with and without the Pi? What are you expecting to get?

What is the purpose of the Pi in your network? Is it doing something special, or just simply that you would rather be using an OpenWrt based router? You have a router from your ISP -- can you remove it and use your Pi in place of the ISP router?

There are still lots of other details that might be relevant, but we first need to get these questions answered to then be able to provide guidance or even guess what else might be going on.

The main router doesn't know the Pi is there, it will send packets returning from the Internet directly to the endpoint PC.

This setup is commonly used when you want a "whole house" VPN client that is a separate box from the main router. However there is nothing to stop a LAN PC from ignoring the DHCP advertised gateway and going directly to the main router instead.

In order to actually, reliably, securely intercept all traffic you need two networks, and two interfaces in the Pi.

Thanks a lot @psherman !


attaching a diagram of the network topology.

I am running the speed tests using the RUN SPEED TEST Button, I get from google on the Chrome when I browse "Speed test", its usually the first result.
its based on Measurement Lab
it looks something like that:
image

Here are the results without having the RPI in the setup:

Download(Mbps) Upload(Mbps)
121.7 49.8
129.1 102.8
115.5 89.2
![image 224x106](upload://d2OpquLnQwQJe1yHa1SZ6IDTS3Z.png)

and here are the results with the RPI setup:

Download(Mbps) Upload(Mbps)
109.1 4.70
114.1 6.30
104.5 6.03
112.3 6.14
![image 224x130](upload://JACfLE0OiVEvb7aOUM9S53BRhZ.png)

Clearly, the Upload speeds are taking a hit.

Coming to the purpose of Pi:
I am trying to build a custom firewall that can see the traffic from all the devices.
the plan is to build the Pi so that, it can be plugged to any ISP router. The devices in the network can continue to connect to the same ISP router SSID over Wifi, even after the introduction of Pi.

i hope this answers your questions.

Thanks a lot @mk24
You are absolutely right in saying that:

However the low Upload speeds are not making sense. The packets going from the devices, have just 1 additional HOP to travel more. But the speed reduction is quite dramatic.
here is the speed comparison with and without this setup:
image

When you say two networks, do you mean, the Pi should create a subnet from the ISP router and then the devices connect to an SSID broadcasted by the Pi?

I also wanted to check the speed and the duplex settings on the connected interface of the Pi (as all the network data passes through that).
can anyone please tell me if there is a command in openwrt to do this?
Also, is it possible for me to configure/change the default values of this configuration?

ethtool does this.

This is an unusual topology, and I'm not entirely certain that this is going to do what you want.

What changes did you make on your main router? Is wifi active on that router? What about the Pi -- do you have wifi enabled there, too? Are your client devices connecting to wifi on the router or on the Pi?

From the Pi...
Please copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

@psherman thanks for your response!
the only change to the main router is Disabling the DHCP Server running on it. (Both for IPV4 and for IPV6).

The Pi doesn't have any wifi capabilities.

The client devices continue to the connect to the Wifi on the main router.

The only thing that happens now is that the DHCP requests from the client devices will now be answered by the DHCP server running on the Pi.

Upon receiving requests, the DHCP server on the Pi configures the IP, the Default Gateway and DNS Server for those devices. All of these values will be set to the static IP of the Pi itself.

This way, the packets from the devices will first arrive at the Pi and then get routed to the main router and from there to the internet.

Here are the details you asked from the Pi:

root@NAGRAScout:~# cat /etc/config/network

config interface 'loopback'
       option ifname 'lo'
       option proto 'static'
       option ipaddr '127.0.0.1'
       option netmask '255.0.0.0'

config globals 'globals'
       option ula_prefix 'auto'

config interface 'lan'
       option type 'bridge'
       option ifname 'eth0'
       option proto 'static'
       option ipaddr '192.168.1.234'
       option netmask '255.255.255.0'
       option ip6assign '60'
       option gateway '192.168.1.1'
       option dns '192.168.1.1'

root@NAGRAScout:~# cat /etc/config/wireless
cat: can't open '/etc/config/wireless': No such file or directory

root@NAGRAScout:~# cat /etc/config/dhcp

config dnsmasq
       option domainneeded '1'
       option boguspriv '1'
       option filterwin2k '0'
       option localise_queries '1'
       option rebind_protection '1'
       option rebind_localhost '1'
       option local '/lan/'
       option domain 'lan'
       option expandhosts '1'
       option nonegcache '0'
       option authoritative '1'
       option readethers '1'
       option leasefile '/tmp/dhcp.leases'
       option resolvfile '/tmp/resolv.conf.auto'
       option nonwildcard '1'
       option localservice '1'
       list server '127.0.0.1#5053'
       option noresolv '1'
       option doh_backup_noresolv '-1'
       list doh_backup_server '127.0.0.1#5053'

config dhcp 'lan'
       option interface 'lan'
       option start '100'
       option limit '150'
       option leasetime '12h'
       option dhcpv6 'server'
       option ra 'server'
       option force '1'
       option ra_management '1'

config dhcp 'wan'
       option interface 'wan'
       option ignore '1'

config odhcpd 'odhcpd'
       option maindhcp '0'
       option leasefile '/tmp/hosts/odhcpd'
       option leasetrigger '/usr/sbin/odhcpd-update'
       option loglevel '4'

root@NAGRAScout:~# cat /etc/config/firewall
config defaults
       option syn_flood        1
       option input            ACCEPT
       option output           ACCEPT
       option forward          REJECT
# Uncomment this line to disable ipv6 rules
#       option disable_ipv6     1

config zone
       option name             lan
       list   network          'lan'
       option input            ACCEPT
       option output           ACCEPT
       option forward          ACCEPT

config zone
       option name             wan
       list   network          'wan'
       list   network          'wan6'
       option input            REJECT
       option output           ACCEPT
       option forward          REJECT
       option masq             1
       option mtu_fix          1

config forwarding
       option src              lan
       option dest             wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
       option name             Allow-DHCP-Renew
       option src              wan
       option proto            udp
       option dest_port        68
       option target           ACCEPT
       option family           ipv4

# Allow IPv4 ping
config rule
       option name             Allow-Ping
       option src              wan
       option proto            icmp
       option icmp_type        echo-request
       option family           ipv4
       option target           ACCEPT

config rule
       option name             Allow-IGMP
       option src              wan
       option proto            igmp
       option family           ipv4
       option target           ACCEPT

# Allow DHCPv6 replies
# see https://dev.openwrt.org/ticket/10381
config rule
       option name             Allow-DHCPv6
       option src              wan
       option proto            udp
       option src_ip           fc00::/6
       option dest_ip          fc00::/6
       option dest_port        546
       option family           ipv6
       option target           ACCEPT

config rule
       option name             Allow-MLD
       option src              wan
       option proto            icmp
       option src_ip           fe80::/10
       list icmp_type          '130/0'
       list icmp_type          '131/0'
       list icmp_type          '132/0'
       list icmp_type          '143/0'
       option family           ipv6
       option target           ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
       option name             Allow-ICMPv6-Input
       option src              wan
       option proto    icmp
       list icmp_type          echo-request
       list icmp_type          echo-reply
       list icmp_type          destination-unreac
       list icmp_type          packet-too-big
       list icmp_type          time-exceeded
       list icmp_type          bad-header
       list icmp_type          unknown-header-typ
       list icmp_type          router-solicitatio
       list icmp_type          neighbour-solicita
       list icmp_type          router-advertiseme
       list icmp_type          neighbour-advertis
       option limit            1000/sec
       option family           ipv6
       option target           ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
       option name             Allow-ICMPv6-Forwa
       option src              wan
       option dest             *
       option proto            icmp
       list icmp_type          echo-request
       list icmp_type          echo-reply
       list icmp_type          destination-unreac
       list icmp_type          packet-too-big
       list icmp_type          time-exceeded
       list icmp_type          bad-header
       list icmp_type          unknown-header-typ
       option limit            1000/sec
       option family           ipv6
       option target           ACCEPT

config rule
       option name             Allow-IPSec-ESP
       option src              wan
       option dest             lan
       option proto            esp
       option target           ACCEPT

config rule
       option name             Allow-ISAKMP
       option src              wan
       option dest             lan
       option dest_port        500
       option proto            udp
       option target           ACCEPT

# include a file with users custom iptables rules
config include
       option path /etc/firewall.user
### EXAMPLE CONFIG SECTIONS
# do not allow a specific ip to access wan
#config rule
#       option src              lan
#       option src_ip   192.168.45.2
#       option dest             wan
#       option proto    tcp
#       option target   REJECT

# block a specific mac on wan
#config rule
#       option dest             wan
#       option src_mac  00:11:22:33:44:66
#       option target   REJECT

# block incoming ICMP traffic on a zone
#config rule
#       option src              lan
#       option proto    ICMP
#       option target   DROP

# port redirect port coming in on wan to lan
#config redirect
#       option src                      wan
#       option src_dport        80
#       option dest                     lan
#       option dest_ip          192.168.16.235
#       option dest_port        80
#       option proto            tcp

# port redirect of remapped ssh port (22001) on wan
#config redirect
#       option src              wan
#       option src_dport        22001
#       option dest             lan
#       option dest_port        22
#       option proto            tcp

### FULL CONFIG SECTIONS
#config rule
#       option src              lan
#       option src_ip   192.168.45.2
#       option src_mac  00:11:22:33:44:55
#       option src_port 80
#       option dest             wan
#       option dest_ip  194.25.2.129
#       option dest_port        120
#       option proto    tcp
#       option target   REJECT

#config redirect
#       option src              lan
#       option src_ip   192.168.45.2
#       option src_mac  00:11:22:33:44:55
#       option src_port         1024
#       option src_dport        80
#       option dest_ip  194.25.2.129
#       option dest_port        120
#       option proto    tcp

Most of these files are with the default values.

I don't really see how this is doing anything productive on your network. None of the data is passing through the firewall because you only have a single zone active (the lan).

@psherman thanks again.

the firewall will be setup later, but the Slow upload speed is something that's bothering me right now.
Do you think any of these configurations can be a reason for that?

Is it possible that because its a Pi (model 3B+) and doesn't have enough processing power, the upload speeds are low?

Is it possible that the network interface is not configured with right parameters?

I have these outputs from ethtool

root@NAGRAScout:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 1
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: pumbag
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
root@NAGRAScout:~# ethtool br-lan
Settings for br-lan:
        Link detected: yes
root@NAGRAScout:~#

Maybe, but the processor isn't really doing much here, from what I can tell.

The 3b+ has a gigabit physical connection, but the bandwidth is actually limited to ~330Mbps because it is connected to a USB 2.0 bus (max theoretical speed of USB 2.0 is 480Mbps). (see this)

I wouldn't even bother with this setup... I don't think it is going to do what you want with the topology you are proposing/using.

Don't just take my word for it... try to configure the firewall to block a specific site or service. For example, if you try to block 8.8.8.8 (google DNS) and then ping it from one of your client computers, you'll find that you probably can't block it based on this topology.

In order to achieve your goals, you would need to have a significantly different configuration. You could look at a bridge firewall, but I don't think this will work in your current setup. You can also simply cascade this device with your other router such that all clients connect through the Pi (or another router) in standard router mode.

This isn't really going to work the way you expect, AFAICT. At the very minimum, you'd need to be able to do some configuration on the ISP router. But really, this requires a situation where the clients connect first to the secondary device (Pi or otherwise) with that secondary device working as a normal NAT router.

1 Like

@psherman thanks a lot Again.

I do appreciate the concerns you are highlighting.
I will try the firewall configuration and see if it really does work.

can you please elaborate a little more on this?
Are you trying to say that the Pi acts here as a standard Router and the devices connect to an SSID advertised by it?

I am saying that if you configure the Pi as a standard NAT router, you could achieve something of what you are trying to do. I wouldn't recommend the Pi 3B+ for this task (a Pi 4 would be better), and I would strongly advise against using the built-in Wifi on any of the Pi devices because it isn't going to produce good results... but that said, here is the theory:

  • eth0 on the pi (built-in ethernet) connects to the upstream/ISP router.
  • eth0 needs to be configured to be the wan interface (typically this would be DHCP client or static IP).
  • the wan interface is associated with the wan zone in the firewall.
  • the wan zone is usually configured with masquerading enabled
  • eth1 (an external USB ethernet adapter connected to one of the USB ports) and/or wifi on the Pi would then be configured as the lan. This will typically be static IP with a DHCP server enabled)
  • devices connect to eth1 or wifi from the Pi.
  • all lan traffic must pass through the routing engine/firewall in order to reach the wan (and therefore the upstream network/internet).

That is how a standard router operates. The firewall can filter the data because it is passing between two independent networks.

1 Like