Connect two devices over Raspberry-Openwrt

Klaus123 · June 29, 2023, 12:36pm

Hi.

I am working on a raspberry with openwrt.

The goal is to route one device to another. My setup is shown below:

With my configuration I can now ping from

Device2 -> Raspberry (192.168.5.3 and 192.168.5.5)
Raspberry -> Device2
But pinging Device1 from Raspberry or from Device2 is not possible.

If I plug off the black adapter, then I can ping Device1 from Raspberry and Raspberry from Device1

In openwrt I created a bridge-device and added both interfaces (black and white).

Which further information do you need to help?

eduperez · June 29, 2023, 12:48pm

I cannot see any image in your post, did you forget to add it, perhaps?

trendy · June 29, 2023, 12:49pm

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru

Klaus123 · June 29, 2023, 12:52pm


root@OpenWrt:~# ubus call system board; \
> uci export network; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
{
        "kernel": "5.10.176",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 400 Rev 1.1",
        "board_name": "raspberrypi,400",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.5",
                "revision": "r20134-5f15225c1e",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 22.03.5 r20134-5f15225c1e"
        }
}
package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdf1:e5b4:5323::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'eth0'
        option proto 'dhcp'

config device
        option name 'usb0'
        option promisc '0'
        option acceptlocal '1'

config interface 'USB0'
        option proto 'static'
        option device 'usb0'
        option ipaddr '192.168.2.2'
        option netmask '255.255.255.0'

config interface 'USBwhite'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.5.5'
        option device 'eth1'

config interface 'USBblack'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.5.3'
        option device 'eth2'

config device
        option name 'eth1'
        option ipv6 '0'
        option acceptlocal '1'

config device
        option name 'eth0'
        option ipv6 '0'

config device
        option type 'bridge'
        option name 'br-blackWhite'
        list ports 'eth1'
        list ports 'eth2'
        option bridge_empty '1'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    inet 192.168.178.75/24 brd 192.168.178.255 scope global eth0
       valid_lft forever preferred_lft forever
8: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    inet 192.168.5.3/24 brd 192.168.5.255 scope global eth2
       valid_lft forever preferred_lft forever
9: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    inet 192.168.5.5/24 brd 192.168.5.255 scope global eth1
       valid_lft forever preferred_lft forever
default via 192.168.178.1 dev eth0  src 192.168.178.75
192.168.5.0/24 dev eth2 scope link  src 192.168.5.3
192.168.5.0/24 dev eth1 scope link  src 192.168.5.5
192.168.178.0/24 dev eth0 scope link  src 192.168.178.75
broadcast 127.0.0.0 dev lo table local scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1
broadcast 192.168.5.0 dev eth2 table local scope link  src 192.168.5.3
broadcast 192.168.5.0 dev eth1 table local scope link  src 192.168.5.5
local 192.168.5.3 dev eth2 table local scope host  src 192.168.5.3
local 192.168.5.5 dev eth1 table local scope host  src 192.168.5.5
broadcast 192.168.5.255 dev eth2 table local scope link  src 192.168.5.3
broadcast 192.168.5.255 dev eth1 table local scope link  src 192.168.5.5
broadcast 192.168.178.0 dev eth0 table local scope link  src 192.168.178.75
local 192.168.178.75 dev eth0 table local scope host  src 192.168.178.75
broadcast 192.168.178.255 dev eth0 table local scope link  src 192.168.178.75
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

iplaywithtoys · June 29, 2023, 12:53pm

Then stick 'em on separate subnets. Right now they're on the same subnet.

Even if your subnet masks are /30 to put them on different subnets, then .3 and .7 are the broadcast addresses, so it still won't work.

Routing 101: every subnet must be different.

trendy · June 29, 2023, 12:58pm

config interface 'usb'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.5.5'
        option device 'br-blackWhite'

Delete interfaces USBwhite/black , and the bridge_empty option from br-blackWhite.

Klaus123 · June 29, 2023, 1:01pm

I did update the post with the image.
Thanks

iplaywithtoys · June 29, 2023, 1:08pm

To elaborate on my Routing 101 comment above, this is because of the binary XOR arithmetic that underpins subnet calculations and routing.

In a nutshell, it works like this:

Device A wants to talk to Device B
Device A looks at Device B's IP address, and performs some simple XOR arithmetic to answer the question: "Is Device B in the same subnet as me, or in a different subnet?"
If the answer is "the same subnet", then Device A spams the broadcast domain with "who has" ARP requests to get the MAC address of Device B.
If the answer is "a different subnet", then Device A sends the traffic to its default gateway for the router to send on to wherever it needs to go.

With a /24 subnet mask, 192.168.5.3 and 192.168.5.5 are in the same subnet so Device A never even tries to send the packets towards the router. It just sits there forlornly calling out into the void, "Device B? What's your MAC address?" and never gets a response.

eduperez · June 29, 2023, 8:28pm

You cannot have a device connected to two separate networks with the same address range.

Klaus123 · July 3, 2023, 8:14am

I did now reset raspi and configured two networks with different subnets

• Raspi reset with new flash of openwrt
• Set new router password

• Login ssh via Putty
• #opkg update
• #opkg install kmod-usb-net-asix-ax88179
• Now have eth1 (white) and eth2 (black)

• Add new interface “white”, static IP 192.168.5.1/24
• Add new interface “black”, static IP 192.168.7.1/24
• Save&Apply

• Ping from Device1 (192.168.5.3) to “black” 192.168.5.1 -> success
• Ping from black toDevice1 -> success
• Ping from Device2 (192.168.7.3) to “black” 192.168.7.1 -> success
• Ping from black toDevice2 -> success

Now I have two interfaces with different subnet.
Changed goal is to forward all traffic send from Device1 to Device2 (at least port 4711).
Do I need a bridge-device? Or a firewall port/nat rule?

root@OpenWrt:~# ubus call system board; \
> uci export network; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
{
        "kernel": "5.10.176",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 400 Rev 1.1",
        "board_name": "raspberrypi,400",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.5",
                "revision": "r20134-5f15225c1e",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 22.03.5 r20134-5f15225c1e"
        }
}
package network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd79:b205:3ef1::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'dhcp'

config interface 'white'
        option device 'eth1'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.5.1'

config interface 'black'
        option proto 'static'
        option device 'eth2'
        option netmask '255.255.255.0'
        option ipaddr '192.168.7.1'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    inet 192.168.178.75/24 brd 192.168.178.255 scope global br-lan
       valid_lft forever preferred_lft forever
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    inet 192.168.7.1/24 brd 192.168.7.255 scope global eth2
       valid_lft forever preferred_lft forever
6: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    inet 192.168.5.1/24 brd 192.168.5.255 scope global eth1
       valid_lft forever preferred_lft forever
default via 192.168.178.1 dev br-lan  src 192.168.178.75
192.168.5.0/24 dev eth1 scope link  src 192.168.5.1
192.168.7.0/24 dev eth2 scope link  src 192.168.7.1
192.168.178.0/24 dev br-lan scope link  src 192.168.178.75
broadcast 127.0.0.0 dev lo table local scope link  src 127.0.0.1
local 127.0.0.0/8 dev lo table local scope host  src 127.0.0.1
local 127.0.0.1 dev lo table local scope host  src 127.0.0.1
broadcast 127.255.255.255 dev lo table local scope link  src 127.0.0.1
broadcast 192.168.5.0 dev eth1 table local scope link  src 192.168.5.1
local 192.168.5.1 dev eth1 table local scope host  src 192.168.5.1
broadcast 192.168.5.255 dev eth1 table local scope link  src 192.168.5.1
broadcast 192.168.7.0 dev eth2 table local scope link  src 192.168.7.1
local 192.168.7.1 dev eth2 table local scope host  src 192.168.7.1
broadcast 192.168.7.255 dev eth2 table local scope link  src 192.168.7.1
broadcast 192.168.178.0 dev br-lan table local scope link  src 192.168.178.75
local 192.168.178.75 dev br-lan table local scope host  src 192.168.178.75
broadcast 192.168.178.255 dev br-lan table local scope link  src 192.168.178.75
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

trendy · July 3, 2023, 8:57am

You can add them both under the lan firewall zone, if it is just for proof of concept. That will allow the traffic from one interface to the other.

Klaus123 · July 3, 2023, 9:20am

Like this:?

But how can Device1 know that 192.168.7.3 is on intervace white?
I cannot ping it from Device1.
Don't I need additional a forward rule from 192.168.5.1 to 192.168.7.3?

iplaywithtoys · July 3, 2023, 9:31am

Now that you've made those changes, what are the current contents of /etc/config/network and /etc/config/firewall ?

Klaus123 · July 3, 2023, 9:38am

After trendys change i get:

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd79:b205:3ef1::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'dhcp'

config interface 'white'
        option device 'eth1'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.5.1'

config interface 'black'
        option proto 'static'
        option device 'eth2'
        option netmask '255.255.255.0'
        option ipaddr '192.168.7.1'

and

root@OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'black'
        list network 'lan'
        list network 'white'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

iplaywithtoys · July 3, 2023, 9:44am

That all looks good. At least, it looks as if the desired traffic should be permitted. So the next task is to try to find out why it appears as if it's not being permitted.

If I understand your test correctly, you're trying to ping from 192.168.5.3 to 192.168.7.3, but you're not seeing the expected response(s). Is that correct?

Does 192.168.5.3 have a default route configured, or a static route for 192.168.7.0/24?
Is 192.168.7.3 active and responding on the network?
Does 192.168.7.3 have any sort of firewall itself which drops (or rejects) ICMP packets?
Does 192.168.7.3 have a default route configured, or a static route for 192.168.5.0/24?

trendy · July 3, 2023, 10:30am

Device1 will either have default gw the .5.1 (from DHCP or static) and device2 .7.1 (again from DHCP or static).
If they don't have default gw, you'll need to add a static route to .5.0/24 via .7.1 and .7.0/24 via .5.1.

Klaus123 · July 3, 2023, 11:17am

Yes, correct.

Set it on windows PC with #route add 192.168.7.0 MASK 255.255.255.0 192.168.5.1

Yes. responding from raspi

ufw disabled on Ubuntu (Device2)

After setting route on Device1 I can now ping the eth2 if with ip 192.168.7.1 This is something new
192.168.7.3 is still not possible,

iplaywithtoys · July 3, 2023, 11:20am

Does tcpdump on 192.168.7.3 show any incoming ICMP packets (or any packets) with a source address of 192.168.5.3?

Also, does tcpdump on OpenWRT (optional package; must be installed first) show packets arriving at one interface and leaving by the other interface?

trendy · July 3, 2023, 11:32am

Windows have default firewall to block anything outside their localnet. Disable that or allow from the other network.

Klaus123 · July 3, 2023, 11:57am

tcpdump is a great tool. I have not seen before.

Result on Device2 is

Also added the route config to screen shot.
==> request comes in, but Ubuntu cannot send answer back.
So it looks like an issue inside Ubuntu