Debugging OpenWrt

[diversity]

Is there some way to debug or dive deeper into the inner workings of OpenWrt? Other than getting the source code of course

I am getting inconsistent behavior and am starting to believe it is personal between OpenWrt and I

[Degenerate]

You would have to tell us what your problem is, then maybe we can tell you what to look at. "Inconsistent behaviour" is not a very enlightening description.

[IanC]

It depends that the problem is. If needed you can hook a remote debug session up with GDB for running binaries that are managing traffic for example, but there's plenty to try before that.

[diversity]

Recently I came to the conclusion that I need to use
'@lan' interface device type

[Solved] Devices on interface not reachable

I have started from scratch by deleting all interfaces, other than lan, wan, wan6, and firewall zones first. Then I added a static ip 10.2.0.1/16 interface called 'virtualization' of device type eth0 and applied the lan firewall zone to it while creating the interface. The devices on the virtualization network segment have gateway 10.2.0.1 and netmask 255.255.0.0 One example is of a device ip is 10.2.1.1 I am unable to reach any of the devices on that network segment while having gotten an i…

but then I ran into what I have been fearing

Interfaces with device '@lan' have access to wan

even though they have their own firewall zone quite recently we solved setting up interfaces and came to the conclusion that it is best to use device '@lan' I am doing the same now for tmp_old_lan interface, in an effort moving IoT devices from a previous network scheme over to the network scheme I am setting up now. the problem is is that setting up such an interface looks like it imidiatly enables devices on that network segment to phone home. uci export network; uci export firewall pa…

So now I am totally clueless about how to proceed. I am getting desperate

[diversity]

I have now the perfect situation setup. An OpenWrt VM with snapshots step by step.

Currently I am looking to create a visual network diagram of what I'd like to setup

[diversity]

On second thought I think debugging is simply too hardcore and potentially not even needed.

All I need to be able to do is following/trace a packet as it leaves a client, and see if it arrives at OpenWrt. And if it does see why it is not getting the love it needs. Perhaps a firewall log.

No need for a diagram as what I am asking for is so generic that is unrelated to a specific network diagram

[vgaetera]

You can simply use tcpdump or insert custom iptables rules for logging.

[diversity]

@vgaetera

You can simply

Thank you brother.

However, please understand that I am not only a newcommer to OpenWrt but to networking In genersl also.

How is one to use a tcp dump (and from the client or the router? )?

No disrespect meant, I'll take what I can get from here and am thankful for it.

But for newcomers it still means a tremendous amount of digging into wiki's that might not always be relevant anymore.

I can help update documentation to not confuse newcomers once I can reproduce my intended setup.

I.o.w. could you please lay out some still relevant resources?

[vgaetera]

You can install and run it like this to capture ping or traceroute:

opkg update
opkg install tcpdump
tcpdump -evni any icmp

[diversity]

I forgot to mention your valid point. It can now seem that I came to the conclusion by myself which is not the case. Your 'plenty to try' statement helped me slow down a notch or two

[diversity]

heheh It goes to show that networking is not for the faint of heart.

So I dove into this tcp -evni any icmp and immediately ran into things again.

14:33:13.522307 Out {max address redacted} ethertype IPv4 (0x0800), length 104: (tos 0xc0, ttl 64, id 12019, offset 0, flags [none], proto ICMP (1), length 88)
    10.2.0.1 > 10.2.1.3: ICMP host 10.3.0.226 unreachable, length 68
	(tos 0x0, ttl 63, id 14897, offset 0, flags [DF], proto TCP (6), length 60)
    10.2.1.3.58002 > 10.3.0.226.139: Flags [S], cksum 0x4371 (correct), seq 1970945678, win 64240, options [mss 1460,sackOK,TS val 2193218004 ecr 0,nop,wscale 7], length 0

Forgive me for not understanding it directly, and yes I did scan through provided links and search results regarding tcpdump.

But does this mean my virtualization interface (10.2.0.1) is talking to one of my hypervisors (10.2.1.3)
"10.2.0.1 > 10.2.1.3: ICMP host 10.3.0.226 unreachable"
And there is no 10.3.0.226 client known to me. It does not show up in the OpenWrt ARP table and, as far as I can tell, has never gotten a DHCP lease.

Also what does the extra address segment (58002) after the first 4 mean?
"10.2.1.3.58002" == hypervisor 3.?????

[diversity]

as a graphical oriented user I am not comfortable yet with a CLI. I tend to mistype a lot and most often simply just do not remember the commands

in a GUI I can more or less always find my way back again even many years not having seen a specific thing I am after.

Hence is there anyway I can ssh into OpenWrt and route the tcpdump output to a wireshark process running on the same client that the ssh session to OpenWrt was instantiated on?

[iplaywithtoys]

@diversity

Also what does the extra address segment (58002) after the first 4 mean?

It's the source (or destination) port for the traffic. In the excerpt you included, it's the source port:

@diversity

10.2.1.3.58002 > 10.3.0.226.139: Flags [S], cksum 0x4371 (correct), seq 1970945678, win 64240, options [mss 1460,sackOK,TS val 2193218004 ecr 0,nop,wscale 7], length 0

That is a record of traffic originating from 10.2.1.3 on port 58002, going to 10.3.0.226 on port 139. The source port is what's known as "ephemeral" and is unlikely to be consistent. The destination port of 139 implies that 10.2.1.3 is attempting to connect to an SMB/CIFS/Microsoft file sharing service hosted by 10.3.0.226.

Now, that doesn't necessarily mean that 10.3.0.226 is actually hosting SMB/CIFS. Windows is very chatty and attempts to query port 139 whether or not the target system actually has anything to offer on that port.

If 10.3.0.226 did have something to offer on port 139, then return traffic would look like this:

10.3.0.226.139 > 10.2.1.3.58002: ...

For the return traffic, 139 would be the source port and 58002 would be the destination port.

@diversity

[...] is there anyway I can ssh into OpenWrt and route the tcpdump output to a wireshark process running on the same client that the ssh session to OpenWrt was instantiated on?

I don't know; I've never tried redirecting tcpdump output interactively. However, you can write the tcpdump output to a file (use the "-w" option) and then import that file into Wireshark for offline analysis.

[diversity]

@iplaywithtoys

That is a record of traffic originating from 10.2.1.3 on port 58002, going to 10.3.0.226 on port 139. The source port is what's known as "ephemeral" and is unlikely to be consistent. The destination port of 139 implies that 10.2.1.3 is attempting to connect to an SMB/CIFS/Microsoft file sharing service hosted by 10.3.0.226

Ohh how I have a lot to learn
Thank you for this info
Yes my hypervisor has a broken link to an old windows share.
I'll get rid of it right now.

[vgaetera]

@diversity

I tend to mistype a lot and most often simply just do not remember the commands

It's a common problem.
Many people including myself use a personal Linux command cheat sheet.

@diversity

ssh into OpenWrt and route the tcpdump output to a wireshark process

The article explains exactly this:
https://openwrt.org/docs/guide-user/firewall/misc/tcpdump_wireshark

[IanC]

@diversity

Recently I came to the conclusion that I need to use
'@lan' interface device type

Can you remind us what you're actually trying to figure out / fix ?

Was it traffic appearing where you didn't expect it to, or not appearing where you did expect it to? Based on MAC, VLAN, port, IP filtering / routing? Or something else?

I think there was a suggestion above to use firewall rules. Setting IP address with ACCEPT should allow you to see it landing and going on its way for example. Not suitable?

EDIT: Found it...

@diversity

All I need to be able to do is following/trace a packet as it leaves a client, and see if it arrives at OpenWrt. And if it does see why it is not getting the love it needs. Perhaps a firewall log.

[diversity]

When looking at the ARP table I see a client listed with ip 10.1.1.1.

 arp
IP address       HW type     Flags       HW address            Mask     Device
{additional lines redacted}
10.1.1.1         0x1         0x2         {mac address redacted}     *        br-lan

I also have an interface 10.1.0.254/16 with the lan firewall zone.

When I ssh into openwrt and ping 10.1.1.1 it hangs indefinitely.

I can reach clients on the virt interface just fine.

funny thing is is that when I manually set my laptop ip address to 10.1.11.11/16 gateway 10.1.0.254
I can reach 10.1.1.1 (and ping it from the laptop) again and all clients in the virt interface. I can also still reach WAN but. I can no longer reach Openwrt via it's name (openwrt.lan) I can now only reach it when using and interface ip address e.g. 10.1.0.254 or 10.0.0.254

uci export network
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd8a:16fa:26f3::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.0.0'
	option ipaddr '10.0.0.254'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'

config interface 'ipmi'
	option proto 'static'
	option ipaddr '10.1.0.254'
	option netmask '255.255.0.0'
	option device 'br-lan'

config interface 'virt'
	option proto 'static'
	option device 'br-lan'
	option ipaddr '10.2.0.254'
	option netmask '255.255.0.0'

uci export firewall
package firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'
	list network 'ipmi'
	list network 'virt'
	option log '1'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled 'false'

config include
	option path '/etc/firewall.user'

I have enabled logging on the LAN firewall zone but for the life of me I can't find where I can actually view that log. Nothing firewall related shows up in the system log and also not in the kernel log.

Where can I find this log or dropped packets to 10.1.1.1? I am assuming they are dropped because it they would have been rejected I would have seen 'destination unreachable' while pinging right?

Any other steps I can take to figure out what's going on?

[diversity]

btw i did manage to pipe the tcpdump to wireshark. I must admit I dont yet have the experience to use it to trace a route and to see where things go wrong. I might revisit this in the future

[IanC]

You're working at a level that I don't understand well, so would be finding my way a bit as well if I was doing what you are doing.

As I mentioned above, I'd put a rule into the firewall for the specific IP address to see if it is being hit.

[IanC]

What is the 10.1.1.1 device and how is its address assigned. For example is it a static address assigned to a device on RJ45 port? Or is it behind some other network device like a hypervisor virtual switch?

EDIT:

@diversity

funny thing is is that when I manually set my laptop ip address to 10.1.11.11/16 gateway 10.1.0.254
I can reach 10.1.1.1 (and ping it from the laptop) again

Have you looked at the routing tables on each device, and compared them for good / bad if you can get different behaviour when changing some other thing?