How to connect two devices' serial ports together as backup console access for each other?

I have two OpenWrt devices. Each have a TTL serial port (only one). By default in OpenWrt, the serial port is used as console.

I want to connect them together RX<>TX, so that each device can be used as backup console access for the other. In case one of them is no longer accessible via SSH, I want to be able to login via serial from the other device.

Is this possible?
How do I "disconnect" the local console from the serial port, so I can start a microcom / minicom on it?
How do I prevent kernel messages from one device to be interpreted as commands on the other device?

Thanks.

1 Like

I don't think it's possible this way.
The much easier way would be, if you connect both devices via a simple toggle or rotary switch with all 3 connections (RX/TX/Gnd) to your TTL-USB-converter.

In that case you wouldn't have the problem you described below:

4 Likes

tl;dr; it's not possible.

By default, your devices tend to be configured to output bootloader- and kernel messages on their UART connection and take input from there. While you can disable the kernel messages (if you build OpenWrt yourself and change it accordingly), you generally can't disable the bootloader's input/ output ending up on the UART (and if you want any recovery means, you really shouldn't look that way anyways). In other words, you can't prevent both devices talking over each other and potentially doing serious harm to each other, with the random chatter happening between them. And having to recover usually doesn't come with pre-announcements, allowing you to reverse the roles between your devices in advance.

What you could do, would be adding a usb2serial to both devices (if they sport a USB port) and connect that one to the native UART of the other one. If you're not in a hurry, you can order them from the slow boat for ~1.5 bucks each (with quick/ local shipping they tend to be around a fiver) - and the beauty of them is that there's no risk of any log messages accidentally being sent by them - they're passive until you want to use them. This procedure might not work on all routers (assuming they have USB ports in the first place) and with all usb2serial adapters, as some might backfeed just enough power (tx) into the SOC to get it into a weird state preventing it from initializing/ booting properly - so you might have to test and tinker a little to get it working reliably.

esp8266/ esp32 solutions with bluetooth or wireless support exist as well.

4 Likes

I want remote access without any manual intervention.

I don't recall how. But it can be done - only one could backup another though.

One of the serials would have to be converted from a console output (DCE) to client serial port (i.e DTE). If you convert this serial port it means that you would loose the ability to troubleshoot this device while OpenWrt is running.

See:

Adding/using a USB-to-serial port as @elder_tinkerer and @slh suggest seems simpler IMHO - but YMMV depending on your use case.

I've never done this in OpenWrt, and TBH I last recall demonstrating this in DOS or Windows 95/98 in 2001. So I cannot provide more details.

In clearer terms - configure OpenWrt custom image to disable the output to console ttyS0 on one of the devices. This makes it available to connect to the console of the other.

Hope this helps.

1 Like

What are the possible esp32 solution for UART to WiFi? I would love something using esphome. So that I use a small OLED SSD1309 to display serial output beside the router

Well, given that and what @lleachii wrote about it:
Lots of luck with it.
I'm curious if and how you'll get this to work the way you want.

Just out of curiosity: How often do you have the case that one of your OpenWrt devices isn't accessible via ssh?
Or do you just want to be prepared if, maybe, one day...?

I've had the same wish as OP, but never gotten around to do anything about it. As mentioned, a USB (or Bluetooth/WiFi) console is superior if it is an option.

But I don't see why it can't be done.

Sure, both devices will send a lot of gibberish to the other end by default. And we can't disable all of that. In particular bootloader output, which obviously is wanted for any recovery situations.

But if we can assume that only one of the devices is booting at a time, then I believe the problem is solvable. We can then concentrate on a userspace solution. The assumption will not hold if e..g both devices share power circuits, so we want to avoid that. Maybe have one of them on a battery backup?

We could probably get away with just connecting two standard gettys together. They will probably loop on their login promplts, accepting the other ends prompts as username/password and constantly fire up /bin/login. But if we ignore that noise then there should be no harm.

A better solution would be to guard the /bin/login invocation behind some magic phrase/password which the other side never/rarely sends unless it is typed. I don't know of any such implementation but I would not be surprised if it existed. In any case - can't be that hard to write.

Then we have the phase when we want to disable the local console and use it to manage the other device instead. This will have to

  • stop the getty/whatever listening
  • silence as much as possible of kernel messages and other spurious console output
  • adjust the speed of the port to match the other device
  • run a terminal application like picocom

This should all be possible to do. The device will obviously be console-less in this state, so we can't handle a double error. But it should be good enough for the occasional emergency fix.

The reverse operation should also be implemented of course - configuring everything back to "console".

Things to look out for in particular is stuff accepting "any input". This could cause issues. I don't know how to deal with that since the proposed solution pretty much depends on the ability to constantly write meaningless data to the console port. We might have to accept that this won't work on all devices with a serial console.

I've been there with a Bluetooth UART dongle attached to the console of a ZTE MF286D. I use "jdy-31" modules instead of USB TTL adapters whenever I can, avoiding all the wires and holes in the case etc. The primitive firmware on these modules is actually an example of such a two-way console solution. The UART port is used to configure the module, but will switch to transparently forwarding between Bluetooth and TTL once Bluetooth is connected. Which makes it work perfectly as a console connection.

However, by default the firmware emits a "CONNECTED" message when it switches modes. This ends up as input on the console of the attached device. Usually not a problem - it's ignored similar to how you can type "CONNECTED" on the console without any harm. Except for the situations where "any input" will do. The U-Boot on the MF286D is nice enough to enter its prompt if you type "any key" on startup. The only problem was that with the Bluetooth module powered by 3.3V from the MF286D and a nearby console server constantly reconnecting to it, it would very often emit that "CONNECTED" message while the MF286D was booting after power-on. And the boot would halt.

In this case, the problem was solved by simply configuring the Bluetooth module firmware to dsable the message. Which BTW was done by simply echoing the command to the console port of the MF286D, avoiding the hassle of openning the case again to disconnect the module.

This does not make much sense on the context of a 3-wire console UART. We connect TX to RX and RX to TX and that's that. There are no other signals involved

I've done this, not on a device running Openwrt though, but regular Linux.
Assigned a term to the BT dev, only drawback was the console was unprotected, no logon required.

Frequently. I build a snapshot Openwrt from the git master once a month.

One of the devices is a raspberry pi 3b+. It was easy to disable the console on the serial port and the bootloader doesn't output anything by default. I may be able to make available the second serial port by disabling BT and some not-properly-documented devicetree overlays.
The other device is normal router. I can connect a USB-serial adapter to it, but it needs a powered USB hub. That's a lot of wires... I would prefer a simpler 1 port solution.

  1. Is it possible to disable kernel messages after boot?
  2. How to disable getty/login respawn?

The protection should be possible to fix.

But IMHO the main problem with this solution is that you have no console unless the OS is running. Bootloader access is important (to me at least) for those situations where things go terribly wrong. Maybe I break things too often :wink:

This is where I like the Bluetooth UART dongles. They're so simple and fast enough that the console is up already by the time the first bootloader stage is executing, even while powered by the device it's attached to. But security is non-existent.... Mostly based on the usable distance being 10 meters or less. Very low power is a double advantage here. Not suitable for public areas though.

A normal OpenWrt instance has:

root@u6-2:~# cat /etc/inittab 
::sysinit:/etc/init.d/rcS S boot
::shutdown:/etc/init.d/rcS K shutdown
::askconsole:/usr/libexec/login.sh
root@u6-2:~# cat /usr/libexec/login.sh
#!/bin/sh

[ "$(uci -q get system.@system[0].ttylogin)" = 1 ] || exec /bin/ash --login

exec /bin/login

You can modify both these files to you're liking. Maybe the nicest solution is to add an additional knob to the login script?

See https://www.kernel.org/doc/html/next/core-api/printk-basics.html for the kernel console messages

Probably, I just didn't bother :slight_smile:

Correct, and OpenWrt has console output set on ttyS0, meaning it's In-Use by default. So that has to be disabled on one device.

It seems very difficult to turn off getty/login and kernel messages on demand (when the other device becomes inaccessible) without rebooting. So, I will use only one-way backup access from RPi to the router.
I turned off the console on RPi and I have tested console access to the router. It works with just 3 wires connected: TX, RX and GND.

1 Like