How to trigger watchdog

robimarko · March 16, 2018, 12:41pm

Hi, I have a project where we will be running a Rambutan dev board on a remote site and it will be running on LTE.
Now we are making a script to act like a ping watchdog so when there is not internet connection it will first reset usb and if that does not help trigger watchdog to reset the device.

I know that watchdog is managed by procd,but I can find how to get procd to stop pinging watchdog so it will trigger reset.
I am trying to use ubus call system watchdog '{"magicclose":"true"}'
but I only get:
{
"status": "running",
"timeout": 30,
"frequency": 5,
"magicclose": false
}

And it seems that you are unable to set anything regarding watchdog over ubus

Any ideas on how to trigger it?

pepe2k · March 16, 2018, 1:57pm

ubus call system watchdog '{"stop": true}'

robimarko · March 16, 2018, 1:58pm

Thanks,I was using ubus call system watchdog '{"stop": "true"}'

Stupid syntax error on my side

pepe2k · March 16, 2018, 2:00pm

Just btw, why do you want to use this approach? Resetting CPU without a clear system shutdown doesn't sound like a good idea for me, especially on board with NAND

robimarko · March 16, 2018, 2:30pm

That will be last resort.
Off course we will do everything before attempting forced reset.

pepe2k · March 16, 2018, 2:42pm

Why not just perform a clean reboot?

robimarko · March 16, 2018, 2:44pm

Like I said we will off course do that first.
But we have to have a way to manually trigger hard reset since we had LTE modems that get stuck in some weird mode in which reseting USB port and rebooting does not help.
Only hard reset,altough rare it sometimes happens

pepe2k · March 16, 2018, 2:59pm

Then you don't have problem with main SOC but with a modem. If resetting main SOC helps, then bringing down USB bus should also help. Plus, most of modems have AT command for a hard reset (assuming that AT port still works) or if it's a miniPCIe one, you should be able to use PERSTn pin for that.

Sorry for OT

robimarko · March 16, 2018, 4:16pm

No need to be sorry.
AT commands don't work as the modem is E3372 from Huawei which uses Hilink mode.
Its USB so no PERSTn pin

valentt · March 16, 2018, 7:00pm

Hi,
i just finished writing a detailed blog post regarding how to use
hardware watchdog and how to manually take control over from procd:

Over the years there were few questions regarding this and first there was no way to do it (missing magicclose feature) and documentation was lacking even after it was implemented.

Hopefully this helps some people.

If you have any suggestions or comments regarding this blog post feel free to write here or contact me directly.

Thanks @robimarko for your help!

Cheers,
Valent.

valentt · March 16, 2018, 7:02pm

I have had situations (rare though) when not even reboot command works.

pepe2k · March 17, 2018, 9:08pm

I've heard that sometimes, on some QC/A WiSoCs, even the built-in watchdog doesn't properly reset CPU. That's probably why some manufacturers add external one.

But either way, using watchodog to reset system still sounds bad for me. Both of you should find the real reason of your problems (in software or hardware) rather than mask it with CPU hard reset. My 2c.

valentt · March 18, 2018, 11:10am

I completely agree with you @pepe2k, but we can't reproduce system freeze in our development lab. System freeze happens once every few months, and on different devices.

If we had a device that constantly froze we would take it in the lab and try to find out the root cause.

But there were other times when just human error was the factor of system freeze and then also hardware watchdog saved us. And you can't take human error out of the equation, at least not for now

mrusedski · April 10, 2018, 5:32pm

Hi,
I have taken manual control of watchdog, as per your manual.
Now, I am able to open, do ioctl etc on that file.
wrote a sample program to test if watchdog will reset/reboot the device.

snippet:
  system("ubus call system watchdog '{\"stop\": true}'");
  system("ubus call system watchdog '{\"magicclose\": true}'");
  system("ubus call system watchdog '{\"stop\": true}'");
  int fd = open("/dev/watchdog",O_WRONLY);
  printf("fd = %d\n", fd);
  int timeout;
  ioctl(fd , WDIOC_GETTIMEOUT,&timeout);
  printf("Get Default Watchdog Time : %d\n",timeout);
  for(;;) sleep(1);

I am not pinging watchdog intentionally to let watchdog reboot.

Reboot is not happening.

Additionally, when I kill this program, i am getting following message

watchdog: watchdog0: watchdog did not stop!

What could be the problem?

UPDATE: My code is working as expected in other openwrt/lede lying around. Tested okay on C7 - TPLink, 940 V4.
NOT WORKING on TPLink C20, AC750.
Do I need a kernel module for this? Can someone help

Thx

denisber · May 18, 2018, 7:08pm

No AT commands? Is there a specific driver for this modem?

In my case I could reset the modem using this command (Huawei E1756)

printf "AT+CFUN=1,0\r" > /dev/ttyUSB2
sleep 3
printf "AT+CFUN=1,1\r" > /dev/ttyUSB2

robimarko · May 18, 2018, 7:09pm

Well,its Hilink based E3372h so no AT.

reinerotto · January 7, 2019, 9:03am

Pls, can you be more detailed. I also have the case that on seldom occasions, a remote openwrt device (WE1026-5G-16M), having internal LTE modem, stops working. And triggering the standard reboot does not help either. Of course, a power cycle solves the probelm. But a software controlled power cycle of the modem might be a simpler solution, I suspect.

pepe2k · January 7, 2019, 9:34am

PIN22 in miniPCIe slot is PERSTn/PERST#, have a look:

If you check datasheet of, for example, Quectel EC25, you will find that this pin can be used as "hardware reset":

Of course, both your router and the modem you use need to support this feature - I don't know WE1026 but if its miniPCIe slot supports also PCIe bus (e.g. WiFi cards) then PERSTn is (or, should be... China vendors sometimes don't care) supported there for 100% (it's required by standard). With small modification in DTS you should be able to export PERSTn as generic GPIO and use it for modem reset.

reinerotto · January 7, 2019, 1:03pm

I also have the EC25 on my WE1026, by chance. Regarding ".. small modification...": Do you know an example of another boards DTS, so I can try to copy/adpt it ?

pepe2k · January 7, 2019, 1:30pm

IIRC, you need to disable pcie node and configure "pcie" group as GPIO. For MT7620x something like this should work:

[...]

&pcie {
        status = "disabled";
};

&pinctrl {
        state_default: pinctrl0 {
                gpio {
                        ralink,group = "pcie";
                        ralink,function = "gpio";
                };
        };
};

[...]

With that, you should be able to export GPIO 36 (GPIO12 in gpio1) in user space and use it for modem reset. You can also do it in DTS (so that the GPIO gets exported automatically for you):

[...]

gpio-export {
	compatible = "gpio-export";
	#size-cells = <0>;

	modem_enable {
		gpio-export,name = "modem_enable";
		gpio-export,output = <1>;
		gpios = <&gpio1 12 GPIO_ACTIVE_HIGH>;
	};
};

[...]

All of that will work only it the PERSTn pin in the slot is connected to MT7620 and it's the dedicated one (vendor could select different pin - in that case, you need to find out which one).