How to get reliable sshd (realtime priority)?

[HectoPascal]

For me its critical to have a reliable remote access to the system. Its not the first time a process stole all my cpu power rendering LuCI and SSHd useless (not responding) whereas the traffic routing still worked.

I want to have these two processes (nginx/sshd) to be realtime priority (SCHED_RR) to have this problem sorted once and for all. Additionally all root (uid 0) processes niceness should be -10.

For the latter problem i've tried to set /etc/security/limits.conf:

root hard priority 29
root hard nice -10

This didn't work though. All root processes were still 'nice 0'.

For the realtime problem i'm out of ideas. The 'chrt' binary is not usable under musl. There are other processes already running as RT/SCHED_FIFO though. They are probably spawned by C API directly.
ps ax --format uname,pid,ppid,tty,cmd,cls,ni,pri,rtprio |grep FF

Any ideas how to get this achieved?

[lleachii]

@HectoPascal

Its not the first time a process stole all my cpu power rendering LuCI and SSHd useless (not responding) whereas the traffic routing still worked.

@HectoPascal

Any ideas how to get this achieved?

I would find out what process(es) cause a router to crash; or otherwise end and halt the processes.

[mk24]

Modify the dropbear init file (and possibly uhttpd *) to start it with high priority. OpenWrt runs most everything as root so you don't want to default all of them.

• LuCI shouldn't be considered bombproof / high reliability, etc. Use ssh for emergencies.

[dl12345]

You can apply this patch to package/utils/util-linux/Makefile. It will make a "renice" command available which you can use to change the priority of a running process.

Then

make package/util-linux/compile V=s

and go install the ipk. You can then change the priority of the process by doing renice -20 <PID>.

I just tested it on my dropbear and it works fine

diff --git a/package/utils/util-linux/Makefile b/package/utils/util-linux/Makefile
index 261a961..aa32c1c 100644
--- a/package/utils/util-linux/Makefile
+++ b/package/utils/util-linux/Makefile
@@ -386,6 +386,16 @@ define Package/partx-utils/description
  contains partx, addpart, delpart
 endef
 
+define Package/renice
+$(call Package/util-linux/Default)
+  TITLE:=Alter the priority of running processes
+endef
+
+define Package/renice/description
+ Alter the priority of running processes
+endef
+
+
 define Package/script-utils
 $(call Package/util-linux/Default)
   TITLE:=make and replay typescript of terminal session
@@ -706,6 +716,11 @@ define Package/partx-utils/install
        $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/sbin/delpart $(1)/usr/sbin/
 endef
 
+define Package/renice/install
+       $(INSTALL_DIR) $(1)/usr/bin
+       $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/renice $(1)/usr/bin/
+endef
+
 define Package/script-utils/install
        $(INSTALL_DIR) $(1)/usr/bin
        $(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/script $(1)/usr/bin/
@@ -788,6 +803,7 @@ $(eval $(call BuildPackage,namei))
 $(eval $(call BuildPackage,nsenter))
 $(eval $(call BuildPackage,prlimit))
 $(eval $(call BuildPackage,rename))
+$(eval $(call BuildPackage,renice))
 $(eval $(call BuildPackage,partx-utils))
 $(eval $(call BuildPackage,script-utils))
 $(eval $(call BuildPackage,setterm))

[HectoPascal]

@lleachii

I would find out what process(es) cause a router to crash; or otherwise end and halt the processes.

I know which process caused that. Its even in an LXC environment that i have full cgroups control over. But its not the point to make the world slave. Its about making SSHd master.

@mk24

LuCI shouldn't be considered bombproof / high reliability, etc. Use ssh for emergencies.

That's what i said. SSHd was not responding either.

@dl12345

It will make a "renice" command available

Yes, i probably will take this over as well. Thanks!

In the meantime i have compiled a static 'chrt' binary against glibc that talks directly to the kernel bypassing musl. Will test that.

It would be nice though if that 'limits.conf' solution would be working so all new root processes are spawned with -10 niceness automatically. It would make a better difference to all the LXC processes. I'm wondering if 'limits.conf' is honored by OpenWrt init system at all.

[HectoPascal]

Static glibc based 'chrt' works properly so i will inject that into init.d for nginx/luci and sshd. This should make them king of the hill.

$ for i in $(pgrep sshd); do chrt -p 90 $i; done
$ ps ax --format uname,pid,ppid,tty,cmd,cls,ni,pri,rtprio |awk 'NR==1 || /sshd/'
USER       PID  PPID TT       CMD                         CLS  NI PRI RTPRIO
root      4633     1 ?        /usr/sbin/sshd -D            RR   - 130     90
root     31889  4633 ?        sshd: root@notty             RR   - 130     90
root     31894 31889 ?        sshd: root@internal-sftp     RR   - 130     90
root     32063  4633 ?        sshd: root@notty             RR   - 130     90
root     32066 32063 ?        sshd: root@internal-sftp     RR   - 130     90
root     32147  4633 ?        sshd: root@pts/0             RR   - 130     90

[dl12345]

limits.conf(5) - Linux man page

Name

limits.conf - configuration file for the pam_limits module

Description

The pam_limits.so module applies ulimit limits, nice priority and number of simultaneous login sessions limit to user login sessions. This description of the configuration file syntax applies to the /etc/security/limits.conf file and *.conf files in the /etc/security/limits.d directory.

The syntax of the lines is as follows:

@HectoPascal

I'm wondering if 'limits.conf' is honored by OpenWrt init system at all.

libpam is supported, but not enabled by default.