LuCI rewrite in ucode - testers wanted

For Developers
1

I am looking for testers for the just completed rewrite of LuCI's server (router) side runtime. The corresponding PR can be found here:

Description

Large parts of LuCI have been moved to client side rendering throughout the last few years leaving mostly request routing (dispatcher) as well as ubus request gateway functionality and a few assorted template rendering and session management tasks on router side.

Those remaining bits were still implemented using the Lua 5.1 language and have now been rewritten using the ucode language used by firewall4 introduced with the 22.03 release.

The Lua parts required to retain compatibility with older LuCI apps or apps that already do client side rendering but still rely on Lua code in the backend (e.g. for custom controllers or rpcd exec plugins) were factored out into a new luci-lua-runtime package.

The aim of the LuCI ucode rewrite is to remain fully compatible with the previous Lua based LuCI.

Motivation

Several considerations led me to the decision to rewrite the LuCI runtime:

  • Storage footprint - since 22.03 now ships with two scripting languages (Lua and ucode) by default I wanted to eliminate the use of Lua in the default images at least to gain back some previously lost space. In early tests here the complete removal of Lua and related libraries saved about 120KB on the x86/64 rootfs after gzip compression.

  • LuCI is stuck on an outdated version 5.1 of Lua and upgrading to the latest version would require significant refactoring of the code, so I figured I could as well spend the extra effort to port it entirely to another environment.

  • Simpler implementation - since ucode has several required functionalities directly built in, mainly templating and JSON support as well as uci, ubus, uloop and filesystem bindings, we require less extra libraries to achieve the same result. Another hope is that the closely JavaScript related syntax will make the code more accessible to future contributors

Implications

Hopefully none :slight_smile: LuCI should function just as before. It should behave and look the same but with a smaller storage footprint. It should also seamlessly support existing LuCI applications still using Lua through the optional luci-lua-runtime package.

Any deviation from the current LuCI would be a bug which need to be addressed.

Building

In order to test this, a very recent OpenWrt master branch including the latest versions of rpcd and ucode is required, as well as the LuCI branch linked in the PR above.

To use it, copy feeds.conf.default to feeds.conf in your buildroot and change the LuCI feed entry to:

src-git-full luci https://github.com/jow-/luci.git;ucode

Proceed with ./scripts/feeds update and ./scripts/feeds install -a -p luci

I advise to start with a fresh .config and just luci selected as package in order to get rid of implicit Lua dependencies still present in existing build configurations.

Afterwards build as usual and test the resulting image.

Roadmap

Once I got some wider testing coverage and can assert the stability of the implementation I would proceed with merging the rewrite into LuCI mainline and start with converting remaining LuCI application Lua code, beginning with the most popular applications.

Once merged, I will also start working on LuCI precompilation support which allows compiling plain ucode sources to packed bytecode at build time. This speeds up the application startup times and further reduces required storage footprint.

28 Likes
2

I compiled this and noticed that it really required disabling lot of apps & themes to get rid of the luci-lua-runtime.

We should maybe consider if all instances of e.g. using opkg/ipkg to check/show the underlying package versions are necessary at the specific LuCI apps.

E.g. luci-app-ddns was converted to JS by @Ansuel in 2019, but the JS implementation kept the practice of fetching the ddns-scripts package version via an ipkg call. Not sure if that info (ddns version is 2.8.2-26) is actually needed.

(the luci-compat dependency was dropped later, but luci-lib-ipkg has been kept there)

3

I guess the theme should be ported to the new ucode template, shouldn't be that hard in theory

4

At the first glance, it seems to work.

However, I noticed that some of the apps do not work although they have no explicit lua dependency. Having lua has been so self-evident, that there may be hidden dependencies in app scripts.

(I disabled everything that officially pulled in lua, but left the other apps in.)

E.g. the /usr/bin/stat-genconfig config generation script in luci-app-statistics uses lua (declared in shebang), although the package does not explicitly declare a lua dependency.

Similarly luci-app-upnp seems to fail for the same reason (and probably also wireguard).

root@router1:# head -n 2 /usr/bin/stat-genconfig
#!/usr/bin/lua


root@router1:# head -n 2 /usr/libexec/rpcd/luci.upnp
#!/usr/bin/env lua


root@router1:# head -n 2 /usr/libexec/rpcd/luci.wireguard
#!/usr/bin/env lua

PS. And sure, you mentioned implicit dependencies :wink:

5

Yeah, the logic is quite naive atm, it simply checks if there's a luasrc/ directory in a package. If there is, a dependency on luci-lua-runtime is added. Independently of this LuCI ucode rewrite we could already start converting rpcd backend scripts to ucode or shell (whatever is preferred).

Could you give me a quick list of packages in your .config which do not explicitly depend on Lua but still require it?

6

I noticed just the statistics and upnp (plus wireguard that I didn't actually trigger).

The implicit dependencies found so far seem to be init/rpcd/status scripts having lua in shebang.

After disabling the packages with explicit dependencies, the build config is small:

perus@ub2204:/Openwrt/r7800$ grep "luci" .config.minimal 
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-app-sqm=y
CONFIG_PACKAGE_luci-app-wol=y
CONFIG_PACKAGE_luci-proto-ppp=y

Implicit dependencies found:

CONFIG_PACKAGE_luci-app-upnp=y
CONFIG_PACKAGE_luci-app-statistics=y
CONFIG_PACKAGE_luci-app-wireguard=y

Disabled due to explicit dependencies:

####CONFIG_PACKAGE_luci-theme-material=y
####CONFIG_PACKAGE_luci-theme-openwrt-2020=y
####CONFIG_PACKAGE_luci-app-adblock=y
####CONFIG_PACKAGE_luci-app-commands=y
####CONFIG_PACKAGE_luci-app-ddns=y
####CONFIG_PACKAGE_luci-app-nlbwmon=y
####CONFIG_PACKAGE_luci-app-uhttpd=y
1 Like
8

I pushed ucode conversions for luci-app-upnp, luci-app-statistics and luci-app-wireguard to the PR branch.

2 Likes
9

Thanks jow,
at the first glance the stats work normally again, without lua in the build :wink:

10

Hmmm, system->startup->initscripts is empty for me, no init scripts listed. Anyone else?

11

Anything reported by ubus call luci getInitList on the cli? If not, any error visible in logread afterwards?

12 
ubus call luci getInitList
Command failed: Unknown error

Nothing in logread.

13

Fixes for conntrack listing and @ldir's init script listing issue pushed. Also converted luci-theme-material to ucode (untested yet).

4 Likes
14

Thanks @jow
The init scripts evaluation (startup page) seems to work, and also the conntrack listing in realtime graphs works.

15

massive work @jow

16

If we rewirte them to ucode what will be L mean from Luci?

17

Lightweight :stuck_out_tongue:

6 Likes
18

On latest master i get this error message now:

Runtime error: Unable to dlopen file '/usr/lib/ucode/luci/core.so': /usr/lib/ucode/luci/core.so: undefined symbol: crypt
In module(), file /usr/share/ucode/luci/dispatcher.uc, line 1, byte 1:
  called from anonymous function (/www/cgi-bin/luci:7:21)

 `// Copyright 2022 Jo-Philipp Wich <jo@mein.io>`
  ^-- Near here

Is this error related to this change?

19

Likely. Is this a custom build? If yes, what's the diffconfig.sh output?

20

Can't post it here because of the char limit.

21

Ah, a glibc build, as suspected. Please try the following fix:

diff --git a/modules/luci-base/src/Makefile b/modules/luci-base/src/Makefile
index 896aeb0a38..ad309e5c6b 100644
--- a/modules/luci-base/src/Makefile
+++ b/modules/luci-base/src/Makefile
@@ -10,7 +10,7 @@ lib/plural_formula.c: lib/plural_formula.y contrib/lemon
 lib/lmo.c: lib/plural_formula.c
 
 core.so: lib/luci.o lib/lmo.o lib/plural_formula.o
-       $(CC) $(LDFLAGS) -shared -o $@ $^
+       $(CC) $(LDFLAGS) -shared -lcrypt -o $@ $^
 
 version.uc:
        echo "export const revision = '$(LUCI_VERSION)', branch = '$(LUCI_GITBRANCH)';" > $@
1 Like