Jsonmg lua JSON parser

For Developers
1

hey all, I have a developed a few new library's for openwrt and would like a developer ... or developers to code review my work before I polish them up and push them ..

The first one, the one in which i think has the most utility for the community. JSONMG is a slightly more feature rich, Lua JSON parser / stringifyer, Inspired by Jows work with the luci.jsonc implementation, I set out to more or less just mimic it using the Mongoose Networking Library, as my other mention library's depend upon it heavily and I just couldn't justify using a JSON parsing library to just parse the json into lua while the mongoose library already had basic json parsing capabilities. Now, with that said, it was missing allot too, so i had to roll allot of my own. Thankfully Mongoose and Lua work/fit together quite naturally so it wasn't a gargantuan task.
the library's are here jsonmg , libmongoose, kasalua

The can all be built in linux using cmake, I have the openwrt makefiles, I just haven't put them on the git yet but i can by request.

everything builds fine, and I have valgrinded evreything with 100% success ..
0 errors 0 blocks in use at exit.

here some sample output of the jsonmg using the lua interperter...

Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio (double int32)

> jsonmg = require "jsonmg"

> stringArray = jsonmg:stringify({"i", "am", "an", "array", "of", "strings"})
> print(stringArray)

["i","am","an","array","of","strings"]  < ---output

> numArray = jsonmg:stringify({1,2,3,4,5})
> print(numArray)

[1,2,3,4,5]

> mixedArray = jsonmg:stringify({"I", "am", "an", "mixed array", 1, 2, 3, "count", "with", "me", 4,5,6})
> print(mixedArray)

["I","am","an","mixed array",1,2,3,"count","with","me",4,5,6]
> object = jsonmg:stringify({ I = "am", An = "object"})
> print(object)

{"I":"am","An":"object"}

> nestedObject = jsonmg:stringify({I = "am", A = { nested = {object = true}}})
> print(nestedObject)

{"I":"am","A":{"nested":{"object":true}}}

It works on the most complex JSON strings i have to throw at it. :slight_smile:

Now, it's un-pollished, little to no error checking, and I am a little concered about the amount of work being performed on the Lua stack ... but in all my testing with bigger (2048 - 4096) byte JSON strings, i have had no complaints or errors and this is more than I need in any of my use cases but perhaps there a better way ... hence why i ask for some reviews.
All questions are welcome

Hostle

2

here is a side by side comparison to the luci.jsonc module, parsing the same JSON string. Notice that, jsonmg retains the true JSON types, instead of converting everything to objects, with string keys

hostle@dani:~/jsonmg-git/test$ ./final-test.lua "[1,2,3,4,5,\"test\",{\"testy\":{\"joe\":\"woods\"}},{\"frank\":\"black\"}]"
JSONC: 	{"1":1,"2":2,"3":3,"4":4,"5":5,"6":"test","7":{"testy":{"joe":"woods"}},"8":{"frank":"black"},"foo3":{"big":"rink"},"foo2":{"dean":{"dallas":"car"},"bill":"ted"},"fun":null,"bar":"foo","foo":"bar"}

JSONMG	["reall??",2,3,4,5,"test",{"testy":{"joe":"woods"}},{"frank":"black"},{"foo3":{"big":"rink"}},{"foo2":{"dean":{"dallas":"car"},"bill":"ted"}},{"arr2":["bill","ted","bogus","journey",[1,2,3,4,5,6,{"cheech":{"chong":2}}]]},{"bar":"foo"},{"foo":"bar"}]

hostle@dani:~/jsonmg-git/test$ ./final-test.lua "{\"key\":\"val\",\"foo\":\"bar\", \"num\": 100, \"test\":{\"joe\":\"woods\"}, \"pstate\":[1,2,3,4,5,6]}";
JSONC: 	{"foo3":{"big":"rink"},"num":100,"foo2":{"dean":{"dallas":"car"},"bill":"ted"},"pstate":[1,2,3,4,5,6],"key":"val","test":{"joe":"woods"},"fun":null,"bar":"foo","foo":"bar"}

JSONMG	{"foo3":{"big":"rink"},"num":100,"foo2":{"dean":{"dallas":"car"},"bill":"ted"},"pstate":[1,2,3,4,5,6],"1":"reall??","key":"val","test":{"joe":"woods"},"arr2":["bill","ted","bogus","journey",[1,2,3,4,5,6,{"cheech":{"chong":2}}]],"bar":"foo","foo":"bar"}

... and doesn't discard anything, except for the function "fun" where as jsonc print the function name for the key and a null for the val

3

I have added the OpenWRT branch in the jsonmg repo that contains the Openwrt Makefile to build the package. Note you'll also need to link against the libmongoose package, to build jsonmg, so I did the same for the libmongoose repo. Both makefiles are up to date and I have built and tested each against 23.05 with 100% success. just drop them into the package directory and do a makemenu config and you should see them in the build menus Utilities-->jsonmg and Libraries --> mongoose

4

I am not 100% on the total footprint of json-c but heres what i got on my sytem when i break down the sizes required for each parser respectively..

luci jsonc
jsonc.so
size = 65.8 kB (65,848 bytes)

libjson-c.so.5.3.0 //--> offers JSON parsing ..??
size = 89.5 kB (89,512 bytes)

total = 155.3kb

jsonmg
jsonmg.so
size = 65.8 kB (65,760 bytes)

libmongoose.so //--> offers http server/s,https server/s ,ws server/s, wss server/s, smtp servers/s, rpc server/s, ssl ..etc
size = 158.3 kB (158,260 bytes)

total = 224kb

if my calculations are correct mongoose offers 10x the bang for bytes and in my breif experinece with ubus and mongoose ... it fits together like a lock and key. A almost turn key solution to ubus websockest communications which I have heard chatter about

** i should note the size of the mongose .so is without openssl support included. and these are compiled sizes

5

AFAIK creating a PR is the way to get developer or developers to review the code.

1 Like
6

Ya, i was just hoping to get some positive criticism before i squeeze it down and polish it up. There's more cases than the supreme court lol many will be combined and squeezed together making it much less readable. Right now, pretty well every case is separate and its very readable and easy to follow, step by step. I will be patient, I have much to do in the meantime.

For those who are interested in the project. everything is available on my git via the links posted above. Last night I managed to pull just the necessary bits from the mongoose library needed for the JSON functionality and compiled a custom shared library mgjson. Now the jsonmg footprint is 47.7 kB (47,728 bytes) less than that of luci.jsonc

jsonmg
jsonmg.so
size = 65.8 kB (65,760 bytes)

libmgjson.so //--> offers JSON parsing
size = 47.7 kB (47,728 bytes)

total = 113kb

7

Now that i have created the mgjson shared library, its size is 44kb, I am thinking that it may be beneficial to compile it statically, rather than having the external dependency. However, this isn't always favourable, so I would like to give the user the option and build it static or to create the shared library and link to it. Can this be achieved in buildroot thru menuconfig or what is the typically way of configuring these types of build options. I am using cmake but I am still very wet around the ears in terms of complex cmake files. Sometimes writing the library is the easy part, any advice is greatly appreciated.

8

I am going create a pr and get these packages added. I guess I should have started with the problem with the current luci.jsonc in my use case, and demonstrate how jsonmg does not have the same issues dealing with complex JSON input.

So you may or may not have notice the kasalua libraby on my git. It is a full blown interfacing library to interact with all (any i have tested) kasa smart devices. You can scan you local network for devices, control kasa smart plugs, wall plugs, smart switches(power bars), dimmer switchs, light switches and smart bulbs (I do not own a light strip to test but they should be work fine aswell). Anyhow, when scanning for or interacting with the devices, the devices communicate thru tcp, the payloads are encrypted json objects, once decrypted the the json data ranges in size from about 500 -> 2048byes. Now i began the project with the intentions of using luci.jsonc to covert the json to lua, perform what ever tasks needed on the data in lua, then convert the lua back to JSON notation to be display on a web page or whatever the case.

This worked great, luci.jsonc performs like a champ in simple parsing of the payload, however, problems began when trying to add data to the converted json payload in lua, or more accurately, well converting the data back from lua to json. Much of the data gets lost ... or is ignored

example 1

hostle@dani:~/kasalua/kasalua/src$ ./test-jsonmg.lua
BROADCASTING ON IP: 10.0.10.174
1       table   0x56f644c82840
2       number  1
3       table   0x56f644c82890
1       table   0x56f644c82840
1       table   0x56f644c82840
BROADCASTING ON IP: 10.0.10.174
1       table   0x56f644c82ce0
2       number  1
3       table   0x56f644c82d30
1       table   0x56f644c82ce0
1       table   0x56f644c82ce0
jsonmg
TP-LINK_Power Strip_0A7D

luci.jsonc
TP-LINK_Power Strip_0A7D

jsonmg
0000000000000000000000000000000000000000 //<-- deviceId 

luci.jsonc
0000000000000000000000000000000000000000 //<--deviceId

jsonmg

{"sysinfo":"{"system":{"get_sysinfo":{"sw_ver":"1.0.8 Build 230804 Rel.172306","hw_ver":"2.0","model":"KP303(US)","deviceId":"0000000000000000000000000000000000000000","oemId":"00000000000000000000000000000000","hwId":"00000000000000000000000000000000","rssi":-52,"latitude_i":-1879048193,"longitude_i":-1879048193,"alias":"TP-LINK_Power Strip_0A7D","status":"new","obd_src":"tplink","mic_type":"IOT.SMARTPLUGSWITCH","feature":"TIM","mac":"AA:BB:CC:DD:EE:FF","updating":0,"led_off":0,"children":[{"id":"00","state":1,"alias":"EXHAUST FAN","on_time":34739,"next_action":{"type":1,"schd_sec":82800,"action":0}},{"id":"01","state":1,"alias":"Light1","on_time":34727,"next_action":{"type":1,"schd_sec":82800,"action":0}},{"id":"02","state":1,"alias":"HUMIDIFIER","on_time":8426,"next_action":{"type":1,"schd_sec":82800,"action":0}}],"child_num":3,"ntc_state":0,"err_code":0}}}","ip":"10.0.10.174"}

luci.jsonc

{"sysinfo":"{\"system\":{\"get_sysinfo\":{\"sw_ver\":\"1.0.8 Build 230804 Rel.172306\",\"hw_ver\":\"2.0\",\"model\":\"KP303(US)\",\"deviceId\":\"0000000000000000000000000000000000000000\",\"oemId\":\"00000000000000000000000000000000\",\"hwId\":\"00000000000000000000000000000000\",\"rssi\":-52,\"latitude_i\":-1879048193,\"longitude_i\":-1879048193,\"alias\":\"TP-LINK_Power Strip_0A7D\",\"status\":\"new\",\"obd_src\":\"tplink\",\"mic_type\":\"IOT.SMARTPLUGSWITCH\",\"feature\":\"TIM\",\"mac\":\"AA:BB:CC:DD:EE:FF\",\"updating\":0,\"led_off\":0,\"children\":[{\"id\":\"00\",\"state\":1,\"alias\":\"EXHAUST FAN\",\"on_time\":34739,\"next_action\":{\"type\":1,\"schd_sec\":82800,\"action\":0}},{\"id\":\"01\",\"state\":1,\"alias\":\"Light1\",\"on_time\":34727,\"next_action\":{\"type\":1,\"schd_sec\":82800,\"action\":0}},{\"id\":\"02\",\"state\":1,\"alias\":\"HUMIDIFIER\",\"on_time\":8426,\"next_action\":{\"type\":1,\"schd_sec\":82800,\"action\":0}}],\"child_num\":3,\"ntc_state\":0,\"err_code\":0}}}","ip":"10.0.10.174"

So far so good. Other than luci.jsonc not un-escaping the string, the results are even ..or both correct, valid json. But there is a subtle isusse, notice the ip is enclosed in(part of the json object) in the jsonmg output. Though its not wrong or very difficult to deal with post conversion, its not correct, or as i intended it to be. jsonmg on the other hand outputs the JSON in the CORRECT format.

now image you, need to add some critical JSON data to the output. lets use a variety of json type this time..
data is the jsonmg converted json and data2 is the jsonc conveted data, respectively.

data.foo = "bar"; //<-- array string
data2.foo = "bar";
data.bar = "foo";
data2.bar = "foo";
data.foo2 = {bill = "ted", dean = {dallas = "car"}}; //<-- a nested object
data2.foo2 = {bill = "ted", dean = {dallas = "car"}};
data.foo3 = {big = "rink"}; //<-- kv pair
data2.foo3 = {big = "rink"};
data.arr2 = {"bill", "ted", "bogus", "journey", {1,2,3,4,5,6, cheech = {chong = 2}}} <-- array with nested array + nested object
data.fun = function() print("hello") end; //<-- simple function to print hello
data[1]="reall??"; //<-- another array string
data2.fun = function() print("hello") end;

results

JSONC:  {"1":1,"2":2,"3":3,"4":4,"5":5,"6":"test","7":{"testy":{"joe":"woods"}},"8":{"frank":"black"},"foo3":{"big":"rink"},"foo2":{"dean":{"dallas":"car"},"bill":"ted"},"fun":null,"bar":"foo","foo":"bar"}

JSONMG   {"1":"reall??","arr2":["bill","ted","bogus","journey",[1,2,3,4,5,6,{"cheech":{"chong":2}}]],"foo3":{"big":"rink"},"foo2":{"dean":{"dallas":"car"},"bill":"ted"},"sysinfo":{"system":{"get_SysInfo":{"obd_src":"tplink","rssi":-46,"feature":"TIM","hw_ver":"2.0","alias":"TP-LINK_Power Strip_0A7D","mac":"AA:BB:CC:DD:EE:FF","err_code":0,"ntc_state":0,"child_num":3,"children":[{"state":1,"next_action":{"schd_sec":82800,"parent_type":1,"action":0},"id":"000000000000000000000000000000000000000000","alias":"EXHAUST FAN","on_time":1},{"state":0,"next_action":{"schd_sec":82800,"parent_type":1,"action":0},"id":"000000000000000000000000000000000000000000","alias":"Light1","on_time":1},{"state":1,"next_action":{"schd_sec":82800,"parent_type":1,"action":0},"id":"000000000000000000000000000000000000000000","alias":"HUMIDIFIER","on_time":2}],"updating":0,"led_off":0,"deviceId":"0000000000000000000000000000000000000000","longitude_i":-1879048193,"latitude_i":-1879048193,"status":"new","model":"KP303(US)","mic_TYPE":"IOT.SMARTPLUGSWITCH","oemId":"00000000000000000000000000000000","hwId":"00000000000000000000000000000000","sw_ver":"1.0.8 Build 230804 Rel.172306"}}},"bar":"foo","foo":"bar","ip":"10.0.10.174"}
9

I created a pr for jsonmg, mgjson, mongoose and kasalua this afternoon.