Package jq has no regex support

Feature Requests
1

Hello. I am trying to write a script to parse a JSON file from mediathekviewweb API to script some movie downloading. However, this is working on my PC but not on my device. After some debugging I stumbled upon an unexpected error:

user@device:~$ cat resp10.txt | jq '.result.results | .[0].title | scan("foo")'
jq: error (at <stdin>:0): jq was compiled without ONIGURUMA regex libary. match/test/sub and related functions are not available.

Unfortunately, installing the oniguruma package does not help. I would like to ask for a second package version with onigurma as a dependency, so jq can perform regular expressions. I am not experienced enough to do this, I never compiled anything, I am used to scripting. I can work around this by using python and its json module, but this is an overkill of resources.

May I ask if someone is so kind to do this for me at some time?
Thank you all!

2

What device is this running on??

ubus call system board
1 Like
3 
{
	"kernel": "5.15.167",
	"hostname": "ArchipelWEI",
	"system": "Qualcomm Atheros QCA9533 ver 2 rev 0",
	"model": "GL.iNet GL-AR300M (NOR)",
	"board_name": "glinet,gl-ar300m-nor",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.5",
		"revision": "r24106-10cc5fcd00",
		"target": "ath79/nand",
		"description": "OpenWrt 23.05.5 r24106-10cc5fcd00"
	}
}
4

You may be able to accomplish your goal with the already installed tools on the router, if you are willing for it to be semi non-portable. Check out jsonfilter:

$ ubus call system board  | jsonfilter -e '$.kernel' -e '$.release.target'
5.15.167
x86/64
5

Thank you for this suggestion.

$ curl -s -H "Content-Type: text/plain" -d '{ "queries" : [ {"fields" : ["topic"], "query" : "Sturm der Liebe"} ] , "size" : 9999 }' "https://mediathekviewweb.de/api/query" > resp.txt
$ cat resp.txt | jsonfilter -e '$.result.results[@.title="Episode 3740"].size'
452984832
$ cat resp.txt | jq '.result.results[] | select(.title=="Episode 3740").size'
452984832

Well, I do not see any advantage over jq or I am too blind to see how I can use regex with jsonfilter. Althouh I could deal without for my specific use case it would be less generic. I prefer the more powerful jq and would like to use regex. I also want to use the functions scan (fails now), select and inside. The possibility to combine them in one statement is way easier and more efficient than doing this in shell loops. I guess this is what you mean with "semi non-portable".

Therefore, I would like to continue this feature request.

6

Json parsing can be done with sed. For complex tasks this may be difficult but probably possible, as a last resort.

7

Yes, absolutely. As I said, I can work around this. Still, jq is a powerfull language and I would like to use it in its full feature-set.

I mean, we do have packages like mtr-json and mtr-nojson and a difference in size of 739 bytes (at least for my platform). The size difference for jq and jq-oniguruma would be sth. like 96641 bytes vs. 96641+170733 bytes. This is still better than higher level languages, enough to provide 2 packages and powerful enough to beat sed/awk in first place.

I may be a rare case, I can understand that. Still I think it is reasonable, is it?

8

You could open an issue in the OpenWrt packages repo and tag the jq mainater who appears to be @ratkaj (on github).

9

That looks like a good idea to me. I just created my first github account and will create an issue the next days - when I am sure not to sth. stupid there.

1 Like
10

There is precedent for having packages with different features versus size tradeoffs. For example, there are 9 different versions of wpad, supporting different TLS libraries and enabled features. So perhaps you can propose a "jq-full" package that has everything enabled.

2 Likes
11

It has been a while since my last attempt in this. After creating my Issue on github there has been no reaction for 3 month now. So I decided to do it myself.

I took openwrt's makefile for jq and changed it to provide 2 different packages: jq and jq-full

new makefile for jq / jq-full 
#
# Copyright (C) 2017 OpenWrt.org
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

include $(TOPDIR)/rules.mk

PKG_NAME:=jq
PKG_VERSION:=1.7.1
PKG_RELEASE:=2

PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://github.com/jqlang/jq/releases/download/$(PKG_NAME)-$(PKG_VERSION)
PKG_HASH:=478c9ca129fd2e3443fe27314b455e211e0d8c60bc8ff7df703873deeee580c2

PKG_MAINTAINER:=Marko Ratkaj <markoratkaj@gmail.com>
PKG_LICENSE:=MIT
PKG_LICENSE_FILES:=COPYING
PKG_CPE_ID:=cpe:/a:jqlang:jq

PKG_INSTALL:=1

include $(INCLUDE_DIR)/package.mk

CONFIGURE_ARGS+= \
	--disable-docs \
	--disable-valgrind \

define Package/jq/Default
  SECTION:=utils
  CATEGORY:=Utilities
  TITLE:=Lightweight and flexible command-line JSON processor
  URL:=https://jqlang.github.io/jq/
  PROVIDES:=jq
endef

define Package/jq
  $(Package/jq/Default)
  TITLE+= without regex support.
  VARIANT:=noregex
  DEFAULT_VARIANT:=1
endef

define Package/jq/description
  Lightweight and flexible command-line JSON processor.
  This package was compiled without ONIGURUMA regex libary. match/test/sub and related functions are not available. 
endef

define Package/jq-full
  $(Package/jq/Default)
  TITLE+= with regex support.
  VARIANT:=regex
  DEPENDS+=+oniguruma
endef

define Package/jq-full/description
  Lightweight and flexible command-line JSON processor.
  This package was compiled with ONIGURUMA regex libary and has full regex support.
endef

ifeq ($(BUILD_VARIANT),noregex)
	CONFIGURE_ARGS += --without-oniguruma
endif

define Package/jq/install/Default
	$(INSTALL_DIR) $(1)/usr/bin
	$(INSTALL_BIN) $(PKG_INSTALL_DIR)/usr/bin/* $(1)/usr/bin/
endef

Package/jq/install = $(Package/jq/install/Default)
Package/jq-full/install = $(Package/jq/install/Default)

$(eval $(call BuildPackage,jq))
$(eval $(call BuildPackage,jq-full))

I successfully tested this makefile locally by building my own image for a testing device (GL-inet AR150) and flashing it to the device.

root@OpenWrt:~# opkg list | grep jq
jq-full - 1.7.1-r1
root@OpenWrt:~# echo -n '"abaabbaaabbb"' |  jq -c 'scan("(a+)(b+)")'
["a","b"]
["aa","bb"]
["aaa","bbb"]

Now, my problem is that I am a noob in relation to github. What would be the best way now? I have an idea about a pull request, but I do not know how to create one. Also, I am afraid there would be no reaction as like in the past 3 months. What would be the best in this case? Forking the jq package and become the maintainer of my own package jq-full? Sounds silly to me. Or can someone force a pull request? I do not have a good feeling with that either.

12

I hope this right way, now I created a pull request:

2 Likes
13

How much larger is the package, when compiled with oniguruma RegExp support?

14

Not that much. From my pull request:

I modified the makefile, so that it will build two packages: jq and jq-full. The former will remain unchanged and the latter will have a dependency to the oniguruma library, so jq-full will have regex functions enabled.

It would not be smart to compile everything in, when the library is already there. The version without oniguruma was just told not to use it. So here are the numbers:

full details for enthusiasts 
$ find . -iname *jq*ipk*
./staging_dir/packages/ath79/jq-full_1.7.1-r1_mips_24kc.ipk
./bin/packages/mips_24kc/packages/jq-full_1.7.1-r1_mips_24kc.ipk
myuser@oldie:~/Temp/openwrt
$ ls -l ./bin/packages/mips_24kc/packages/jq-full_1.7.1-r1_mips_24kc.ipk
-rw-r--r-- 1 myuser myuser 120677 22. Feb 13:30 ./bin/packages/mips_24kc/packages/jq-full_1.7.1-r1_mips_24kc.ipk
myuser@oldie:~/Temp/openwrt
$ find . -iname *onigu*ipk*
./staging_dir/packages/ath79/oniguruma5_6.9.9-r1_mips_24kc.ipk
./bin/packages/mips_24kc/packages/oniguruma5_6.9.9-r1_mips_24kc.ipk
myuser@oldie:~/Temp/openwrt
$ ls -la ./bin/packages/mips_24kc/packages/oniguruma5_6.9.9-r1_mips_24kc.ipk
-rw-r--r-- 1 myuser myuser 174963 22. Feb 13:26 ./bin/packages/mips_24kc/packages/oniguruma5_6.9.9-r1_mips_24kc.ipk

root@Ar150Van:~# opkg info jq
Package: jq
Version: 1.7.1-r1
Depends: libc
Status: install ok installed
Section: utils
Architecture: mips_24kc
Size: 118360
Filename: jq_1.7.1-r1_mips_24kc.ipk
Description: Lightweight and flexible command-line JSON processor.
root@Ar150Van:~# opkg info oniguruma5
Package: oniguruma5
Version: 6.9.9-r1
Depends: libc
Provides: oniguruma
Status: unknown ok not-installed
Section: libs
Architecture: mips_24kc
Size: 174816
Filename: oniguruma5_6.9.9-r1_mips_24kc.ipk
Description: Oniguruma is a modern and flexible regular expressions library.
 It encompasses features from different regular expression implementations that
 traditionally exist in different languages.

 Character encoding can be specified per regular expression object.

The restricted jq version is 118360 bytes and the jq-full version is 120677 large. That is 2317 bytes difference.

Of course you have to add the oniguruma package, which is a dependency in jq-full. Its size is 174963 bytes in on my testing system (which is an ath79/generic GL.iNet GL-AR150). That is the real difference.

1 Like
15

Thank you for answering the question I wanted to ask (How much more flash-usage does your mod cause?), but did not quite phrase properly.

1 Like
16

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.