Given a .config is it possible to know the independent, (not dependent) packages?

I am trying to create a simple shell script that could return the list of independent packages (without parent) or not preselected as part of the target.

In order to do so I have created this script that runs in bash in the device (install bash if you want to run it):

#!/bin/bash
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=$(gzip -c -d /tmp/opkg-lists/openwrt_base)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_core)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_luci)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_packages)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_routing)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_telephony)"\n"

# ldb: contains all the packages in the downloaded database in a separated line
# followed by its dependencies
ldb=$(echo -e "$db" | awk '/Depends:/||/Package:/{print}' | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g;s/([^ ]* //ig' | sort)

# manifest: contains the local list of installed packages, one per line
manifest=$(opkg list-installed | awk '{print $1}' | sort)

dep=
for p in $manifest; do
    echo -n "processing $p ..."
    line=$(echo -e "$ldb" | grep "^$p ")
    [ -n "$line" ] && d=" "$(echo "$line" | cut -d ' ' -f 2-) || d=$(opkg info $p | awk '/Depends:/ { $1=""; print}' | sed 's/,//g')
    [ -n "$d" ] && echo "$d" || echo "NOT FOUND *****************"
    dep+="$d"
done
# dep: contains all the dependencies of the installed packages.

. /etc/openwrt_release
TARGET=$(echo $DISTRIB_TARGET | awk -F"/" '{print $1}')
SUBT=$(echo $DISTRIB_TARGET | awk -F"/" '{print $2}')
URL=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBT/openwrt-$TARGET-$SUBT.manifest

# targets: contains the list of packages depending on the target architecture
targets=$(wget $URL -O - | awk '{print $1}' | sort -u)

#  dept: contains the list of all dependent packages, local dependents and target dependents
dept=$(echo $dep" "$targets | tr ' ' '\n' | sort -u)

_dep_=$(echo " "$dept" " | sed 's/ / \n /g;s/ [ ]*/ /g' | sort -u)
parents=
for p in $manifest; do
    if ! grep -q " $p " <<<"$_dep_"; then
        parents+=$p" "
    fi
done
for p in $targets; do
    if ! grep -q "$p" <<<"$manifest"; then
        parents+="-"$p" "
    fi
done

# parents: contains a list of removed and added packages
# (those independent packages or removed from the target default)
parents=$(echo $parents | tr ' ' '\n' | sort -u)

echo $(echo $manifest | wc -w)" packages installed."
echo $(echo $_dep_ | wc -w)" packages are dependents."
echo $(echo $parents | wc -w)" packages are independents."
echo -e "\nParent Packages:"
echo "$parents"

but....
There are some packages listed as independent that are actually dependents such as libjpeg-turbo that that in my case depends on libgd. But libgd depends on libjpeg, so there is no way to know the actual parent of libjpeg-turbo.

To solve this, I can run it from the OpenWrt kitchen before building the cake (the image) from the final .config

Any suggestion is welcome, such as ideas on how to handle the files under Openwrt/tmp/ to remove default target packages and what final packages I have actually added.

Basically I would be happy if I could get a list of packages to be installed (or already installed) followed by one of these 4 possibilities: target | removed | independent | dependent.
Target: package included as part of the target architecture
Removed: package deselected from the default target
Independent: package not required by any other installed package
Dependent: package required by at least one installed package.

Running my script in the device returns in my case:

-libustream-wolfssl20201210
-libwolfssl5.1.1.31258522
-wpad-basic-wolfssl
6in4
6rd
6to4
auc
bash
blockd
ca-certificates
ccrypt
collectd-mod-conntrack
collectd-mod-cpufreq
collectd-mod-dhcpleases
collectd-mod-entropy
collectd-mod-ping
collectd-mod-sqm
collectd-mod-thermal
collectd-mod-uptime
collectd-mod-wireless
cryptsetup
curl
ddns-scripts-noip
diffutils
e2fsprogs
f2fs-tools
gdbserver
hostapd-utils
htop
ip-tiny
ip6tables-mod-nat
ip6tables-nft
iperf3
iptables-nft
irqbalance
kmod-fs-cifs
kmod-fs-exfat
kmod-fs-ext4
kmod-fs-f2fs
kmod-fs-hfs
kmod-fs-hfsplus
kmod-fs-msdos
kmod-fs-nfs-v3
kmod-fs-nfs-v4
kmod-mt7915e
kmod-nls-cp1250
kmod-nls-cp850
kmod-nls-iso8859-15
kmod-tun
kmod-usb-storage-uas
kmod-usb3
libjpeg-turbo
luci-app-adblock
luci-app-argon-config
luci-app-attendedsysupgrade
luci-app-banip
luci-app-bcp38
luci-app-commands
luci-app-ddns
luci-app-nlbwmon
luci-app-openvpn
luci-app-sqm
luci-app-statistics
luci-app-ttyd
luci-app-uhttpd
luci-app-upnp
luci-app-vnstat2
luci-app-wireguard
luci-ssl-openssl
luci-theme-argon
mc
miniupnpd-nftables
nano-full
ncdu
nfs-utils
ntfs-3g
patch
ppp-mod-pptp
tc-mod-iptables
tc-tiny
tcpdump-mini
tree
wget-ssl
wpad-openssl

As you can see, there are some clear problems with the correct dependency detection of packages such as libjpeg-turbo or the kmod-mt7915e.

Any help is welcome.

Thanks

You are evaluating the target default packages, but not the device specific packages (like WiFi drivers, firmware, special button drivers) that vary inside a target.

You should probably take the PROVIDES lines into account when evaluating the dependency chain. Libjpeg-turbo provides libjpeg and gets pulled in due to that.

But note that several packages can provide a defined package, and the end result may vary depending on the installation order, so I don't think that there is any clear way to 100% backtrack from .config to the original singular package selections. There will sometimes be corner cases that need to be pruned by hand.

1 Like

See if running opkg list-installed works for you.

Nope. Not a solution for this.

He is trying to figure out which packages have been installed due to being a dependency, and only list independent top selections. (Plus the excluded/removed defaults)

1 Like

I am only building for a few different devices, so I have created my device recipes (like my R7800 recipe that your config seems to be a derivative of) by manually pruning the diffconfig output. Once you always add the new packages via the recipe, the things stay manageable.

Your script seems to already work pretty ok, but like I said, there will possibly be a few corner cases with more complex dependencies and concurrent variants ( or alternatives) providing a dependency.

1 Like

Testing this script I wrote to see what the independent packages will look like in the output...

for PKG in $(opkg list-installed)
do
	opkg info $PKG | grep -E "Package:|Depends:" 
done

@hnyman, where can I retrieve that information from?

Thanks for pointing me to the Provides field. Yes, it is a nice way to share code, but it is really a nightmare from the database point of view. I believe I can retrieve the original package from a built image. I will try it this afternoon.
However, getting that before building the rom.... seems quite difficult because, as you said, there are several packages providing the same package dependency.

Good catch! Actually it is based on a diffconfig file you shared for the E8450. And I am using it also for an R7800.

@anon89577378, I see you thought the same way. Actually that is more or less the first loop in my code. I do all the previous steps —retrieving the opkg database— to speedup the process and relieve the router from un-gzipping hundreds of times the same files, those you download when doing a opkg update and ungzip when you do an opkg info.

You can see the packages in the manifest compiled along your own build, but they are not specified as device packages in any way. Buildbot does not store that anywhere.

And you can see that from the device Makefile (or actually the image recipe) in source code.
Example where also negative exclusions from the target defaults...

It was a quick proof of concept...nothing I would put in "production". :wink:

This piece of code from the wiki is quicker...

opkg list-installed | awk -e '{print $1}' | tr '\n' ' '

Hi guys,

I have reworked the code a little bit and now it is way faster and more clear. I have taken into account the provided packages, thanks @hnyman for pointing me in that direction, and the final list resembles a lot more the actual minimal set.
@anon89577378, the problem was not getting the local manifest or the list of dependents. That was easy and fast. So far this is the new version of the code with the provided packages included:

## MANIFEST: contains the local list of installed packages, one per line
echo -n "Acquiring local package manifest..."
MANIFEST=$(opkg list-installed | awk '{print $1}' | sort) && echo "success."

## TDEFAULTS: contains the list of the default packages for the target architecture
echo -n "Acquiring default packages for this architecture..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
URL=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
TDEFAULTS=$(wget -q $URL -O - | awk '{print $1}' | sort -u) && echo "success."

## REMOVED: contains the list of the removed default packages
echo -n "Identifying removed default packages..."
REMOVED=$(sed 's|^|/^|; s|$|\$/d|' <<<$MANIFEST | sed -f- <(echo "$TDEFAULTS")) && echo "success."

## DEFAULTS: contains the list of installed default packages
echo -n "Identifying installed default packages..."
DEFAULTS=$(sed 's|^|/^|; s|$|\$/d|' <<<$REMOVED | sed -f- <(echo "$TDEFAULTS")) && echo "success."

## db: contains the OpenWrt package info
echo -n "Building global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in $(echo "base core luci packages routing telephony"); do
    db+=$(gzip -c -d /tmp/opkg-lists/openwrt_$section)$'\n\n'
done && echo "success."

## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Simplifying the dependents database..."
depdb=$(echo "$db" | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' | sort) && echo "success."

## DEPENDENTS_STEP1: contains all the dependencies identified in the OpenWrt db of installeed packages
echo -n "Calculating dependents from OpenWrt database..."
DEPENDENTS_STEP1=$(sed 's/^/^/; s/ / |^/g; s/$/ /' <<<$(echo $MANIFEST) | grep -E -f- <(echo "$depdb") | cut -d ' ' -f 2- | tr ' ' '\n' | sort -u) && echo "success."

## EXTRANEOUS: contains packages not listed in OpenWrt package info
echo -n "Identifying local packages not tracked in OpenWrt database..."
EXTRANEOUS=$(sed 's|^|/^|; s|$|\$/d|' <(echo "$db" | awk '/Package: / {print $2}') | sed -f- <(echo "$MANIFEST")) && echo "success."

## DEPENDENTS_STEP2: contains contains all the dependencies from local packages, including extraneous packages
echo -n "Adding dependents of extraneous packages..."
DEPENDENTS_STEP2="$DEPENDENTS_STEP1"
for p in $EXTRANEOUS; do
    DEPENDENTS_STEP2+=$(opkg info $p | awk '/Depends: / { $1=""; print}' | sed 's/,//g')
done
DEPENDENTS_STEP2=$(echo "$DEPENDENTS_STEP2" | tr ' ' '\n' | sort -u) && echo "success."

## DEPENDENTS_STEP3: contains all the dependents from local packages and target architecture
echo -n "Adding dependents of target architecture defaults..."
DEPENDENTS_STEP3=$(echo -e "$DEPENDENTS_STEP2\n$DEFAULTS" | sed '/(/d;/)/d' | sort -u) && echo "success."

## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' | sort) && echo "success."

## DEPENDENTS_STEP4: contains the list of all dependent packages, acrhitecture's defaults, their providers, and provided packages.
echo -n "Adding provided packages to dependents..."
DEPENDENTS_STEP4=$(echo "$DEPENDENTS_STEP3$(sed 's/ / | /g; s/^/ /; s/$/ /' <<<$(echo $DEPENDENTS_STEP3) | grep -E -f- <(echo "$provdb" | sed 's/^/ /g;s/$/ /g') | tr ' ' '\n')" | sort -u) && echo "success."

## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(sed 's|^|/^|; s|$|\$/d|' <<<$DEPENDENTS_STEP4 | sed -f- <(echo "$MANIFEST")) && echo "success."

## DEPENDENTS: Contains the list of all local dependent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$(sed 's|^|/^|; s|$|\$/d|' <<< $(echo -e "$INDEPENDENTS\n$DEFAULTS") | sed -f- <(echo "$MANIFEST")) && echo "success."

## OUTPUT

R=$(wc -w <<<$REMOVED)
M=$(wc -w <<<$MANIFEST)
T=$(wc -w <<<$DEFAULTS)
I=$(wc -w <<<$INDEPENDENTS)
D=$(wc -w <<<$DEPENDENTS)
echo -e "\n$M packages installed:\n"$MANIFEST
echo -e "\n$T target default packages:\n"$DEFAULTS
echo -e "\n$R packages removed from defaults:\n"$REMOVED
echo -e "\n$I independent packages:\n"$INDEPENDENTS
echo -e "\n$D dependent packages:\n"$DEPENDENTS

echo -e "\nMinimal set of packages:"
echo "$REMOVED"| sed 's/^/-/g'
echo "$INDEPENDENTS"

As you can see is way faster than the previous version. But I am still facing two issues:

  1. The problem now is detecting packages that are added as part of the device, not the target architecture. Those are now identified as independent. I will dig into it with the info provided by @hnyman here.
  2. There are some dependencies that are actually replaced from the target default packages that are now detected as dependents. I will have to dig into this too.

opkg list-installed | cut -f 1 -d ' ' | sed ':a;N;$!ba;s/\n/ /g'

opkg list-installed | cut -f 1 -d ' ' | sed ':a;N;$!ba;s/\n/ /g' > liste.txt

Is there an easy way to get the device's OpenWrt profile ID? Right now I am getting it from /etc/build.config using:

. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
DEVICE=$( cat /etc/build.config | grep -E '^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_' | sed '/^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_//;s/=y$//' )

Relying on the provided /etc/build.config doesn't seem quite reliable. Do images in the download section also have this file or do only customized images have it?

@F5BJR, thanks for the suggestion, there are many ways to retrieve the local package manifest, but that is always fast. You won't notice the difference between your proposal and, for instance this other:
echo $(opkg list-installed | cut -f 1 -d ' ')

But opkg is slow. I am thinking on ditching opkg completely and rely on the opkg status file. For instance the local manifest could be retrieved using:
echo $(cat /usr/lib/opkg/status| grep Package |cut -d ' ' -f 2-)
The slower part is etrieving the dependents because at this time, the original data comes from the opkg general database available in /tmp/opkg, the files you download when doing an opkg update. I have removed all but one loop from version 1 to version 2 of the code, but as you can see, it is still quite slow, even for shell script. I will try using that local status file instead to accelerate the process. This way I can even process in the same loop the packages I am now identifying as EXTRANOEUS.

I was curious as to what I would see...

I have done another step forward. In this third version I have added the device specific packets and have improved even more the code for speed. In an R7800 this is the improvement in execution time:

VERSION  LINES    TIME 
-------  -----    ---- 
   v1      59      58s
   v2      85      21s
   v3     107       5s

And here is the v3

#!/bin/bash

function ADD() { echo -e "$1\n$2" | sort -u ;}
function INTERSECT() { sed 's/^/^/; s/ /$|^/g; s/$/$/' <<<$(echo $1) | grep -E -f- <(echo "$2") ;} 
function AnotB() { sed 's|^|/^|; s|$|\$/d|' <<<$2 | sed -f- <(echo "$1") ;}
function dependentsof() { sed 's/^/^/; s/ / |^/g; s/$/ /' <<<$(echo $1) | grep -E -f- <(echo "$depdb") | cut -d ' ' -f 2- | tr ' ' '\n' |awk 'NF' | sort -u ;}

reset
## ORIGDEVICE: list of device specific packages
echo -n "Downloading list of device specific packages..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
DEVICE=$(cat /etc/build.config | grep -E '^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_' | sed 's/^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_//;s/=y$//')
URL1=https://raw.githubusercontent.com/openwrt/openwrt/master/target/linux/$TARGET/image/$SUBTARGET.mk
ORIGDEVICE=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | sed '/^-/d'|awk 'NF' | sort) && echo "success."

## ORIGTARGET: list of target architecture specific packages
echo -n "Downloading list of target architecture scpecific packages..."
URL2=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
ORIGTARGET=$(wget -q $URL2 -O - | awk '{print $1}' | sort -u) && echo "success."

## DEVREMOVES: list of the packages removed per device
echo -n "Downloading list of default removed target packages..."
DEVREMOVES=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | grep '^-' |cut -c2- |awk 'NF' |sort) && echo "success."

## DEFAULTS: list of installed default packages for the target and device
echo -n "Identifying installed default packages..."
DEFAULTS=$( AnotB "$(ADD "$ORIGTARGET" "$ORIGDEVICE")" "$DEVREMOVES" ) && echo "success."

## MANIFEST: list of installed packages
echo -n "Acquiring local package manifest..."
MANIFEST=$(cat /usr/lib/opkg/status | grep "^Package" | cut -d ' ' -f 2- | sort -u) && echo "success."

## USERREMOVED: list of manually removed default packages
echo -n "Identifying default packages removed by user..."
USERREMOVED=$( AnotB "$DEFAULTS" "$MANIFEST") && echo "success."

## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Retrieving the dependents database..."
depdb=$(cat /usr/lib/opkg/status | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' |sed 's/([^)]*)//g' |tr -s ' '|sort) && echo "success."

## DEPENDENTS_STEP1: contains all the dependencies identified in the OpenWrt db of installeed and device dependent packages
DEPENDENTS_STEP1=$( ADD "$(dependentsof "$MANIFEST")"  "$DEFAULTS" )

## db: contains the OpenWrt package info
echo -n "Retrieving local copy of global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in $(echo "base core luci packages routing telephony"); do
    db+=$(gzip -c -d /tmp/opkg-lists/openwrt_$section)$'\n\n'
done && echo "success."

## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' |sed 's/([^)]*)//g' | sort) && echo "success."

## DEPENDENTS_STEP2: contains the list of all dependent packages, acrhitecture's dependents, and its provided packages.
echo -n "Adding provided packages to dependents..."
DEPENDENTS_STEP2=$(ADD "$DEPENDENTS_STEP1" "$(sed 's/ / | /g; s/^/ /; s/$/ /' <<<$(echo $DEPENDENTS_STEP1) | grep -E -f- <(echo "$provdb" | sed 's/^/ /g;s/$/ /g') | tr ' ' '\n')"|awk 'NF') && echo "success."

## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(AnotB "$MANIFEST" "$DEPENDENTS_STEP2") && echo "success."

## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$( AnotB "$MANIFEST" "$( ADD "$INDEPENDENTS" "$DEFAULTS" )" ) && echo "success."


## TARGETREMOVED: contains the list of removed target architecture specific packages
echo -n "Identifying removed target packages..."
TARGETREMOVED=$( AnotB "$ORIGTARGET" "$MANIFEST") && echo "success."

## DEVREMOVED: contains the list of removed device specific packages
echo -n "Identifying removed device packages..."
DEVREMOVED=$( AnotB "$ORIGDEVICE" "$MANIFEST") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
TARGETINSTALLED=$( INTERSECT "$ORIGTARGET" "$MANIFEST") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
DEVICEINSTALLED=$( AnotB "$ORIGDEVICE" "$REMOVED") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Calculating minimal device packages..."
MINIMUMDEVICE=$( AnotB "$( dependentsof "$DEFAULTS")" "$DEFAULTS" ) && echo "success."

# OUTPUT

function report() { echo -e "\n$(wc -w <<<$2) $1:\n"$2; }
report "INSTALLED PACKAGES" "$MANIFEST"
report "ORIGINAL TARGET PACKAGES" "$ORIGTARGET"
report "ORIGINAL DEVICE PACKAGES" "$ORIGDEVICE"
report "PACKAGES REMOVED BY DEVICE" "$DEVREMOVES"
report "PACKAGES REMOVED BY USER" "$USERREMOVED"
# report "TARGET DEPENDENT INSTALLED PACKAGES" "$TARGETINSTALLED"
# report "DEVIDE DEPENDENT INSTALLED PACKAGES" "$DEVICEINSTALLED"
report "INDEPENDENT INSTALLED PACKAGES (Selected by user)" "$INDEPENDENTS"
report "DEPENDENT INSTALLED PACKAGES (Automatically retrieved)" "$DEPENDENTS"
report "MINIMUM DEVICE PACKAGES" "$MINIMUMDEVICE"

echo -e "\nMINIMAL SET OF DIFF PACKAGES:"
echo "$USERREMOVED" | sed 's/^/-/g'
echo "$INDEPENDENTS"

I still don't know how to manage how to differentiate sourced alternative packages providing certain other packages from independent packages. Any idea over there?

And, returning to the initial idea of getting all this info from a given .config, I think it is possible.

You seem to be using non-standard bash shell for some reason, instead of the normal ash shell (with /bin/sh shebang)

Well, I am using bash and some of its benefits compared to standard sh/ash. Bash offers more flexibility than the minimal implementation of ash in busybox.
Other than that, I am used to bash and zsh. ash has some limitations that would require extra time for me to adapt. In my case, it was a lot faster installing bash and using it for familiar string management. I have plenty of free storage and memory in my devices so bash is ok for starting this. Later on, if it is interesting, it can be properly backported to be compatible with the minimal sh. I actually started in sh, and ditched the use of arrays, but at certain point, I got lost with all the string management and redirection limitations of ash. So I decided to install bash and speedup the process.

In any case, I do not understand what do you mean by "non-standard bash shell" as I am using standard bash. Maybe the formatting is not the best, compact pipelines instead of per line instructions, etc. but for an average eye, I believe the code is simple enough.

That OpenWrt by default uses the busybox ash shell, so scripts using bash extensions are unusable for most OpenWrt users. (I tried your script yesterday, and it naturally failed on line 4)

1 Like

I was aware of that, in the OP I was stressing the need to install bash in order to execute these first versions of the script.
Any help is very welcome.

OK, I understand the need for making it fully compliant with at least the default shell, busybox ash.

So I focused on removing the non compliances with POSIX shell. The result is not only a fully compliant script, but also faster — not much, only 20% faster, from 5 seconds to 4 seconds— and more compact as the code is much simpler.

This is the 4th version of the code you can test in your base OpenWrt:

#!/bin/sh

ADD () { echo -e "$1\n$2" | sort -u ;}
INTERSECT () { echo "$1" |grep -Ex "$2";} 
AnotB () { echo "$1" |grep -Evx "$2" ;}
dependentsof () { echo "$depdb" |grep -E "$( echo $1 |sed 's/^/^/; s/ / |^/g; s/$/ /')" |cut -d ' ' -f 2- |tr ' ' '\n' |awk 'NF' |sort -u ;}
xpandprovided () { echo "$1 "$(echo -e "$provdb" |grep  -Ew "$1") | tr ' ' '\n' |awk 'NF' | sort -u ;}

reset
## ORIGDEVICE: list of device specific packages
echo -n "Downloading list of device specific packages..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
DEVICE=$(cat /etc/build.config | grep -E '^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_' | sed 's/^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_//;s/=y$//')
URL1=https://raw.githubusercontent.com/openwrt/openwrt/master/target/linux/$TARGET/image/$SUBTARGET.mk
ORIGDEVICE=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | sed '/^-/d'|awk 'NF' | sort) && echo "success."

## ORIGTARGET: list of target architecture specific packages
echo -n "Downloading list of target architecture scpecific packages..."
URL2=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
ORIGTARGET=$(wget -q $URL2 -O - | awk '{print $1}' | sort -u) && echo "success."

## DEVREMOVES: list of the packages removed per device
echo -n "Downloading list of default removed target packages..."
DEVREMOVES=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | grep '^-' |cut -c2- |awk 'NF' |sort) && echo "success."

## DEFAULTS: list of installed default packages for the target and device
echo -n "Identifying installed default packages..."
DEFAULTS=$( AnotB "$(ADD "$ORIGTARGET" "$ORIGDEVICE")" "$DEVREMOVES" ) && echo "success."

## MANIFEST: list of installed packages
echo -n "Acquiring local package manifest..."
MANIFEST=$(cat /usr/lib/opkg/status | grep "^Package" | cut -d ' ' -f 2- | sort -u) && echo "success."

## USERREMOVED: list of manually removed default packages
echo -n "Identifying default packages removed by user..."
USERREMOVED=$( AnotB "$DEFAULTS" "$MANIFEST") && echo "success."

## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Retrieving the dependents database..."
depdb=$(cat /usr/lib/opkg/status | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' |sed 's/([^)]*)//g' |tr -s ' '|sort) && echo "success."

## db: contains the OpenWrt package info
echo -n "Retrieving local copy of global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in base core luci packages routing telephony; do
    db=$db$'\n\n'$(gzip -c -d /tmp/opkg-lists/openwrt_$section)
done && echo "success."

## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' |sed 's/([^)]*)//g' | sort) && echo "success."

## DEPENDENTSPOOL: contains the list of all dependent packages, acrhitecture's dependents, and its provided packages.
echo -n "Generating dependents pool..."
DEPENDENTSPOOL=$(xpandprovided "$( ADD "$(dependentsof "$MANIFEST")"  "$DEFAULTS" )" )

## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(AnotB "$MANIFEST" "$DEPENDENTSPOOL") && echo "success."

## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$( AnotB "$MANIFEST" "$( ADD "$INDEPENDENTS" "$DEFAULTS" )" ) && echo "success."

## TARGETREMOVED: contains the list of removed target architecture specific packages
echo -n "Identifying removed target packages..."
TARGETREMOVED=$( AnotB "$ORIGTARGET" "$MANIFEST") && echo "success."

## DEVREMOVED: contains the list of removed device specific packages
echo -n "Identifying removed device packages..."
DEVREMOVED=$( AnotB "$ORIGDEVICE" "$MANIFEST") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
TARGETINSTALLED=$( INTERSECT "$ORIGTARGET" "$MANIFEST") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed device specific packages..."
DEVICEINSTALLED=$( AnotB "$ORIGDEVICE" "$REMOVED") && echo "success."

## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Calculating minimum package set for this device..."
MINIMUMDEVICE=$( AnotB "$DEFAULTS" "$( dependentsof "$DEFAULTS")") && echo "success."

# OUTPUT

function report() { echo -e "\n$(echo "$2" |wc -w) $1:\n"$2; }
report "INSTALLED PACKAGES" "$MANIFEST"
report "ORIGINAL TARGET PACKAGES" "$ORIGTARGET"
report "ORIGINAL DEVICE PACKAGES" "$ORIGDEVICE"
report "PACKAGES REMOVED BY DEVICE REQUIREMENTS" "$DEVREMOVES"
report "PACKAGES MANUALLY REMOVED FROM DEFAULT SET" "$USERREMOVED"
# report "Target installed packages" "$TARGETINSTALLED"
# report "Device installed packages" "$DEVICEINSTALLED"
report "INDEPENDENT INSTALLED PACKAGES (Selected by user)" "$INDEPENDENTS"
report "DEPENDENT INSTALLED PACKAGES (Automatically retrieved)" "$DEPENDENTS"
report "MINIMUM DEVICE PACKAGES" "$MINIMUMDEVICE"

echo -e "\nMINIMAL SET OF DIFF PACKAGES:"
echo "$USERREMOVED" | sed 's/^/-/g'
echo "$INDEPENDENTS"