I am trying to create a simple shell script that could return the list of independent packages (without parent) or not preselected as part of the target.
In order to do so I have created this script that runs in bash in the device (install bash if you want to run it):
#!/bin/bash
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=$(gzip -c -d /tmp/opkg-lists/openwrt_base)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_core)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_luci)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_packages)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_routing)
db+="\n"$(gzip -c -d /tmp/opkg-lists/openwrt_telephony)"\n"
# ldb: contains all the packages in the downloaded database in a separated line
# followed by its dependencies
ldb=$(echo -e "$db" | awk '/Depends:/||/Package:/{print}' | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g;s/([^ ]* //ig' | sort)
# manifest: contains the local list of installed packages, one per line
manifest=$(opkg list-installed | awk '{print $1}' | sort)
dep=
for p in $manifest; do
echo -n "processing $p ..."
line=$(echo -e "$ldb" | grep "^$p ")
[ -n "$line" ] && d=" "$(echo "$line" | cut -d ' ' -f 2-) || d=$(opkg info $p | awk '/Depends:/ { $1=""; print}' | sed 's/,//g')
[ -n "$d" ] && echo "$d" || echo "NOT FOUND *****************"
dep+="$d"
done
# dep: contains all the dependencies of the installed packages.
. /etc/openwrt_release
TARGET=$(echo $DISTRIB_TARGET | awk -F"/" '{print $1}')
SUBT=$(echo $DISTRIB_TARGET | awk -F"/" '{print $2}')
URL=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBT/openwrt-$TARGET-$SUBT.manifest
# targets: contains the list of packages depending on the target architecture
targets=$(wget $URL -O - | awk '{print $1}' | sort -u)
# dept: contains the list of all dependent packages, local dependents and target dependents
dept=$(echo $dep" "$targets | tr ' ' '\n' | sort -u)
_dep_=$(echo " "$dept" " | sed 's/ / \n /g;s/ [ ]*/ /g' | sort -u)
parents=
for p in $manifest; do
if ! grep -q " $p " <<<"$_dep_"; then
parents+=$p" "
fi
done
for p in $targets; do
if ! grep -q "$p" <<<"$manifest"; then
parents+="-"$p" "
fi
done
# parents: contains a list of removed and added packages
# (those independent packages or removed from the target default)
parents=$(echo $parents | tr ' ' '\n' | sort -u)
echo $(echo $manifest | wc -w)" packages installed."
echo $(echo $_dep_ | wc -w)" packages are dependents."
echo $(echo $parents | wc -w)" packages are independents."
echo -e "\nParent Packages:"
echo "$parents"
but....
There are some packages listed as independent that are actually dependents such as libjpeg-turbo that that in my case depends on libgd. But libgd depends on libjpeg, so there is no way to know the actual parent of libjpeg-turbo.
To solve this, I can run it from the OpenWrt kitchen before building the cake (the image) from the final .config
Any suggestion is welcome, such as ideas on how to handle the files under Openwrt/tmp/ to remove default target packages and what final packages I have actually added.
Basically I would be happy if I could get a list of packages to be installed (or already installed) followed by one of these 4 possibilities: target | removed | independent | dependent.
Target: package included as part of the target architecture
Removed: package deselected from the default target
Independent: package not required by any other installed package
Dependent: package required by at least one installed package.
Running my script in the device returns in my case:
You are evaluating the target default packages, but not the device specific packages (like WiFi drivers, firmware, special button drivers) that vary inside a target.
You should probably take the PROVIDES lines into account when evaluating the dependency chain. Libjpeg-turbo provides libjpeg and gets pulled in due to that.
But note that several packages can provide a defined package, and the end result may vary depending on the installation order, so I don't think that there is any clear way to 100% backtrack from .config to the original singular package selections. There will sometimes be corner cases that need to be pruned by hand.
He is trying to figure out which packages have been installed due to being a dependency, and only list independent top selections. (Plus the excluded/removed defaults)
I am only building for a few different devices, so I have created my device recipes (like my R7800 recipe that your config seems to be a derivative of) by manually pruning the diffconfig output. Once you always add the new packages via the recipe, the things stay manageable.
Your script seems to already work pretty ok, but like I said, there will possibly be a few corner cases with more complex dependencies and concurrent variants ( or alternatives) providing a dependency.
@hnyman, where can I retrieve that information from?
Thanks for pointing me to the Provides field. Yes, it is a nice way to share code, but it is really a nightmare from the database point of view. I believe I can retrieve the original package from a built image. I will try it this afternoon.
However, getting that before building the rom.... seems quite difficult because, as you said, there are several packages providing the same package dependency.
Good catch! Actually it is based on a diffconfig file you shared for the E8450. And I am using it also for an R7800.
@anon89577378, I see you thought the same way. Actually that is more or less the first loop in my code. I do all the previous steps —retrieving the opkg database— to speedup the process and relieve the router from un-gzipping hundreds of times the same files, those you download when doing a opkg update and ungzip when you do an opkg info.
You can see the packages in the manifest compiled along your own build, but they are not specified as device packages in any way. Buildbot does not store that anywhere.
And you can see that from the device Makefile (or actually the image recipe) in source code.
Example where also negative exclusions from the target defaults...
I have reworked the code a little bit and now it is way faster and more clear. I have taken into account the provided packages, thanks @hnyman for pointing me in that direction, and the final list resembles a lot more the actual minimal set. @anon89577378, the problem was not getting the local manifest or the list of dependents. That was easy and fast. So far this is the new version of the code with the provided packages included:
## MANIFEST: contains the local list of installed packages, one per line
echo -n "Acquiring local package manifest..."
MANIFEST=$(opkg list-installed | awk '{print $1}' | sort) && echo "success."
## TDEFAULTS: contains the list of the default packages for the target architecture
echo -n "Acquiring default packages for this architecture..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
URL=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
TDEFAULTS=$(wget -q $URL -O - | awk '{print $1}' | sort -u) && echo "success."
## REMOVED: contains the list of the removed default packages
echo -n "Identifying removed default packages..."
REMOVED=$(sed 's|^|/^|; s|$|\$/d|' <<<$MANIFEST | sed -f- <(echo "$TDEFAULTS")) && echo "success."
## DEFAULTS: contains the list of installed default packages
echo -n "Identifying installed default packages..."
DEFAULTS=$(sed 's|^|/^|; s|$|\$/d|' <<<$REMOVED | sed -f- <(echo "$TDEFAULTS")) && echo "success."
## db: contains the OpenWrt package info
echo -n "Building global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in $(echo "base core luci packages routing telephony"); do
db+=$(gzip -c -d /tmp/opkg-lists/openwrt_$section)$'\n\n'
done && echo "success."
## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Simplifying the dependents database..."
depdb=$(echo "$db" | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' | sort) && echo "success."
## DEPENDENTS_STEP1: contains all the dependencies identified in the OpenWrt db of installeed packages
echo -n "Calculating dependents from OpenWrt database..."
DEPENDENTS_STEP1=$(sed 's/^/^/; s/ / |^/g; s/$/ /' <<<$(echo $MANIFEST) | grep -E -f- <(echo "$depdb") | cut -d ' ' -f 2- | tr ' ' '\n' | sort -u) && echo "success."
## EXTRANEOUS: contains packages not listed in OpenWrt package info
echo -n "Identifying local packages not tracked in OpenWrt database..."
EXTRANEOUS=$(sed 's|^|/^|; s|$|\$/d|' <(echo "$db" | awk '/Package: / {print $2}') | sed -f- <(echo "$MANIFEST")) && echo "success."
## DEPENDENTS_STEP2: contains contains all the dependencies from local packages, including extraneous packages
echo -n "Adding dependents of extraneous packages..."
DEPENDENTS_STEP2="$DEPENDENTS_STEP1"
for p in $EXTRANEOUS; do
DEPENDENTS_STEP2+=$(opkg info $p | awk '/Depends: / { $1=""; print}' | sed 's/,//g')
done
DEPENDENTS_STEP2=$(echo "$DEPENDENTS_STEP2" | tr ' ' '\n' | sort -u) && echo "success."
## DEPENDENTS_STEP3: contains all the dependents from local packages and target architecture
echo -n "Adding dependents of target architecture defaults..."
DEPENDENTS_STEP3=$(echo -e "$DEPENDENTS_STEP2\n$DEFAULTS" | sed '/(/d;/)/d' | sort -u) && echo "success."
## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' | sort) && echo "success."
## DEPENDENTS_STEP4: contains the list of all dependent packages, acrhitecture's defaults, their providers, and provided packages.
echo -n "Adding provided packages to dependents..."
DEPENDENTS_STEP4=$(echo "$DEPENDENTS_STEP3$(sed 's/ / | /g; s/^/ /; s/$/ /' <<<$(echo $DEPENDENTS_STEP3) | grep -E -f- <(echo "$provdb" | sed 's/^/ /g;s/$/ /g') | tr ' ' '\n')" | sort -u) && echo "success."
## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(sed 's|^|/^|; s|$|\$/d|' <<<$DEPENDENTS_STEP4 | sed -f- <(echo "$MANIFEST")) && echo "success."
## DEPENDENTS: Contains the list of all local dependent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$(sed 's|^|/^|; s|$|\$/d|' <<< $(echo -e "$INDEPENDENTS\n$DEFAULTS") | sed -f- <(echo "$MANIFEST")) && echo "success."
## OUTPUT
R=$(wc -w <<<$REMOVED)
M=$(wc -w <<<$MANIFEST)
T=$(wc -w <<<$DEFAULTS)
I=$(wc -w <<<$INDEPENDENTS)
D=$(wc -w <<<$DEPENDENTS)
echo -e "\n$M packages installed:\n"$MANIFEST
echo -e "\n$T target default packages:\n"$DEFAULTS
echo -e "\n$R packages removed from defaults:\n"$REMOVED
echo -e "\n$I independent packages:\n"$INDEPENDENTS
echo -e "\n$D dependent packages:\n"$DEPENDENTS
echo -e "\nMinimal set of packages:"
echo "$REMOVED"| sed 's/^/-/g'
echo "$INDEPENDENTS"
As you can see is way faster than the previous version. But I am still facing two issues:
The problem now is detecting packages that are added as part of the device, not the target architecture. Those are now identified as independent. I will dig into it with the info provided by @hnymanhere.
There are some dependencies that are actually replaced from the target default packages that are now detected as dependents. I will have to dig into this too.
Relying on the provided /etc/build.config doesn't seem quite reliable. Do images in the download section also have this file or do only customized images have it?
@anon69880279, thanks for the suggestion, there are many ways to retrieve the local package manifest, but that is always fast. You won't notice the difference between your proposal and, for instance this other: echo $(opkg list-installed | cut -f 1 -d ' ')
But opkg is slow. I am thinking on ditching opkg completely and rely on the opkg status file. For instance the local manifest could be retrieved using: echo $(cat /usr/lib/opkg/status| grep Package |cut -d ' ' -f 2-)
The slower part is etrieving the dependents because at this time, the original data comes from the opkg general database available in /tmp/opkg, the files you download when doing an opkg update. I have removed all but one loop from version 1 to version 2 of the code, but as you can see, it is still quite slow, even for shell script. I will try using that local status file instead to accelerate the process. This way I can even process in the same loop the packages I am now identifying as EXTRANOEUS.
I have done another step forward. In this third version I have added the device specific packets and have improved even more the code for speed. In an R7800 this is the improvement in execution time:
VERSION LINES TIME
------- ----- ----
v1 59 58s
v2 85 21s
v3 107 5s
And here is the v3
#!/bin/bash
function ADD() { echo -e "$1\n$2" | sort -u ;}
function INTERSECT() { sed 's/^/^/; s/ /$|^/g; s/$/$/' <<<$(echo $1) | grep -E -f- <(echo "$2") ;}
function AnotB() { sed 's|^|/^|; s|$|\$/d|' <<<$2 | sed -f- <(echo "$1") ;}
function dependentsof() { sed 's/^/^/; s/ / |^/g; s/$/ /' <<<$(echo $1) | grep -E -f- <(echo "$depdb") | cut -d ' ' -f 2- | tr ' ' '\n' |awk 'NF' | sort -u ;}
reset
## ORIGDEVICE: list of device specific packages
echo -n "Downloading list of device specific packages..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
DEVICE=$(cat /etc/build.config | grep -E '^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_' | sed 's/^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_//;s/=y$//')
URL1=https://raw.githubusercontent.com/openwrt/openwrt/master/target/linux/$TARGET/image/$SUBTARGET.mk
ORIGDEVICE=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | sed '/^-/d'|awk 'NF' | sort) && echo "success."
## ORIGTARGET: list of target architecture specific packages
echo -n "Downloading list of target architecture scpecific packages..."
URL2=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
ORIGTARGET=$(wget -q $URL2 -O - | awk '{print $1}' | sort -u) && echo "success."
## DEVREMOVES: list of the packages removed per device
echo -n "Downloading list of default removed target packages..."
DEVREMOVES=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | grep '^-' |cut -c2- |awk 'NF' |sort) && echo "success."
## DEFAULTS: list of installed default packages for the target and device
echo -n "Identifying installed default packages..."
DEFAULTS=$( AnotB "$(ADD "$ORIGTARGET" "$ORIGDEVICE")" "$DEVREMOVES" ) && echo "success."
## MANIFEST: list of installed packages
echo -n "Acquiring local package manifest..."
MANIFEST=$(cat /usr/lib/opkg/status | grep "^Package" | cut -d ' ' -f 2- | sort -u) && echo "success."
## USERREMOVED: list of manually removed default packages
echo -n "Identifying default packages removed by user..."
USERREMOVED=$( AnotB "$DEFAULTS" "$MANIFEST") && echo "success."
## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Retrieving the dependents database..."
depdb=$(cat /usr/lib/opkg/status | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' |sed 's/([^)]*)//g' |tr -s ' '|sort) && echo "success."
## DEPENDENTS_STEP1: contains all the dependencies identified in the OpenWrt db of installeed and device dependent packages
DEPENDENTS_STEP1=$( ADD "$(dependentsof "$MANIFEST")" "$DEFAULTS" )
## db: contains the OpenWrt package info
echo -n "Retrieving local copy of global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in $(echo "base core luci packages routing telephony"); do
db+=$(gzip -c -d /tmp/opkg-lists/openwrt_$section)$'\n\n'
done && echo "success."
## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' |sed 's/([^)]*)//g' | sort) && echo "success."
## DEPENDENTS_STEP2: contains the list of all dependent packages, acrhitecture's dependents, and its provided packages.
echo -n "Adding provided packages to dependents..."
DEPENDENTS_STEP2=$(ADD "$DEPENDENTS_STEP1" "$(sed 's/ / | /g; s/^/ /; s/$/ /' <<<$(echo $DEPENDENTS_STEP1) | grep -E -f- <(echo "$provdb" | sed 's/^/ /g;s/$/ /g') | tr ' ' '\n')"|awk 'NF') && echo "success."
## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(AnotB "$MANIFEST" "$DEPENDENTS_STEP2") && echo "success."
## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$( AnotB "$MANIFEST" "$( ADD "$INDEPENDENTS" "$DEFAULTS" )" ) && echo "success."
## TARGETREMOVED: contains the list of removed target architecture specific packages
echo -n "Identifying removed target packages..."
TARGETREMOVED=$( AnotB "$ORIGTARGET" "$MANIFEST") && echo "success."
## DEVREMOVED: contains the list of removed device specific packages
echo -n "Identifying removed device packages..."
DEVREMOVED=$( AnotB "$ORIGDEVICE" "$MANIFEST") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
TARGETINSTALLED=$( INTERSECT "$ORIGTARGET" "$MANIFEST") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
DEVICEINSTALLED=$( AnotB "$ORIGDEVICE" "$REMOVED") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Calculating minimal device packages..."
MINIMUMDEVICE=$( AnotB "$( dependentsof "$DEFAULTS")" "$DEFAULTS" ) && echo "success."
# OUTPUT
function report() { echo -e "\n$(wc -w <<<$2) $1:\n"$2; }
report "INSTALLED PACKAGES" "$MANIFEST"
report "ORIGINAL TARGET PACKAGES" "$ORIGTARGET"
report "ORIGINAL DEVICE PACKAGES" "$ORIGDEVICE"
report "PACKAGES REMOVED BY DEVICE" "$DEVREMOVES"
report "PACKAGES REMOVED BY USER" "$USERREMOVED"
# report "TARGET DEPENDENT INSTALLED PACKAGES" "$TARGETINSTALLED"
# report "DEVIDE DEPENDENT INSTALLED PACKAGES" "$DEVICEINSTALLED"
report "INDEPENDENT INSTALLED PACKAGES (Selected by user)" "$INDEPENDENTS"
report "DEPENDENT INSTALLED PACKAGES (Automatically retrieved)" "$DEPENDENTS"
report "MINIMUM DEVICE PACKAGES" "$MINIMUMDEVICE"
echo -e "\nMINIMAL SET OF DIFF PACKAGES:"
echo "$USERREMOVED" | sed 's/^/-/g'
echo "$INDEPENDENTS"
I still don't know how to manage how to differentiate sourced alternative packages providing certain other packages from independent packages. Any idea over there?
And, returning to the initial idea of getting all this info from a given .config, I think it is possible.
Well, I am using bash and some of its benefits compared to standard sh/ash. Bash offers more flexibility than the minimal implementation of ash in busybox.
Other than that, I am used to bash and zsh. ash has some limitations that would require extra time for me to adapt. In my case, it was a lot faster installing bash and using it for familiar string management. I have plenty of free storage and memory in my devices so bash is ok for starting this. Later on, if it is interesting, it can be properly backported to be compatible with the minimal sh. I actually started in sh, and ditched the use of arrays, but at certain point, I got lost with all the string management and redirection limitations of ash. So I decided to install bash and speedup the process.
In any case, I do not understand what do you mean by "non-standard bash shell" as I am using standard bash. Maybe the formatting is not the best, compact pipelines instead of per line instructions, etc. but for an average eye, I believe the code is simple enough.
That OpenWrt by default uses the busybox ash shell, so scripts using bash extensions are unusable for most OpenWrt users. (I tried your script yesterday, and it naturally failed on line 4)
I was aware of that, in the OP I was stressing the need to install bash in order to execute these first versions of the script.
Any help is very welcome.
OK, I understand the need for making it fully compliant with at least the default shell, busybox ash.
So I focused on removing the non compliances with POSIX shell. The result is not only a fully compliant script, but also faster — not much, only 20% faster, from 5 seconds to 4 seconds— and more compact as the code is much simpler.
This is the 4th version of the code you can test in your base OpenWrt:
#!/bin/sh
ADD () { echo -e "$1\n$2" | sort -u ;}
INTERSECT () { echo "$1" |grep -Ex "$2";}
AnotB () { echo "$1" |grep -Evx "$2" ;}
dependentsof () { echo "$depdb" |grep -E "$( echo $1 |sed 's/^/^/; s/ / |^/g; s/$/ /')" |cut -d ' ' -f 2- |tr ' ' '\n' |awk 'NF' |sort -u ;}
xpandprovided () { echo "$1 "$(echo -e "$provdb" |grep -Ew "$1") | tr ' ' '\n' |awk 'NF' | sort -u ;}
reset
## ORIGDEVICE: list of device specific packages
echo -n "Downloading list of device specific packages..."
. /etc/openwrt_release
TARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $1}')
SUBTARGET=$(echo "$DISTRIB_TARGET" | awk -F"/" '{print $2}')
DEVICE=$(cat /etc/build.config | grep -E '^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_' | sed 's/^CONFIG_TARGET_'$TARGET'_'$SUBTARGET'_DEVICE_//;s/=y$//')
URL1=https://raw.githubusercontent.com/openwrt/openwrt/master/target/linux/$TARGET/image/$SUBTARGET.mk
ORIGDEVICE=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | sed '/^-/d'|awk 'NF' | sort) && echo "success."
## ORIGTARGET: list of target architecture specific packages
echo -n "Downloading list of target architecture scpecific packages..."
URL2=https://downloads.openwrt.org/snapshots/targets/$TARGET/$SUBTARGET/openwrt-$TARGET-$SUBTARGET.manifest
ORIGTARGET=$(wget -q $URL2 -O - | awk '{print $1}' | sort -u) && echo "success."
## DEVREMOVES: list of the packages removed per device
echo -n "Downloading list of default removed target packages..."
DEVREMOVES=$(wget -q $URL1 -O - | sed ':a;N;$!ba;s/\\\n//g;s/\t//g' | grep -E '^define Device/|DEVICE_PACKAGES :=' | sed 's/DEVICE_PACKAGES := //g' | tr '\n' ' ' | tr -s ' ' | sed 's|define Device/|\n|g;s/ &//g' | grep "^$DEVICE " | cut -d ' ' -f 2- | tr ' ' '\n' | grep '^-' |cut -c2- |awk 'NF' |sort) && echo "success."
## DEFAULTS: list of installed default packages for the target and device
echo -n "Identifying installed default packages..."
DEFAULTS=$( AnotB "$(ADD "$ORIGTARGET" "$ORIGDEVICE")" "$DEVREMOVES" ) && echo "success."
## MANIFEST: list of installed packages
echo -n "Acquiring local package manifest..."
MANIFEST=$(cat /usr/lib/opkg/status | grep "^Package" | cut -d ' ' -f 2- | sort -u) && echo "success."
## USERREMOVED: list of manually removed default packages
echo -n "Identifying default packages removed by user..."
USERREMOVED=$( AnotB "$DEFAULTS" "$MANIFEST") && echo "success."
## depdb: contains all the packages in the downloaded database followed by its dependencies
echo -n "Retrieving the dependents database..."
depdb=$(cat /usr/lib/opkg/status | grep -E "^Package:|^Depends:" | tr '\n' ' ' | sed 's/Package: /\n/g;s/Depends: //g;s/,//g' |sed 's/([^)]*)//g' |tr -s ' '|sort) && echo "success."
## db: contains the OpenWrt package info
echo -n "Retrieving local copy of global package database..."
[ -e "/tmp/opkg-lists/openwrt_base" ] || opkg update
db=
for section in base core luci packages routing telephony; do
db=$db$'\n\n'$(gzip -c -d /tmp/opkg-lists/openwrt_$section)
done && echo "success."
## provdb: each line contains a package in the OpenWrt database followed by its provided packages
echo -n "Simplifying the provided packages database..."
provdb=$(echo -e "$db" | grep -E "^Package:|^Provides:" | tr '\n' ' ' | sed 's/Package: /\n/g' | grep 'Provides:' | sed 's/Provides: //g;s/,//g' |sed 's/([^)]*)//g' | sort) && echo "success."
## DEPENDENTSPOOL: contains the list of all dependent packages, acrhitecture's dependents, and its provided packages.
echo -n "Generating dependents pool..."
DEPENDENTSPOOL=$(xpandprovided "$( ADD "$(dependentsof "$MANIFEST")" "$DEFAULTS" )" )
## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
INDEPENDENTS=$(AnotB "$MANIFEST" "$DEPENDENTSPOOL") && echo "success."
## INDEPENDENTS: Contains the list of all local independent packages
echo -n "Adding provided packages to dependents..."
DEPENDENTS=$( AnotB "$MANIFEST" "$( ADD "$INDEPENDENTS" "$DEFAULTS" )" ) && echo "success."
## TARGETREMOVED: contains the list of removed target architecture specific packages
echo -n "Identifying removed target packages..."
TARGETREMOVED=$( AnotB "$ORIGTARGET" "$MANIFEST") && echo "success."
## DEVREMOVED: contains the list of removed device specific packages
echo -n "Identifying removed device packages..."
DEVREMOVED=$( AnotB "$ORIGDEVICE" "$MANIFEST") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed target architecture specific packages..."
TARGETINSTALLED=$( INTERSECT "$ORIGTARGET" "$MANIFEST") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Identifying installed device specific packages..."
DEVICEINSTALLED=$( AnotB "$ORIGDEVICE" "$REMOVED") && echo "success."
## TARGETINSTALLED: contains the list of installed target architecture specific packages
echo -n "Calculating minimum package set for this device..."
MINIMUMDEVICE=$( AnotB "$DEFAULTS" "$( dependentsof "$DEFAULTS")") && echo "success."
# OUTPUT
function report() { echo -e "\n$(echo "$2" |wc -w) $1:\n"$2; }
report "INSTALLED PACKAGES" "$MANIFEST"
report "ORIGINAL TARGET PACKAGES" "$ORIGTARGET"
report "ORIGINAL DEVICE PACKAGES" "$ORIGDEVICE"
report "PACKAGES REMOVED BY DEVICE REQUIREMENTS" "$DEVREMOVES"
report "PACKAGES MANUALLY REMOVED FROM DEFAULT SET" "$USERREMOVED"
# report "Target installed packages" "$TARGETINSTALLED"
# report "Device installed packages" "$DEVICEINSTALLED"
report "INDEPENDENT INSTALLED PACKAGES (Selected by user)" "$INDEPENDENTS"
report "DEPENDENT INSTALLED PACKAGES (Automatically retrieved)" "$DEPENDENTS"
report "MINIMUM DEVICE PACKAGES" "$MINIMUMDEVICE"
echo -e "\nMINIMAL SET OF DIFF PACKAGES:"
echo "$USERREMOVED" | sed 's/^/-/g'
echo "$INDEPENDENTS"