Custom group of packages for offline installation?

I know about the custom firmware builder and all but wondering if there's a way of picking some packages to group together that would also include any and all dependencies into one bigger file.

Also, where can I find a list of default packages for a specific router? I know about the manifest file but isn't that generic for the all the routers that the firmware line covers? And for the custom firmware creator, if you pick a package, does it automatically include dependencies or are you expected to know and add them on your own?

In addition to the marked solution, I want to give credit to these two posts as well:

Yes

Do you mean a bigger ipk? Or baked into the firmware bin file? The firmware selector will bake all the packages into the bin file.

Not being a ipk guru, I can only assume that it can only be a single contained package and not an archive of packages that all get installed. So to answer the question, more like a zip/gzip file that can be extracted so that the individual packages can be installed locally (including the dependencies) without needing an internet connection.

If you goto

https://firmware-selector.openwrt.org/

Find your hardware, then click

Customize installed packages and/or first boot script

You can type the name of the packages you want to bundle into the firmware image. When you flash this image onto your router/hardware, it will have all the packages you've specified including dependencies. No need for an internet connection.

I'm talking about a separate bundle that isn't a firmware package though. Sort of like a firmware file to flash, but more for just installing extra packages after a flash.

If I understand correctly, the following might help.

From directory /etc/opkg/ on my device, I see distfeeds.conf file which lists the source for the package groups ( base, routing, target specific etc ).

In each of those directories on the downloads.openwrt.org server is a file named Packages.
Those files list packages and info about each including a list of dependencies. You can go through the files and make a list of the packages you want and their dependencies and then iteratively look at the package info of dependencies for sub dependencies to make your aggregate list.

It worked for me.

The opkg command has search tools built in to do this as well. You just need to fetch the package lists from the download site with opkg update and then search away. See opkg without any arguments for more info.
The package lists are put in /var/opkg-lists/ which is the same as /tmp/ so they disappear on a reboot and need to be fetched again.

I suppose I could just run opkg with the command to not actually do anything (just 'pretend' to) with a list of desired packages and see what all it shows it would be installing in addition to those packages. Then I would have a list I could use to download all the files and zip them up myself. :grin:

Smart idea, thanks.

Can be useful if I want certain packages to be available to a router that won't have internet access right away, or that I connect long enough to do the above (lol) when wanting to do test installs/settings, making it easier to do with less effort.

What about being able to get a list of packages that come in the firmware package for a particular router?
Wait, I went looking and maybe I figured it out...

Would I take the list from the manifest file (openwrt_version_target_subtarget.manifest) and then look at the profiles.json file to get router specific files to add to the list in order to know everything that is included in the premade firmware files?

I don't know about the manifest and profiles.json files.
I recall going through the data in
/usr/lib/opkg/status
and
/usr/lib/opkg/info/*
on a couple of my device types and the packages with the same date as a few known to be in the pre-made distribution all seemed be just that.

The packages with a different date were the ones I added after, or their dependencies.

I cobbled together a script to parse it all but my goal was to get a list without dependencies since opkg determines dependencies and installs them automatically.

Attended sysupgrade is much easier, though I think I did have to do a diff with the list from "Customize installed packages and/or first boot script" on the firmware selector site.

Sorry, that was a year ago or so.

I just did a manual comparison of packages from the default "status" file to those in the ".manifest" list as well as the "profiles.json" file (looking for my router and seeing what extra packages were mentioned" and it matched up. I've noticed that when using the firmware builder (unless I did something wrong), it doesn't install all the same packages that come in a premade firmware package file. So knowing how to put together a list to get the same packages (and then be able to tweak from there) is quite handy. Replace dnsmasq with dnsmasq-full, swap out the basic SSL package with a full package, add in a preferred theme, obviously the unique packages for the device, as well as any extras that you know you'll want. Like certain "luci-app-*" files (ttyd and advanced-reboot for example) and extra commands (less, htop, etc.).

Your script would also be useful to parse through the default list just to get a list of the main packages (and know which are only dependencies). Like kernel and libc are dependencies, but busybox and dropbear aren't.

1 Like

It has been mentioned that there is only a little difference in packages installed by the firmware builder compared to the regular builds. Specifically, luci or luci-ssl need to be added manually in order to have the package list match. Adding either "luci" package also adds quite a few dependencies.

I invested several hours today working on a new script to try to do what you want. It has not gone as well as I hoped. I haven't given up yet though.

Give up, it can't be done. :grin:

Just kidding.

I just tried it, where only the base packages (those that aren't a dependency to other packages) were added to the firmware builder, along with wanted extras (again, only the base ones that aren't dependents). Compared to an earlier custom build where the complete list was used and the only difference was one package that I forgot to remove from the first custom build. In other words, only difference was a mistake on my part.

Did this using the snapshot for the Linksys WRT1200AC (so LuCI isn't auto-included). I'm counting 25 core packages in order to get the firmware version of the package (specific to the router), and to include LuCI only requires including the luci-ssl package. The rest of the packages are included via dependencies.

1 Like

On two of my devices, the package list on the custom area of the firmware-selector site (same as what you are calling firmware builder I think) matches the manifest and json files you pointed out to me. Thanks for that by the way.

I hacked this script together to determine what depends on each package provided in a list.
This process depends on packages being installed but not having a current opkg update list.

Create the input list with one entry per line in file /tmp/my-wd-input.
The input file can have other data per line following the package name like from the output of opkg list-installed. (space separated fields.)

The script:

for i in $( cat /tmp/my-wd-input | cut -d\  -f 1 | sort -u )
  do 
    echo $i
    k=$( opkg whatdepends $i | grep  "^\t" | sed 's/^\t//g' | cut -d\  -f1 | sed 's/\n/ /g' )
    j=$( echo "$k" | wc -w )
    printf "$j\t$i\t $( echo $k )\n" >>/tmp/my-wd-output
    printf "$j\t$i\t $( echo $k )\n"
  done
echo
echo Done

The output goes to /tmp/my-wd-output .
Move the output to a new location before running a new data set or you will get a larger combined output data set.

The form of the output is:
F1=count F2=package_name F3 is a space separated list of what installed packages depend on the searched-for package.
count package-name what depends list (space separated)
Sample output snippet:

0	ath10k-firmware-qca9888-ct	 
0	base-files	 
0	busybox	 
0	ca-bundle	 
0	dnsmasq	 
0	dropbear	 
1	fstools	 base-files
3	kmod-nft-offload	 firewall4 luci-app-firewall luci
1	libustream-wolfssl	 luci-ssl
1	netifd	 base-files
3	nftables	 firewall4 luci-app-firewall luci
2	opkg	 luci-app-opkg luci
3	uclient-fetch	 opkg luci-app-opkg luci

The output can be sorted, grepped and cut to provide interesting sub lists.
For instance, to get the list of what you call base packages, run something like this (assuming the output is still at /tmp/my-wd-output).
cat /tmp/my-wd-output | grep "^0" |cut -f 2
to get a list of packages that nothing depends on, strpped of the number, tabs, and what-depends lists.
To get a space separated list (like for custom firmware selector) convert the new lines to spaces
with |tr '\n' ' '
cat /tmp/my-wd-out | grep "^0" |cut -f 2 | tr '\n' ' '

I hope this is useful!

In a few days I may get back to the script to list all the dependencies and sub-dependencies given a list of packages. Usefull for your idea of local installation on an off-line OpenWrt device.

One more for today.
This script should list the packages installed since the image installation/upgrade:

#!/bin/sh
#
out_fqn=$(mktemp)
  install_time=`opkg status kernel | awk '$1 == "Installed-Time:" { print $2 }'`
  opkg status | awk '$1 == "Package:" {package = $2} \
  $1 == "Status:" { user_inst = / user/ && / installed/ } \
  $1 == "Installed-Time:" && $2 != '$install_time' && user_inst { print package }' | \
  sort >> $out_fqn 2>> $out_fqn

cat $out_fqn

I called mine getuserinstalled.sh
Manually remove the temp file or modify the script to do it.

How about this little tweak so that you can specify the input filename and it will output to the same name except adding "output" and the current date/time to it, so no need to delete files?

Also sorting the file so that it's in order from least to most dependencies?

And if there is no input name specified, generate a list of installed packages and use that?

time=$(date +'%Y%m%d_%H%M%S')
opkg="~/installed_${time}"
file=$(echo $1 $opkg|cut -d' ' -f 1)
temp="/tmp/temp_${time}"
for i in $( cat $file | cut -d\  -f 1 | sort -u )
  do 
    echo $i
    k=$( opkg whatdepends $i | grep  "^\t" | sed 's/^\t//g' | cut -d\  -f1 | sed 's/\n/ /g' )
    j=$( echo "$k" | wc -w )
    printf "$j\t$i\t $( echo $k )\n" | tee -a $temp
  done
sort -h $temp > "${file}_output_${time}"
rm -f $temp $opkg
echo
echo Done
1 Like

Nice!
I thought I'd provide the "bones" of a script and let people adapt it to their own use.

May as well put
#!/bin/sh
at the top, save it to file. chmod +x the file and have a callable script file to use. :-)

Couple of more tweaks...

Displays the current package, then clears the line and overwrites with the output, so instead of name, then next line having count/name/dependencies, it'll re-use the name line. Excuse the crude implementation of clearing the line, I'm not very fluent with linux scripting. Scratch that, I googled it and found a better way.

#!/bin/sh
#Script name: depends.sh (or adult-diapers.sh)

time=$(date +'%Y%m%d_%H%M%S')
opkg=~/installed_${time}
opkg list-installed >>$opkg
file=$(echo $1 $opkg|cut -d' ' -f 1)
temp="/tmp/temp_${time}"
for i in $( cat $file | cut -d\  -f 1 | sort -u )
  do 
    echo -ne $i \\r
    k=$( opkg whatdepends $i | grep  "^\t" | sed 's/^\t//g' | cut -d\  -f1 | sort -h | sed 's/\n/ /g' )
    j=$( echo "$k" | wc -w )
    echo -ne \\r\\003[K
    printf "$j\t$i\t $( echo $k )\n" | tee -a $temp
  done
sort -h $temp > "${file}_output_${time}"
rm -f $temp $opkg
echo -e \\nDone

1 Like

Any chance you could whip up something that would take the "opkg status" output and convert it into a package_name \t dependents list?

The from that, reverse sort (so similar names have the longest name first), find lines that have packages with it as a dependent, and removing itself from the list of that depends (so similar shorter names won't accidentally match it). Provide the same count, name, list of packages that need it (similar to what you did and I tweaked). Would that happen to be any faster?

It seems like opkg whatdepends takes while each time and thinking that it might be faster to generate a simple package \t depends list, reverse sort, then have it go line by line finding lines that need it. Or if there is a way to have it look at each dependency for each package and add to each package count that way (like in an array of some kind).

What about situations when two or more packages have the same dependency?

Do a fresh install on the router in question and run:

opkg list-installed

I might. :-) ...but I'm sure it will take me a while.

By the way -
" opkg status " is the contents of /usr/lib/opkg/status .
The .control files in /usr/lib/opkg/info/*.control is the same info with data for each package broken out into a separate file. Only "installed" packages though.
" opkg depends " and " opkg whatdepends " both take lists of packages but they don't iterate/loop like my script.

I wound up working on it today.

The screen output has the first level dependency list for each item in the input list and then it loops through the items in the sub-dependency list and adds each "new" item to the depsList and only looks into sub-dependencies if not seen yet. The individual item deps lists only go to the screen.
With tabs between fields and the deps list field being space separated, the list pastes into a spreadsheet nicely.
The first number is the sub-loop number or sub-dependency level. The second number is the count of direct dependencies for that package as listed. Both numbers are for debugging but interesting to me.

The space separated "depsList" (shown at the bottom) is accumulated in a variable and can be written to a file.

Here is a sample of the output using luci-ssl for input:

Edit: I added a copy of the depsList for package "luci-ssl" here at the top for reference:

depsList:
luci-ssl libc luci libustream-wolfssl20201210 px5g-wolfssl libgcc uhttpd uhttpd-mod-ubus luci-mod-admin-full luci-theme-bootstrap luci-app-firewall luci-app-opkg luci-proto-ppp libiwinfo-lua luci-proto-ipv6 rpcd-mod-rrdns libubox20220515 libwolfssl5.5.4.ee39414e libwolfssl5.5.4.ee39414e libblobmsg-json20220515 libjson-script20220515 libjson-c5 libubus20220601 luci-base luci-mod-status luci-mod-system luci-mod-network uci-firewall libiwinfo20210430 liblua5.1.5 rpcd luci-lib-nixio luci-lib-ip libubus-lua luci-lib-jsonc liblucihttp-lua luci-lib-base rpcd-mod-file rpcd-mod-luci cgi-io rpcd-mod-iwinfo libnl-tiny1 libuci20130104 libiwinfo-data liblucihttp0 libiwinfo

...and sorted:

cgi-io libblobmsg-json20220515 libc libgcc libiwinfo libiwinfo-data libiwinfo-lua libiwinfo20210430 libjson-c5 libjson-script20220515 liblua5.1.5 liblucihttp-lua liblucihttp0 libnl-tiny1 libubox20220515 libubus-lua libubus20220601 libuci20130104 libustream-wolfssl20201210 libwolfssl5.5.4.ee39414e luci luci-app-firewall luci-app-opkg luci-base luci-lib-base luci-lib-ip luci-lib-jsonc luci-lib-nixio luci-mod-admin-full luci-mod-network luci-mod-status luci-mod-system luci-proto-ipv6 luci-proto-ppp luci-theme-bootstrap px5g-wolfssl rpcd rpcd-mod-file rpcd-mod-iwinfo rpcd-mod-luci rpcd-mod-rrdns uci-firewall uhttpd uhttpd-mod-ubus

.
.
Original output:

inlist:luci-ssl

luci-ssl	0	4	libc luci libustream-wolfssl20201210 px5g-wolfssl 

libc	1	1	libgcc 
luci	1	11	libc uhttpd uhttpd-mod-ubus luci-mod-admin-full luci-theme-bootstrap luci-app-firewall luci-app-opkg luci-proto-ppp libiwinfo-lua luci-proto-ipv6 rpcd-mod-rrdns 
libustream-wolfssl20201210	1	3	libc libubox20220515 libwolfssl5.5.4.ee39414e 
px5g-wolfssl	1	2	libc libwolfssl5.5.4.ee39414e 
libgcc	2	0	
uhttpd	2	5	libc libubox20220515 libblobmsg-json20220515 libjson-script20220515 libjson-c5 
uhttpd-mod-ubus	2	4	libc uhttpd libubus20220601 libblobmsg-json20220515 
luci-mod-admin-full	2	5	libc luci-base luci-mod-status luci-mod-system luci-mod-network 
luci-theme-bootstrap	2	1	libc 
luci-app-firewall	2	2	libc uci-firewall 
luci-app-opkg	2	2	libc opkg 
luci-proto-ppp	2	1	libc 
libiwinfo-lua	2	3	libc libiwinfo20210430 liblua5.1.5 
luci-proto-ipv6	2	1	libc 
rpcd-mod-rrdns	2	4	libc rpcd libubox20220515 libubus20220601 
libubox20220515	2	1	libc 
libwolfssl5.5.4.ee39414e	2	1	libc 
libwolfssl5.5.4.ee39414e	2	1	libc 
libblobmsg-json20220515	3	3	libc libjson-c5 libubox20220515 
libjson-script20220515	3	2	libc libubox20220515 
libjson-c5	3	1	libc 
libubus20220601	3	2	libc libubox20220515 
luci-base	3	12	libc lua luci-lib-nixio luci-lib-ip rpcd libubus-lua luci-lib-jsonc liblucihttp-lua luci-lib-base rpcd-mod-file rpcd-mod-luci cgi-io 
luci-mod-status	3	4	libc luci-base libiwinfo20210430 libiwinfo-lua 
luci-mod-system	3	2	libc luci-base 
luci-mod-network	3	4	libc luci-base libiwinfo-lua rpcd-mod-iwinfo 
uci-firewall	3	0	
libiwinfo20210430	3	5	libc libnl-tiny1 libuci20130104 libubus20220601 libiwinfo-data 
liblua5.1.5	3	1	libc 
rpcd	3	6	libc libubus20220601 libubox20220515 libuci20130104 libblobmsg-json20220515 libjson-c5 
luci-lib-nixio	4	2	libc liblua5.1.5 
luci-lib-ip	4	3	libc liblua5.1.5 libnl-tiny1 
libubus-lua	4	3	libc libubus20220601 liblua5.1.5 
luci-lib-jsonc	4	3	libc liblua5.1.5 libjson-c5 
liblucihttp-lua	4	3	libc liblucihttp0 liblua5.1.5 
luci-lib-base	4	6	libc lua luci-lib-nixio luci-lib-ip luci-lib-jsonc liblucihttp-lua 
rpcd-mod-file	4	4	libc libubus20220601 libubox20220515 rpcd 
rpcd-mod-luci	4	5	libc rpcd libubox20220515 libubus20220601 libnl-tiny1 
cgi-io	4	3	libc libubox20220515 libubus20220601 
rpcd-mod-iwinfo	4	8	libiwinfo (>= 2022-12-15) libc libubus20220601 libubox20220515 rpcd libiwinfo20210430 
libnl-tiny1	4	1	libc 
libuci20130104	4	2	libc libubox20220515 
libiwinfo-data	4	1	libc 
liblucihttp0	5	1	libc 
libiwinfo	5	0	

depsList:
luci-ssl libc luci libustream-wolfssl20201210 px5g-wolfssl libgcc uhttpd uhttpd-mod-ubus luci-mod-admin-full luci-theme-bootstrap luci-app-firewall luci-app-opkg luci-proto-ppp libiwinfo-lua luci-proto-ipv6 rpcd-mod-rrdns libubox20220515 libwolfssl5.5.4.ee39414e libwolfssl5.5.4.ee39414e libblobmsg-json20220515 libjson-script20220515 libjson-c5 libubus20220601 luci-base luci-mod-status luci-mod-system luci-mod-network uci-firewall libiwinfo20210430 liblua5.1.5 rpcd luci-lib-nixio luci-lib-ip libubus-lua luci-lib-jsonc liblucihttp-lua luci-lib-base rpcd-mod-file rpcd-mod-luci cgi-io rpcd-mod-iwinfo libnl-tiny1 libuci20130104 libiwinfo-data liblucihttp0 libiwinfo

Only one duplicate and I'm baffled why it shows up. Oooh! I just had an idea how to fix it...

The idea would be that all calculated dependencies would get included, not each package with all of its dependencies with no regards to everything else.

Think of it as taking all the wanted packages, calculating all of their dependencies, grabbing those, then going through the list of dependencies for that group and continuing on until all needed files are included. Or to simplify it, if a dependency is already included, don't try to include it again.

Actually I found the answer I was looking for. In the (version)/targets/(target)/(subtarget) download folder, there is a file called "profiles.json" which lists all the default packages for each of the supported devices, and then a list of additional files for the specific devices.

For example, in 22.03.2/targets/ramips/mt7621/, the profiles.json file has 30 default packages, then for a router I have (Linksys EA8100v2), there is another 4 packages listed. So combine those two lists and you have the default list for that router.

Yeah, but I figure that it'd be smarter to make opkg do the work of providing the list, instead of risking the possibility of the folder structure changing and being moved somewhere else. Since opkg will always know where it is (built-in), easier to just call on it.

Perhaps a bunch of temporary files could be made inside of a temporary folder, where the filename would be the package name (or a simple hash of it), with a list of packages that depend on it, one to a line. After the primary list is done (and a simple list of package names is left), it could then go through each time, combine all the lines into one (if any), then do the count \t name \t dependencies list that way. Might be faster since it would strip down to only the needed information up front and then only process that.