Detecting user installed pkgs

Thanks for the reply @spence - what you describe is how I understand the custom firmware image builder to work. Given an existing installation, gather up all of the 'opkg' modules installed, and create me a new firmware with the latest version of these (say at level 23.03.05) and deliver that to me.

However, this doesn't solve my problem.

When 24.xx.xx is available, the base modules will have (likely) changed. Plus, there isn't as far as I can tell a way to use the Attended Sysupgrade process to move up major versions.

This puts me in a situation where I do not have a way to inspect the current install, and figure out which 'user installed' packages I want to overlay on top of the base install for 24.xx.xx.

Is there a way for me to take a 'stock' firmware image and enumerate the packages in it?

Let me outline a specific scenario.

  1. I start by installing an Archer C7 v5 with the 21.02.04 image https://downloads.openwrt.org/releases/21.02.4/targets/ath79/generic/openwrt-21.02.4-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin
  2. I install some user packages (prometheus, sqm, vnstat)
  3. I then (later) use the Attended Sysupgrade to update that device to 21.02.07
  4. I now want to perform an upgrade to 22.03.05 - but alas, I no longer remember what packaged specifically I installed in step 2 - but I want to have the same function with the new version.

If you imagine months passing between each of the steps above, you can see that I've got a problem. If I had avoided using the Attended Sysupgrade (3) and instead prior to upgrading figured out the user installed packages, then the user installed packaged would always have a different install time.

I suppose, with enough spare hardware - I could install the 21.02.07 image https://downloads.openwrt.org/releases/21.02.7/targets/ath79/generic/openwrt-21.02.7-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin - then dump the list of installed packages, and use a diff between that and the list that installed on my production 21.02.07 version -- since the delta would be the packages that I'd installed beyond the standard.

I'm wondering if there is a way to do this without needing spare hardware.

I guess, I could with more downtime - first gather the list of packages on my attended sysupgrade 21.02.07 device - flash the 'stock' 21.02.07 firmware, then gather the list of packages - then flash the 22.03.05 upgraded version and apply the different packages between the upgraded 21.02.07 and the stock one.. but what a pain that would be.

Is there a package manifest for the https://firmware-selector.openwrt.org firmwares? That's really what I'm trying to find.. but would like to know it without installing the firmware.

A short answer is that the list of packages is listed on the firmware selector. Click on the triangle next to " Customize installed packages and/or first boot script" to show the list. It is just missing the 2 packages for luci.
Compare the list between your old version and the new one you are upgrading to. That should reveal any differences in the base package set, except for dependencies.

The manifest file for the standard build for the C7 v5 is at https://downloads.openwrt.org/releases/21.02.4/targets/ath79/generic/openwrt-21.02.4-ath79-generic.manifest but I'm not sure that is exact for all sub-targets. Hopefully someone else can clarify. Change the path to be the right version you want to compare the second version with.
I'm unaware of a manifest file that includes dependencies but I'm just a user and not a dev.

Sorry, I'm short on time right now.

1 Like

The general problem with auc/attendedsysupgrade is that dependencies change over time, and it is easy to accumulate unnecessary crust via old dependencies. Also dependencies of "user-installed" packages may change, not only those in the core selection.

The problem with all manifests (and .config files) is that there is no data regarding why a package was included in a build: was it an intentional user selection or an automatic dependency. Additionally, dependencies are left installed if you remove a package. So, if you try package A that requires B, and later remove A, you will still have B installed.

Thus the viable clean alternative in the long run is related to the "step 2" above: Keeping list of the intentionally added "top-level" packages like luci-app-sqm, prometheus etc. (No need to keep list on the dependencies even if they seem user installed.)

Using a .config recipe works well for all the build options:

make image \
 PROFILE="dynalink_dl-wrx36" \
 PACKAGES="ccrypt diffutils gdbserver htop irqbalance mtr-nojson nano-full \
  openssh-sftp-server patch tcpdump-mini tree wget-ssl \
  block-mount kmod-usb-storage kmod-fs-cifs kmod-fs-exfat libblkid \
  kmod-fs-ext4 kmod-fs-msdos kmod-fs-ntfs3 kmod-nls-cp437 kmod-nls-iso8859-1 \
  kmod-nls-utf8 hostapd-utils wpad-openssl ca-certificates \
  luci-ssl-openssl \
  luci-app-adblock luci-app-banip luci-app-bcp38 luci-app-commands \
  luci-app-nlbwmon luci-app-opkg luci-app-sqm luci-app-uhttpd \
  luci-app-statistics collectd-mod-conntrack collectd-mod-cpufreq \
  collectd-mod-ping collectd-mod-thermal collectd-mod-uptime \
  kmod-tun luci-proto-wireguard unetd unet-cli \
  iptables-nft ip6tables-nft ipset \
  qosify kmod-sched-bpf \
  -wpad-basic-mbedtls -libustream-mbedtls -libmbedtls" \
 FILES="../files"
  • firmware selector: keep a short text file on your PC with the package selection. Keep it updated and apply it as modification when using firmware selector
2 Likes

Using a script like the one I shared in post 12 of this thread Custom group of packages for offline installation? - #12 by spence should do the hard work of what you want.

I should have time tonight or tomorrow to add specifics to get the list of installed packages but it is basically taking the list items produced from my script referenced above that have zero dependencies and doing a diff with the list of default packages. It uses a couple temp files.

I think this may be a reasonable solution for you:

Please see the new EDIT 2023-10-16 sections for an addition that my make this work fully.

Here is a solution for when you need to determine the list of installed packages beyond the default packages and not including packages that are just dependencies and you used the firmware builder etc. Rather than just provide a manual recipe to use the script I referenced a couple days ago, I modified it to minimize manual work.

You need to provide the list of default packages. That can easily be obtained from the list provided in the firmware selector web site in the section called "Customize installed packages and/or first boot script" or from the default_packages section of the profiles.json file for your target from the downloads.openwrt.org site.
Put the list, one name per line in file /tmp/my-def-pkgs .

Copy the following script to your openwrt device, possibly in /tmp/ or /root/ , set it as executable (chmod + x) and run it.

EDIT 2023-10-16(1) - added a proposed file name in the script text.

#!/bin/sh

# proposed script file name: listUserInstalled.sh
# added 2023-10-16

if test ! -s /tmp/my-def-pkgs 
 then 
   echo " /tmp/my-def-pkgs was not found or is empty."
   echo " Please add list of default packages there and re-run this script."
   echo
   exit
 fi

opkg list-installed >/tmp/my-wd-input

echo "made temp file for what-depends list at /tmp/my-wd-input"
echo "generating list of packages and what depends on each to file /tmp/my-wd-output..."

for i in $( cat /tmp/my-wd-input | cut -d\  -f 1 | sort -u )
  do 
    #echo $i
    printf "."
    k=$( opkg whatdepends $i | grep  "^\t" | sed 's/^\t//g' | cut -d\  -f1 | sed 's/\n/ /g' )
    j=$( echo "$k" | wc -w )
    printf "$j\t$i\t $( echo $k )\n" >>/tmp/my-wd-output
    # printf "$j\t$i\t $( echo $k )\n"
    printf "."
done
echo ""

echo "preparing input list for user installed list..."
cat /tmp/my-wd-output | grep "^0" | cut -f2 >/tmp/my-user-inst-input

echo "The list of user installed packages is being written to /tmp/my-user-installed-packages"
echo "as well as on screen..."

for i in $( cat /tmp/my-user-inst-input)
  do
    if !(grep -Fqw $i /tmp/my-def-pkgs)
     then 
       echo $i >>/tmp/my-user-installed-packages
       echo $i
    fi
  done

echo
echo Done
echo "Clean up temp files /tmp/my-wd-output and /tmp/my-user-installed-packages in /tmp/ before re-running script."
echo

The opkg whatdepends function is slow so this process may take several minutes to run on slow processor but the script provides a progress indicator.

Once you have the list of user installed packages on your existing installation, you will probably have to do some manual research to determine what can be added in the new version as-is and what you might need to make accommodations for changes for.

EDIT 2023-10-16(2) - @R00 - Per your reply about finding 4 packages unaccounted for with the output of the script I provided above, I'm fairly sure I found the answer. :slight_smile:
The profiles.json file in the web directory for each target not only has the list of default packages but it also lists device specific packages in the profiles section. The few I checked which includes tagets different than your C7, like mips24 and aarch64 all matched.

I wrote a script that can either just provides the URL or can fetch it and then parse it to generate the "full" list of standard packages in /tmp/my-def-pkgs all ready to run the script from May 31.
I added a note in the above script as a suggested script name of listUserInstalled.sh .

I propose a name of prepToListUserInstalled.sh for this new script.
prepToListUserInstalled.sh gathers the board_name version, and target from ubus call system board and generates what should be a valid URL for the profiles.json file. It works on the 3 devices I tested with.

If you run prepToListUserInstalled.sh by itself, it prints some variable info and the URL so you could manually wget it.
If you add -full to the script call, it will wget the profiles.json file using the generated URL, extract the device and default packages and write the /tmp/my-def-pkgs file.
./prepToListUserInstalled.sh -full

If /tmp/my-def-pkgs already exists, the script quits without fetching or writing any files.
If /tmp/profiles.json aleady exists, the script skips the fetch and uses the existing file.
Manually delete or rename either files as desired to re-run the script and control input/output.

Here is proposed script prepToListUserInstalled.sh


# proposed script name: prepToListUserInstalled.sh

# also install script listUserInstalled.sh and chmod +x if needed
# and call it after running this script and checking output.

myFullRun="false"

if [ "$1" = "-full" ]
  then
    printf "\n Asked for full run!\n\n"
    myFullRun="true"
  else
    printf "\n Just running prep.\n\n"
fi

myDeviceName=$(ubus call system board | jsonfilter -e '@.board_name' | tr ',' '_')
printf "myDeviceName is "%s"\n" $myDeviceName

myDeviceTarget=$(ubus call system board | jsonfilter -e '@.release.target')
printf "myDeviceTarget is %s\n" $myDeviceTarget

myDeviceVersion=$(ubus call system board | jsonfilter -e '@.release.version')
printf "myDeviceVersion is %s\n" $myDeviceVersion

myDeviceJFilterString=@[\"profiles\"][\"$myDeviceName\"][\"device_packages\"]
printf "myDeviceJFilterString is %s\n" $myDeviceJFilterString

myDefaultJFilterString=@[\"default_packages\"]
printf "myDefaultJFilterString is %s\n" $myDefaultJFilterString

#myDeviceProfilesURL="https://downloads.openwrt.org/releases/$myDeviceVersion/targets/$myDeviceTarget/profiles.json"
if [ "$myDeviceVersion" = 'SNAPSHOT' ] ; then
    myDeviceProfilesURL="https://downloads.openwrt.org/snapshots/targets/$myDeviceTarget/profiles.json"
else
    myDeviceProfilesURL="https://downloads.openwrt.org/releases/$myDeviceVersion/targets/$myDeviceTarget/profiles.json"
fi

# 2023-10-17: Potential better way to get URL:
#myDeviceProfilesURL=$(grep openwrt_core /etc/opkg/distfeeds.conf / | grep -o "https.*[/]")profiles.json

printf "\nmyDeviceProfilesURL is %s\n\n" $myDeviceProfilesURL

if ! [ "$myFullRun" = "true" ]                                                                   
  then                                                                                           
    printf " You can manually wget the profiles.json file at the URL above\n"
    printf " to /tmp/profiles.json\n"
    printf " or\n"
    printf " Add -full to the command line to fetch the json file and output /tmp/my-def-pkgs.\n\n"

    exit 0                                                                                        
fi                                                                                               
                                                                                                 
                                                                                                 
if test -s /tmp/my-def-pkgs                                                                      
  then                                                                                           
    printf "\n File /tmp/my-def-pkgs already exists!\n"                                          
    printf " Since this script creates it, please remove it and run this script again.\n\n"      

    exit 0                                                                                 
fi                                                                                         
    
if test ! -s /tmp/profiles.json 
  then 
    printf "\n /tmp/profiles.json file was not found.  Fetching it.\n\n"
    wget -O /tmp/profiles.json "$myDeviceProfilesURL"
  else
    printf "\n /tmp/profiles.json was found.  Skipping the fetch!\n\n"
fi

if test ! -s /tmp/profiles.json
  then
    printf "\n\n OOPS! /tmp/profiles.json did not get created.\n\n"
    printf " Exiting!\n\n"
    exit 0
  else
    jsonfilter -i /tmp/profiles.json -e $myDeviceJFilterString | sed s/\"/''/g | tr '[' ' ' | tr ']' ' ' | sed s/\ /''/g | tr ',' '\n' > /tmp/my-def-pkgs

    jsonfilter -i /tmp/profiles.json -e $myDefaultJFilterString | sed s/\"/''/g | tr '[' ' ' | tr ']' ' ' | sed s/\ /''/g | tr ',' '\n' >> /tmp/my-def-pkgs

    printf "\n%s\n" " The list of packages, including device specific and default is:"
    cat /tmp/my-def-pkgs

    printf "\n%s\n" " The list was also written to /tmp/my-def-pkgs "
    printf "%s\n\n" " so the script to list user installed packages can be run now."
    printf "%s\n\n" " Manual cleanup of temp files in /tmp/ may be needed if you run this script again."
    printf "%s\n\n" " Manually run the script to list user installed packages next."
    printf "%s\n\n" " End of prep script."
fi
# end of script

Maybe this is a valid solution now.
End EDIT 2023-10-16(2)

I hope this is useful to you.

Please mark as solution if it provides what you need or ask for clarification on anything that isn't clear.

1 Like

I want to claim success here - it's very very close - and thank you for writing the script which saved me a bunch of time figuring it out myself. That is much appreciated.

I particularly liked the fact that the my-def-pkgs is simply a copy of the XXX.manifest file.

So here is what I did.

  1. Visit the https://firmware-selector.openwrt.org/ and find my firmware (archer c7 v2)

  2. Click on the 'folder' link to take me to the download directory

  3. Go to the bottom of the page to locate the XXX.manifest file

  4. log into the router - use the wget command to grab the manifest file (into /tmp)

wget https://downloads.openwrt.org/releases/22.03.5/targets/ath79/generic/openwrt-22.03.5-ath79-generic.manifest

  1. rename that file to my-def-pkgs

  2. Copy your script above into a file in /tmp - make it executable

  3. run your magic script..

ath10k-firmware-qca988x-ct
kmod-ath10k-ct
kmod-usb-ledtrig-usbport
kmod-usb2
libpcap1
luci-app-attendedsysupgrade
prometheus-node-exporter-lua-nat_traffic
prometheus-node-exporter-lua-netstat
prometheus-node-exporter-lua-openwrt
prometheus-node-exporter-lua-wifi
prometheus-node-exporter-lua-wifi_stations
rsync

This is super close to the right answer.

If we look at my blog post on upgrade from 21.02 to 22.03 - the list I captured there was

opkg install prometheus-node-exporter-lua-netstat
opkg install prometheus-node-exporter-lua-wifi
opkg install prometheus-node-exporter-lua-openwrt
opkg install prometheus-node-exporter-lua-wifi_stations
opkg install rsync
opkg install prometheus-node-exporter-lua-nat_traffic
opkg install libpcap1

(oh, but I've since added luci-app-attendedsysupgrade - so that difference I can explain)

This leaves a small delta between what I'd consider the "right" list and what your script gave me..

ath10k-firmware-qca988x-ct
kmod-ath10k-ct
kmod-usb-ledtrig-usbport
kmod-usb2

Thank you again for your time and attention here. What you've provided is probably plenty to make this work.. but I'd love to solve the mystery of these additional pkgs.

It should be easy to find where/why that discrepancy occurs.

I suspect that they show up as dependencies to one or more packages even though you manually added them.

The script I provided lists all the installed packages that depend on each installed package in file /tmp/my-wd-output.
Search it for each of those packages:

grep "ath10k-firmware-qca988x-ct" /tmp/my-wd-output
grep "kmod-ath10k-ct" /tmp/my-wd-output
grep "kmod-usb-ledtrig-usbport" /tmp/my-wd-output
grep "kmod-usb2" /tmp/my-wd-output

They each should show up in the second column and the remaining columns show what packages depend on it.

If any don't show up, look for similar names such as removing the trailing "2" from "kmod-usb2". Some packages have installed names that differ from the package name, often with differing trailing info such as version info.

If any are still missing, make sure they are actually installed.

Please report back what you find. :slight_smile:

Hmm - I'm not sure if I'm embarrassed, or humbled here. I probably should have dug further into what the script was actually doing and figured out why I had this mystery. However, after looking at this for 10 minutes I'm more lost and humbled at yours (and others) much deeper grasp of OpenWRT.

# grep "ath10k-firmware-qca988x-ct" /tmp/my-wd-output
1	ath10k-board-qca988x	 ath10k-firmware-qca988x-ct
0	ath10k-firmware-qca988x-ct	 
113	libc	 iwinfo liblucihttp0 cgi-io luci-lib-base opkg luci-app-opkg ubus iw libuci20130104 rpcd busybox luci-lib-ip libwolfssl5.5.4.ee39414e libubus-lua libiwinfo-lua swconfig luci-mod-system libnl-tiny1 libustream-wolfssl20201210 getrandom prometheus-node-exporter-lua-netstat ucode-mod-ubus luci-theme-bootstrap prometheus-node-exporter-lua-wifi wpad-basic-wolfssl procd-ujail base-files ucode-mod-uci netifd libubus20220601 firewall4 uboot-envtools dnsmasq procd ubusd prometheus-node-exporter-lua px5g-wolfssl luci-mod-status libjson-script20220515 luasocket luci-app-firewall libmnl0 jansson4 odhcp6c fstools uclient-fetch uci lua ucode-mod-fs luci-ssl dropbear prometheus-node-exporter-lua-openwrt libnftnl11 rpcd-mod-file mtd odhcpd-ipv6only procd-seccomp ath10k-firmware-qca988x-ct libiwinfo-data ucode rpcd-mod-luci urandom-seed luci-proto-ppp luci-mod-admin-full ppp luci-base logd prometheus-node-exporter-lua-wifi_stations rsync libblobmsg-json20220515 luci-proto-ipv6 openwrt-keyring luci-app-attendedsysupgrade prometheus-node-exporter-lua-nat_traffic jshn nftables-json attendedsysupgrade-common uhttpd-mod-lua libpcap1 libiwinfo20210430 libjson-c5 uhttpd ath10k-board-qca988x usign liblua5.1.5 zlib luci-lib-nixio libubox20220515 rpcd-mod-rpcsys ca-bundle liblucihttp-lua libuclient20201210 luci-lib-jsonc luci libucode20220812 ubox kernel rpcd-mod-iwinfo luci-mod-network kmod-nft-core kmod-nls-base uhttpd-mod-ubus fwtool jsonfilter hostapd-common libpopt0 kmod-ath9k-common wireless-regdb urngd kmod-slhc kmod-cfg80211 rpcd-mod-rrdns ppp-mod-pppoe
111	libgcc1	 libc opkg luci-app-opkg libpthread ubus iw libuci20130104 rpcd busybox luci-lib-ip libwolfssl5.5.4.ee39414e libubus-lua libiwinfo-lua swconfig luci-mod-system libnl-tiny1 libustream-wolfssl20201210 getrandom prometheus-node-exporter-lua-netstat ucode-mod-ubus luci-theme-bootstrap prometheus-node-exporter-lua-wifi wpad-basic-wolfssl procd-ujail base-files ucode-mod-uci netifd libubus20220601 firewall4 uboot-envtools dnsmasq procd ubusd prometheus-node-exporter-lua px5g-wolfssl luci-mod-status libjson-script20220515 luasocket luci-app-firewall libmnl0 jansson4 odhcp6c fstools uclient-fetch uci lua ucode-mod-fs luci-ssl dropbear prometheus-node-exporter-lua-openwrt libnftnl11 rpcd-mod-file mtd odhcpd-ipv6only procd-seccomp ath10k-firmware-qca988x-ct libiwinfo-data ucode rpcd-mod-luci urandom-seed luci-proto-ppp luci-mod-admin-full ppp luci-base logd prometheus-node-exporter-lua-wifi_stations rsync libblobmsg-json20220515 luci-proto-ipv6 openwrt-keyring luci-app-attendedsysupgrade prometheus-node-exporter-lua-nat_traffic jshn nftables-json attendedsysupgrade-common uhttpd-mod-lua libpcap1 libiwinfo20210430 libjson-c5 uhttpd ath10k-board-qca988x usign liblua5.1.5 zlib luci-lib-nixio libubox20220515 rpcd-mod-rpcsys ca-bundle liblucihttp-lua libuclient20201210 luci-lib-jsonc luci libucode20220812 ubox kernel rpcd-mod-iwinfo luci-mod-network kmod-nft-core kmod-nls-base uhttpd-mod-ubus fwtool jsonfilter hostapd-common libpopt0 kmod-ath9k-common wireless-regdb urngd kmod-slhc kmod-cfg80211 rpcd-mod-rrdns ppp-mod-pppoe

Let's look at that first grep - because while it generates a lot of output, there are only 4 rows of data. I think the way to read this is "thing in column 1, is a dependency of column 2 onwards". This would make sense as libc will have a lot of dependent packages, as well libgcc1.

If I do opkg whatdepends XXX on things I can get some other interesting data:

# opkg whatdepends ath10k-board-qca988x
Root set:
  ath10k-board-qca988x
What depends on root set
	ath10k-firmware-qca988x-ct 2020-11-08-1	depends on ath10k-board-qca988x

and

# opkg whatdepends ath10k-firmware-qca988x-ct
Root set:
  ath10k-firmware-qca988x-ct
What depends on root set

These two whatdepends seem to logically map to the first two rows from the grep - but I'm just more confused now.

Maybe if I better understood the format of the /tmp/my-wd-output file.. this would make sense to me?

EDIT

Ah, I see - you are doing a whatdepends on each thing installed - and then counting the number of 'words' aka packages that depend on it. That is the first column (the number) followed by the package, then the list of packages.

So - what we have here is ath10k-firmware-qca988x-ct is installed - and has zero dependencies. It also doesn't appear in the manifest (but I can't explain why).

My guess is that either you installed it or it is missing from the dependencies list of something that is installed. I thought you were indicating that those 4 items you had installed but were not showing in the output of the script. I guess it is the other way around then?

Here is my reply I wrote before seeing your edit:
The format of /tmp/my-wd-output comes from the loop:

for i in $( cat /tmp/my-wd-input | cut -d\  -f 1 | sort -u )
  do 
    #echo $i
    printf "."
    k=$( opkg whatdepends $i | grep  "^\t" | sed 's/^\t//g' | cut -d\  -f1 | sed 's/\n/ /g' )
    j=$( echo "$k" | wc -w )
    printf "$j\t$i\t $( echo $k )\n" >>/tmp/my-wd-output
    # printf "$j\t$i\t $( echo $k )\n"
    printf "."
done

For each item i from the output of opkg list-installed read from /tmp/my-wd-input
run opkg whatdepends $i and assign the space separated list of whatdepends to $k.
$j is the count of what depends on package $i.
Column 1 is the count $j of list $k.
tab
Column 2 is each package name $i.
tab
The remaining items from list $k are any packages found from opkg whatdepends $i cleaned up with grep, sed and cut.

I call libgcc1 the root package and each package that doesn't have anything depending on it a leaf. The packages in between I call branches.
Any line with zero in column 1 is a leaf and is likely a user installed package or a default package.

Don't feel bad about learning something new. I certainly invest a lot of time to understand just the stuff in my script.

Some thoughts on those 4 items:
The ath10k files are wifi drivers I think and might be left over from switching or trying different wifi setups. Removing a package, even in the process of installing a replacement may leave dependent packages orphaned on a system. I could see the same issue for the usb kernel modules.

Try looking through file /usr/lib/opkg/status to see if you find those 4 mystery modules, what their install times are and what else was installed close to the same time.

Actually, ls -ltr /usr/lib/opkg/info/ might make that an easier task. :slight_smile:

Let me outline the details and history of this device.

I've been running OpenWRT for years. When I upgrade from 19.07 to 21.02 I blogged about it. My blogging is mostly for future me, so I can go back and re-learn what I learned.

Then when I did the upgrade from 21.02 -> 22.03 I also blogged about it. This time I actually captured in the blog post the packages that I installed to customize my setup.

It was around then that I discovered the attended upgrade script. So I then used this process to upgrade from 22.03.02 -> 22.03.05..

Then I thought, let me check the script for catching user installed packages -- and discovered that ah.. look - because the attended sysupgrade uses the custom firmware builder I'm now running with all of the packages installed on the same date!

Grr.. well - I suspect that I can repeat my actions and figure this out the 'hard' way. If I were to re-install the stock firmware at a 22.03.02 level - and then install the user installed packages that I know are the right list.. then I'll get back to a stock state + user installed scripts which have the right timestamp.

From there - I can validate that your script works - because you're not using the install date that my script(s) copied from the doc were using to figure out user installed.

Then I can re-do the attended sysupgrade -- and see where in the process the ath10k packages were installed. I really don't think I manually installed them - same for the usb support..

I think there are really two possibilities here

  1. The .manifest isn't actually the complete file for the firmware I have installed - despite coming from the page - maybe the archer c7 has slightly different packages
  2. There is some magic package installation as part of the stock install, that detects the specific hardware and installs the right drivers (much less likely really)

Oh.. and I have two nearly identical devices - which I have upgraded / installed at slightly different times. Both show the ath10k .. so it's not an error, it's on both.

1 Like

Use this in terminal

opkg list-installed | awk -F' - ' '{printf $1 " "}'

I finally remembered this thread, here's a rewrite of the script I did a few weeks back and a couple notes.

The opkg whatdepends x command computes the transitive closure of dependencies, so you get not just those packages that depend directly on x, but those packages that depend on the packages that depend on the packages... You get the idea, try opkg whatdepends terminfo for an example. As you've probably already guessed, this is phenomenally slow.

All of the information about packages is in /usr/lib/opkg/info/, with the *.control files being the most interesting. This is where opkg subcommands look for any information about installed packages, and you can just ls that directory and see what is installed.

Replacing whatdepends with appropriate grep + awk drops run time of the script from ~180+ seconds to ~5 seconds on my edge router (an apu2 x86), but it changes the dependency count reported on many of the non-top-level packages (i.e., those with zero dependencies), because it skips the transitivity search. If you can live with that, you can save some CPU cycles.

Also of note is that there is an alias-like (or maybe more symbolic-link-like?) thing that opkg supports. You can see it with grep 'Provides:' /usr/lib/opkg/info/*.control. It's easy to see it in action with the packages dnsmasq and dnsmasq-full, where the 'Provides:' field in both says (duh) dnsmasq. The opkg subcommands parse through these and make sure to do the right thing, whereas my script sometimes sticks in both the "real" package name and an alias for it in the output.

This manifests as some packages being reported as "top-level" even though they are not (for example, you'll see firewall4 reported as a top-level package, when in fact it "supplies" uci-firewall, which has a bunch of dependents). So far, I've only seen firewall4 and tc-tiny reported this way, and anyway these false positives won't break anything, they're just annoying.

Enough of my blah blah blah, here's the script.

#!/bin/sh
# vim: set expandtab softtabstop=4 shiftwidth=4:
#-------------------------------------------------------------------------------

log_all=false  # Log everything, or just those with count == 0.

package_list=./installed-packages
rm -f $package_list

examined=0
for pkg in $(opkg list-installed | awk '{print $1}') ; do
    examined=$((examined + 1))
    printf '%5d - %-40s\r' "$examined" "$pkg"
    #deps=$(opkg whatdepends "$pkg" | awk '/^\t/{printf $1" "}')
    deps=$(
        cd /usr/lib/opkg/info/ &&
        grep -lE "Depends:.* ${pkg}([, ].*|)$" -- *.control | awk -F'\.control' '{printf $1" "}'
    )
    count=$(echo "$deps" | wc -w)
    if $log_all || [ "$count" -eq 0 ] ; then
        printf '%d\t%s\t%s\n' "$count" "$pkg" "$deps" >> $package_list
    fi
done

n_logged=$(wc -l < $package_list)
printf 'Done, logged %d of %d entries\n' "$n_logged" "$examined"
1 Like

Hi @R00 ,
That is true!
The profiles.json file lists those four packages in a device specific section.

Please see my edits to post 8 above and try my new "pre" script and let me know how it works.

Thanks!

Hey Spence, lookin' good! The profiles for snapshots live at a different path, you need something like this to point to them (I only tested it on one platform, x86/64, and it worked there).

if [ "$myDeviceVersion" = 'SNAPSHOT' ] ; then
    myDeviceProfilesURL="https://downloads.openwrt.org/snapshots/targets/$myDeviceTarget/profiles.json"
else
    myDeviceProfilesURL="https://downloads.openwrt.org/releases/$myDeviceVersion/targets/$myDeviceTarget/profiles.json"
fi

Thanks @efahl .

Please share the output of ubus call system board for that device running the snapshot version.
I was hoping $.release.version would match what is needed in the URL.

Thanks.

PS. I wonder if version strings for release candidates will also differ between system board data and downloads url path.

Snapshot updated this morning, with auto-hacked model and board_name (both originally just default-string). The target field is correct for use in finding the snapshot's profile.json, though.

$ ubus call system board
{
        "kernel": "5.15.135",
        "hostname": "rtr02",
        "system": "Intel(R) Celeron(R) N5105 @ 2.00GHz",
        "model": "CWWK 4x2.5G v.5",
        "board_name": "x86_64,combined",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "SNAPSHOT",
                "revision": "r24124-518923178c",
                "target": "x86/64",
                "description": "OpenWrt SNAPSHOT r24124-518923178c"
        }
}
1 Like

Thanks for that.

The following is not a reply to you but is more information relating to x86 targets.
I replied to keep it tied to the thread of consciousness. :slight_smile:

I did a bit of analysis and it looks like the x86 builds may be
too generalized to get the device specific package list in the
one case of target x86/geode.
The script could map to generic in all x86 cases and provide
a note about manual intervention for the x86/geode case.

With the possibility of more changes in the snapshot ecosystem,
I'm inclined to not do a lot of work to customize the script for it.

Some info from my analysis:

On https://downloads.openwrt.org/snapshots/targets/

There are 4 sub-directories for x86 which would be nice if they match
the json $.release.target value from ubus call system board
so the script can build a functional URL.
x86/64/
x86/generic/
x86/geode/
x86/legacy/

To get the device specific package list with jsonfilter of the profiles.json file,
the system board value with json key $.board_name should match
one of the keys under json path $.profiles[*]

https://downloads.openwrt.org/snapshots/targets/x86/64/profiles.json has 32 default packages listed and has $profiles[*] genericwith 12 device specific packages listed and nothing else from yourubuscall example that seems to align but the script could do a static mapping togeneric`.

https://downloads.openwrt.org/snapshots/targets/x86/generic/profiles.json has 32 default packages listed and has $profiles[*] genericwith 12 device specific packages listed but the script could do a static mapping togeneric`.

https://downloads.openwrt.org/snapshots/targets/x86/geode/profiles.json has 32 default packages listed and has $profiles[*] generic geos`

profile generic has 4 device specific packages listed
profile geos has 16 device specific packages listed
I do not know how the user or installer chooses the package list.

https://downloads.openwrt.org/snapshots/targets/x86/legacy/profiles.json has 32 default packages listed and has $profiles[*] genericwith 13 device specific packages listed but the script could do a static mapping togeneric`.

Looking at profiles.json file data for non-x86 targets under
https://downloads.openwrt.org/snapshots/targets/
it looks like the "normal" names are used so the script should work fine.

Looking at profiles.json file data for x86 targets under
non-snapsot versions, it looks like the same situation
as the snapshots versions, so the issue is not with
snapshots and it is with x86. At least there are only 4
profiles.json files to possibly manually get data from
and only one has more than one device profile to use.

That's it for now.

1 Like

Thanks for reviving this thread - I've tagged the post that I think has the right solution in it. There may be a minor issue where luci-ssl was flagged as a user installed package -- but the comment that you have to combine the manifest + the package.json board specifics is correct... if I manually create that file it works fine. (so maybe there are still some minor bugs here -- but this gets you close enough to identifying the user installed packages AFTER an "Attended Sysupgrade" has run..