Hi,
I have about 30~50 OpenWrt installations on x86-based computers, ranging from Intel 1U-servers to PCEngines APU. Yesterday I found a bug in my application and distributed a new version, this is done automatically within my software. I can send it a command to perform the upgrade. I have done this many times before. Because in this fix I changed some firewall rules I decided to ask all devices to do a reboot, so I could be sure the changes would be there.
About 14 of the devices did come back on-line after the reboot.
I check one of the devices and noticed that all the files that are in my pkg are 0 bytes!
I did an opkg update, followed by a --force-reinstall and that solved the issue on this machine.
This morning one of my techs visited 2 other locations. The issue was slightly different. The files still showed correct sizes, but have no content. No opkg update possible, it downloads then complains about missing newlines. Upon further checking /usr/lib/opkg/status is corrupt, so are the files for my package in /usr/lib/opkg/info. They all seem to be empty.
To remediate I copied a valid status files, plus the correct info files from a running system, and did a --force-reinstall, now those are working again.
But can anyone explain to me why all these files were corrupted. I could understand if one system had a disk issue, but 14? And to make it even more bizarre, the reboot command was sent to the latest version, so the one I installed that is now corrupted.
I am using ext4, could it have been some kind of sync issue?
My images is based on r18297-7e89421a7c of OpenWrt.