How to upgrade x86 OpenWrt?

No you can't expand squashfs but you can build yourself at any size.

I wouldn't go so far as to say openwrt just isn't designed for x86 at all. Personally (and I've no desire to start a debate on this) I'd say it's the best platform for it because of its flexibility and CPU power - nothing comes close if you're running openvpn or a web server.

You can also consider pfsense or opensense... Or multiboot with them... And then return back to openwrt because of its great support and multitude of packages and functionality.

I think the message should be that x86 is a bit different, there's a whole bunch of different opportunities and you should use openwrt in a slightly different way to a commercial plastic box router.

I had been using extroot on OpenWrt 18.06 x86_64, but the squashfs image I tried for 19.07 had not enough free space to utilize extroot, although I didn't test the latest minor release.

@hgl I run OpenWrt x86_64 in a VM as my main firewall/router. It works incredibly well and to be honest, I quit pfSense in favor of OpenWrt. Happy to elaborate more on the "why" if anyone is interested.

But to your questions I upgrade my VM all the time because I run tweaked master snapshot builds and update quite a bit. I run the ext4 filesystem and upgrades are just as easy as on a standard router. Given my VM runs on an SSD, the process takes just under 20 seconds to update, reboot, and get back to functional.

If you are using the ext4 filesystem, you just upload the *-generic-ext4-combined.img.gz file via the web interface (or scp + sysupgrade if you prefer CLI). That's it. :slight_smile:

2 Likes

Yes, it should be fine if you avoid customizing builds and upgrading packages.
Otherwise, this thread would not exist.

Could you elaborate on that? I customize builds, yet still upgrade my x86_64 VM with the updated builds with no trouble. As for upgrading packages, it’s the same situation as on a non-x86 unit. There are some packages that can be upgraded via opkg, but anything that is tightly integrated with the kernel likely will have to “upgraded” through a rebuild.

1 Like

It is not the same for other general-purpose Linux distros.
APT/DNF typically respect existing symlinks and do not overwrite them unlike OPKG.

Based on my understand of @hgl's OP, it sounds as though his whole issue is not feeding the right type of upgrade image to sysupgrade to begin with.

@hgl, could you reply with the full path.img value that you are passing to sysupgrade?

2 Likes

I think I know why I failed to sysupgrade: I passed the uncompressed image. I guess that's also why it complained the image had no metadata. Thank you so much for showing me the file name you used. I'll try again some time later.

Also a couple follow up questions if you don't mind me asking:

  1. Do you flash the MBR or UEFI version of the image? Will flashing a image also upgrade the boot partition? I'm asking because I have an UEFI only system, and I have to swap out the boot partition in the 19.07.03 image with the one in the snapshot image for the flashed openwrt to boot. I wonder if the next version still doesn't have official UEFI image, will I be able to directly flash the MBR image.
  2. Once flashed, does your ext4 partition shrink to the size specified in the image?
  3. Once flashed, what happens to the outdated packages and kernel modules? Are they automatically removed or disabled? Is executing opkg upgrade enough to bring everything back to normal?
  4. Once flashed, do files in locations other than /etc/config preserved?

If I could just sysupgrade the (correct) ex4 image and everything just works, that would be dreamy.

I'm quite also interested in your VM set up. Do you mind share a little bit?

I'm under the impression that a router is both the first and last defense regarding network security. Wouldn't running OpenWrt in a VM make the attack surface bigger?

But my x86 router is pretty powerful and has much space left after OpenWrt is installed. I don want to somehow run a full distro alongside it. I'm also eyeing for the docker support in OpenWrt 20, which seems to make it possible to run a full distro directly in it.

Do you use something like ESXi as the hypervisor?

I will do the same trial next weekend, as right now I have Installed my AtomicPi with stable version 19.07.3 by tweaking the boot partition and Grub partition. Both partitions (root and boot) are EXT4 and the grub is fat 16, but the WIFI driver is not working on kernel 4.XX and also USB ethernet I am using, is not in default installed. My plan to use 5.xx Kernel with snapshot image and I have expanded my root partition to the max around 15GB (internal eMMC).
just wondering if the setup what @_FailSafe doing, can be applied to my AtomicPi without too much reconfiguration. Thanks

I do run ESXi 6.7 and multiple network-related VMs (DNS, grafana + influxdb for stats collection, etc.) in addition to my OpenWrt VM. The OpenWrt VM itself has four vCPUs, though it can actually run current workload with two vCPUs on my core-i7@3.4Ghz. I give it 512MB RAM, though it honestly could run comfortably at 384MB or so.

I added an Intel 1Gb quad-port NIC into my ESXi box (I use the onboard NIC for ESXi Mgmt) and currently assign one of the quad-port interfaces to a LAN vSwitch (which trunks multiple VLANs to my upstream switch). A second quad-port interface goes to my WAN vSwitch, which is only attached to my OpenWrt VM's WAN vNIC and on the physical side, my cable modem. Obviously, my LAN vSwitch connects to a LAN vNIC on the OpenWrt VM. I also have a separate DNS vSwitch which is used for the vNICs on my DNS VMs and connected to a third vNIC on my OpenWrt VM.

The DNS vSwitch configuration is not really necessary, but it's a little cleaner in my head and easy enough to add in a virtual environment.

The main downside to a configuration like this is that my ESXi box can't be remotely managed if the OpenWrt VM goes down for any reason. But from a stability standpoint, that VM never crashes, so the only real "outage" situation I encounter is a true internet down (i.e. ISP goes offline) scenario. In that case, even a physical router won't save me. But, I do have all my critical VMs set to auto-start in the proper sequence and my ESXi host machine is set to return to previous power-state upon power loss (via BIOS setting). Any time I have had my ESXi host go down (power outage, for example), it just powers back on by itself and all my VMs come back online shortly thereafter.

Feel free to read up on a configuration like this and draw your own conclusions, but personally, I am good with the separation of traffic via vSwitches and consider that to be secure enough in my case. Having used ESX/ESXi in professional contexts, this configuration is actually quite common.

2 Likes

Thanks for the delineation, it's very interesting, and made me want to give ESXi a try.

I'm actually considering switching to pfsense, with a linux distro running alongside it to run docker, etc, to make use of the x86 system more thoroughly.

Curiosity question... what is behind your motivation for using pfSense?

I'm not trying to start a flame war, but it's mostly the fact that OpenWRT gears more toward wireless routers, and the result of that includes:

  • most commands it offers are of stripped down versions.
  • aggressively clean up files, like package index, etc.
  • no first-class support for IDS/IPS systems.

Which aren't necessarily benefits for x86 systems. But these aren't major, I'm still quite ambivalent.

1 Like

Understood and I was not asking the question to set you up for a war on behalf of OpenWrt or pfSense. I was genuinely curious as one who has used both, fairly extensively, and ended up sticking with OpenWrt even on x86.

Objectively, what commands does pfSense offer that you wish you had (or had more thorough versions of) on OpenWrt? I have found that a lot of the full and proper commands/libs can be had through a custom OpenWrt build.

OpenWrt does favor “temp” stuff a lot, which makes sense given its history as a router based distro. Historically memory-bound devices and the desire to minimize writes to disk have, and still are, shaping the way it functions as an OS. I have overcome some of that by altering the locations (via symlinks or modifying paths in configs) of some files, like the DHCP lease file, so they survive reboots.

On the IDS/IPS front, you are correct. The gold medal goes to pfSense on that one from a feature availability perspective. That is, assuming you have the horsepower in your x86 box to turn all that on. I found pfSense to be very memory heavy for the additional packages you enable. Especially with things like pfBlockerNG. (I prefer Pi-hole to pfBlockerNG now anyway, FWIW)

A couple more thoughts between the two...

I found the traffic shaping and QoS configuration in pfSense to be downright maddening at times. It affords you a ton of configuration options (as does the whole of pfSense), but requires a ton of knowledge to know how to tweak all the pulleys and levers. Even after weeks of tuning, I still could not achieve the same level of consistent bufferbloat reduction in pfSense that I get with OpenWrt SQM (CAKE) with about 30 minutes of tuning.

I much prefer pfSense’s GUI version of firewall rule configuration to OpenWrt’s GUI. It’s much easier to visually see what you’ve got set up hierarchically with rule precedence in pfSense because each interface has a separate tab with just its rules. OpenWrt’s firewall GUI, while more “pretty”, is far less functional to me. Along those lines, pfSense makes it so much easier via either web console or pftop to quickly view firewall logs in real-time.

The available packages for pfSense are significantly smaller in variety and availability. There are pros and cons to that, of course. A pro is that you get well-curated and functionally tested/stable packages that just work. The cons are that there are a limited number of pfSense devs who have time to test new packages and approve them. So the package selection tends to stay pretty static and they very much favor lagging version upgrades in favor of stability (not necessarily a con). It’s definitely a more tightly controlled environment as opposed to the very open, community driven approach of OpenWrt.

Lastly, one of pfSense’s strengths is also one of its biggest deterrents. It provides one with hundreds of textboxes, dropdowns, checkboxes, and radio buttons to tweak about anything you can dream up on a firewall/router. Even with decently advanced network knowledge, I was finding myself having to refer to pfSense docs a lot to get clarity on many of the settings. A lot of times that led me down the road of landing on pfSense forums. In order to not offend anyone, let me just say that OpenWrt’s community forums are top notch in the level of friendly support offered. OpenWrt forums are generally (sure, there are some exceptions) filled with people who care to help others understand concepts without belittling them. That cannot be understated, and it is one of the main reasons I love to stick with OpenWrt. It has a great community that I haven’t found a lot of other places.

Anyway, I’ll wrap this up here because I just realized I wrote a book. Strictly from a stable network perspective, both pfSense and OpenWrt will get you there. It’s the value-adds that have to be weighed. :slight_smile:

EDIT 1
I lied... I wasn’t done. :joy: Another thing I just remembered that I meant to call out is that OpenWrt has been [thankfully] very accepting of WireGuard. It has not been well accepted by pfSense to date. In fact, here is an example of both their WireGuard stance and what is a very common style of forum response.

3 Likes

this is a good question: the short answer is yes, however, hypervisor stacks have come a long way...

providing you pay attention to not inadvertently exposing host or parallel guestOS's over a bridged wan or "untrusted" link, the risks are as high as the hypervisor vendors exposure + code quality + update schedule... the prevalence of the vmware and virtualbox brands ( amongst others ) speaks volumes on that front...

as with most things exposure related... risk and risk assessment/management are commensurate to your attack surface... data centre level protection is on another level...

3 Likes

Thanks for taking the time to give such a rich response.

I, for example, tried swapping dropbear with openssh in a custom build (for ed25519 support), but it seemed luci was designed to operate with dropbear. It's good to know you are having good results. Maybe I had done something wrong, will give it another try soon.

IDS/IPS and traffic shaping are more of educational than practical purpose for me, so I wasn't really complaining :slight_smile:, and thanks for bringing SQM to my attention.

I haven't used pfSense intensively, so wasn't familiar with its community culture. Looks like I made good choice by starting with OpenWrt.

All in all, I think you just manage to swing me toward OpenWrt.

A daydream question though, from this Wikipedia list, it's apparent that there are quite few Linux-based router systems that focus on x86. Do you have any experience with them? I personally never heard any of them. Assuming I wasn't living under a rock, I wonder why they never gain as much traction as OpenWrt does? I guess it's either pfSense absorbed most x86 router users, or OpenWrt already fits the needs quite well?

That's a good point.

I actually don't really have a powerful x86 machine as a router (I did, but then I changed to a much less powerful one). I guess I'll just move most "server services" to a dedicate box in the LAN.

Also to respond to some of my own questions for prosperity:

Will flashing an image also upgrade the boot partition?

Yes. My grub wait time was reverted after flashing

Once flashed, does your ext4 partition shrink to the size specified in the image?

Yes.

Once flashed, what happens to the outdated packages and kernel modules?

All gone.

Once flashed, do files in locations other than /etc/config preserved?

Nope.

1 Like

This should depend on the sysupgrade configuration:
https://openwrt.org/docs/guide-user/troubleshooting/backup_restore#customize_and_verify1

2 Likes