OpenWRT x86-64 running on AWS

After multiple years of using OpenWRT, I built trust to this great product. That was the reason for me to use it also in AWS environment.
For now I have running OpenWrt 19.07.5 r11257-5090152ae3 as AWS EC2 t2.micro instance, all is OK.
I would like to migrate from t.2 to t3.micro instance, but during execution of the VM, it freeze on message "Waiting for root device /dev/xvda2..."
The difference between t2 and t3 is, that t3 does use Amazon EFA (Elastic Fabric Adapter) and ENA(Elastic Network Adapter), which I think is the problem during boot to connect to the xvda2 device.

Amazon provides drivers for EFA anf ENA on https://github.com/amzn/amzn-drivers/tree/master/kernel/linux but I am not experienced in drivers compilation.

Please, would it be possible to compile these drivers for OpenWrt so it will be possible to run on AWS t3 instances?

Thank you and hopefully I will be able to use it for more years.

interesting! can you provide more details about costs, performances, step-by-step installation guide or link to howto..?

what exactly is OpenWrt doing for you? Are you running a bunch of containers inside one AWS or something?

That says Linux is waiting for root device which is Xen volume but can't find it. You should check with initramfs image (connect to it with serial console if it's provided by AWS) what's populated in /dev and modify GRUB config file to point root= to proper device. That should fix it. Good luck debugging.

Step-by-step guide would be for full blog, but in top level steps

For testing/development you can use free tier (t2.micro or t3.micro) for one year, which is at no cost.
For production you can use multiple cost optimization programs which brings significant cost savings.

I am running academic project related to security research with 50 honeynets/honeypots worldwide. Openwrt is very valuable here not only from its capabilities but also from its very low footprint in terms of storage (AWS low limit for system drive is 1GB which is for no additional costs and OpenWrt fits it very well). Also its requirements for RAM and CPU is very cost effective.
By migrating to t3.nano I will spare a lot of funds (decreasing cost by ~50%)

2 Likes

I tried to point root= from /dev/xd.. to partition UUID produced the same error.
In general I think that the problem is in the storage/network (EFA/ENA) driver. EC2 T2 instances use EBS-only as storage, not local storage. Migrating to any other T2 instance profile works without any problem, there s no EFA/ENA in use. T3 instances does use EFA/ENA for storage/network optimization and require to install appropriate drivers and when I migrate to T3 profile, mentioned error is returned.

I will try to play with debugging, but I am not yet sure what are the options in AWS environment.
Anyway thank you very much for your hints.

If the missing drivers would be the case, take look at package/kernel/mwlwifi/Makefile or package/kernel/rtl8812au-ct/Makefile for basics how to add external kernel modules to build system. Good luck.

That's high science if not alchemy for me, so I rather not to try it and let it for experts. :slight_smile: I was hoping for installable package delivered via opkg.
I appreciate your hints and interest.

Just to be more specific, this my problem is more like feature request either ad-hoc for me, or for whole community. I don't feel comfortable to go deeper into development area of OpenWrt. I am user of the product without any kernel or system component development capabilities.

should be easy enough as they are already upstreamed in Linux kernel


so if you want packages it's just a matter of adding a makefile that defines a package with them, and I can do that. (adding an out-of-tree driver is way harder and I can't do that)

But I see the error "waiting for root device". That means the kernel cannot find the root filesystem so making a package is useless (as packages are installed in root filesystem, which is not accessible until the driver is loaded).

I think both drivers must be compiled-in aka enabled in kernel config of 64bit, which is even easier as I think I only need to add

CONFIG_INFINIBAND_EFA=y
CONFIG_NET_VENDOR_AMAZON=y
CONFIG_ENA_ETHERNET=y

in the x86_64 target kernel config file here https://github.com/openwrt/openwrt/blob/master/target/linux/x86/64/config-5.4

Of course I cannot test this, if I compile images with it, and give you a download link, will you test them?
Do you want Luci web interface in the test images?

Note that even if all this works and I open a PR with this contribution it will take months before someone merges it and can be used without setting up your own Linux VM to run build locally. It's not hard to just build OpenWrt, it's very automated.

2 Likes

@bobafetthotmail
wow, that would be awesome if you could help me with this. And hopefully more people will find this useful.

I will be very glad to test the images, of course. To have Luci web in the test image will be helpful.
You made my day, really. Thank you very much.

ok I added the config options, I compiled and I didn't get any error. There is Luci web interface.
Let's see if they work

download link

3 Likes

OK, I have downloaded it, installed on virtualbox and created .ova. I will try to upload to AWS asap, not sure if today. I will keep you updated latest tomorrow evening. Let's see if magic happens.

Many thanks for all your effort so far.

@bobafetthotmail
Bingo! Your image works without any flaw. I can confirm that it works on AWS t3 instances very well.
While comparing it with standard openwrt image, I see that you also implemented new kernel 5.4.87, while standard uses 4.14.209. That's great, but many opkg packages can't be installed.
But of course, this exercise is mainly for proof of concept, which was successful.
If you would need, for any kind of confirmation/verification/curiosity, I can give you details to ssh and luci web connection.
Many thanks for your help! :clap:

1 Like

eyyyy! Very good.

I did not implement that, I used OpenWrt master branch, aka the latest source code. It's roughly equivalent to "OpenWrt development snapshot". or a "nightly build".
That error about kernel versions when installing packages is normal for it. The packages in the repos are built by the official build bots every few days and may be too old or too new for what I compiled.

I used that because I will now open a PR to contribute this change in the master branch, so I wanted to make sure it works for that.

Since you will probably want to use it as a normal person I'll do the same thing with OpenWrt stable release 19.xxx source.
This way the kernel is the same version as normal stable release and it should let you install everything.
I'll be back in a bit with a new download link

1 Like

this is an OpenWrt stable 19.07.5 (latest stable at the moment I post this) image of the same thing as above, Luci web interface is included.

download link

Tell me if there are problems with it.

1 Like

Today evening I will test it and let you know.

Again, many many thanks for your help and when I will verify its functionality I will make generalized snapshot as Community AMI within AWS, so other people can install and benefit from it. :sparkler:

1 Like

yeah, just warn people using that "Community AMI" to NOT upgrade their OpenWrt system as that would "upgrade" the kernel too and remove the ability to use t3 instances. You can update packages all you want, but no kernel updates.

If all goes well and OpenWrt members accept my contribution (will open a PR in a bit), this should get merged in time for the next official stable release (which at this time is unknown, fun fact: they skipped 20.xxx release already)

But it will NOT go in official OpenWrt 19.07 stable releases.

Hmm, something is not OK, I am getting again the error "waiting for root....". The difference now is, that first image was ext4, now it is squashfs. And also difference is kernel version.
Could you please generate ext4 image?

that's strange, both squashfs and ext4 should work. In the "development snapshot" there is also a squashfs image, can you try that too?

Anyway, this is OpenWrt stable with both images, squashfs and ext4 download link