I recently went through the week-long ordeal of installing a full Build System described here for the Build System running on a 32-bit ARM virtual machine in QEMU (armhf armvirt) without cross-compiling. It is not a process that I would like to repeat.
How would I go about installing a minimal Build System running on the armhf armvirt virtual machine in QEMU, that would allow me to natively compile only Loadable Kernel Modules (LKMs), which can be directly insmod'ed into the stable OpenWRT distribution v18.06.5, which is based on the Kernel v4.14.151 (for armhf, too) ?
...with debugging symbols for the kernel and for any hardware agnostic LKMs, which are normally included in the OpenWRT v18.06.5 stable releases, as well as my own custom LKMs.
I emphasize, that I want to avoid dealing with downloading, compiling / building / installing of any user-mode packages, or their dependencies or tools/headers needed to make them...unless they are needed for compiling the LKMs.
I am also not interested in compiling/building any hardware-specific kernel-mode drivers.
If you followed that procedure, you may have built OpenWrt in an emulated ARMv7 environment - but you have still cross compiled OpenWrt, because that's how OpenWrt's buildsystem is laid out. That means even if you would take a native ARMv7 system (e.g. RPi, sunxi or similar ARMv7 SBCs), running a native linux distribution (Debian, Fedora, arch, gentoo, mandriva, OpenSuSE, Ubuntu, ...) without any emulation, the buildsystem would still (have to-) cross-compile to arm32-musl on ARMv7.
You can't.
As mentioned above, OpenWrt's buildsystem is modelled around the buildroot approach and will always cross-compile, regardless of the host architecture - and it has to, given that Debian armhf (glibc based by nature) is not compatible to OpenWrt's musl based (e.g.) arm_cortex-a15_neon-vfpv4. Admittedly, the kernel is slightly special in this regard, considering that it (upstream linux) technically has less dependencies on the host system than userspace programs, but that doesn't change the fact that OpenWrt's buildroot environment is not made for that. Changing this is possible in the sense that 'everything is possible', but it's not technically viable (as you'd have to spent months on splitting it out, recreating a suitable environment and manually re-integrating stuff).
There are two 'easy' alternatives, using the SDK which does provide some leeway to build leaf kernel modules (cross-building using a precompiled toolchain for x86_64) or using the normal buildroot approach from source (still cross-compiling, but from source and on any host architecture you like; for performance reasons it does not make sense to do a full system emulation for a non-native architecture here).
--
Yes, there are some facilties in the buildsystem to use an external kernel tree, but that is barely supported at all - and requires initimate knowledge of the system and its inner workings. Given that OpenWrt isn't self-hosting, it won't avoid the necessity to cross-compile either and does involve a heap of manual work to present the patched external tree to the rest of the buildsystem in a way to get it accepted by it (yes, we're talking about weeks- to months of very low level development. If you have to ask, you're probably not going to make it).
What about debugging the kernel and LKM's that come with the standard stable OpenWRT distribution?
Me thinks that debugging on a different CPU/hardware plattform (real or virtual) does not make much sense, even when the source code and symbols are available to the debugger, because inspecting the disassembly of an LKM's machine code is often indispensable.
Most embedded systems run a debugger on a full-featured OS. I’m not sure what else you’d want, especially as most routers don’t have even 2k displays for a multi-pane, modern IDE and debugger.
@jeff
The Debugger client can run on a fully-featured OS, while the Debugee runs only the debugging server stub on the abridged OpenWRT OS. i.e. gdb allows debugging over LAN or even over a UART.
This is how I'd debug an OpenWRT armvirt system running inside QEMU - not on some router.
I do not want to run a full-featured debugger client GUI on an OpenWRT system, nor do I want to compile/build/debug Kernel code on a real router hardware.
@slh
OK, I understood that OpenWRT is not self-hosting, which means that it is incapable of compiling/building its own kernel without doing a lot of development work first, ...in other words: OpenWRT cannot build itself even if the build system is installed and running on the same CPU platform as the target system. Bummer
I don't think this statement is always correct. An example "use-case" in which the build system for OpenWRT (armhf) is installed on a virtualized fully-featured Linux such as Debian (armhf) running inside QEMU as armvirt, would make sense if the OS hosting the QEMU and the virtual machine, was NOT a Linux system.
This would also allow debugging of the kernel code running in the armhf armvirt machine via gdb over a virtual network (or UART) by the Non-Linux system, which hosts the QEMU virtual machine. I mentioned the kernel debugging issues in my previous post to Jeff.
Would the kernel of a stable OpenWRT (armhf) distro be able to directly load the LKMs built by this x86_64 SDK ? What about the Kernels mismatch ?
Anyway, doing it this way would force me to have 2 virtual machines:
One x86_64 virtual machine running a fully-fledged Linux (e.g. Debian) for cross-building LKMs with the SDK, so the generated LKMs can be loaded by the stable release of OpenWRT's kernel for armhf.
Second armhf armvirt machine running a stable OpenWRT kernel and acting as a gdb server which would allow for native debugging of the armhf kernel code, as well as my LKM's code built by the SDK above, so I can observe how the OpenWRT's kernel code loads my LKM's code and interacts with it.
Isn't there a simpler way ?
Doesn't this link contain the SDK compiled for the armhf architecture ?
Compiling on a QEMU emulated arch is a bad idea (it is also a bad idea if it is a true ARM device) as performance is going to be garbage if compared to any half-decent x86_64
Running compilation on a test system is also a bad idea.
So yes you cannot use the same VM, but there are no good reasons to do so anyway.
No, it contains the SDK that creates packages FOR the target architecture, i.e. the right crosscompile toolchain for armhf in this case.
The Image Builder and SDK are supposed to be run in a x86_64 system.
Compiling and debugging on the same system is a very good reason IMHO.
Unless you are a fan of iterative Compiling -> copying the built files to Host -> rebooting into an armhf VM -> copying the built files to the armhf VM -> Debugging -> Rebooting into the x86_64 VM with the Buld System ...over and over
Having a proven preconfigured Build System encapsulated in a VM is also a very portable method, that can be just downloaded and executed on any host OS instead of its cumbersome and lengthy configuration & building from source. ...and having a Build System in a VM is the only way to compile LKMs on non-Linux systems !
In such scenario, the low performance of the Build/Debug System encapsulated in a VM does not matter, because the alternative is no performance at all.
It would be much more helpful, if you could suggest what "make menuconfig" options should be chosen here and here to create a minimal Build System just for building stock and custom LKMs as well as the kernel for armhf armvirt machine (with debug symbols and headers).
Compiling INSIDE the VM that is software-emulating a CPU architecture is an excuse for slacking off.
We are talking of software emulating a CPU. A modern x86_64 processor will have the same performance of a smartphone CPU. You really don't want to compile anything on a smartphone CPU, much less a kernel or a whole OpenWrt system.
recompiling a kernel on an armvirt takes many hours, copying files over is annoying but takes a few minutes. If your VM host does not completely suck it can have "shared folders" so it's not hard to copy stuff over.
That said, you can use a single x86_64 Linux VM to run everything (be your compile system and your QEMU host) and follow the same armvirt workflow that people on a Linux host follow, as QEMU isn't virtualization but software CPU emulation, so it does not care about being run in a VM.
You download all stuff you need to run the full OpenWrt buildsystem and QEMU on a single Linux VM (Debian for example).
Then you compile a firmware as normal, when you are done with it you spin up the armivirt system inside the same Linux VM with QEMU.
If that does not work (but I think it should work as we are not using KVM but pure CPU emulation with QEMU) you need to enable nested virtualization in your host virtualization system (VMWare can do it, KVM can do it, Virtualbox should also support that).
I'm not aware of any options to build a smaller buildsystem, in any case it will create its own toolchain.
To get it to use an external kernel, you need to enable the option about that under Advanced configuration options (for developers) or under the kernel configuration options.
I don't know much more than that, sorry. I never had to use a custom kernel.
Yes, software CPU emulation is slow, but it is still faster than than NESTED emulation of a virtual armhf CPU by a virtual Linux OS running on a virtual x86_84 CPU.
BTW: VMware and VirtualBox are incapable of emulating an ARM CPU.
No, according to benchmarks x86_64 CPUs used in desktops and servers are still faster than the ARM/MIPS/OpenRISC CPUs used in smartphones.
Yes, but it needs to be done only once, because I will not be changing the kernel's sources but only changing the sources of my own LKM that interacts with the static kernel.
Yes, but constantly changing VMs disrupts my workflow. Which would slow me down even more than copying files between VMs. Especially that my LKM is so small that it compiles in seconds so compiling on an emulated CPU slows me down much less than copying files and switching VMs ...not to mention the debugging of my LKM, which MUST be done on an emulated armhf CPU anyway.
Unless your virtualization software on x86_64 is utter trash you are using hardware-assisted virtualization (Intel VT-x or AMD Secure Virtualization) and your x86_64 Linux system will have only a 5% CPU performance penality.
QEMU is software emulating a CPU, and there yes you get a big performance penalty.
Not what I said. I said that if you run QEMU armvirt on a x86_64 CPU the emulated ARM CPU will be as fast as a smartphone.
This is because SOFTWARE EMULATING an ARM CPU on a x86_64 CPU with QEMU has a significant performance hit, while HARDWARE ASSISTED VIRTUALIZATION on x86_64 to run another x86_64 system has a tiny performance hit.
Afaik the OpenWrt build system recompiles the whole kernel to create the kernel module(s) so I'm not sure you can "just recompile the LKM".
Unless the ultimate target is QEMU, doing low-level kernel testing on an emulation of a target is a disaster waiting to happen, if even possible at all (for example, the hardware on the board is not present or is emulated).
Anyone doing high-level kernel work would have an x86_64 available as that is effectively the "reference implementation" of Linux.
Claims that flashing the device are too slow don't fly, given that 10 or even 100 MB can be copied to a loop device in seconds, on a bad day. Since the OP is working with kernel modules, any sane test would involve a reboot of the target. TFTP boot of an initramfs image eliminates the need to "flash" the image.
The OP seems unaware that thousands, if not tens of thousands of embedded-systems developers are able to build an image, flash it to the target device, run on the target device with a debugger attached every working day, as SOP for the industry.
Aha!, now I understand why you are so strongly opposed to compiling LKMs on a slow "smartphone-like CPU", and I will have to agree with you IF it is really impossible to compile just my own custom LKM as described here, without recompiling the entire kernel.
That would be true if I did not write specifically that I am not interested in building any hardware-specific kernel modules or drivers.
As long as I am building and debugging hardware-agnostic modules, I don't see this becoming a problem.
...and uploading and flashing the built binaries to the real-hardware target acting as a gdb debugging server, is not always feasible, e.g. especially when this hardware target is accessed remotely and any bug in the custom LKM will result in kernel panic or lockup....and loss of remote access.
In this scenario, the custom-built LKMs really need to be debugged and pretested on a virtual armhf armvirt machine, before being transferred and insmod'ed on the real armhf hardware.
I know you can rebuild kernel modules on a normal Linux system (my PCs are running OpenSUSE Tumbleweed, for crying out loud), i'm just saying I think this is not supported by OpenWrt's build system so if you want to do that you are on your own.
It's easy to find out if this is the case, try rebuilding a single kmod with the SDK environment (from a normal x86_64 Linux system/VM) with V=ss added to the make command, and see what happens. If you see that it's rebuilding the kernel whole you have your answer.
