Build failed with SELinux

Hi. I'm coming to ask about the problems I met when trying to build and run OpenWRT with SELinux.

I have turned on the following options related to SELinux in menuconfig:

  • Base System -> selinux-policy
  • Development -> gcc, m4, make
  • Global Build Settings -> default SELinux type -> targeted
  • Utilities -> libselinux tools -> all the utilities
  • Utilities -> selinux-audit2allow, selinux-chcat, selinux-semanage, semodule-utils, setools, secilc, policycoreutils

When I tried to turn on the "Support NSA Security Enhanced Linux" option (located at "Base system"->"Customize busybox options"->"Settings" in menuconfig), the compiling process failed with the following notification:

/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans0.ltrans.o: in function `setfscreatecon_or_die':
<artificial>:(.text.setfscreatecon_or_die+0x6): undefined reference to `setfscreatecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans0.ltrans.o: in function `procps_scan':
<artificial>:(.text.procps_scan+0xd1): undefined reference to `getpidcon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans0.ltrans.o: in function `update_passwd':
<artificial>:(.text.update_passwd+0x10f): undefined reference to `is_selinux_enabled'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x122): undefined reference to `getprevcon_raw'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x174): undefined reference to `string_to_security_class'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x191): undefined reference to `string_to_av_perm'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x19d): undefined reference to `selinux_check_passwd_access'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x22d): undefined reference to `fgetfilecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.update_passwd+0x263): undefined reference to `freecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans0.ltrans.o: in function `copy_file':
<artificial>:(.text.copy_file+0xde): undefined reference to `is_selinux_enabled'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0xf0): undefined reference to `lgetfilecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0xff): undefined reference to `setfscreatecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0x491): undefined reference to `is_selinux_enabled'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0x4a0): undefined reference to `getfscreatecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0x4c6): undefined reference to `setfilecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0x4e9): undefined reference to `freecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.copy_file+0x4f7): undefined reference to `freecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans2.ltrans.o: in function `my_stat':
<artificial>:(.text.my_stat+0x4c): undefined reference to `getfilecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: <artificial>:(.text.my_stat+0x78): undefined reference to `lgetfilecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans2.ltrans.o: in function `display_single':
<artificial>:(.text.display_single+0x17a): undefined reference to `freecon'
/home/aurora/iot/openwrt-try2/openwrt/staging_dir/toolchain-x86_64_gcc-8.4.0_musl/lib/gcc/x86_64-openwrt-linux-musl/8.4.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/aurora/iot/openwrt-try2/openwrt/tmp/ccV977D8.ltrans2.ltrans.o: in function `ls_main':
<artificial>:(.text.ls_main+0x7a): undefined reference to `is_selinux_enabled'
...
collect2: error: ld returned 1 exit status
Note: if build needs additional libraries, put them in CONFIG_EXTRA_LDLIBS.
Example: CONFIG_EXTRA_LDLIBS="pthread dl tirpc audit pam"
make[4]: *** [Makefile:718: busybox_unstripped] Error 1
make[4]: Leaving directory '/home/aurora/iot/openwrt-try2/openwrt/build_dir/target-x86_64_musl/busybox-default/busybox-1.33.0'
make[3]: *** [Makefile:160: /home/aurora/iot/openwrt-try2/openwrt/build_dir/target-x86_64_musl/busybox-default/busybox-1.33.0/.built] Error 2
make[3]: Leaving directory '/home/aurora/iot/openwrt-try2/openwrt/package/utils/busybox'
time: package/utils/busybox/default/compile#21.77#3.04#5.13
    ERROR: package/utils/busybox failed to build (build variant: default).
make[2]: *** [package/Makefile:114: package/utils/busybox/compile] Error 1
make[2]: Leaving directory '/home/aurora/iot/openwrt-try2/openwrt'
make[1]: *** [package/Makefile:108: /home/aurora/iot/openwrt-try2/openwrt/staging_dir/target-x86_64_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/aurora/iot/openwrt-try2/openwrt'
make: *** [/home/aurora/iot/openwrt-try2/openwrt/include/toplevel.mk:230: world] Error 2

Then I tried to turn off the "Support NSA Security Enhanced Linux" option, successfully built the OpenWRT. I compiled a Busybox statically with the "Support NSA Security Enhanced Linux" option outside the OpenWRT building process seperately, then I replaced the BusyBox in the OpenWRT with the seperately built one.

Through such compilation process, the SELinux component cannot work functionally in OpenWRT. When I tried to execute:

semanage login -l

The output was:

ValueError: SELinux policy is not managed or store cannot be accessed.

Edit the context of a file with:

semanage fcontext -a -t tmp_fs file1

The output:

ValueError: SELinux policy is not managed or store cannot be accessed.

Tried to load the SELinux policy again with:

load_policy

The log was:

...
Permission bpf in cap_userns not defined in policy.
the above unknown classes and permissions will be denied.
Converting 23 SID table entries...
policy capability network_peer_controls=1
...
audit: type=1403 audit(1618283426.064:7): audit=4294967295 ses=4294967295 lsm=selinux res=1

Apparently, some components related to SELinux are missing in the OpenWRT. And I am aware that such BusyBox replacement is not the most appropriate way to build OpenWRT with SELinux. But the normal build option cannot proceed.

Could you please give me some advice about compiling OpenWRT with SELinux? Was there any libraries or components missed in the building process described above? Thank you!

1 Like

Building busybox with just enabling the SELinux options in that way will not work with the way OpenWrt does it -- we have an extra variant called busybox-selinux which is exactly that: busybox built with SELinux enabled against libselinux.
I recommend you to start over (ie. with clean new .config) and first try it like this:

Global build settings  --->
[*] Enable SELinux
    default SELinux type (dssp)  --->

You don't need to make any other changes to configuration, SELinux-enabled variants of procd and busybox will be selected automatically if you select Enable SELinux (ie. CONFIG_SELINUX=y).

Regarding your problems when using targeted refpolicy:
This is expected, refpolicy has not been adapted to work with OpenWrt and it would be a lot of work to do so. It exists for historic reasons as it was useful to bring up the SELinux tooling in OpenWrt, maybe we should mark it as BROKEN to avoid having people wasting their time with it.

The good news is that there is dssp, which is a minimal policy design specifically for OpenWrt.

In other words: Just keep the default selection dssp instead of getting into the mess of trying to use refpolicy (targeted) on OpenWrt.

Please let me know if you succeeded and feel free to ask for further advise, if needed.

2 Likes

I've followed your advice and the compilation process seems great. The default file contexts and process contexts in the squashfs image also worked very well. (Although the ext4 image seems problematic)
But "semanage" is a quite common tool used in SELinux system. It seems that "semanage" is a neccessary tool to set file context and check whether SELinux actually works or not. So I came back and turn on the "Utilities -> selinux-semanage" option in menuconfig.

The compilation succeeded. The "sestatus" seems fine. But I still failed to execute "semanage login -l":

ValueError: SELinux policy is not managed or store cannot be accessed.

Failed to change a file context when I executed "semanage fcontext -a -t tmpfile.conftmpfile /tmp/file1":

ValueError: SELinux policy is not managed or store cannot be accessed.

btw, after I turned on all the SELinux-related options in "Utilities" like "libselinux-tools", the compilation worked fine, but the above mentioned commands still cannot work.

So, did the SELinux actually worked? Or did I missed something or used the wrong commands?

Thank you so much for your time! Looking forward to your reply.

Zijing

We are not using semanage, there is nothing to configure when using the dssp (which is the only truly supported way of using SELinux on OpenWrt).
It's really more like SELinux on Android than SELinux refpolicy running on RHEL. So no semanage, sebool, ...
Check if sestatus telling you it's 'enforcing', if so you succeeded.

Thank you for your reply!

Sad to hear that dssp policy cannot be customized by users. I was planning to try out some SELinux policies, doing experiment to make OpenWRT more secure.

If I want to figure out some new policies, is "targeted" mode(which is highly unusable) the only way to do so? Is there any chance that I can edit the policies contained in dssp? If so, how could I achieve that?

Can I change the policy inside the OpenWRT system after booting(without using semanage)? Or can I only change the policy outside the system, and then substitute the policy file contained in the original OpenWRT img, then flash that into the device?

Thank you so much for your time! Looking forward to your reply!

Zijing

You can customize dssp: here is a tutorial (although might be outdated) https://github.com/doverride/openwrt-selinux-policy/blob/master/README.md

The only difference is that changes are made at compile time and not at run time.

You can still use tools like chcon to set contexts on files, but semanage is a tool to customize the policy at run-time and that functionality is not supported. It cannot reasonably be supported due to the nature of the system (policy is loaded from read-only squashfs, wireless router devices generally lack resources to compile policy at run-time)

What issues did you encounter with dssp on ext4 image? SELinux will print its feedback to the kernel ring buffer, so you can grep that (dmesg | grep -i denied) and report those so that they can be interpreted and addressed.

You can surely customize dssp as well, just that happens at build-time and not dynamically during run-time as you may be used on non-embedded platforms.

Yes, something like that. Substituting only the policy file itself is usually not enough, as also labels are applied from the policy. I use CONFIG_SRC_TREE_OVERRIDE to build using a local git checkout of dssp policy I'm working on and then also labels from there are applied on the rootfs.

And yes, only squashfs is supported for now as that's what OpenWrt is mostly all about. Using rootfs filesystems other than squashfs is only intended for development/debugging.

Is there any kind of target restriction, I have not been able to get by thread

Thank you for your help!~
I've tried out some experiment. The basic allow/deny worked fine on ext4 image. The only problem encountered in ext4 was the insufficient default file context / process context.

For example, the default context of the root path in squashfs is:

But the default context of the root path in ext4 image is:

Same issue occurred in process context. This doesn't affect my study and some little experiment at the moment, since the basic allow/deny works fine. But I'm still a little bit worry about the reason why such default contexts cannot be applied correctly on files in ext4 image.

Could you please tell me the reason? Will this affect other SELinux utilities?

Thank you!

Zijing

Thank you so much for your help! That explains a lot! I'm going to try out some simple self-customized policies to start.

I truly appreciate your timely help!

Zijing

I'm not an expert on this, but I've encountered this problem before when I was trying to build images for MIPS architecture.

You can try turning off the "busybox" option in "Base system" in menuconfig, keep the auto-selected "busybox-selinux" only.

Turn off the "procd" in "Base system", keep only the auto-selected "procd-selinux":

Although I'm not an expert, and these are just some dummy experience-based suggestions, you can still give it a try~

Zijing

Seems that the build-system did not label the ext4 filesystem. I think this is a known issue that also happens when you use a tmpfs rootfs. As Daniel stated, the focus is on squashfs with overlay currently (but I do consider this to be a loose end that should eventually be addressed).

Obviously this will affect SELinux enforcement because the labels are incorrect. So, i guess for now it is safe to say that SELinux does not work reliably on images with ext4 and tmpfs rootfs.

The labels on your squashfs installation look fine, with the exception of the /boot mountpoint. Not sure why you even have a /boot mountpoint but regardless that minor issue should not affect operations. Do you have any filesystem mounted on /boot, if so: which? and why?

Thank you for your time!

I didn't mount anything on purpose. I was just using the squashfs-combined type of image. Seems like the loader component like "grub" is mounted on that. I guess this will not cause any extra problems, right?

Thanks for your help!

Zijing

should not cause any issues, but its not really pretty either.

is there any point to having that stuff though? I mean it boots from a read-only squashfs so you cannot edit /boot/grub/grub.cfg anyway and expect that to take affect on the next boot since at that point the overlay is not mounted yet.

Actually I'm a little bit confused too :joy::joy: But it just be mounted automatically when I use qemu to boot OpenWRT with something like:

sudo qemu-system-x86_64 -drive file=openwrt-x86-64-generic-squashfs-combined.img,format=raw,if=virtio -m 256 -nographic

and /boot is already mounted.

Zijing

Hi. I customized dssp based on the tutorial, which is well-written and easy to follow.

Specifically, I added some auditallow statements to observe the behavior of my program.
Due to the massive logs, I want to change the context of the running program (a process actually) to a new label/context auditing fewer operations, when the audit log reaches the amount limitation.

How can I do this——I mean how to change the context of a running process?
Any command or interesting idea? I really need your advice.
Thank you!

Contexts transitions generally either only happen on execve or create with automatic transition rules, manually with the runcon command or programmatically with the setcon() function: https://linux.die.net/man/3/setcon

I guess closest to what you are looking for would be setcon().

Depending on what you need this for there may be better solutions, like dontaudit and typepermissive statements, or customization to your logging configuration.

Thank you for this timely reply.

So as you said, I can only re-run the program with a different context, but can not change the context of the already running program. Do I understand right?

Besides, I fail to execute the ausearch command in the OpenWrt and get the following hint:

Config file /etc/audit/auditd.conf doesn't exist, skipping
Error opening /var/log/audit/audit.log (No such file or directory)

It seems that SELinux denial messages are logged to /var/log/audit/audit.log, but this file is missing in the OpenWrt.

If I can‘t using ausearch, where can I get the original audit log file in the OpenWrt? In fact, I wanted a more detailed parsing of the raw logs output by the dmesg command.

1 Like

Right, unless you use the setcon() function in your program (but with various limitations as mentioned in the link to man setcon(3))

You should reconsider whether you actually want or need audit on your device because that comes with overhead that might not be worth it. Unless you know that you need audit you probably should not use it.

When you do not have audit installed then SELinux will log to the kernel ring buffer. You can use the dmesg command to read and manage it. One of the nice aspects of this feature is that it is just memory and you can define how much memory it can use. The logd component also reads the kernel ring buffer I believe and so you can use that as well (for example you could use logread with all its features.

If you really need audit for some reason then, I suppose, you have to create those /etc/audit/auditd.conf and /var/log/audit/audit.log files yourself.

I recommend you not use audit.