APK: fetch causes busybox "ls -la" to segfault

bigsmile · December 2, 2024, 10:04am

Yay! I found a segfault , that's a first.

Running OpenWrt SNAPSHOT r28214-5a4eb56a7b on a NanoPi-R6C with squashfs

Something goes wrong when fetching a package with apk. I recommend when verifying this, to do this in a in-memory-filesystem like /dev/shm , just in case it causes corruption.

# cd /dev/shm
# ls -la
drwxrwxrwt    2 root     root            40 Dec  2 09:41 .
drwxrwxrwt   18 root     root           460 Dec  2 09:41 ..
# apk fetch uboot-envtools
Downloading uboot-envtools-2024.07-r1
# ls -la
drwxrwxrwt    2 root     root            60 Dec  2 09:43 .
drwxrwxrwt   18 root     root           460 Dec  2 09:41 ..
Segmentation fault <--
#

# cp uboot-envtools-2024.07-r1.apk uboot-envtools-2024.07-r1.apk.copy

# stat uboot-envtools-2024.07-r1.apk
  File: uboot-envtools-2024.07-r1.apk
  Size: 16304     	Blocks: 32         IO Block: 4096   regular file
Device: 0,22	Inode: 236         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 8921497-01-09 02:32:28.000000000 +0000	<--
Modify: 8921546-03-08 18:19:04.000000000 +0000	<--
Change: 2024-12-02 09:43:56.774702254 +0000
 Birth: 2024-12-02 09:43:56.314686749 +0000

Look at those access and modify dates, that ain't right.

# stat uboot-envtools-2024.07-r1.apk.copy 
  File: uboot-envtools-2024.07-r1.apk.copy
  Size: 16304     	Blocks: 32         IO Block: 4096   regular file
Device: 0,22	Inode: 237         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-12-02 09:44:58.406779586 +0000
Modify: 2024-12-02 09:44:58.406779586 +0000
Change: 2024-12-02 09:44:58.406779586 +0000
 Birth: 2024-12-02 09:44:58.406779586 +0000

brada4 · December 2, 2024, 11:34am

What is logged in dmesg ?

bigsmile · December 2, 2024, 12:57pm

Nothing:

# dmesg | sha256sum
c7d163199a4d1fde5e29ff1e9b3a86f87d188a2b4630620661607bafc743204d  -
# apk fetch uboot-envtools
Downloading uboot-envtools-2024.07-r1
# dmesg | sha256sum
c7d163199a4d1fde5e29ff1e9b3a86f87d188a2b4630620661607bafc743204d  -
# ls -la
drwxrwxrwt    2 root     root            60 Dec  2 12:54 .
drwxrwxrwt   18 root     root           460 Dec  2 09:41 ..
Segmentation fault
#

brada4 · December 2, 2024, 1:30pm

logged during crash "pid XXX received SEGV" or so.

appelsin · December 2, 2024, 1:35pm

Wild .

appelsin · December 2, 2024, 1:40pm

... and strange that the shell doesn't crash, thought ls was a built-in.

Perhaps strace busybox ls -la could give some clues, hmm.

bigsmile · December 2, 2024, 1:40pm

Found something:

# logread | grep apk
Fri Nov 29 20:11:35 2024 kern.warn kernel: [ 1006.583541] apk[2537]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set
#

bigsmile · December 2, 2024, 1:46pm

Here ya go:

execve("/bin/busybox", ["/bin/busybox", "ls", "-la"], 0xffffcb58c3e0 /* 16 vars */) = 0
set_tid_address(0xffffa5d24e88)         = 6198
brk(NULL)                               = 0x28add000
brk(0x28adf000)                         = 0x28adf000
mmap(0x28add000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x28add000
openat(AT_FDCWD, "/etc/ld-musl-aarch64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/libgcc_s.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=131088, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 200704, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xffffa5c51000
mmap(0xffffa5c80000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1f000) = 0xffffa5c80000
close(3)                                = 0
mprotect(0xffffa5c80000, 4096, PROT_READ) = 0
mprotect(0x47f000, 4096, PROT_READ)     = 0
prctl(PR_GET_NAME, "busybox")           = 0
ioctl(0, TIOCGWINSZ, {ws_row=44, ws_col=94, ws_xpixel=0, ws_ypixel=0}) = 0
ioctl(1, TIOCGWINSZ, 0xffffe2caeeb8)    = -1 ENOTTY (Not a tty)
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, ".", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa5d1b000
getdents64(3, 0xffffa5d1b038 /* 4 entries */, 2048) = 128
newfstatat(AT_FDCWD, "./.", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=80, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "./..", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=460, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "./uboot-envtools-2024.07-r1.apk", {st_mode=S_IFREG|0644, st_size=16304, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "./log", {st_mode=S_IFREG|0644, st_size=1935, ...}, AT_SYMLINK_NOFOLLOW) = 0
getdents64(3, 0xffffa5d1b038 /* 0 entries */, 2048) = 0
close(3)                                = 0
munmap(0xffffa5d1b000, 8192)            = 0
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa5d1c000
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:0:root:/root:/bin/ash\nd"..., 1024) = 429
lseek(3, -398, SEEK_CUR)                = 31
close(3)                                = 0
munmap(0xffffa5d1c000, 4096)            = 0
openat(AT_FDCWD, "/etc/group", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffa5d1c000
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:\ndaemon:x:1:\nadm:x:4:\nm"..., 1024) = 233
lseek(3, -223, SEEK_CUR)                = 10
close(3)                                = 0
munmap(0xffffa5d1c000, 4096)            = 0
openat(AT_FDCWD, "/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
mmap(NULL, 4, PROT_READ, MAP_SHARED, 3, 0) = 0xffffa5d1c000
close(3)                                = 0
ioctl(1, TIOCGWINSZ, 0xffffe2caed98)    = -1 ENOTTY (Not a tty)
writev(1, [{iov_base="drwxrwxrwt    2 root     root   "..., iov_len=58}, {iov_base="\n", iov_len=1}], 2drwxrwxrwt    2 root     root            80 Dec  2 13:42 .
) = 59
munmap(0xffffa5d1c000, 4)               = 0
openat(AT_FDCWD, "/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
mmap(NULL, 4, PROT_READ, MAP_SHARED, 3, 0) = 0xffffa5d1c000
close(3)                                = 0
munmap(0xffffa5d1c000, 4)               = 0
openat(AT_FDCWD, "/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
mmap(NULL, 4, PROT_READ, MAP_SHARED, 3, 0) = 0xffffa5d1c000
close(3)                                = 0
munmap(0xffffa5d1c000, 4)               = 0
openat(AT_FDCWD, "/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
mmap(NULL, 4, PROT_READ, MAP_SHARED, 3, 0) = 0xffffa5d1c000
close(3)                                = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault

edit: missed the busybox in the cmdline, now its proper.

appelsin · December 2, 2024, 2:09pm

Thanks

Seems like it gets as far as beginning to print stuff...

Maybe crash happens during formatting of the bad timestamp?

(I imagine at this point it has already sorted the entries it wants to show the user, and figured out how wide the columns should be etc.)

Just for reference, here's the timestamp printing code, far as I can tell (click to expand):

Summary

561   #if ENABLE_FEATURE_LS_TIMESTAMPS
562   		/* long listing: show {m,c,a}time */
563   		if (opt & OPT_full_time) { /* --full-time */
564   			/* coreutils 8.4 ls --full-time prints:
565   			 * 2009-07-13 17:49:27.000000000 +0200
566   			 * we don't show fractional seconds.
567   			 */
568   			char buf[sizeof("YYYY-mm-dd HH:MM:SS TIMEZONE")];
569   			strftime(buf, sizeof(buf), "%Y-%m-%d %H:%M:%S %z",
570   					localtime(&dn->dn_time));
571   			column += printf("%s ", buf);
572   		} else { /* ordinary time format */
573   			/* G.current_time_t is ~== time(NULL) */
574   			char *filetime = ctime(&dn->dn_time);
575   			/* filetime's format: "Wed Jun 30 21:49:08 1993\n" */
576   			time_t age = G.current_time_t - dn->dn_time;
577   			if (age < 3600L * 24 * 365 / 2 && age > -15 * 60) {
578   				/* less than 6 months old */
579   				/* "mmm dd hh:mm " */
580   				printf("%.12s ", filetime + 4);
581   			} else {
582   				/* "mmm dd  yyyy " */
583   				/* "mmm dd yyyyy " after year 9999 :) */
584   				strchr(filetime + 20, '\n')[0] = ' ';
585   				printf("%.7s%6s", filetime + 4, filetime + 20);
586   			}
587   			column += 13;
588   		}
589   #endif

(from https://github.com/mirror/busybox/blob/master/coreutils/ls.c)

That's a null pointer dereference, right?

Perhaps if it's not too much trouble, opkg install gdb and gdb --args busybox ls -la would also be nice to see

(type "r" and hit to run)

bigsmile · December 2, 2024, 2:20pm

Did you look at the apk error?
apk[2537]: memfd_create() called without MFD_EXEC or MFD_NOEXEC_SEAL set

No problem:

# gdb --args /bin/busybox ls -la <<EOF
> r
> q
> EOF
GNU gdb (GDB) 15.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /bin/busybox...
(No debugging symbols found in /bin/busybox)
(gdb) Starting program: /bin/busybox ls -la
warning: Unable to determine the number of hardware watchpoints available.
warning: Unable to determine the number of hardware breakpoints available.
drwxrwxrwt    2 root     root            60 Dec  2 14:19 .
drwxrwxrwt   18 root     root           460 Dec  2 09:41 ..

Program received signal SIGSEGV, Segmentation fault.
0x0000fffff7fc8070 in ?? ()
(gdb) A debugging session is active.

	Inferior 1 [process 6922] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]
#

appelsin · December 2, 2024, 2:35pm

Yep certainly looks like a failure to allocate memory of some sort, although I'm not sure exactly what the message means (haven't seen it before).

An allocation failure would vibe well with a subsequent null pointer derefence though, so my personal guess would be that you found the smoking gun there...

Great, thanks!

Now we just need someone who knows how to turn that address into a line number in the source code, humn...

Hoping @brada4 knows all the things as usual

appelsin · December 2, 2024, 2:38pm

Guessing either somewhere around here or in localtime() or strftime().

On my box, ldd /bin/busybox says that it links to "musl" so could be where those functions are hiding out.

5    struct tm *__localtime_r(const time_t *restrict t, struct tm *restrict tm)
6    {
7    	/* Reject time_t values whose year would overflow int because
8    	 * __secs_to_zone cannot safely handle them. */
9    	if (*t < INT_MIN * 31622400LL || *t > INT_MAX * 31622400LL) {
10   		errno = EOVERFLOW;
11   		return 0;
12   	}
13   	__secs_to_zone(*t, 0, &tm->tm_isdst, &tm->__tm_gmtoff, 0, &tm->__tm_zone);
14   	if (__secs_to_tm((long long)*t + tm->__tm_gmtoff, tm) < 0) {
15   		errno = EOVERFLOW;
16   		return 0;
17   	}
18   	return tm;
19   }

(from https://github.com/bminor/musl/blob/master/src/time/localtime_r.c)

Looks like there is some sort of overflow handling in localtime() at least it can return EOVERFLOW.

musl strftime() is a bit long to list verbatim here, but link: https://github.com/bminor/musl/blob/master/src/time/strftime.c

Perhaps the ls applet was supposed to check the return value from localtime() for EOVERFLOW before it blindly calls strftime?

bigsmile · December 2, 2024, 2:57pm

Thanks, this was most insightful

appelsin · December 2, 2024, 3:51pm

Of course, a pleasure!

Here's a minimal reproduction test case:

~# busybox touch -d 8921497-01-09 bob
~# busybox ls -l bob
Illegal instruction

I think you might actually have found 3 bugs though...

busybox ls applet does a SIGSEGV.
apk plows crazy timestamps into the filesystem.
apk doesn't know how to memfd_create()

So.... good job?

Makes you wonder how many devices out there have a broken ls on them haha. Decent odds you could go to the nearest driveway and spot a modern car that contains this bug in some embedded system inside it. Wouldn't be too surprised if the local hospital's X-ray machine or some of the space rockets have a bit of busybox too. And so on...

I'd wager a guess that perhaps the crazy timestamp comes from uninitialized memory, judging from the dmesg output. But then again I don't know anything about apk.

brada4 · December 2, 2024, 3:59pm

9999-x-y ok
10000-x-y ko

appelsin · December 2, 2024, 4:04pm

Great find.

Pesky 5-digit years

brada4 · December 2, 2024, 4:05pm

Im surprised it goes past 2038 though

robimarko · December 2, 2024, 4:52pm

memfd is not the issue, its just a warning and it is fixed upstream:

I will either update APK or simply backport it to extinguish the warning.

bigsmile · December 2, 2024, 7:53pm

Submitted it to busybox:
https://bugs.busybox.net/show_bug.cgi?id=16267

brada4 · December 2, 2024, 9:52pm

                    char buf[sizeof("YYYY-mm-dd HH:MM:SS TIMEZONE")];