I am experiencing OpenSSL libcrypto segmentation faults on OpenWRT 24.10 branch, triggered by browsing LuCI pages (served by uhttpd).
[18521.543248] uhttpd[23671]: segfault at 7fdc30fee050 ip 00007fdc312494d0 sp 00007fff93910538 error 4 in libcrypto.so.3[7fdc3120b000+26a000] likely on CPU 2 (core 2, socket 0)
[18521.558742] Code: 00 00 31 c0 5b c3 0f 1f 40 00 31 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 77 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 <48> 8b 47 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 89 77 28 c3 66 2e
[18610.413906] uhttpd[1893]: segfault at 7fd7c26d0440 ip 00007fd7c28e94d0 sp 00007fff0a02e688 error 4 in libcrypto.so.3[7fd7c28ab000+26a000] likely on CPU 4 (core 4, socket 0)
[18610.429321] Code: 00 00 31 c0 5b c3 0f 1f 40 00 31 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 77 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 <48> 8b 47 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 89 77 28 c3 66 2e
[18624.719284] uhttpd[2168]: segfault at 7f319b174b90 ip 00007f319b39b4d0 sp 00007ffdcf8eca08 error 4 in libcrypto.so.3[7f319b35d000+26a000] likely on CPU 1 (core 1, socket 0)
[18624.734686] Code: 00 00 31 c0 5b c3 0f 1f 40 00 31 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 77 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 <48> 8b 47 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 89 77 28 c3 66 2e
[18650.676000] uhttpd[2461]: segfault at 7f318af87050 ip 00007f318b1f64d0 sp 00007ffd2f13c418 error 4 in libcrypto.so.3[7f318b1b8000+26a000] likely on CPU 0 (core 0, socket 0)
[18650.691500] Code: 00 00 31 c0 5b c3 0f 1f 40 00 31 c0 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 48 89 77 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 <48> 8b 47 40 c3 66 2e 0f 1f 84 00 00 00 00 00 90 89 77 28 c3 66 2e
Tend to occur when requesting a new LuCI page while the previous one is still loading. Front end symptom is XHR Request aborted by browser
24.10 branch was updated to OpenSSL 3.0.17 about 2 months ago and there is a known regression with concurrent SSL connections in 3.0.17 which will be fixed in 3.0.18. https://github.com/openssl/openssl/issues/28171
Can someone with the right powers update the package version of OpenSSL in the 24.10 branch to the fixed version (at least 3.0.18).
Thanks!
EDIT - just realized 3.0.18 is not yet released, so probably best to revert to 3.0.16 for now.
root@router:~# ubus call system board
{
"kernel": "6.6.104",
"hostname": "router",
"system": "Intel(R) Xeon(R) D-2145NT CPU @ 1.90GHz",
"model": "DELL VEP-4600",
"board_name": "dell-vep-4600",
"rootfs_type": "ext4",
"release": {
"distribution": "OpenWrt",
"version": "24.10-SNAPSHOT",
"revision": "r28863-f3a27d09e2",
"target": "x86/64",
"description": "OpenWrt 24.10-SNAPSHOT r28863-f3a27d09e2",
"builddate": "1757697958"
}
}
root@router:~# dmesg | grep segfault
[ 1038.101914] uhttpd[4161]: segfault at 7f4e41f44170 ip 00007f4e4215d4d0 sp 00007ffe402e1ec8 error 4 in libcrypto.so.3[7f4e4211f000+26a000] likely on CPU 6 (core 6, socket 0)
[ 2257.542487] uhttpd[15737]: segfault at 7f8f89fca050 ip 00007f8f8a2314d0 sp 00007ffd38f8f988 error 4 in libcrypto.so.3[7f8f8a1f3000+26a000] likely on CPU 1 (core 1, socket 0)
[ 2941.085695] uhttpd[17255]: segfault at 7f5611cea0e0 ip 00007f5611f004d0 sp 00007ffe8e9d9be8 error 4 in libcrypto.so.3[7f5611ec2000+26a000] likely on CPU 7 (core 7, socket 0)
[ 2980.476647] uhttpd[23144]: segfault at 7f6f68665c20 ip 00007f6f688bd4d0 sp 00007ffe0399e668 error 4 in libcrypto.so.3[7f6f6887f000+26a000] likely on CPU 7 (core 7, socket 0)
[18521.543248] uhttpd[23671]: segfault at 7fdc30fee050 ip 00007fdc312494d0 sp 00007fff93910538 error 4 in libcrypto.so.3[7fdc3120b000+26a000] likely on CPU 2 (core 2, socket 0)
[18610.413906] uhttpd[1893]: segfault at 7fd7c26d0440 ip 00007fd7c28e94d0 sp 00007fff0a02e688 error 4 in libcrypto.so.3[7fd7c28ab000+26a000] likely on CPU 4 (core 4, socket 0)
[18624.719284] uhttpd[2168]: segfault at 7f319b174b90 ip 00007f319b39b4d0 sp 00007ffdcf8eca08 error 4 in libcrypto.so.3[7f319b35d000+26a000] likely on CPU 1 (core 1, socket 0)
[18650.676000] uhttpd[2461]: segfault at 7f318af87050 ip 00007f318b1f64d0 sp 00007ffd2f13c418 error 4 in libcrypto.so.3[7f318b1b8000+26a000] likely on CPU 0 (core 0, socket 0)
[118966.384011] uhttpd[2547]: segfault at 7f06a7a4f320 ip 00007f06a7c944d0 sp 00007ffec829da38 error 4 in libcrypto.so.3[7f06a7c56000+26a000] likely on CPU 4 (core 4, socket 0)
[120924.147027] uhttpd[29017]: segfault at 7eff4b24a4d0 ip 00007eff4b4634d0 sp 00007fffd0f1e248 error 4 in libcrypto.so.3[7eff4b425000+26a000] likely on CPU 6 (core 6, socket 0)
[120959.004978] uhttpd[3090]: segfault at 7f94567fe290 ip 00007f9456a254d0 sp 00007ffc60647ae8 error 4 in libcrypto.so.3[7f94569e7000+26a000] likely on CPU 3 (core 3, socket 0)
root@router:~#
Why not? I thought the decision tree of risk was 1. Stable Release, 2. Stable Release Branch, 3. master.
I’m sure I read somewhere that release branches (i.e. 24.10) had only ‘carefully considered backports from master’.
And in any case, if no-one used SNAPSHOT, no-one would catch things like this and OpenSSL 3.0.17 might very well end up in OpenWRT 24.10.3 - it seems to have caused a lot of problems for other projects (Debian, MariaDB, Python, Node etc).
Might not be worthwhile to revert to 3.0.16 (if 3.0.18 would be expected in nearby future).
The bug occurence must be rather rare, as the openssl 3.0.17 has been on both master and 24.10 code for over two months, and we do not have a flood of bug reports. Note also that OpenSSL is not the default SSL lib, so it doesn't get into any official release images. Only to private builds.
I am using openssl based OpenWrt on all my routers, and I have not seen the bug. I managed now to get trigger the XHR warning (in master), but did not manage to get crashes into log.