When I build an image for x86/64 using PR#16547 which enables the 6.12 kernel, starting nfsd causes a kernel triggers a kernel panic whereas running it on a snapshot with the 6.6 kernel does not.
I'm looking for a way to disable nfs callbacks to see if that can circumvent but am not finding a way. Does anyone have some knowledge with our nfs kmods/package?
dmesg
Kernel BUG at nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd] [verbose debug info unavailable]
Oops: invalid opcode: 0000 [#1] SMP NOPTI
CPU: 2 UID: 0 PID: 17091 Comm: rpc.nfsd Not tainted 6.12.19 #0
Hardware name: iKOOLCORE R2Max/R2Max, BIOS v1.1 12/17/2024
RIP: 0010:nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd]
Code: 89 de 48 c7 c7 e0 20 95 a0 e8 08 fc ff ff 41 89 c4 85 c0 0f 85 ce 2d 00 00 48 c7 c7 58 dd 95 a0 45 31 e4 e8 8e 7a 0b e1 eb 08 <0f> 0b 41 bc f4 ff ff ff 48 83 c4 08 44 89 e0 5b 41 5c 41 5d 41 5e
RSP: 0018:ffffc9000613bc98 EFLAGS: 00010282
RAX: 0000000000000049 RBX: ffff888105e06000 RCX: 0000000000000000
RDX: ffff88846fb21920 RSI: ffff88846fb1d980 RDI: ffff88846fb1d980
RBP: ffffc9000613bcc0 R08: 0000000000000000 R09: ffffffff824b56c8
R10: ffffffff8249d688 R11: 0000000000000003 R12: ffffffff82736400
R13: ffff888105e06000 R14: ffff888105e06000 R15: ffffc9000613bcd0
FS: 00007f1a404e5b28(0000) GS:ffff88846fb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a40411000 CR3: 000000010d47a006 CR4: 0000000000370ef0
Call Trace:
<TASK>
? show_regs.part.0+0x1d/0x20
? __die+0x52/0x91
? die+0x2a/0x50
? do_trap+0x103/0x110
? do_error_trap+0x6c/0x90
? nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd]
? exc_invalid_op+0x4f/0x70
? nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd]
? asm_exc_invalid_op+0x1b/0x20
? nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd]
? nfsd4_probe_callback_sync+0xff6/0x25b0 [nfsd]
nfsd4_client_tracking_init+0x39/0x150 [nfsd]
nfs4_state_start_net+0x2ce/0x370 [nfsd]
nfsd_svc+0x1a0/0x2d0 [nfsd]
nfssvc_encode_voidres+0x19a1/0x1be0 [nfsd]
? simple_transaction_get+0xb7/0xe0
? nfssvc_encode_voidres+0x18f0/0x1be0 [nfsd]
nfssvc_encode_voidres+0x1a3/0x1be0 [nfsd]
vfs_write+0xcb/0x390
? putname+0x4c/0x60
ksys_write+0x57/0xd0
__x64_sys_write+0x14/0x20
x64_sys_call+0x79/0x1780
do_syscall_64+0x7b/0x190
entry_SYSCALL_64_after_hwframe+0x67/0x6f
RIP: 0033:0x7f1a404c9659
Code: c3 8b 07 85 c0 75 24 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> e9 82 d8 ff ff 41 54 b8 02 00 00 00 55 48 89 f5 be 00 88 08 00
RSP: 002b:00007fff99aeb2c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f1a404c9659
RDX: 0000000000000003 RSI: 000000000040d360 RDI: 0000000000000003
RBP: 00007f1a404e5b28 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 0000000000020000 R14: 0000000000409112 R15: 00007f1a40430dd0
</TASK>
Modules linked in: xt_connlimit pppoe ppp_async nf_conncount i915 xt_state xt_helper xt_conntrack xt_connmark xt_connlabel xt_connbytes xt_CT wireguard video pppox ppp_mppe ppp_generic nft_redir nft_nat nft_masq nft_flow_offload nft_fib_inet nft_ct nft_chain_nat nf_nat nf_flow_table_inet nf_flow_table nf_conntrack_netlink nf_conntrack libchacha20poly1305 ipt_REJECT iavf i40e curve25519_x86_64 chacha_x86_64 cdc_ncm cdc_ether aqc111 zstd xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_quota xt_pkttype xt_owner xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY xt_CHECKSUM wmi usbnet unix_diag ts_kmp ts_fsm ts_bm slhc sch_cake poly1305_x86_64 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_queue nft_numgen nft_log nft_limit nft_hash nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_compat nfnetlink_queue nf_tables nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 macvlan lzo_rle lzo libie
libeth libcurve25519_generic libchacha iptable_mangle iptable_filter ipt_ECN ip_tables forcedeth e1000e drm_display_helper drm_buddy crc_ccitt cdc_acm bnx2 atlantic af_packet_diag sch_teql sch_sfq sch_multiq sch_gred sch_fq sch_ets sch_codel em_text em_nbyte em_meta em_cmp act_skbmod act_simple act_pedit act_csum sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs evdev drivetemp i2c_piix4 i2c_smbus i2c_dev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink dwmac_intel ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 dwmac_generic
stmmac_platform stmmac nfsd nfs bonding ixgbe igbvf e1000 amd_xgbe ifb sctp udp_tunnel ip6_udp_tunnel mdio netlink_diag udp_diag tcp_diag raw_diag inet_diag rpcsec_gss_krb5 auth_rpcgss oid_registry veth lockd sunrpc grace dns_resolver nls_utf8 pcs_xpcs macsec ena ecdh_generic ecc xxhash_generic crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_ssse3 sha512_generic sha1_ssse3 sha1_generic seqiv sha3_generic jitterentropy_rng drbg md5 kpp hmac geniv rng des_generic libdes cts chacha20poly1305 blake2b_generic authencesn authenc arc4 crypto_acompress nls_iso8859_1 nls_cp437 r8169 mdio_devres xhci_plat_hcd iTCO_wdt iTCO_vendor_support fsl_mph_dr_of ehci_platform ehci_fsl igb igc vfat fat btrfs zstd_decompress zstd_compress zstd_common xxhash xor raid6_pq lzo_decompress lzo_compress libcrc32c dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax tg3 realtek phylink mii libphy tpm cbc sha256_ssse3 sha256_generic encrypted_keys trusted
---[ end trace 0000000000000000 ]---
RIP: 0010:nfsd4_probe_callback_sync+0x1094/0x25b0 [nfsd]
Code: 89 de 48 c7 c7 e0 20 95 a0 e8 08 fc ff ff 41 89 c4 85 c0 0f 85 ce 2d 00 00 48 c7 c7 58 dd 95 a0 45 31 e4 e8 8e 7a 0b e1 eb 08 <0f> 0b 41 bc f4 ff ff ff 48 83 c4 08 44 89 e0 5b 41 5c 41 5d 41 5e
RSP: 0018:ffffc9000613bc98 EFLAGS: 00010282
RAX: 0000000000000049 RBX: ffff888105e06000 RCX: 0000000000000000
RDX: ffff88846fb21920 RSI: ffff88846fb1d980 RDI: ffff88846fb1d980
RBP: ffffc9000613bcc0 R08: 0000000000000000 R09: ffffffff824b56c8
R10: ffffffff8249d688 R11: 0000000000000003 R12: ffffffff82736400
R13: ffff888105e06000 R14: ffff888105e06000 R15: ffffc9000613bcd0
FS: 00007f1a404e5b28(0000) GS:ffff88846fb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a40411000 CR3: 000000010d47a006 CR4: 0000000000370ef0
brada4
March 22, 2025, 11:56am
2
You mix xt and nf conntracks, none the wiser, but it is documented to cause unpredictable effects.
Not sure why that is... I do not remembering selecting any specific kmod, guess dependencies of other packages. As I test, I renamed all of the xt_conn*.ko to xf_conn*.no and rebooted but got the same panic.
# cd /lib/modules/6.12.1
# ls -lh | grep xt_conn
-rw-r--r-- 1 root root 17K Mar 22 08:38 nf_conncount.ko
-rw-r--r-- 1 root root 146K Mar 22 08:38 nf_conntrack.ko
-rw-r--r-- 1 root root 47K Mar 22 08:38 nf_conntrack_netlink.ko
-rw-r--r-- 1 root root 5.5K Mar 22 08:38 xt_connbytes.ko.no
-rw-r--r-- 1 root root 5.9K Mar 22 08:38 xt_connlabel.ko.no
-rw-r--r-- 1 root root 6.0K Mar 22 08:38 xt_connlimit.ko.no
-rw-r--r-- 1 root root 7.7K Mar 22 08:38 xt_connmark.ko.no
-rw-r--r-- 1 root root 6.9K Mar 22 08:38 xt_conntrack.ko.no
dmesg_with_xf_conn_kmods_renamed.log
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.130421] __x64_sys_write+0x14/0x20
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.134728] x64_sys_call+0x79/0x1780
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.138921] do_syscall_64+0x7b/0x190
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.143115] entry_SYSCALL_64_after_hwframe+0x67/0x6f
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.148864] RIP: 0033:0x7f3087ed2659
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.152972] Code: c3 8b 07 85 c0 75 24 49 89 fb 48 89 f0 48 89 d7 48 89 ce 4c 89 c2 4d 89 ca 4c 8b 44 24 08 4c 8b 4c 24 10 4c 89 5c 24 08 0f 05 <c3> e9 82 d8 ff ff 41 54 b8 02 00 00 00 55 48 89 f5 be 00 88 08 00
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.174268] RSP: 002b:00007ffcbf58f058 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.182838] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3087ed2659
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.190931] RDX: 0000000000000003 RSI: 000000000040d360 RDI: 0000000000000003
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.199025] RBP: 00007f3087eeeb28 R08: 0000000000000000 R09: 0000000000000000
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.207105] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.215194] R13: 0000000000020000 R14: 0000000000409112 R15: 00007f3087e39dd0
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.223274] </TASK>
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.225820] Modules linked in: pppoe ppp_async nf_conncount i915 xt_state xt_helper xt_CT wireguard video pppox ppp_mppe ppp_generic nft_redir nft_nat nft_masq nft_flow_offload nft_fib_inet nft_ct nft_chain_nat nf_nat nf_flow_table_inet nf_flow_table nf_conntrack_netlink nf_conntrack libchacha20poly1305 ipt_REJECT iavf curve25519_x86_64 chacha_x86_64 cdc_ncm cdc_ether aqc111 zstd xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_quota xt_pkttype xt_owner xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY xt_CHECKSUM wmi usbnet unix_diag ts_kmp ts_fsm ts_bm slhc sch_cake poly1305_x86_64 nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_queue nft_numgen nft_log nft_limit nft_hash nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_compat nfnetlink_queue nf_tables nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 macvlan lzo_rle lzo libeth libcurve25519_generic libchacha it87(O) iptable_mangle iptable_filter
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.225853] ipt_ECN ip_tables forcedeth e1000e drm_display_helper drm_buddy crc_ccitt cdc_acm bnx2 atlantic af_packet_diag sch_teql sch_sfq sch_multiq sch_gred sch_fq sch_ets sch_codel em_text em_nbyte em_meta em_cmp act_skbmod act_simple act_pedit act_csum sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact configs evdev drivetemp i2c_piix4 i2c_smbus i2c_dev cryptodev(O) xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink dwmac_intel ip6t_rt ip6t_mh ip6t_ipv6header ip6t_hbh ip6t_frag ip6t_eui64 ip6t_ah ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 dwmac_generic stmmac_platform stmmac nfsd nfs bonding ixgbe igbvf e1000 amd_xgbe ifb
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.327740] sctp udp_tunnel ip6_udp_tunnel mdio netlink_diag udp_diag tcp_diag raw_diag inet_diag rpcsec_gss_krb5 auth_rpcgss oid_registry veth lockd sunrpc grace dns_resolver nls_utf8 pcs_xpcs macsec ena ecdh_generic ecc xxhash_generic crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha512_ssse3 sha512_generic sha1_ssse3 sha1_generic seqiv sha3_generic jitterentropy_rng drbg md5 kpp hmac geniv rng des_generic libdes cts chacha20poly1305 blake2b_generic authencesn authenc arc4 crypto_acompress nls_iso8859_1 nls_cp437 r8169 mdio_devres xhci_plat_hcd sp5100_tco iTCO_wdt iTCO_vendor_support fsl_mph_dr_of ehci_platform ehci_fsl i40e libie igb igc vfat fat btrfs zstd_decompress zstd_compress zstd_common xxhash xor raid6_pq lzo_decompress lzo_compress libcrc32c dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax button_hotplug(O) tg3 realtek phylink mii libphy tpm cbc sha256_ssse3 sha256_generic encrypted_keys trusted
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.523361] ---[ end trace 0000000000000000 ]---
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.528722] RIP: 0010:nfsd4_probe_callback_sync+0x10c4/0x25e0 [nfsd]
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.536008] Code: 89 de 48 c7 c7 60 df a0 a0 e8 08 fc ff ff 41 89 c4 85 c0 0f 85 ae 2d 00 00 48 c7 c7 58 1d a2 a0 45 31 e4 e8 ce 4a ff e0 eb 08 <0f> 0b 41 bc f4 ff ff ff 48 83 c4 08 44 89 e0 5b 41 5c 41 5d 41 5e
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.557426] RSP: 0018:ffffc90008a47c98 EFLAGS: 00010282
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.563463] RAX: 0000000000000049 RBX: ffff888103e36000 RCX: 0000000000000000
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.571631] RDX: ffff8886c3d618e0 RSI: ffff8886c3d5d980 RDI: ffff8886c3d5d980
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.579792] RBP: ffffc90008a47cc0 R08: 0000000000000000 R09: ffffffff822b5608
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.587958] R10: ffffffff8229d5c8 R11: 0000000000000003 R12: ffffffff82535540
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.596113] R13: ffff888103e36000 R14: ffff888103e36000 R15: ffffc90008a47cd0
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.604270] FS: 00007f3087eeeb28(0000) GS:ffff8886c3d40000(0000) knlGS:0000000000000000
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.613503] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sat Mar 22 12:41:58 2025 kern.warn kernel: [ 21.620117] CR2: 00007fb3902c4ea0 CR3: 000000010f574000 CR4: 0000000000350ef0
brada4
March 22, 2025, 5:51pm
4
Run nft list ruleset | grep xt and add nft modules replacing xt passthrough according to dictionary https://wiki.nftables.org/wiki-nftables/index.php/Supported_features_compared_to_xtables
The output of that command is nothing:
# nft list ruleset | grep xt
# echo $?
1
brada4
March 22, 2025, 6:34pm
6
full upper backtrace part please. Task info is of mini help
Happy to report that this was an upstream issue, see: https://bugzilla.kernel.org/show_bug.cgi?id=219911#c1
2 Likes
system
April 1, 2025, 6:56pm
8
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.