NFS with kerberos (krb5) not working

justnulling · December 22, 2024, 10:02pm

Hi all,

I have BPI-R3 running OpenWrt 23.05.5 (nfs.example.com) and would like to export nfs share with kerberos, but it fails with Permission denied (err -13).

Here I am exporting share with sec=sys.

root@NFS:~# cat /etc/exports
/data/export 192.168.0.0/16(fsid=0,ro,sec=sys,sync,no_subtree_check,no_root_squash)

And mont works

root@edge:~# mount -v -t nfs -o vers=4.2,sec=sys nfs.example.com:/ /mnt/mounted
mount.nfs: timeout set for Sun Dec 22 16:04:56 2024
mount.nfs: trying text-based options 'vers=4.2,sec=sys,addr=192.168.120.1,clientaddr=192.168.20.50'

But after exporting it with sec=krb5.

root@NFS# cat /etc/exports
/data/export 192.168.0.0/16(fsid=0,ro,sec=krb5,sync,no_subtree_check,no_root_squash)

I am not able to mount it:

root@edge# mount -v -t nfs -o vers=4.2,sec=krb5 nfs.example.com:/ /mnt/mounted
mount.nfs: timeout set for Sun Dec 22 15:59:23 2024
mount.nfs: trying text-based options 'vers=4.2,sec=krb5,addr=192.168.120.1,clientaddr=192.168.20.50'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting nfs.example.com:/

/data is ext4 partition on nvm

root@R3-LYN:~# mount | grep /data
/dev/nvme0n1p2 on /data type ext4 (rw,relatime)

Looks like all needed models are loaded:

root@NFS:~# lsmod |grep "krb\|nfs\|gss"
auth_rpcgss            53248  2 nfsd,rpcsec_gss_krb5
dns_resolver           12288  1 nfsv4
grace                  12288  2 nfsd,lockd
lockd                  65536  2 nfsd,nfs
nfs                   139264  1 nfsv4
nfsd                  249856  3
nfsv4                 258048  0
oid_registry           12288  2 auth_rpcgss,trusted
rpcsec_gss_krb5        24576  0
sunrpc                192512 16 nfsv4,nfsd,nfs,rpcsec_gss_krb5,auth_rpcgss,lockd

Here are relevant logs from client machine (edge):

2024-12-22T15:57:24.011330-05:00 edge rpc.gssd[81116]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
2024-12-22T15:57:24.011353-05:00 edge rpc.gssd[81116]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
2024-12-22T15:57:24.011386-05:00 edge rpc.gssd[81116]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
...
2024-12-22T15:57:24.015498-05:00 edge rpc.gssd[81116]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
2024-12-22T15:57:24.015520-05:00 edge rpc.gssd[81116]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
2024-12-22T15:57:24.015541-05:00 edge rpc.gssd[81116]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
2024-12-22T15:57:24.015570-05:00 edge rpc.gssd[81116]: do_error_downcall(0x7c9e33e006c0): uid 0 err -13

Kerberos seems to be working fine as I am able to mount folder from another ubuntu machine (server.example.com) using krb5 with kerberos EXAMPLE.COM realm running on OpenWRT.

root@edge:~# mount -v -t nfs -o vers=4.2,sec=krb5 server.example.com:/ /mnt/mounted
mount.nfs: timeout set for Sun Dec 22 16:51:21 2024
mount.nfs: trying text-based options 'vers=4.2,sec=krb5,addr=192.168.120.7,clientaddr=192.168.20.50'

Tickets for both nfs and server machines are available on client machine

root@edge:~# klist /tmp/krb5ccmachine_EXAMPLE.COM
Ticket cache: FILE:/tmp/krb5ccmachine_EXAMPLE.COM
Default principal: root/edge@EXAMPLE.COM

Valid starting       Expires              Service principal
12/21/2024 23:58:26  12/22/2024 23:58:26  krbtgt/EXAMPLE.COM@EXAMPLE.COM
12/21/2024 23:58:26  12/22/2024 23:58:26  nfs/nfs.example.com@EXAMPLE.COM
12/22/2024 16:26:39  12/22/2024 23:58:26  nfs/server.example.com@EXAMPLE.COM

Both hosts are resolvable from client machine

root@edge:~# cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       edge
192.168.120.1   nfs.example.com
192.168.120.7   server.example.com

Does anyone have any pointers what might be wrong?

greem · December 23, 2024, 8:11am

That looks very much like your krb5.conf is slightly wrong. According to your other log lines, the ticket should be nfs/**nfs.example.com@EXAMPLE.COM** - server name is highlighted with asterisks.

Make sure your local DNS names, default search domain in resolv.conf and krb5.conf are the same on all of the machines involved or make sure you're setting the correct realm and mappings in krb5.conf.

justnulling · December 24, 2024, 4:17am

Thanks Graeme, this should resolve as EXAMPLE.COM is default realm.

here is my /etc/krb5.conf

[libdefaults]
        default_realm = EXAMPLE.COM
        dns_lookup_realm = false
        dns_lookup_kdc = false
        ticket_lifetime = 24h
        forwardable = yes
        rdns=false
        permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

[appdefaults]
    MACHINE_TICKET = false

[realms]
        EXAMPLE.COM = {
                database_name = /etc/krb5kdc/principal
                admin_keytab = /etc/krb5kdc/kadm5.keytab
                acl_file = /etc/krb5kdc/kadm5.acl
                dict_file = /usr/share/dict/words
                key_stash_file = /etc/krb5kdc/stash
                kdc_ports = 88
                kdc_tcp_ports = 88
                kdc = kerberos.example.com:88
                admin_server = kerberos.example.com:749
                default_domain = example.com
        }

[domain_realm]
        .example.com = EXAMPLE.COM
        example.com = EXAMPLE.COM

[logging]
        kdc = FILE:/opt/var/log/krb5/krb5kdc.log
        admin_server = FILE:/opt/var/log/krb5/kadmin.log
        default = FILE:/opt/var/log/krb5/krb5lib.log

greem · December 24, 2024, 9:14am

I think you'll have to find the docs for the user of KRB5_DEBUG and use that on the client.

It's very noisy but it'll show you how your client, the NFS server and the KDC are interacting.

justnulling · December 25, 2024, 5:36am

Found KRB5_TRACE, but that doesn't add logs anything. I think the issue is with NFSD on OPENWRT, as I am able to mount export from another server and all 3 machines have the same /etc/krb5.conf and klist has nfs/nfs.example.com@EXAMPLE.COM so it doesn't seem to be krb issue.

Here is mount with KRB5_TRACE enabled.

root@edge:~# env KRB5_TRACE=/dev/stdout mount -v -t nfs -o vers=4.2,sec=krb5 nfs.example.com:/ /mnt/mounted
mount.nfs: timeout set for Wed Dec 25 00:18:48 2024
mount.nfs: trying text-based options 'vers=4.2,sec=krb5,addr=192.168.20.1,clientaddr=192.168.20.50'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting nfs.example.com:/

klist looks good

root@edge:~# klist /tmp/krb5ccmachine_EXAMPLE.COM
Ticket cache: FILE:/tmp/krb5ccmachine_EXAMPLE.COM
Default principal: root/edge@EXAMPLE.COM

Valid starting       Expires              Service principal
12/24/2024 22:39:30  12/25/2024 22:39:30  krbtgt/EXAMPLE.COM@EXAMPLE.COM
12/24/2024 22:39:30  12/25/2024 22:39:30  nfs/server.example.com@EXAMPLE.COM
12/25/2024 00:16:48  12/25/2024 22:39:30  nfs/nfs.example.com@EXAMPLE.COM

complete logs

root@NFS# tail -f -n0 /opt/var/log/krb5/*
==> /opt/var/log/krb5/krb5-trace.log <==
[11300] 1735103808.700166: AP-REQ ticket: root/edge@EXAMPLE.COM -> krbtgt/EXAMPLE.COM@EXAMPLE.COM, session key aes256-cts/591B
[11300] 1735103808.700167: Negotiated enctype based on authenticator: aes256-cts
[11300] 1735103808.700168: Authenticator contains subkey: aes256-cts/95DB

==> /opt/var/log/krb5/krb5kdc.log <==
Dec 25 00:16:48 R3-LYN krb5kdc[11300](info): TGS_REQ (2 etypes {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17)}) 192.168.20.50: ISSUE: authtime 1735097970, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, root/edge@EXAMPLE.COM for nfs/nfs.example.com@EXAMPLE.COM

root@edge# journalctl -f
edge rpc.gssd[2361]: watchdog: sleeping 30 secs
edge rpc.gssd[2361]: inotify event for topdir (nfs) - ev->wd (8) ev->name (clnt45) ev->mask (0x40000100)
edge rpc.gssd[2361]: creating client nfs/clnt45
edge rpc.gssd[2361]: scanning client nfs/clnt45
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (idmap) ev->mask (0x00000100)
edge rpc.gssd[2361]:
                                     handle_gssd_upcall(0x72fb43b71740): 'mech=krb5 uid=0 service=* enctypes=20,19,26,25,18,17' (nfs/clnt45)
edge rpc.gssd[2361]: start_upcall_thread(0x72fb43b71740): created thread id 0x72fb33e006c0
edge rpc.gssd[2361]: krb5_use_machine_creds(0x72fb33e006c0): uid 0 tgtname (null)
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb33e006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb33e006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb33e006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb33e006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb33e006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a5b80
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340a9a50
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aacc0:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aacc0:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a5b80
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb33e006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb33e006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb33e006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb33e006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb33e006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a7340
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340a4930
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aacc0:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aacc0:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a7340
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
edge rpc.gssd[2361]: do_error_downcall(0x72fb33e006c0): uid 0 err -13
edge rpc.gssd[2361]:
                                     handle_gssd_upcall(0x72fb43b71740): 'mech=krb5 uid=0 service=* enctypes=20,19,26,25,18,17' (nfs/clnt45)
edge rpc.gssd[2361]: start_upcall_thread(0x72fb43b71740): created thread id 0x72fb42c006c0
edge rpc.gssd[2361]: krb5_use_machine_creds(0x72fb42c006c0): uid 0 tgtname (null)
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb42c006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb42c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb42c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb42c006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb42c006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a8420
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340a4de0
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aa080:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aa080:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a8420
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb42c006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb42c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb42c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb42c006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb42c006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340ad330
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb3402a800
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aa080:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aa080:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340ad330
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
edge rpc.gssd[2361]: do_error_downcall(0x72fb42c006c0): uid 0 err -13
edge rpc.gssd[2361]:
                                     handle_gssd_upcall(0x72fb43b71740): 'mech=krb5 uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt45)
edge rpc.gssd[2361]: start_upcall_thread(0x72fb43b71740): created thread id 0x72fb41c006c0
edge rpc.gssd[2361]: krb5_use_machine_creds(0x72fb41c006c0): uid 0 tgtname (null)
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb41c006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb41c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb41c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb41c006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb41c006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a5b80
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb3402a740
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb34039c60:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb34039c60:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a5b80
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb41c006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb41c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb41c006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb41c006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb41c006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a7930
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340ad960
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb34039c60:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb34039c60:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a7930
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
edge rpc.gssd[2361]: do_error_downcall(0x72fb41c006c0): uid 0 err -13
edge rpc.gssd[2361]:
                                     handle_gssd_upcall(0x72fb43b71740): 'mech=krb5 uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt45)
edge rpc.gssd[2361]: start_upcall_thread(0x72fb43b71740): created thread id 0x72fb412006c0
edge rpc.gssd[2361]: krb5_use_machine_creds(0x72fb412006c0): uid 0 tgtname (null)
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb412006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb412006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb412006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb412006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb412006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340a8f20
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340a4c00
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aea50:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aea50:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340a8f20
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb412006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb412006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb412006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb412006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb412006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb3402a640
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340acf40
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aea50:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aea50:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb3402a640
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
edge rpc.gssd[2361]: do_error_downcall(0x72fb412006c0): uid 0 err -13
edge rpc.gssd[2361]: inotify event for topdir (nfs) - ev->wd (8) ev->name (clnt46) ev->mask (0x40000100)
edge rpc.gssd[2361]: creating client nfs/clnt46
edge rpc.gssd[2361]: scanning client nfs/clnt46
edge rpc.gssd[2361]: inotify event for topdir (nfs) - ev->wd (8) ev->name (clnt47) ev->mask (0x40000100)
edge rpc.gssd[2361]: creating client nfs/clnt47
edge rpc.gssd[2361]: scanning client nfs/clnt47
edge rpc.gssd[2361]:
                                     handle_gssd_upcall(0x72fb43b71740): 'mech=krb5 uid=0 enctypes=20,19,26,25,18,17' (nfs/clnt45)
edge rpc.gssd[2361]: start_upcall_thread(0x72fb43b71740): created thread id 0x72fb3be006c0
edge rpc.gssd[2361]: krb5_use_machine_creds(0x72fb3be006c0): uid 0 tgtname (null)
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb3be006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb3be006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb3be006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb3be006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb3be006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb34039d00
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340a8d40
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aad20:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aad20:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb34039d00
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: WARNING: Machine cache prematurely expired or corrupted trying to recreate cache for server nfs.example.com
edge rpc.gssd[2361]: No key table entry found for edge$@EXAMPLE.COM while getting keytab entry for 'edge$@EXAMPLE.COM'
edge rpc.gssd[2361]: No key table entry found for EDGE$@EXAMPLE.COM while getting keytab entry for 'EDGE$@EXAMPLE.COM'
edge rpc.gssd[2361]: find_keytab_entry(0x72fb3be006c0): Success getting keytab entry for 'root/edge@EXAMPLE.COM'
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb3be006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: gssd_get_single_krb5_cred(0x72fb3be006c0): Credentials in CC 'FILE:/tmp/krb5ccmachine_EXAMPLE.COM' are good until Wed Dec 25 22:17:36 2024
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb3be006c0): creating tcp client for server nfs.example.com
edge rpc.gssd[2361]: DEBUG: port already set to 2049
edge rpc.gssd[2361]: create_auth_rpc_client(0x72fb3be006c0): creating context with server nfs@nfs.example.com
edge rpc.gssd[2361]: in authgss_create_default()
edge rpc.gssd[2361]: in authgss_create()
edge rpc.gssd[2361]: authgss_create: name is 0x72fb340ae320
edge rpc.gssd[2361]: authgss_create: gd->name is 0x72fb340abc10
edge rpc.gssd[2361]: in authgss_refresh()
edge rpc.gssd[2361]: The token being sent (length 837):
edge rpc.gssd[2361]: in authgss_marshal()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success ((nil):0)
edge rpc.gssd[2361]: xdr_rpc_gss_cred: encode success (v 1, proc 1, seq 0, svc 1, ctx (nil):0)
edge rpc.gssd[2361]: in authgss_wrap()
edge rpc.gssd[2361]: xdr_rpc_gss_buf: encode success (0x72fb340aad20:837)
edge rpc.gssd[2361]: xdr_rpc_gss_init_args: encode success (token 0x72fb340aad20:837)
edge rpc.gssd[2361]: authgss_refresh: RPC: Unable to receive errno: Connection reset by peer
edge rpc.gssd[2361]: authgss_create_default: freeing name 0x72fb340ae320
edge rpc.gssd[2361]: WARNING: Failed to create krb5 context for user with uid 0 for server nfs@nfs.example.com
edge rpc.gssd[2361]: WARNING: Failed to create machine krb5 context with cred cache FILE:/tmp/krb5ccmachine_EXAMPLE.COM for server nfs.example.com
edge rpc.gssd[2361]: ERROR: Failed to create machine krb5 context with any credentials cache for server nfs.example.com
edge rpc.gssd[2361]: do_error_downcall(0x72fb3be006c0): uid 0 err -13
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt47) - ev->wd (82) ev->name (info) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt47) - ev->wd (82) ev->name (<?>) ev->mask (0x00008000)
edge rpc.gssd[2361]: destroying client nfs/clnt47
edge rpc.gssd[2361]: freeing client nfs/clnt47
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (idmap) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (krb5) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (gssd) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt46) - ev->wd (81) ev->name (info) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt46) - ev->wd (81) ev->name (<?>) ev->mask (0x00008000)
edge rpc.gssd[2361]: destroying client nfs/clnt46
edge rpc.gssd[2361]: freeing client nfs/clnt46
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (info) ev->mask (0x00000200)
edge rpc.gssd[2361]: inotify event for clntdir (nfs/clnt45) - ev->wd (80) ev->name (<?>) ev->mask (0x00008000)
edge rpc.gssd[2361]: destroying client nfs/clnt45
edge rpc.gssd[2361]: freeing client nfs/clnt45