NFS Crashes under heavy load?

Is this a thing? Seems I'm experiancing this from time to time... Running NFS v4 on openwrt few clients have NFS mounted and two of which are doing right and readfile transfer operations.

Any tips I can do to improve stability? I do have alot of tweaks installed but memory seems availble.

Which device?

What shows in the logs?

Depending on device and perspective on what "a lot" entails, all that is installed over "stock" could be a significant factor.

1 Like

Okay, i'm running WRT1900ACS. Most of my tweaks are disabled in system startup because I do not use them often.

Sun Feb 24 08:31:00 2019 kern.err kernel: [ 2632.134099] sd 5:0:0:0: timing out command, waited 180s
Sun Feb 24 08:31:00 2019 kern.info kernel: [ 2632.139361] sd 5:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sun Feb 24 08:31:00 2019 kern.info kernel: [ 2632.147581] sd 5:0:0:0: [sdd] tag#0 Sense Key : 0x4 [current]
Sun Feb 24 08:31:00 2019 kern.info kernel: [ 2632.153439] sd 5:0:0:0: [sdd] tag#0 ASC=0x44 ASCQ=0x0
Sun Feb 24 08:31:00 2019 kern.info kernel: [ 2632.158605] sd 5:0:0:0: [sdd] tag#0 CDB: opcode=0x28 
Sun Feb 24 08:31:00 2019 kern.err kernel: [ 2632.166210] print_req_error: I/O error, dev sdd, sector 262375384
Sun Feb 24 08:31:04 2019 kern.err kernel: [ 2635.904011] sd 5:0:0:0: timing out command, waited 180s
Sun Feb 24 08:31:04 2019 kern.info kernel: [ 2635.909273] sd 5:0:0:0: [sdd] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sun Feb 24 08:31:04 2019 kern.info kernel: [ 2635.917493] sd 5:0:0:0: [sdd] tag#0 Sense Key : 0x4 [current]
Sun Feb 24 08:31:04 2019 kern.info kernel: [ 2635.923351] sd 5:0:0:0: [sdd] tag#0 ASC=0x44 ASCQ=0x0
Sun Feb 24 08:31:04 2019 kern.info kernel: [ 2635.928518] sd 5:0:0:0: [sdd] tag#0 CDB: opcode=0x28 
Sun Feb 24 08:31:04 2019 kern.err kernel: [ 2635.936123] print_req_error: I/O error, dev sdd, sector 262148624

The </> button in the editor is a gret way to format "code"

Looks like your drive or its driver is failing from what I see there. Messages like I/O error, dev sdd, sector 262375384 are sometimes indicative of a bad drive.

Have you tried to see what badblocks on a "desktop" Linux-based OS reports for that drive? It's probably sufficient for a quick check to only do a read check, with something like

badblocks -v /dev/sdXN
1 Like

smartmontools would probably also be useful or at least probably faster

1 Like

Thanks for the response guys, I think that log message is misguiding; it only pops up after NFS crash.

Figured it out (sort of) one of the two device writing to the NFS share is the issue; If I unplug it, NFS shares come back up. I'll try restricting the amount of writes it can do. You guys run into this before?

It's called a broken storage device for whatever reason.

1 Like

Don't think the drive is the issue.

Its more likely that i/o error is due to soft mount option used and is the result of the client that was writing to that drive halting.

I read alot about NFS over last 24hrs, think I've got a good config now. Fingers crossed.

If the kernel can't read data from the source device how is application/service X supposed to supply your client with it?

Instead of guessing, you could run a long test using smartmontools and/or badblocks as suggested above to rule out hardware issues.

1 Like

Will do, thanks!