So that would make it stay in panic and not automatically reboot. But would I be able to get any information from it after it experienced a kernel panic?
Not sure, but that would be the idea. I would guess access would be borked at that point, so up and running serial for a console dump? Or an already established ssh connection, but that might timeout so...
1 Like
Swap will slow doom down hopefully managing to log something more.
1 Like
This was the problem. I haven't 100% confirmed this, but this was essentially a memory leak, because it was spawning these processes to scp and waiting for a password. I must have cleared the ssh keys when I did the upgrade, so the scp couldn't finish. There were dozens of these scp processes running when it was low on memory. Each one was eating 1% of the memory.
I just figured this out, and I cleared them all out and the free swap went from <50% to 100%. I won't know for sure for a week or so of it not rebooting. But I am pretty confident this is the issue.
Thank you all for helping me think this through and sharing your expertise. This helped a ton and I have a satisfying answer. 
.
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.