Kernel panics with OpenWRT x86-64 when using docker

For Developers
1

Hi everybody!

I have been setting up openwrt for an i7 server. I have it booting from uefi, with dual rootfs partition so I can upgrade with sysupgrade safely from openwrt itself. It's really cool building openwrt from inside itself.

A netgear r7800 was being the bottleneck on my home lan, so I wanted to replace it entirely with the x86 server, which formerly was running an ubuntu server.

All seems to work wonderfully, but one important detail, which renders the whole thing unusable. The kernel panics again and again when using docker. The back traces are so different that I have no clue what the problem really is.

All I know for sure is it happens when docker is running. Since they work wonderfully from ubuntu on the same hardware, I suspect the kernel, but as I said, I haven't been able to stabilize it at all. I tried both 4.14 and 4.19, and I even tried to build openwrt with a newer kernel, but that's not a 2 hours job, it's way more involved.

You can see some of the panics here: kernel panics logs

Any idea what might be causing these?

I want to complete the dual partition system for x86-64 and integrate it in luci-app-advancedreboot, then share it, but I need to make it stable first.

2

Hello,

can you discribe your setup a bit?
Self compiled, snapshot, 19.07-rc?
What commands did your run befor the exeption occurs?

I only see this always the same panics:

[   32.654286] Kernel BUG at kfree+0x10c/0x120 [verbose debug info unavailable]
[   32.662562] invalid opcode: 0000 [#1] SMP PTI

invalid opcode: 0000
Faulty hardware?

3

It's all self built, from master. I'm trying to find all the original logs with all the faults, but I didn't see them all the same. My bad it seems.

Anyhow, faulty hardware? It's been rock solid for 6 years, an i7-3770. It's been my nas, build machine, ... for quite long. I'm now running it with the same dockers I tried on openwrt with no issue at all. In 5 years I might have seen a couple of them when installing the last server or maybe when playing with OC settings in the bios, other than that 100% rock solid. To say more, the entire server, with its 5 3TB disks, and its case survived a fire where everything around it melted. That was a couple years ago.
Hence, I seriously doubt the hardware issue.

The last builds I made didn't start dockerd on init, so I could launch it manually after it booted (openwrt). Last time I tried, it booted ok the first time (after flashing the .img to the usb stick I'm booting from). I then started docker, and seconds later the kernel panicked. The weird part comes on the next boot, it again panics while booting, so it didn't even start docker.

After this two panics in succession I've been playing with openwrt inside a docker and use arch as the host system, but I really prefer to have openwrt as the host so I'd love to see these panics go away.

There are a couple more details, even though I tried with different storage systems for docker, they were always over a btrfs drive, I didn't get to try on an ext4 for example, there might be some issue with btrfs too, I saw it in several of the backtraces.

Besides that, the x86-64 kernel configuration needed MM_CONFIG=y for my r8169 network card to work fully, and I also added btrfs in as well as mdraid. Once I got the panics I tried to replicate a more general desktop linux kernel configuration, since the openwrt kernel is quite clean, but nothing fixed the panics.

Other than that, I'll try to locate all the logs I thought I had saved, otherwise I'll have to try again.

1 Like
4

Here you can see better the usual panics I see. Most of them are general protection faults, but the hardware is rock solid with an ubuntu kernel.

https://pastebin.com/yjmfHFWP

https://nextcloud.marcet.info/s/2pCGX5psj5swgo6