Openwrt PXE server works, distros hang

reconfigurator · November 15, 2019, 1:14pm

Can anyone help solve this, please.

I have set up a PXE server using OPENWRT OpenWrt 18.06.4
on Linksys WRT 32X closely following these instructions:
https://openwrt.org/docs/guide-user/services/tftp.pxe-server
And it’s serving files.

But the process hangs and the screen is corrupted just displaying blue background and instead of normal installation process and incomplete output

Client machine is satellite P20 which is 32 bit & has nVIDIA GeForce FX Go5200 graphics controller. The distros are legacy/32 bit.

I have tftp on a 62Gb Kingston datatraveller attached to usb port formatted f2fs

Here are it’s contents:

root@OpenWrt:~# ls /mnt/sda1/tftp

images ldlinux.c32 libcom32.c32 libutil.c32 pxelinux.0 pxelinux.cfg vesamenu.c32

Here is default file:

root@OpenWrt:~# cat /mnt/sda1/tftp/pxelinux.cfg/default
DEFAULT vesamenu.c32
PROMPT 0
MENU TITLE OpenWrt PXE-Boot Menu

label lubuntu
MENU LABEL lubuntu
KERNEL images/lubuntu/casper/vmlinuz
APPEND boot=casper ide=nodma netboot=nfs nfsroot=192.168.0.1:/mnt/sda1/tftp/images/lubuntu initrd=images/lubuntu/casper/initrd.lz
TEXT HELP
Starts the Lubuntu 18.04 LTS i386
ENDTEXT

label bodhilinux
MENU LABEL bodhilinux
KERNEL images/bodhilinux/casper/vmlinuz
APPEND boot=casper ide=nodma netboot=nfs nfsroot=192.168.0.1:/mnt/sda1/tftp/images/bodhilinux initrd=images/bodhilinux/casper/initrd.lz
TEXT HELP
Starts the Bodhi Linux 5.0.0 Legacy
ENDTEXT

label fedora
MENU LABEL fedora
KERNEL images/Fedora-KDE-Live-31-1-9/images/pxeboot/vmlinuz
APPEND boot=pxeboot ide=nodma netboot=nfs nfsroot=192.168.0.1:/mnt/sda1/tftp/images/Fedora-KDE-Live-31-1-9/images/pxeboot/initrd.img
TEXT HELP
Starts the Fedora-KDE-Live-31-1-9
ENDTEXT

I cp -r the files to the /images directories from the isos
Because it wasn’t finding them in the subdirectories which had spaces in the names, corresponding to version

so images/lubuntu looks like this:

root@OpenWrt:~# ls /mnt/sda1/tftp/images/lubuntu
README.diskdefines casper install md5sum.txt pool ubuntu
boot dists isolinux pics preseed
root@OpenWrt:~# ls -l /mnt/sda1/tftp/images/lubuntu
-r--r--r-- 1 root root 229 Nov 14 16:42 README.diskdefines
dr-xr-xr-x 3 root root 3488 Nov 14 16:29 boot
dr-xr-xr-x 2 root root 3488 Nov 14 16:42 casper
dr-xr-xr-x 3 root root 3488 Nov 14 16:42 dists
dr-xr-xr-x 2 root root 3488 Nov 14 16:42 install
dr-xr-xr-x 2 root root 8192 Nov 14 16:42 isolinux
-r--r--r-- 1 root root 20664 Nov 14 16:50 md5sum.txt
dr-xr-xr-x 2 root root 3488 Nov 14 16:42 pics
dr-xr-xr-x 4 root root 3488 Nov 14 16:42 pool
dr-xr-xr-x 2 root root 3488 Nov 14 16:42 preseed
lrwxrwxrwx 1 root root 1 Nov 14 16:42 ubuntu -> .

I might have cp the ubuntu directory above in error but nothing is calling it and I get exactly the same hang with the bodhilinux distro, which looks like this:

root@OpenWrt:~# ls -l /mnt/sda1/tftp/images/bodhilinux
drwxr-xr-x 3 root root 3488 Nov 14 16:52 EFI
drwxr-xr-x 2 root root 3488 Nov 14 16:52 ISOLINUX
-rw-r--r-- 1 root root 205 Nov 14 16:52 README.diskdefines
drwxr-xr-x 3 root root 3488 Nov 14 16:52 boot
drwxr-xr-x 2 root root 3488 Nov 14 16:52 casper
drwxr-xr-x 3 root root 3488 Nov 14 16:52 dists
drwxr-xr-x 2 root root 3488 Nov 14 16:52 install
drwxr-xr-x 2 root root 3488 Nov 14 16:52 isolinux
-rw-r--r-- 1 root root 20664 Nov 14 16:52 md5sum.txt
drwxr-xr-x 2 root root 3488 Nov 14 16:52 preseed
-rw-r--r-- 1 root root 0 Nov 14 16:52 ubuntu

When I choose lubuntu the process loads with normal full screen output & unpacks initramfs
then screen goes blue
and appears to be showing quite a lot of partial output, some stretching off-screen and parts of the screen do not appear to be displaying.

Output including like this:

..premount done
no lock begin trying linux cdrom
connection refused

Then the process halts saying.

27.2-2ubuntu3) built-in shell (ash)
system on network
init done

Openwrt syslog looks like this:

Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: failed sending /mnt/sda1/tftp/pxelinux.0 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/pxelinux.0 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/ldlinux.c32 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/pxelinux.cfg/default to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/vesamenu.c32 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/libcom32.c32 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/libutil.c32 to 192.168.0.181
Fri Nov 15 13:08:27 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/pxelinux.cfg/default to 192.168.0.181
Fri Nov 15 13:08:32 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/images/bodhilinux/casper/vmlinuz to 192.168.0.181
Fri Nov 15 13:08:42 2019 daemon.info dnsmasq-tftp[3370]: sent /mnt/sda1/tftp/images/bodhilinux/casper/initrd.lz to 192.168.0.181

I am questioning:

Should I have used dd instead of cp?

Should I have copied the entire .iso file instead of burning it and cp ing individual the files within it?

Does PXE for some or no reason require a sub directory, not just /lubuntu?

But, as stated above, pxe can’t find the files if they have spaces in their names, it reads each section of the filename and says no such file or directory (when it is there)

And, PXE is working, it’s the booting of the distros that is hanging...

So is it something else?

tmomas · November 15, 2019, 1:21pm

Please use the "Preformatted text </>" button for logs, scripts, configs and general console output.

Please edit your posting accordingly.
Thanks!

anon50098793 · November 15, 2019, 1:33pm

lets assume that everything is using traditional non-efi...

what you have yet to demonstrate in the output you've provided... is testing simple nfs operation from a client or how you've setup nfs...

apart from that... everything is distro specific... aka.... whether to place the iso.... copy the iso contents.... download a specific initrd ( over tftp ) with possibly net drivers and bootstrapped installers within.... cmdline nfsversion parameters... / loop setup parameters.... ( redhat based distro's usually handle nfs-iso very well )

if you test nfs from a client.... then all you need to do is follow up with the relevant distro documentation / version as to which syntax/method is best to try/use...

reconfigurator · November 15, 2019, 4:48pm

Thanks for responding Wulfy

Yes client machine is non-efi and I called vmlinuz & initrd.lz from within casper directory, both distros are 32 bit

The files are being sent to the PXE client machine according to syslog
and client machine seems to be receiving them
but the boot process goes bent after initramfs loads normally
are you saying this could still be a problem with NFS?

If you think the NFS implementation could be to blame how should I test that from another client machine, please, this is the 1st time I have used NFS?

Has anyone else successfully run PXE server off Openwrt like this to serve lubuntu / bodhilinux / ubuntu who can suggest what to do if the problem is distro-based?

reconfigurator · November 15, 2019, 4:57pm

root@OpenWrt:~# cat /etc/exports
/mnt    *(ro,all_squash,insecure,sync)
/mnt/sda1/tftp/images  *(ro,async,no_subtree_check)

bolvan · November 16, 2019, 12:24pm

then screen goes blue
and appears to be showing quite a lot of partial output, some stretching off-screen and parts of the screen do not appear to be displaying.

Try parameters : nomodeset nosplash
May be will see something useful

reconfigurator · November 16, 2019, 5:06pm

Thank you very much that worked to make output visible

Pity I can’t pause it or capture it just have to try re-typing it after multiple reboots

everything seems to go OK until it hangs like this:

trying to netboot 192.168.0.1:/mnt/sda1/tftp/images/lubuntu/cdrom

begin nfs mount -0 nolock 0 ro 192.168.0.1:/mnt/sda1/tftp/images/lubuntu/cdrom

connection refused

(Loads of times)

Then

Busybox v1.27.2 (Ubuntu 1:1.27.2-2Ubuntu3) built-in shell (ash)

(initramfs) done.

Unable to find a live file system on the network

[107.884047] randon: crng init done

So something is looking for subdirectory /cdrom when there isn’t one:

ls -l /mnt/sda1/tftp/images/lubuntu
-r--r--r--    1 root     root           229 Nov 14 16:42 README.diskdefines
dr-xr-xr-x    3 root     root          3488 Nov 14 16:29 boot
dr-xr-xr-x    2 root     root          3488 Nov 14 16:42 casper
dr-xr-xr-x    3 root     root          3488 Nov 14 16:42 dists
dr-xr-xr-x    2 root     root          3488 Nov 14 16:42 install
dr-xr-xr-x    2 root     root          8192 Nov 14 16:42 isolinux
-r--r--r--    1 root     root         20664 Nov 14 16:50 md5sum.txt
dr-xr-xr-x    2 root     root          3488 Nov 14 16:42 pics
dr-xr-xr-x    4 root     root          3488 Nov 14 16:42 pool
dr-xr-xr-x    2 root     root          3488 Nov 14 16:42 preseed
lrwxrwxrwx    1 root     root             1 Nov 14 16:42 ubuntu -> .

At least I'm assuming connection refused = no such directory

Am I on the right track and can I solve this by renaming on of the other directories to 'cdrom'?
Maybe isolinux?

Or could it be something else?

Why is it looking for a directory that doesn't exist in the distro?

What is looking for it?

bolvan · November 16, 2019, 7:57pm

I suspect connection refused mean NFS server is not running.
Try mounting manually first

mk24 · November 17, 2019, 12:35am

Also you need to know if it is NFS3 or NFS4 because they are quite different and incompatible.

reconfigurator · November 17, 2019, 10:04am

Thanks for your help guys

It looks like it's mounted (last line):

root@OpenWrt:~# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime,ubi=0,vol=1)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sda1 on /mnt/sda1 type f2fs (rw,lazytime,relatime,background_gc=on,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6)

I don't know how to interrogate NFS from command line but Luci says installed packages are 4:
kmod-fs-nfs 4.14.131-1
kmod-fs-nfs-common 4.14.131-1
kmod-fs-nfs-common-rpcsec 4.14.131-1
kmod-fs-nfs-v4 4.14.131-1
kmod-fs-nfsd 4.14.131-1
nfs-kernel-server 2.3.3-3

Is there some command I can use to check it's running correctly because I was assuming it must be as the files are received

lulcat · May 10, 2020, 4:44pm

It's been a long time since I did both general booting over pxe, and initrd mount hacks, which i did years ago, before much of it became widely known or popular. But, from the little thing I would say from that, is
it seems to me to be a NFS support issue in the ramdisks.. you NEED support for that.. which has nothing to do with the pxe server, openwrt , nfs as such but the ramdisk. Especially things like "mint", bodhi, aka not real distros but flavours will rarely have the tech in there to do it, unless their 'daddy/mummy' has it. You can easily write some litle initrd scripts to add that functionality though. That usually would include some kernel parameter too, to pick up the nfs root.

Bare with me, I might have completely forgotten everything, as I tend to boot from my own scripts which usually do loop isos or loop image files.

But, pxe gets you to the initrd, it unwraps you go, it hits the root, on NFS , and unless your initrd has support for that (or http say).. then.... what? This is what I think is where you are hitting a snag. with that said, I did used to boot all the things in the past , including all kinds of hobby flavours over pxe, and I don't think I have to double check the nfs support. but ya. Just wanted to throw in a consideration to check. I might even look up my own notes or sap a pxe up quickly and solve the issue for ya, (but it's not on my list to do right now, so might not get a chance). Been months, so for sure you have sorted it.