FreeBSD 13 remote install on IBM x3200 M3

Trying to install OS on a remote server have multiple constraints, especially if the machine does not have full function BMC to boot from a remote disk, or IP-KVM to enter BIOS/UEFI setup screen. This is not a step-to-step guide, but a summary of pitfalls to avoid when installing FreeBSD 13 on IBM x3200 M3 over PXE with limited BMC functionality, without a “virtual media key” on the motherboard.

This article assumes that you have access to another FreeBSD machine (physical or VM) at the remote site that can be configured as a TFTP server and NFS server. Also able to modify DHCP server config at the remote site for PXE boot.

x3200 M3 capabilities and constraints

IBM x3200 M3 have the following capabilities and constraints.

IBM Integrated Management Module (IMM) supports:

Power on/off the machine
Serial console (use comconsole_port="0x2F8" in /boot/looader.conf) over SSH session
Can configure the machine to boot via PXE on the next boot

Constraints:

Because the “virtual media key” is not plugged into the motherboard, IMM remote control (remote disk and IP-KVM) is not supported
Have no access to the keyboard and VGA console because it’s located at a remote site.

Steps

I’ll describe what to do and what to avoid. For details, check the “details and pitfalls” section.

PXE server setup

Use UEFI PXE boot (FreeBSD/boot/loader.efi), don’t use legacy PXE (pxeboot)
Configure serial console (by modifying /tftpboot/FreeBSD/boot/loader.conf)
Copy loader.efi by adding 0xff (one-byte, output of printf "\255") at the end of the filename

On x3200 M3

Go to IMM web console and tick “attempt PXE on next reboot”, and click “save”
SSH to IMM, connect to the serial console (console 1), and power on x3200 M3 (power on)
Install FreeBSD. I choose root-on-ZFS, GPT (UEFI) boot. Don’t reboot after installation.
Enter shell to make additional modifications, edit /boot/loader.conf to setup serial console
Use efibootmgr -v to see boot options, and use efibootmgr -o to change the boot sequence. Do not use Legacy Only boot item.

Detail and pitfalls

UEFI PXE works, but legacy PXE does not

Don’t know why. I tried boot/pxeboot and monitor the network, it finished downloading the file over TFTP but never tried to mount NFS.
In comparison, UEFI PXE using boot/loader.efi works.

When IBM x3200 M3 tries to get file via TFTP, it adds 0xff suffix

I don’t know why. The easy workaround is to make an additional copy of loader.efi with 0xff (1-byte) suffix

Legacy disk boot might work, but take 30 minutes to boot

This affects many IBM servers, and I have tried the following combinations

Legacy disk boot, root (32G) on ZFS: Take about 5 minutes to boot.
Legacy disk boot, root (32G) on ZFS, with 3T partition behind (another ZFS pool): Nothing from serial console, not even bootloader.
Legacy disk boot, root (32G) on UFS: Take about 10 minutes to boot. At bootloader it prints a lot of disk0: Read 1 sector(s) from 5860533167 to 0xffffe000 (0x8000): 0x1
Legacy disk boot, root (32G) on UFS, with 3T partition behind (another ZFS pool): Similar to above, but take about 30 minutes to boot

So UEFI boot is recommended for FreeBSD 13 on IBM x3200 M3.

Although it’s not possible to get into the UEFI setup screen, UEFI boot order can be configured with efibootmgr. And by excluding the Legacy Only boot option, we can avoid lengthy legacy boot.