Background
A buddy of mine is building a customized Linux OS using Buildroot. This sounded like a neat project, so I asked him to keep me posted on his progress. His ultimate goal is to distribute his OS to some hardware, but he wants to test in emulation first. This makes sense to me. He uses xcg-ng virtual machines, and apparently xcg-ng and wanted to boot from Optical Disk Image (ISO)
There is a problem. Booting is hard, and it is hard for me to help him without going through this process myself.
I build filesystems and kernels very often, but I never made a bootable ISO image before. Typically I use QEMU and do the following for testing kernels:
1qemu-system-x86_64 -enable-kvm -kernel ./bzImage -initrd ./rootfs.cpio -nographic -append 'console=ttyS0' -m 4G
This isn’t good enough for distribution. To do that, I’d usually build a raw image with partitions and distribute that. I’m fairly certain ISO works the same way, and that it’s possible to convert a raw image into a ISO image.
Preparation
I wont go into using Buildroot, but I ended up with this structure:
1➜ iso tree -h images
2[ 72] images
3├── [ 13M] bzImage
4├── [ 6] efi-part
5│ └── [ 8] EFI
6│ └── [ 38] BOOT
7│ ├── [ 608K] bootx64.efi
8│ └── [ 117] grub.cfg
9├── [ 96M] rootfs.cpio
10└── [ 104M] rootfs.tar
11
124 directories, 5 files
The above QEMU command successfully boots this, and that’s a good first sign I have a working system.
I decided to include GRUB bootloader as part of the build, and I noticed something interesting:
efi-part/EFI/BOOT/grub.cfg
set default="0"
set timeout="5"
menuentry "Buildroot" {
linux /boot/bzImage root=/dev/sda1 rootwait console=tty1
}
The GRUB config seemed… barren… It’s expecting that our root filesystem
exists in the /dev/sda1 device. In our QEMU command, I use initrd as
the filesystem that’s loaded into ram. But our entry doesn’t want that.
Next, I’ve never seen the rootwait option before. So I looked it up in the
Admin guide - kernel parameters:
1rootwait [KNL] Wait (indefinitely) for root device to show up.
2 Useful for devices that are detected asynchronously
3 (e.g. USB and MMC devices).
I can test the rootwait functionality with QEMU, but I don’t have much experience toying with hotplug devices, and I didn’t have much luck with setting up a usb-storage device to do so. It’s safe to ignore this option.
Before making a bootable ISO, let’s make a bootable image first.
Bootable disk
The setup I want here is:
- Partition 1: Ext4 1G
- Partition 2: UEFI 10M
- Partition 3: /boot ~1G
Modern systems use UEFI for boot, and should be able to detect the boot partition regardless of location, and our menu expects our first device to be /dev/sda1. No MBR for me!
1➜ iso qemu-img create rootfs.img 2G
2
3➜ iso fdisk rootfs.img
4
5Welcome to fdisk (util-linux 2.40.4).
6Changes will remain in memory only, until you decide to write them.
7Be careful before using the write command.
8
9Device does not contain a recognized partition table.
10Created a new DOS (MBR) disklabel with disk identifier 0x2dd10ff3.
11
12Command (m for help): g
13Created a new GPT disklabel (GUID: E2E0BA42-2265-42CE-A1AA-FB9BCA8AD8ED).
14
15Command (m for help): n
16Partition number (1-128, default 1):
17First sector (2048-4194270, default 2048):
18Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-4194270, default 4192255): +1G
19
20Created a new partition 1 of type 'Linux filesystem' and of size 1 GiB.
21
22Command (m for help): t
23Selected partition 1
24Partition type or alias (type L to list all): L
25Type of partition 1 is unchanged: Linux filesystem.
26
27Command (m for help): n
28Partition number (2-128, default 2):
29First sector (2099200-4194270, default 2099200):
30Last sector, +/-sectors or +/-size{K,M,G,T,P} (2099200-4194270, default 4192255): +10M
31
32Created a new partition 2 of type 'Linux filesystem' and of size 10 MiB.
33
34Command (m for help): i
35Partition number (1,2, default 2): 2
36
37 Device: rootfs.img2
38 Start: 2099200
39 End: 2119679
40 Sectors: 20480
41 Size: 10M
42 Type: Linux filesystem
43 Type-UUID: 0FC63DAF-8483-4772-8E79-3D69D8477DE4
44 UUID: E251FA43-7073-4EB5-850F-3F96573E7CAD
45
46Command (m for help): t
47Partition number (1,2, default 2): 2
48Partition type or alias (type L to list all): L
49Partition type or alias (type L to list all): 1
50
51Changed type of partition 'Linux filesystem' to 'EFI System'.
52
53Command (m for help): n
54Partition number (3-128, default 3):
55First sector (2119680-4194270, default 2119680):
56Last sector, +/-sectors or +/-size{K,M,G,T,P} (2119680-4194270, default 4192255):
57
58Created a new partition 3 of type 'Linux filesystem' and of size 1012 MiB.
59
60Command (m for help): t
61Partition number (1-3, default 3): 3
62Partition type or alias (type L to list all): L
63Partition type or alias (type L to list all): 142
64
65Changed type of partition 'Linux filesystem' to 'Linux extended boot'.
66
67Command (m for help): p
68Disk rootfs.img: 2 GiB, 2147483648 bytes, 4194304 sectors
69Units: sectors of 1 * 512 = 512 bytes
70Sector size (logical/physical): 512 bytes / 512 bytes
71I/O size (minimum/optimal): 512 bytes / 512 bytes
72Disklabel type: gpt
73Disk identifier: E2E0BA42-2265-42CE-A1AA-FB9BCA8AD8ED
74
75Device Start End Sectors Size Type
76rootfs.img1 2048 2099199 2097152 1G Linux filesystem
77rootfs.img2 2099200 2119679 20480 10M EFI System
78rootfs.img3 2119680 4192255 2072576 1012M Linux extended boot
79
80Command (m for help): w
81The partition table has been altered.
Then format & populate the partitions:
1➜ iso mkdir -p boot efi fs
2➜ iso sudo mkfs.ext4 /dev/loop0p1
3mke2fs 1.47.2 (1-Jan-2025)
4Discarding device blocks: done
5Creating filesystem with 262144 4k blocks and 65536 inodes
6Filesystem UUID: 19096c35-e1f0-4ec6-9fd2-2e73956ffbfc
7Superblock backups stored on blocks:
8 32768, 98304, 163840, 229376
9
10Allocating group tables: done
11Writing inode tables: done
12Creating journal (8192 blocks): done
13Writing superblocks and filesystem accounting information: done
14
15➜ iso sudo mkfs.ext4 /dev/loop0p3
16mke2fs 1.47.2 (1-Jan-2025)
17Discarding device blocks: done
18Creating filesystem with 259072 4k blocks and 64768 inodes
19Filesystem UUID: 35db06b8-5ab6-4785-9f97-adae766a6aa0
20Superblock backups stored on blocks:
21 32768, 98304, 163840, 229376
22
23Allocating group tables: done
24Writing inode tables: done
25Creating journal (4096 blocks): done
26Writing superblocks and filesystem accounting information: done
27
28➜ iso sudo mkfs.vfat /dev/loop0p2
29mkfs.fat 4.2 (2021-01-31)
30
31➜ iso sudo mount /dev/loop0p1 fs
32➜ iso sudo mount /dev/loop0p2 efi
33➜ iso sudo mount /dev/loop0p3 boot
34
35➜ iso sudo tar -xpvf images/rootfs.tar -C ./fs
36...
37
38➜ iso sudo cp -r images/efi-part/* efi
39➜ iso sudo mkdir -p boot/boot
40➜ iso sudo cp images/bzImage boot/boot
Buildroot certainly has options to make this process seamless. However, I used nearly all defaults when making the image. This means I need to modify the GRUB boot entry to work with my kernel and disk setup.
QEMU is setup on ttyS0, I don’t have VGA enabled, therefore I need to set the correct console output. GRUB also doesn’t know what the boot partition is, so I set the root to be that partition.
efi/EFI/BOOT/grub.cfg
1set default="0"
2set timeout="5"
3set root=(hd0,gpt3)
4
5menuentry "Buildroot" {
6 linux /boot/bzImage root=/dev/sda1 console=ttyS0
7}
1➜ iso sudo umount ./fs ./efi ./boot
2➜ iso sudo losetup -D
With QEMU:
1qemu-system-x86_64 \
2 -enable-kvm \
3 -m 4G \
4 -nographic \
5 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd \
6 -device virtio-scsi-pci,id=scsi \
7 -drive id=img,if=none,format=raw,file=rootfs.img \
8 -device scsi-hd,drive=img
BOOOT!
The big thing I added here is a bios option to ensure QEMU uses UEFI
BIOS firmware to boot our disk.
ISO
I made an assumption before starting this process that it should be trivial to convert a disk image into a ISO. That was bold of me. I didn’t even do the research first. The formats are completely incompatible and is a rabbit hole of a mess to sift through.
Most people are interested in extracting information from ISO’s than creating bootable ISOs. Fortunately, Debian saved me!
1➜ iso mkdir -p testiso
2
3➜ iso sudo cp images/bzImage testiso/boot
4
5➜ iso sudo cp -r images/efi-part/* testiso/
6
7➜ iso sudo mkdir -p testiso/boot/grub
8
9➜ iso sudo qemu-img create testiso/boot/grub/efi.img 8M
10Formatting 'testiso/boot/grub/efi.img', fmt=raw size=8388608
11
12➜ iso sudo mkfs.vfat testiso/boot/grub/efi.img
13mkfs.fat 4.2 (2021-01-31)
14
15➜ iso mkdir efi-img
16➜ iso sudo mount testiso/boot/grub/efi.img ./efi-img
17➜ iso sudo mkdir -p ./efi-img/efi/boot
18➜ iso sudo grub2-mkimage \
19 -C xz \
20 -O x86_64-efi \
21 -p /boot/grub \
22 -o ./efi-img/efi/boot/bootx64.efi \
23 boot linux search normal configfile \
24 part_gpt btrfs ext2 fat iso9660 loopback \
25 test keystatus gfxmenu regexp probe \
26 efi_gop efi_uga all_video gfxterm font \
27 echo read ls cat png jpeg halt reboot
28
29➜ iso sudo umount ./efi-img
30
31➜ iso cat testiso/EFI/BOOT/grub.cfg
32set default="0"
33set timeout="5"
34
35menuentry "Buildroot" {
36 linux /boot/bzImage root=/dev/sr0 console=ttyS0
37}
38
39➜ iso sudo xorriso -as mkisofs \
40 -iso-level 3 \
41 -r -V "TEST" \
42 -J -joliet-long \
43 -no-emul-boot \
44 -e boot/grub/efi.img \
45 -o testiso.iso \
46 testiso
47
48➜ iso qemu-system-x86_64 \
49 -enable-kvm \
50 -m 4G \
51 -nographic \
52 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd \
53 -device virtio-scsi-pci \
54 -drive id=cd,if=none,file=testiso.iso,media=cdrom \
55 -device scsi-cd,drive=cd
BOOOT!
What’s interesting about bootable ISOs is that they’re read-only. When I live boot, there’s not much I can do. This makes live ISOs very good for recovery, but you’re on your own to mount filesystems onto disks to have some RW capabilities.
Lessons learned
While a fun experiment, I feel it’s better for my buddy to not worry about distributing his OS via ISO. At least not without some installer to go with it.
Best to write a disk image to a USB stick. The reverse is true too, ISO can be byte-for-byte written to USB. USB the disks are bootable and writable, after all.
This is exactly what Unraid does. They ship Slackware linux.
Here’s how we can boot from USB in QEMU:
1qemu-system-x86_64 \
2 -enable-kvm \
3 -m 4G \
4 -nographic \
5 -bios /usr/share/edk2/ovmf/OVMF_CODE.fd \
6 -drive id=img,if=none,format=raw,file=rootfs.img \
7 -usb \
8 -device nec-usb-xhci,id=xhci \
9 -device usb-storage,bus=xhci.0,drive=img
BOOOT!
All that said, making a bootable ISO is still good because now I know how to make media in which I can run debugging tooling from.
I’m still on the fence of which approach is easier for me. Maybe ISO with more practice…