19 Creating Partitioned Images Using Wic
Creating an image for a particular hardware target using the OpenEmbedded build system does not necessarily mean you can boot that image as is on your device. Physical devices accept and boot images in various ways depending on the specifics of the device. Usually, information about the hardware can tell you what image format the device requires. Should your device require multiple partitions on an SD card, flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to create the properly partitioned image.
The wic command generates partitioned images from existing
OpenEmbedded build artifacts. Image generation is driven by partitioning
commands contained in an OpenEmbedded kickstart file (.wks)
specified either directly on the command line or as one of a selection
of canned kickstart files as shown with the wic list images command
in the
“Generate an Image using an Existing Kickstart File”
section. When you apply the command to a given set of build artifacts, the
result is an image or set of images that can be directly written onto media and
used on a particular system.
Note
For a kickstart file reference, see the “OpenEmbedded Kickstart (.wks) Reference” Chapter in the Yocto Project Reference Manual.
The wic command and the infrastructure it is based on is by
definition incomplete. The purpose of the command is to allow the
generation of customized images, and as such, was designed to be
completely extensible through a plugin interface. See the
“Using the Wic Plugin Interface” section
for information on these plugins.
This section provides some background information on Wic, describes what you need to have in place to run the tool, provides instruction on how to use the Wic utility, provides information on using the Wic plugins interface, and provides several examples that show how to use Wic.
19.1 Background
This section provides some background on the Wic utility. While none of this information is required to use Wic, you might find it interesting.
The name “Wic” is derived from OpenEmbedded Image Creator (oeic). The “oe” diphthong in “oeic” was promoted to the letter “w”, because “oeic” is both difficult to remember and to pronounce.
Wic is loosely based on the Meego Image Creator (
mic) framework. The Wic implementation has been heavily modified to make direct use of OpenEmbedded build artifacts instead of package installation and configuration, which are already incorporated within the OpenEmbedded artifacts.Wic is a completely independent standalone utility that initially provides easier-to-use and more flexible replacements for an existing functionality in OE-Core’s image-live class. The difference between Wic and those examples is that with Wic the functionality of those scripts is implemented by a general-purpose partitioning language, which is based on Redhat kickstart syntax.
19.2 Requirements
In order to use the Wic utility with the OpenEmbedded Build system, your system needs to meet the following requirements:
The Linux distribution on your development host must support the Yocto Project. See the “Supported Linux Distributions” section in the Yocto Project Reference Manual for the list of distributions that support the Yocto Project.
The standard system utilities, such as
cp, must be installed on your development host system.You must have sourced the build environment setup script (i.e. oe-init-build-env) found in the Build Directory.
You need to have the build artifacts already available, which typically means that you must have already created an image using the OpenEmbedded build system (e.g.
core-image-minimal). While it might seem redundant to generate an image in order to create an image using Wic, the current version of Wic requires the artifacts in the form generated by the OpenEmbedded build system.You must build several native tools, which are built to run on the build system:
$ bitbake wic-tools
Include “wic” as part of the IMAGE_FSTYPES variable.
Include the name of the wic kickstart file as part of the WKS_FILE variable. If multiple candidate files can be provided by different layers, specify all the possible names through the WKS_FILES variable instead.
19.3 Getting Help
You can get general help for the wic command by entering the wic
command by itself or by entering the command with a help argument as
follows:
$ wic -h
$ wic --help
$ wic help
Currently, Wic supports seven commands: cp, create, help,
list, ls, rm, and write. You can get help for all these
commands except “help” by using the following form:
$ wic help command
For example, the following command returns help for the write
command:
$ wic help write
Wic supports help for three topics: overview, plugins, and
kickstart. You can get help for any topic using the following form:
$ wic help topic
For example, the following returns overview help for Wic:
$ wic help overview
There is one additional level of help for Wic. You can get help on
individual images through the list command. You can use the list
command to return the available Wic images as follows:
$ wic list images
genericx86 Create an EFI disk image for genericx86*
edgerouter Create SD card image for Edgerouter
beaglebone-yocto Create SD card image for Beaglebone
qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
systemd-bootdisk Create an EFI disk image with systemd-boot
mkhybridiso Create a hybrid ISO image
mkefidisk Create an EFI disk image
sdimage-bootpart Create SD card image with a boot partition
directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
directdisk Create a 'pcbios' direct disk image
directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
qemuriscv Create qcow2 image for RISC-V QEMU machines
directdisk-gpt Create a 'pcbios' direct disk image
efi-bootdisk
Once you know the list of available
Wic images, you can use help with the command to get help on a
particular image. For example, the following command returns help on the
“beaglebone-yocto” image:
$ wic list beaglebone-yocto help
Creates a partitioned SD card image for Beaglebone.
Boot files are located in the first vfat partition.
19.4 Operational Modes
You can use Wic in two different modes, depending on how much control you need for specifying the OpenEmbedded build artifacts that are used for creating the image: Raw and Cooked:
Raw Mode: You explicitly specify build artifacts through Wic command-line arguments.
Cooked Mode: The current MACHINE setting and image name are used to automatically locate and provide the build artifacts. You just supply a kickstart file and the name of the image from which to use artifacts.
Regardless of the mode you use, you need to have the build artifacts ready and available.
19.4.1 Raw Mode
Running Wic in raw mode allows you to specify all the partitions through
the wic command line. The primary use for raw mode is if you have
built your kernel outside of the Yocto Project Build Directory.
In other words, you can point to arbitrary kernel, root filesystem locations,
and so forth. Contrast this behavior with cooked mode where Wic looks in the
Build Directory (e.g. tmp/deploy/images/machine).
The general form of the wic command in raw mode is:
$ wic create wks_file options ...
Where:
wks_file:
An OpenEmbedded kickstart file. You can provide
your own custom file or use a file from a set of
existing files as described by further options.
optional arguments:
-h, --help show this help message and exit
-o OUTDIR, --outdir OUTDIR
name of directory to create image in
-e IMAGE_NAME, --image-name IMAGE_NAME
name of the image to use the artifacts from e.g. core-
image-sato
-r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
path to the /rootfs dir to use as the .wks rootfs
source
-b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
path to the dir containing the boot artifacts (e.g.
/EFI or /syslinux dirs) to use as the .wks bootimg
source
-k KERNEL_DIR, --kernel-dir KERNEL_DIR
path to the dir containing the kernel to use in the
.wks bootimg
-n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
path to the native sysroot containing the tools to use
to build the image
-s, --skip-build-check
skip the build check
-f, --build-rootfs build rootfs
-c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
compress image with specified compressor
-m, --bmap generate .bmap
--no-fstab-update Do not change fstab file.
-v VARS_DIR, --vars VARS_DIR
directory with <image>.env files that store bitbake
variables
-D, --debug output debug information
Note
You do not need root privileges to run Wic. In fact, you should not run as root when using the utility.
19.4.2 Cooked Mode
Running Wic in cooked mode leverages off artifacts in the
Build Directory. In other words, you do not have to specify kernel or
root filesystem locations as part of the command. All you need to provide is
a kickstart file and the name of the image from which to use artifacts
by using the “-e” option. Wic looks in the Build Directory (e.g.
tmp/deploy/images/machine) for artifacts.
The general form of the wic command using Cooked Mode is as follows:
$ wic create wks_file -e IMAGE_NAME
Where:
wks_file:
An OpenEmbedded kickstart file. You can provide
your own custom file or use a file from a set of
existing files provided with the Yocto Project
release.
required argument:
-e IMAGE_NAME, --image-name IMAGE_NAME
name of the image to use the artifacts from e.g. core-
image-sato
19.5 Using an Existing Kickstart File
If you do not want to create your own kickstart file, you can use an existing file provided by the Wic installation. As shipped, kickstart files can be found in the Yocto Project Source Repositories in the following two locations:
poky/meta-yocto-bsp/wic
poky/scripts/lib/wic/canned-wks
Use the following command to list the available kickstart files:
$ wic list images
genericx86 Create an EFI disk image for genericx86*
beaglebone-yocto Create SD card image for Beaglebone
edgerouter Create SD card image for Edgerouter
qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
directdisk-gpt Create a 'pcbios' direct disk image
mkefidisk Create an EFI disk image
directdisk Create a 'pcbios' direct disk image
systemd-bootdisk Create an EFI disk image with systemd-boot
mkhybridiso Create a hybrid ISO image
sdimage-bootpart Create SD card image with a boot partition
directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
When you use an existing file, you
do not have to use the .wks extension. Here is an example in Raw
Mode that uses the directdisk file:
$ wic create directdisk -r rootfs_dir -b bootimg_dir \
-k kernel_dir -n native_sysroot
Here are the actual partition language commands used in the
genericx86.wks file to generate an image:
# short-description: Create an EFI disk image for genericx86*
# long-description: Creates a partitioned EFI disk image for genericx86* machines
part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
part swap --ondisk sda --size 44 --label swap1 --fstype=swap
bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
19.6 Using the Wic Plugin Interface
You can extend and specialize Wic functionality by using Wic plugins. This section explains the Wic plugin interface.
Note
Wic plugins consist of “source” and “imager” plugins. Imager plugins are beyond the scope of this section.
Source plugins provide a mechanism to customize partition content during
the Wic image generation process. You can use source plugins to map
values that you specify using --source commands in kickstart files
(i.e. *.wks) to a plugin implementation used to populate a given
partition.
Note
If you use plugins that have build-time dependencies (e.g. native tools, bootloaders, and so forth) when building a Wic image, you need to specify those dependencies using the WKS_FILE_DEPENDS variable.
Source plugins are subclasses defined in plugin files. As shipped, the Yocto Project provides several plugin files. You can see the source plugin files that ship with the Yocto Project here. Each of these plugin files contains source plugins that are designed to populate a specific Wic image partition.
Source plugins are subclasses of the SourcePlugin class, which is
defined in the poky/scripts/lib/wic/pluginbase.py file. For example,
the BootimgEFIPlugin source plugin found in the bootimg-efi.py
file is a subclass of the SourcePlugin class, which is found in the
pluginbase.py file.
You can also implement source plugins in a layer outside of the Source
Repositories (external layer). To do so, be sure that your plugin files
are located in a directory whose path is
scripts/lib/wic/plugins/source/ within your external layer. When the
plugin files are located there, the source plugins they contain are made
available to Wic.
When the Wic implementation needs to invoke a partition-specific
implementation, it looks for the plugin with the same name as the
--source parameter used in the kickstart file given to that
partition. For example, if the partition is set up using the following
command in a kickstart file:
part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
The methods defined as class
members of the matching source plugin (i.e. bootimg-pcbios) in the
bootimg-pcbios.py plugin file are used.
To be more concrete, here is the corresponding plugin definition from
the bootimg-pcbios.py file for the previous command along with an
example method called by the Wic implementation when it needs to prepare
a partition using an implementation-specific function:
.
.
.
class BootimgPcbiosPlugin(SourcePlugin):
"""
Create MBR boot partition and install syslinux on it.
"""
name = 'bootimg-pcbios'
.
.
.
@classmethod
def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
oe_builddir, bootimg_dir, kernel_dir,
rootfs_dir, native_sysroot):
"""
Called to do the actual content population for a partition i.e. it
'prepares' the partition to be incorporated into the image.
In this case, prepare content for legacy bios boot partition.
"""
.
.
.
If a
subclass (plugin) itself does not implement a particular function, Wic
locates and uses the default version in the superclass. It is for this
reason that all source plugins are derived from the SourcePlugin
class.
The SourcePlugin class defined in the pluginbase.py file defines
a set of methods that source plugins can implement or override. Any
plugins (subclass of SourcePlugin) that do not implement a
particular method inherit the implementation of the method from the
SourcePlugin class. For more information, see the SourcePlugin
class in the pluginbase.py file for details:
The following list describes the methods implemented in the
SourcePlugin class:
do_prepare_partition(): Called to populate a partition with actual content. In other words, the method prepares the final partition image that is incorporated into the disk image.do_configure_partition(): Called beforedo_prepare_partition()to create custom configuration files for a partition (e.g. syslinux or grub configuration files).do_install_disk(): Called after all partitions have been prepared and assembled into a disk image. This method provides a hook to allow finalization of a disk image (e.g. writing an MBR).do_stage_partition(): Special content-staging hook called beforedo_prepare_partition(). This method is normally empty.Typically, a partition just uses the passed-in parameters (e.g. the unmodified value of
bootimg_dir). However, in some cases, things might need to be more tailored. As an example, certain files might additionally need to be taken frombootimg_dir + /boot. This hook allows those files to be staged in a customized fashion.Note
get_bitbake_var()allows you to access non-standard variables that you might want to use for this behavior.
You can extend the source plugin mechanism. To add more hooks, create
more source plugin methods within SourcePlugin and the corresponding
derived subclasses. The code that calls the plugin methods uses the
plugin.get_source_plugin_methods() function to find the method or
methods needed by the call. Retrieval of those methods is accomplished
by filling up a dict with keys that contain the method names of
interest. On success, these will be filled in with the actual methods.
See the Wic implementation for examples and details.
19.7 Wic Examples
This section provides several examples that show how to use the Wic
utility. All the examples assume the list of requirements in the
“Requirements” section have been met. The
examples assume the previously generated image is
core-image-minimal.
19.7.1 Generate an Image using an Existing Kickstart File
This example runs in Cooked Mode and uses the mkefidisk kickstart
file:
$ wic create mkefidisk -e core-image-minimal
INFO: Building wic-tools...
.
.
.
INFO: The new image(s) can be found here:
./mkefidisk-201804191017-sda.direct
The following build artifacts were used to create the image(s):
ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
INFO: The image(s) were created using OE kickstart file:
/home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
The previous example shows the easiest way to create an image by running
in cooked mode and supplying a kickstart file and the “-e” option to
point to the existing build artifacts. Your local.conf file needs to
have the MACHINE variable set
to the machine you are using, which is “qemux86” in this example.
Once the image builds, the output provides image location, artifact use, and kickstart file information.
Note
You should always verify the details provided in the output to make sure that the image was indeed created exactly as expected.
Continuing with the example, you can now write the image from the
Build Directory onto a USB stick, or whatever media for which you
built your image, and boot from the media. You can write the image by using
bmaptool or dd:
$ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
or
$ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
Note
For more information on how to use the bmaptool
to flash a device with an image, see the
“Flashing Images Using bmaptool”
section.
19.7.2 Using a Modified Kickstart File
Because partitioned image creation is driven by the kickstart file, it
is easy to affect image creation by changing the parameters in the file.
This next example demonstrates that through modification of the
directdisk-gpt kickstart file.
As mentioned earlier, you can use the command wic list images to
show the list of existing kickstart files. The directory in which the
directdisk-gpt.wks file resides is
scripts/lib/image/canned-wks/, which is located in the
Source Directory (e.g. poky).
Because available files reside in this directory, you can create and add
your own custom files to the directory. Subsequent use of the
wic list images command would then include your kickstart files.
In this example, the existing directdisk-gpt file already does most
of what is needed. However, for the hardware in this example, the image
will need to boot from sdb instead of sda, which is what the
directdisk-gpt kickstart file uses.
The example begins by making a copy of the directdisk-gpt.wks file
in the scripts/lib/image/canned-wks directory and then by changing
the lines that specify the target disk from which to boot:
$ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
/home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Next, the example modifies the directdisksdb-gpt.wks file and
changes all instances of “--ondisk sda” to “--ondisk sdb”. The
example changes the following two lines and leaves the remaining lines
untouched:
part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
Once the lines are changed, the
example generates the directdisksdb-gpt image. The command points
the process at the core-image-minimal artifacts for the Next Unit of
Computing (nuc) MACHINE the
local.conf:
$ wic create directdisksdb-gpt -e core-image-minimal
INFO: Building wic-tools...
.
.
.
Initialising tasks: 100% |#######################################| Time: 0:00:01
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
INFO: Creating image(s)...
INFO: The new image(s) can be found here:
./directdisksdb-gpt-201710090938-sdb.direct
The following build artifacts were used to create the image(s):
ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
INFO: The image(s) were created using OE kickstart file:
/home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
Continuing with the example, you can now directly dd the image to a
USB stick, or whatever media for which you built your image, and boot
the resulting media:
$ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
140966+0 records in
140966+0 records out
72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
$ sudo eject /dev/sdb
19.7.3 Using a Modified Kickstart File and Running in Raw Mode
This next example manually specifies each build artifact (runs in Raw
Mode) and uses a modified kickstart file. The example also uses the
-o option to cause Wic to create the output somewhere other than the
default output directory, which is the current directory:
$ wic create test.wks -o /home/stephano/testwic \
--rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
--bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
--kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
--native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
INFO: Creating image(s)...
INFO: The new image(s) can be found here:
/home/stephano/testwic/test-201710091445-sdb.direct
The following build artifacts were used to create the image(s):
ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
INFO: The image(s) were created using OE kickstart file:
test.wks
For this example,
MACHINE did not have to be
specified in the local.conf file since the artifact is manually
specified.
19.7.4 Using Wic to Manipulate an Image
Wic image manipulation allows you to shorten turnaround time during image development. For example, you can use Wic to delete the kernel partition of a Wic image and then insert a newly built kernel. This saves you time from having to rebuild the entire image each time you modify the kernel.
Note
In order to use Wic to manipulate a Wic image as in this example,
your development machine must have the mtools package installed.
The following example examines the contents of the Wic image, deletes the existing kernel, and then inserts a new kernel:
List the Partitions: Use the
wic lscommand to list all the partitions in the Wic image:$ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic Num Start End Size Fstype 1 1048576 25041919 23993344 fat16 2 25165824 72157183 46991360 ext4
The previous output shows two partitions in the
core-image-minimal-qemux86.wicimage.Examine a Particular Partition: Use the
wic lscommand again but in a different form to examine a particular partition.Note
You can get command usage on any Wic command using the following form:
$ wic help command
For example, the following command shows you the various ways to use the wic ls command:
$ wic help ls
The following command shows what is in partition one:
$ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1 Volume in drive : is boot Volume Serial Number is E894-1809 Directory for ::/ libcom32 c32 186500 2017-10-09 16:06 libutil c32 24148 2017-10-09 16:06 syslinux cfg 220 2017-10-09 16:06 vesamenu c32 27104 2017-10-09 16:06 vmlinuz 6904608 2017-10-09 16:06 5 files 7 142 580 bytes 16 582 656 bytes freeThe previous output shows five files, with the
vmlinuzbeing the kernel.Note
If you see the following error, you need to update or create a
~/.mtoolsrcfile and be sure to have the line “mtools_skip_check=1” in the file. Then, run the Wic command again:ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0 output: Total number of sectors (47824) not a multiple of sectors per track (32)! Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
Remove the Old Kernel: Use the
wic rmcommand to remove thevmlinuzfile (kernel):$ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
Add In the New Kernel: Use the
wic cpcommand to add the updated kernel to the Wic image. Depending on how you built your kernel, it could be in different places. If you useddevtooland an SDK to build your kernel, it resides in thetmp/workdirectory of the extensible SDK. If you usedmaketo build the kernel, the kernel will be in theworkspace/sourcesarea.The following example assumes
devtoolwas used to build the kernel:$ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \ poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuzOnce the new kernel is added back into the image, you can use the
ddcommand or bmaptool to flash your wic image onto an SD card or USB stick and test your target.Note
Using
bmaptoolis generally 10 to 20 times faster than usingdd.