Yocto Project Reference Manual

Currently, the Yocto Project is supported on the following distributions:

The list of packages you need on the host development system can be large when covering all build scenarios using the Yocto Project. This section provides required packages according to Linux distribution and function.

     $ sudo apt-get build-dep qemu
     $ sudo apt-get remove oss4-dev
                    

In order to use the build system, your host development system must meet the following version requirements for Git, tar, and Python:

If your host development system does not meet all these requirements, you can resolve this by installing a buildtools tarball that contains these tools. You can get the tarball one of two ways: download a pre-built tarball or use BitBake to build the tarball.

In the development environment you will need to build an image whenever you change hardware support, add or change system libraries, or add or change services that have dependencies.

Building an Image

The first thing you need to do is set up the OpenEmbedded build environment by sourcing an environment setup script (i.e. oe-init-build-env or oe-init-build-env-memres). Here is an example:

     $ source oe-init-build-env [build_dir]
            

The build_dir argument is optional and specifies the directory the OpenEmbedded build system uses for the build - the Build Directory. If you do not specify a Build Directory, it defaults to a directory named build in your current working directory. A common practice is to use a different Build Directory for different targets. For example, ~/build/x86 for a qemux86 target, and ~/build/arm for a qemuarm target.

Once the build environment is set up, you can build a target using:

     $ bitbake target
            

The target is the name of the recipe you want to build. Common targets are the images in meta/recipes-core/images, meta/recipes-sato/images, etc. all found in the Source Directory. Or, the target can be the name of a recipe for a specific piece of software such as BusyBox. For more details about the images the OpenEmbedded build system supports, see the "Images" chapter.

When building an image using GPL components, you need to maintain your original settings and not switch back and forth applying different versions of the GNU General Public License. If you rebuild using different versions of GPL, dependency errors might occur due to some components not being rebuilt.

You can find the log for a task in the file ${WORKDIR}/temp/log.do_taskname. For example, the log for the do_compile task of the QEMU minimal image for the x86 machine (qemux86) might be in tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile. To see the commands BitBake ran to generate a log, look at the corresponding run.do_taskname file in the same directory.

log.do_taskname and run.do_taskname are actually symbolic links to log.do_taskname.pid and log.run_taskname.pid, where pid is the PID the task had when it ran. The symlinks always point to the files corresponding to the most recent run.

BitBake's -e option is used to display variable values after parsing. The following command displays the variable values after the configuration files (i.e. local.conf, bblayers.conf, bitbake.conf and so forth) have been parsed:

     $ bitbake -e
            

The following command displays variable values after a specific recipe has been parsed. The variables include those from the configuration as well:

     $ bitbake -e recipename
            

In the output of bitbake -e, each variable is preceded by a description of how the variable got its value, including temporary values that were later overriden. This description also includes variable flags (varflags) set on the variable. The output can be very helpful during debugging.

Variables that are exported to the environment are preceded by export in the output of bitbake -e. See the following example:

     export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
            

In addition to variable values, the output of the bitbake -e and bitbake -e recipe commands includes the following information:

You can use the oe-pkgdata-util command-line utility to query PKGDATA_DIR and display various package-related information. When you use the utility, you must use it to view information on packages that have already been built.

Following are a few of the available oe-pkgdata-util subcommands.

For more information on the oe-pkgdata-util command, use the help facility:

     $ oe-pkgdata-util ‐‐help
     $ oe-pkgdata-util subcommand --help
            

Sometimes it can be hard to see why BitBake wants to build other recipes before the one you have specified. Dependency information can help you understand why a recipe is built.

To generate dependency information for a recipe, run the following command:

     $ bitbake -g recipename
            

This command writes the following files in the current directory:

The graphs are in DOT format and can be converted to images (e.g. using the dot tool from Graphviz).

You can use a different method to view dependency information by using the following command:

     $ bitbake -g -u taskexp recipename
            

This command displays a GUI window from which you can view build-time and runtime dependencies for the recipes involved in building recipename.

As mentioned in the "Checksums (Signatures)" section of the BitBake User Manual, BitBake tries to automatically determine what variables a task depends on so that it can rerun the task if any values of the variables change. This determination is usually reliable. However, if you do things like construct variable names at runtime, then you might have to manually declare dependencies on those variables using vardeps as described in the "Variable Flags" section of the BitBake User Manual.

If you are unsure whether a variable dependency is being picked up automatically for a given task, you can list the variable dependencies BitBake has determined by doing the following:

There is also a bitbake-diffsigs command for comparing two siginfo or sigdata files. This command can be helpful when trying to figure out what changed between two versions of a task. If you call bitbake-diffsigs with just one file, the command behaves like bitbake-dumpsig.

You can also use BitBake to dump out the signature construction information without executing tasks by using either of the following BitBake command-line options:

     ‐‐dump-signatures=SIGNATURE_HANDLER
     -S SIGNATURE_HANDLER
            

Using BitBake with either of these options causes BitBake to dump out sigdata files in the stamps directory for every task it would have executed instead of building the specified target package.

Any given recipe consists of a set of tasks. The standard BitBake behavior in most cases is: do_fetch, do_unpack, do_patch, do_configure, do_compile, do_install, do_package, do_package_write_*, and do_build. The default task is do_build and any tasks on which it depends build first. Some tasks, such as do_devshell, are not part of the default build chain. If you wish to run a task that is not part of the default build chain, you can use the -c option in BitBake. Here is an example:

     $ bitbake matchbox-desktop -c devshell
            

The -c option respects task dependencies, which means that all other tasks (including tasks from other recipes) that the specified task depends on will be run before the task. Even when you manually specify a task to run with -c, BitBake will only run the task if it considers it "out of date". See the "Stamp Files and the Rerunning of Tasks" section for how BitBake determines whether a task is "out of date".

If you want to force an up-to-date task to be rerun (e.g. because you made manual modifications to the recipe's WORKDIR that you want to try out), then you can use the -f option.

The following example shows one way you can use the -f option:

     $ bitbake matchbox-desktop
               .
               .
     make some changes to the source code in the work directory
               .
               .
     $ bitbake matchbox-desktop -c compile -f
     $ bitbake matchbox-desktop
            

This sequence first builds and then recompiles matchbox-desktop. The last command reruns all tasks (basically the packaging tasks) after the compile. BitBake recognizes that the do_compile task was rerun and therefore understands that the other tasks also need to be run again.

Another, shorter way to rerun a task and all normal recipe build tasks that depend on it is to use the -C option.

Using this option invalidates the given task and then runs the do_build task, which is the default task if no task is given, and the tasks on which it depends. You could replace the final two commands in the previous example with the following single command:

     $ bitbake matchbox-desktop -C compile
            

Internally, the -f and -C options work by tainting (modifying) the input checksum of the specified task. This tainting indirectly causes the task and its dependent tasks to be rerun through the normal task dependency mechanisms.

     WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
                

     $ bitbake matchbox-desktop -c clean
     $ bitbake matchbox-desktop
                

You can view a list of tasks in a given package by running the do_listtasks task as follows:

     $ bitbake matchbox-desktop -c listtasks
            

The results appear as output to the console and are also in the file ${WORKDIR}/temp/log.do_listtasks.

You can see debug output from BitBake by using the -D option. The debug output gives more information about what BitBake is doing and the reason behind it. Each -D option you use increases the logging level. The most common usage is -DDD.

The output from bitbake -DDD -v targetname can reveal why BitBake chose a certain version of a package or why BitBake picked a certain provider. This command could also help you in a situation where you think BitBake did something unexpected.

Sometimes issues on the host development system can cause your build to fail. Following are known, host-specific problems. Be sure to always consult the Release Notes for a look at all release-related issues.

To build a specific recipe (.bb file), you can use the following command form:

     $ bitbake -b somepath/somerecipe.bb
            

This command form does not check for dependencies. Consequently, you should use it only when you know existing dependencies have been met.

The Yocto Project provides several logging functions for producing debugging output and reporting errors and warnings. For Python functions, the following logging functions exist. All of these functions log to ${T}/log.do_task, and can also log to standard output (stdout) with the right settings:

The same logging functions are also available in shell functions, under the names bbplain, bbnote, bbdebug, bbwarn, bberror, and bbfatal. The logging class implements these functions. See that class in the meta/classes folder of the Source Directory for information.

     python do_listtasks() {
         bb.debug(2, "Starting to figure out the task list")
         if noteworthy_condition:
             bb.note("There are 47 tasks to run")
         bb.debug(2, "Got to point xyz")
         if warning_trigger:
             bb.warn("Detected warning_trigger, this might be a problem later.")
         if recoverable_error:
             bb.error("Hit recoverable_error, you really need to fix this!")
         if fatal_error:
             bb.fatal("fatal_error detected, unable to print the task list")
         bb.plain("The tasks present are abc")
         bb.debug(2, "Finished figuring out the tasklist")
     }
                

     do_my_function() {
         bbdebug 2 "Running do_my_function"
         if [ exceptional_condition ]; then
             bbnote "Hit exceptional_condition"
         fi
         bbdebug 2  "Got to point xyz"
         if [ warning_trigger ]; then
             bbwarn "Detected warning_trigger, this might cause a problem later."
         fi
         if [ recoverable_error ]; then
             bberror "Hit recoverable_error, correcting"
         fi
         if [ fatal_error ]; then
             bbfatal "fatal_error detected"
         fi
         bbdebug 2 "Completed do_my_function"
     }
                

Here are some other tips that you might find useful:

Build history is disabled by default. To enable it, add the following INHERIT statement and set the BUILDHISTORY_COMMIT variable to "1" at the end of your conf/local.conf file found in the Build Directory:

     INHERIT += "buildhistory"
     BUILDHISTORY_COMMIT = "1"
            

Enabling build history as previously described causes the build process to collect build output information and commit it to a local Git repository.

You can disable build history by removing the previous statements from your conf/local.conf file.

Build history information is kept in ${TOPDIR}/buildhistory in the Build Directory as defined by the BUILDHISTORY_DIR variable. The following is an example abbreviated listing:

At the top level, there is a metadata-revs file that lists the revisions of the repositories for the layers enabled when the build was produced. The rest of the data splits into separate packages, images and sdk directories, the contents of which are described below.

     PV = 1.22.1
     PR = r32
     RPROVIDES =
     RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
     RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
     PKGSIZE = 540168
     FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
        /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
        /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
        /usr/share/pixmaps /usr/share/applications /usr/share/idl \
        /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
     FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
        /etc/busybox.links.nosuid /etc/busybox.links.suid
                

     PV = 1.22.1
     PR = r32
     DEPENDS = initscripts kern-tools-native update-rc.d-native \
        virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
        virtual/libc virtual/update-alternatives
     PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
        busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
        busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
                

     # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
     SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
                

     $ buildhistory-collect-srcrevs -a
     # i586-poky-linux
     SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
     SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
     # x86_64-linux
     SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
     SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
     SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
     SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
     SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
     SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
     SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
     SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
     SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
     # qemux86-poky-linux
     SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
     SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
     # all-poky-linux
     SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
                

     DISTRO = poky
     DISTRO_VERSION = 1.7
     USER_CLASSES = buildstats image-mklibs image-prelink
     IMAGE_CLASSES = image_types
     IMAGE_FEATURES = debug-tweaks
     IMAGE_LINGUAS =
     IMAGE_INSTALL = packagegroup-core-boot run-postinsts
     BAD_RECOMMENDATIONS =
     NO_RECOMMENDATIONS =
     PACKAGE_EXCLUDE =
     ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
        write_image_manifest ; buildhistory_list_installed_image ; \
        buildhistory_get_image_installed ; ssh_allow_empty_password;  \
        postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
     IMAGE_POSTPROCESS_COMMAND =   buildhistory_get_imageinfo ;
     IMAGESIZE = 6900
                

     INHERIT += "buildhistory"
     BUILDHISTORY_COMMIT = "0"
     BUILDHISTORY_FEATURES = "image"
                

     DISTRO = poky
     DISTRO_VERSION = 1.3+snapshot-20130327
     SDK_NAME = poky-glibc-i686-arm
     SDK_VERSION = 1.3+snapshot
     SDKMACHINE =
     SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
     BAD_RECOMMENDATIONS =
     SDKSIZE = 352712
                

2.4.2.5. Examining Build History Information¶

You can examine build history output from the command line or from a web interface.

To see any changes that have occurred (assuming you have BUILDHISTORY_COMMIT = "1"), you can simply use any Git command that allows you to view the history of a repository. Here is one method:

      $ git log -p
                

You need to realize, however, that this method does show changes that are not significant (e.g. a package's size changing by a few bytes).

A command-line tool called buildhistory-diff does exist, though, that queries the Git repository and prints just the differences that might be significant in human-readable form. Here is an example:

     $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
     Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
        /etc/anotherpkg.conf was added
        /sbin/anotherpkg was added
        * (installed-package-names.txt):
        *   anotherpkg was added
     Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
        anotherpkg was added
     packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
        * PR changed from "r0" to "r1"
        * PV changed from "0.1.10" to "0.1.12"
     packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
        * PR changed from "r0" to "r1"
        * PV changed from "0.1.10" to "0.1.12"
                

Note

The buildhistory-diff tool requires the GitPython package. Be sure to install it using Pip3 as follows:

   $ pip3 install GitPython --user
                    

Alternatively, you can install python3-git using the appropriate distribution package manager (e.g. apt-get, dnf, or zipper).

To see changes to the build history using a web interface, follow the instruction in the README file here. http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/.

Here is a sample screenshot of the interface:

The distribution layer provides policy configurations for your distribution. Best practices dictate that you isolate these types of configurations into their own layer. Settings you provide in conf/distro/distro.conf override similar settings that BitBake finds in your conf/local.conf file in the Build Directory.

The following list provides some explanation and references for what you typically find in the distribution layer:

The BSP Layer provides machine configurations. Everything in this layer is specific to the machine for which you are building the image or the SDK. A common structure or form is defined for BSP layers. You can learn more about this structure in the Yocto Project Board Support Package (BSP) Developer's Guide.

The BSP Layer's configuration directory contains configuration files for the machine (conf/machine/machine.conf) and, of course, the layer (conf/layer.conf).

The remainder of the layer is dedicated to specific recipes by function: recipes-bsp, recipes-core, recipes-graphics, and recipes-kernel. Metadata can exist for multiple formfactors, graphics support systems, and so forth.

The software layer provides the Metadata for additional software packages used during the build. This layer does not include Metadata that is specific to the distribution or the machine, which are found in their respective layers.

This layer contains any new recipes that your project needs in the form of recipe files.

Upstream project releases exist anywhere in the form of an archived file (e.g. tarball or zip file). These files correspond to individual recipes. For example, the figure uses specific releases each for BusyBox, Qt, and Dbus. An archive file can be for any released product that can be built using a recipe.

Local projects are custom bits of software the user provides. These bits reside somewhere local to a project - perhaps a directory into which the user checks in items (e.g. a local directory containing a development source tree used by the group).

The canonical method through which to include a local project is to use the externalsrc class to include that local project. You use either the local.conf or a recipe's append file to override or set the recipe to point to the local directory on your disk to pull in the whole source tree.

For information on how to use the externalsrc class, see the "externalsrc.bbclass" section.

Another place the build system can get source files from is through an SCM such as Git or Subversion. In this case, a repository is cloned or checked out. The do_fetch task inside BitBake uses the SRC_URI variable and the argument's prefix to determine the correct fetcher module.

When fetching a repository, BitBake uses the SRCREV variable to determine the specific revision from which to build.

Two kinds of mirrors exist: pre-mirrors and regular mirrors. The PREMIRRORS and MIRRORS variables point to these, respectively. BitBake checks pre-mirrors before looking upstream for any source files. Pre-mirrors are appropriate when you have a shared directory that is not a directory defined by the DL_DIR variable. A Pre-mirror typically points to a shared directory that is local to your organization.

Regular mirrors can be any site across the Internet that is used as an alternative location for source code should the primary site not be functioning for some reason or another.

The first stages of building a recipe are to fetch and unpack the source code:

The do_fetch and do_unpack tasks fetch the source files and unpack them into the work directory.

By default, everything is accomplished in the Build Directory, which has a defined structure. For additional general information on the Build Directory, see the "build/" section.

Unpacked source files are pointed to by the S variable. Each recipe has an area in the Build Directory where the unpacked source code resides. The name of that directory for any given recipe is defined from several different variables. You can see the variables that define these directories by looking at the figure:

Once source code is fetched and unpacked, BitBake locates patch files and applies them to the source files:

The do_patch task processes recipes by using the SRC_URI variable to locate applicable patch files, which by default are *.patch or *.diff files, or any file if "apply=yes" is specified for the file in SRC_URI.

BitBake finds and applies multiple patches for a single recipe in the order in which it finds the patches. Patches are applied to the recipe's source files located in the S directory.

For more information on how the source directories are created, see the "Source Fetching" section.

After source code is patched, BitBake executes tasks that configure and compile the source code:

This step in the build process consists of three tasks:

After source code is configured and compiled, the OpenEmbedded build system analyzes the results and splits the output into packages:

The do_package and do_packagedata tasks combine to analyze the files found in the D directory and split them into subsets based on available packages and files. The analyzing process involves the following as well as other items: splitting out debugging symbols, looking at shared library dependencies between packages, and looking at package relationships. The do_packagedata task creates package metadata based on the analysis such that the OpenEmbedded build system can generate the final packages. Working, staged, and intermediate results of the analysis and package splitting process use these areas:

The FILES variable defines the files that go into each package in PACKAGES. If you want details on how this is accomplished, you can look at the package class.

Depending on the type of packages being created (RPM, DEB, or IPK), the do_package_write_* task creates the actual packages and places them in the Package Feed area, which is ${TMPDIR}/deploy. You can see the "Package Feeds" section for more detail on that part of the build process.

Once packages are split and stored in the Package Feeds area, the OpenEmbedded build system uses BitBake to generate the root filesystem image:

The image generation process consists of several stages and depends on several tasks and variables. The do_rootfs task creates the root filesystem (file and directory structure) for an image. This task uses several key variables to help create the list of packages to actually install:

With IMAGE_ROOTFS pointing to the location of the filesystem under construction and the PACKAGE_INSTALL variable providing the final list of packages to install, the root file system is created.

Package installation is under control of the package manager (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or not package management is enabled for the target. At the end of the process, if package management is not enabled for the target, the package manager's data files are deleted from the root filesystem. As part of the final stage of package installation, postinstall scripts that are part of the packages are run. Any scripts that fail to run on the build host are run on the target when the target system is first booted. If you are using a read-only root filesystem, all the post installation scripts must succeed during the package installation phase since the root filesystem is read-only.

The final stages of the do_rootfs task handle post processing. Post processing includes creation of a manifest file and optimizations.

The manifest file (.manifest) resides in the same directory as the root filesystem image. This file lists out, line-by-line, the installed packages. The manifest file is useful for the testimage class, for example, to determine whether or not to run specific tests. See the IMAGE_MANIFEST variable for additional information.

Optimizing processes run across the image include mklibs, prelink, and any other post-processing commands as defined by the ROOTFS_POSTPROCESS_COMMAND variable. The mklibs process optimizes the size of the libraries, while the prelink process optimizes the dynamic linking of shared libraries to reduce start up time of executables.

After the root filesystem is built, processing begins on the image through the do_image task. The build system runs any pre-processing commands as defined by the IMAGE_PREPROCESS_COMMAND variable. This variable specifies a list of functions to call before the OpenEmbedded build system creates the final image output files.

The do_image task dynamically creates other do_image_* tasks as needed, which include compressing the root filesystem image to reduce the overall size of the image. The process turns everything into an image file or a set of image files. The formats used for the root filesystem depend on the IMAGE_FSTYPES variable.

The final task involved in image creation is the do_image_complete task. This task completes the image by applying any image post processing as defined through the IMAGE_POSTPROCESS_COMMAND variable. The variable specifies a list of functions to call once the OpenEmbedded build system has created the final image output files.

The OpenEmbedded build system uses BitBake to generate the Software Development Kit (SDK) installer script for both the standard and extensible SDKs:

Like image generation, the SDK script process consists of several stages and depends on many variables. The do_populate_sdk and do_populate_sdk_ext tasks use these key variables to help create the list of packages to actually install. For information on the variables listed in the figure, see the "Application Development SDK" section.

The do_populate_sdk task helps create the standard SDK and handles two parts: a target part and a host part. The target part is the part built for the target hardware and includes libraries and headers. The host part is the part of the SDK that runs on the SDKMACHINE.

The do_populate_sdk_ext task helps create the extensible SDK and handles host and target parts differently than its counter part does for the standard SDK. For the extensible SDK, the task encapsulates the build system, which includes everything needed (host and target) for the SDK.

Regardless of the type of SDK being constructed, the tasks perform some cleanup after which a cross-development environment setup script and any needed configuration files are created. The final output is the Cross-development toolchain installation script (.sh file), which includes the environment setup script.

For each task that completes successfully, BitBake writes a stamp file into the STAMPS_DIR directory. The beginning of the stamp file's filename is determined by the STAMP variable, and the end of the name consists of the task's name and current input checksum.

To determine if a task needs to be rerun, BitBake checks if a stamp file with a matching input checksum exists for the task. If such a stamp file exists, the task's output is assumed to exist and still be valid. If the file does not exist, the task is rerun.

Since STAMPS_DIR is usually a subdirectory of TMPDIR, removing TMPDIR will also remove STAMPS_DIR, which means tasks will properly be rerun to repopulate TMPDIR.

If you want some task to always be considered "out of date", you can mark it with the nostamp varflag. If some other task depends on such a task, then that task will also always be considered out of date, which might not be what you want.

For details on how to view information about a task's signature, see the "Viewing Task Variable Dependencies" section.

The description of tasks so far assumes that BitBake needs to build everything and there are no prebuilt objects available. BitBake does support skipping tasks if prebuilt objects are available. These objects are usually made available in the form of a shared state (sstate) cache.

The idea of a setscene task (i.e do_taskname_setscene) is a version of the task where instead of building something, BitBake can skip to the end result and simply place a set of files into specific locations as needed. In some cases, it makes sense to have a setscene task variant (e.g. generating package files in the do_package_write_* task). In other cases, it does not make sense, (e.g. a do_patch task or do_unpack task) since the work involved would be equal to or greater than the underlying task.

In the OpenEmbedded build system, the common tasks that have setscene variants are do_package, do_package_write_*, do_deploy, do_packagedata, and do_populate_sysroot. Notice that these are most of the tasks whose output is an end result.

The OpenEmbedded build system has knowledge of the relationship between these tasks and other tasks that precede them. For example, if BitBake runs do_populate_sysroot_setscene for something, there is little point in running any of the do_fetch, do_unpack, do_patch, do_configure, do_compile, and do_install tasks. However, if do_package needs to be run, BitBake would need to run those other tasks.

It becomes more complicated if everything can come from an sstate cache because some objects are simply not required at all. For example, you do not need a compiler or native tools, such as quilt, if there is nothing to compile or patch. If the do_package_write_* packages are available from sstate, BitBake does not need the do_package task data.

To handle all these complexities, BitBake runs in two phases. The first is the "setscene" stage. During this stage, BitBake first checks the sstate cache for any targets it is planning to build. BitBake does a fast check to see if the object exists rather than a complete download. If nothing exists, the second phase, which is the setscene stage, completes and the main build proceeds.

If objects are found in the sstate cache, the OpenEmbedded build system works backwards from the end targets specified by the user. For example, if an image is being built, the OpenEmbedded build system first looks for the packages needed for that image and the tools needed to construct an image. If those are available, the compiler is not needed. Thus, the compiler is not even downloaded. If something was found to be unavailable, or the download or setscene task fails, the OpenEmbedded build system then tries to install dependencies, such as the compiler, from the cache.

The availability of objects in the sstate cache is handled by the function specified by the BB_HASHCHECK_FUNCTION variable and returns a list of the objects that are available. The function specified by the BB_SETSCENE_DEPVALID variable is the function that determines whether a given dependency needs to be followed, and whether for any given relationship the function needs to be passed. The function returns a True or False value.

BitBake is the tool at the heart of the OpenEmbedded build system and is responsible for parsing the Metadata, generating a list of tasks from it, and then executing those tasks.

This section briefly introduces BitBake. If you want more information on BitBake, see the BitBake User Manual.

To see a list of the options BitBake supports, use either of the following commands:

     $ bitbake -h
     $ bitbake --help
            

The most common usage for BitBake is bitbake packagename, where packagename is the name of the package you want to build (referred to as the "target" in this manual). The target often equates to the first part of a recipe's filename (e.g. "foo" for a recipe named foo_1.3.0-r0.bb). So, to process the matchbox-desktop_1.2.3.bb recipe file, you might type the following:

     $ bitbake matchbox-desktop
            

Several different versions of matchbox-desktop might exist. BitBake chooses the one selected by the distribution configuration. You can get more details about how BitBake chooses between different target versions and providers in the "Preferences" section of the BitBake User Manual.

BitBake also tries to execute any dependent tasks first. So for example, before building matchbox-desktop, BitBake would build a cross compiler and glibc if they had not already been built.

A useful BitBake option to consider is the -k or --continue option. This option instructs BitBake to try and continue processing the job as long as possible even after encountering an error. When an error occurs, the target that failed and those that depend on it cannot be remade. However, when you use this option other dependencies can still be processed.

Files that have the .bb suffix are "recipes" files. In general, a recipe contains information about a single piece of software. This information includes the location from which to download the unaltered source, any source patches to be applied to that source (if needed), which special configuration options to apply, how to compile the source files, and how to package the compiled output.

The term "package" is sometimes used to refer to recipes. However, since the word "package" is used for the packaged output from the OpenEmbedded build system (i.e. .ipk or .deb files), this document avoids using the term "package" when referring to recipes.

Class files (.bbclass) contain information that is useful to share between Metadata files. An example is the autotools class, which contains common settings for any application that Autotools uses. The "Classes" chapter provides details about classes and how to use them.

The configuration files (.conf) define various configuration variables that govern the OpenEmbedded build process. These files fall into several areas that define machine configuration options, distribution configuration options, compiler tuning options, general common configuration options, and user configuration options in local.conf, which is found in the Build Directory.

When determining what parts of the system need to be built, BitBake works on a per-task basis rather than a per-recipe basis. You might wonder why using a per-task basis is preferred over a per-recipe basis. To help explain, consider having the IPK packaging backend enabled and then switching to DEB. In this case, the do_install and do_package task outputs are still valid. However, with a per-recipe approach, the build would not include the .deb files. Consequently, you would have to invalidate the whole build and rerun it. Rerunning everything is not the best solution. Also, in this case, the core must be "taught" much about specific tasks. This methodology does not scale well and does not allow users to easily add new tasks in layers or as external recipes without touching the packaged-staging core.

The shared state code uses a checksum, which is a unique signature of a task's inputs, to determine if a task needs to be run again. Because it is a change in a task's inputs that triggers a rerun, the process needs to detect all the inputs to a given task. For shell tasks, this turns out to be fairly easy because the build process generates a "run" shell script for each task and it is possible to create a checksum that gives you a good idea of when the task's data changes.

To complicate the problem, there are things that should not be included in the checksum. First, there is the actual specific build path of a given task - the WORKDIR. It does not matter if the work directory changes because it should not affect the output for target packages. Also, the build process has the objective of making native or cross packages relocatable.

The checksum therefore needs to exclude WORKDIR. The simplistic approach for excluding the work directory is to set WORKDIR to some fixed value and create the checksum for the "run" script.

Another problem results from the "run" scripts containing functions that might or might not get called. The incremental build solution contains code that figures out dependencies between shell functions. This code is used to prune the "run" scripts down to the minimum set, thereby alleviating this problem and making the "run" scripts much more readable as a bonus.

So far we have solutions for shell scripts. What about Python tasks? The same approach applies even though these tasks are more difficult. The process needs to figure out what variables a Python function accesses and what functions it calls. Again, the incremental build solution contains code that first figures out the variable and function dependencies, and then creates a checksum for the data used as the input to the task.

Like the WORKDIR case, situations exist where dependencies should be ignored. For these cases, you can instruct the build process to ignore a dependency by using a line like the following:

     PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
            

This example ensures that the PACKAGE_ARCHS variable does not depend on the value of MACHINE, even if it does reference it.

Equally, there are cases where we need to add dependencies BitBake is not able to find. You can accomplish this by using a line like the following:

      PACKAGE_ARCHS[vardeps] = "MACHINE"
            

This example explicitly adds the MACHINE variable as a dependency for PACKAGE_ARCHS.

Consider a case with in-line Python, for example, where BitBake is not able to figure out dependencies. When running in debug mode (i.e. using -DDD), BitBake produces output when it discovers something for which it cannot figure out dependencies. The Yocto Project team has currently not managed to cover those dependencies in detail and is aware of the need to fix this situation.

Thus far, this section has limited discussion to the direct inputs into a task. Information based on direct inputs is referred to as the "basehash" in the code. However, there is still the question of a task's indirect inputs - the things that were already built and present in the Build Directory. The checksum (or signature) for a particular task needs to add the hashes of all the tasks on which the particular task depends. Choosing which dependencies to add is a policy decision. However, the effect is to generate a master checksum that combines the basehash and the hashes of the task's dependencies.

At the code level, there are a variety of ways both the basehash and the dependent task hashes can be influenced. Within the BitBake configuration file, we can give BitBake some extra information to help it construct the basehash. The following statement effectively results in a list of global variable dependency excludes - variables never included in any checksum:

     BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
         SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
         USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
         PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
         CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
            

The previous example excludes WORKDIR since that variable is actually constructed as a path within TMPDIR, which is on the whitelist.

The rules for deciding which hashes of dependent tasks to include through dependency chains are more complex and are generally accomplished with a Python function. The code in meta/lib/oe/sstatesig.py shows two examples of this and also illustrates how you can insert your own policy into the system if so desired. This file defines the two basic signature generators OE-Core uses: "OEBasic" and "OEBasicHash". By default, there is a dummy "noop" signature handler enabled in BitBake. This means that behavior is unchanged from previous versions. OE-Core uses the "OEBasicHash" signature handler by default through this setting in the bitbake.conf file:

     BB_SIGNATURE_HANDLER ?= "OEBasicHash"
            

The "OEBasicHash" BB_SIGNATURE_HANDLER is the same as the "OEBasic" version but adds the task hash to the stamp files. This results in any Metadata change that changes the task hash, automatically causing the task to be run again. This removes the need to bump PR values, and changes to Metadata automatically ripple across the build.

It is also worth noting that the end result of these signature generators is to make some dependency and hash information available to the build. This information includes:

Checksums and dependencies, as discussed in the previous section, solve half the problem of supporting a shared state. The other part of the problem is being able to use checksum information during the build and being able to reuse or rebuild specific components.

The sstate class is a relatively generic implementation of how to "capture" a snapshot of a given task. The idea is that the build process does not care about the source of a task's output. Output could be freshly built or it could be downloaded and unpacked from somewhere - the build process does not need to worry about its origin.

There are two types of output, one is just about creating a directory in WORKDIR. A good example is the output of either do_install or do_package. The other type of output occurs when a set of data is merged into a shared directory tree such as the sysroot.

The Yocto Project team has tried to keep the details of the implementation hidden in sstate class. From a user's perspective, adding shared state wrapping to a task is as simple as this do_deploy example taken from the deploy class:

     DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
     SSTATETASKS += "do_deploy"
     do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
     do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"

     python do_deploy_setscene () {
         sstate_setscene(d)
     }
     addtask do_deploy_setscene
     do_deploy[dirs] = "${DEPLOYDIR} ${B}"
            

The following list explains the previous example:

Behind the scenes, the shared state code works by looking in SSTATE_DIR and SSTATE_MIRRORS for shared state files. Here is an example:

     SSTATE_MIRRORS ?= "\
     file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
     file://.* file:///some/local/dir/sstate/PATH"
            

The shared state package validity can be detected just by looking at the filename since the filename contains the task checksum (or signature) as described earlier in this section. If a valid shared state package is found, the build process downloads it and uses it to accelerate the task.

The build processes use the *_setscene tasks for the task acceleration phase. BitBake goes through this phase before the main execution code and tries to accelerate any tasks for which it can find shared state packages. If a shared state package for a task is available, the shared state package is used. This means the task and any tasks on which it is dependent are not executed.

As a real world example, the aim is when building an IPK-based image, only the do_package_write_ipk tasks would have their shared state packages fetched and extracted. Since the sysroot is not used, it would never get extracted. This is another reason why a task-based approach is preferred over a recipe-based approach, which would have to install the output from every task.

The code in the build system that supports incremental builds is not simple code. This section presents some tips and tricks that help you work around issues related to shared state code.

This Yocto Project release supports the final specifications of x32 psABI. Support for x32 psABI exists as follows:

Future Plans for the x32 psABI in the Yocto Project include the following:

Follow these steps to use the x32 spABI:

The Wayland protocol libraries and the reference Weston compositor ship as integrated packages in the meta layer of the Source Directory. Specifically, you can find the recipes that build both Wayland and Weston at meta/recipes-graphics/wayland.

You can build both the Wayland and Weston packages for use only with targets that accept the Mesa 3D and Direct Rendering Infrastructure, which is also known as Mesa DRI. This implies that you cannot build and use the packages if your target uses, for example, the Intel® Embedded Media and Graphics Driver (Intel® EMGD) that overrides Mesa DRI.

To enable Wayland, you need to enable it to be built and enable it to be included in the image.

     DISTRO_FEATURES_append = " wayland"
                

     CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
                

To run Weston inside X11, enabling it as described earlier and building a Sato image is sufficient. If you are running your image under Sato, a Weston Launcher appears in the "Utility" category.

Alternatively, you can run Weston through the command-line interpretor (CLI), which is better suited for development work. To run Weston under the CLI, you need to do the following after your image is built:

The license of an upstream project might change in the future. In order to prevent these changes going unnoticed, the LIC_FILES_CHKSUM variable tracks changes to the license text. The checksums are validated at the end of the configure step, and if the checksums do not match, the build will fail.

     LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
                         file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
                         file://licfile2.txt;endline=50;md5=zzzz \
                         ..."
                

     LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
                                         md5=bb14ed3c4cda583abc85401304b5cd4e"
     LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
                

By default, the OpenEmbedded build system disables components that have commercial or other special licensing requirements. Such requirements are defined on a recipe-by-recipe basis through the LICENSE_FLAGS variable definition in the affected recipe. For instance, the poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly recipe contains the following statement:

     LICENSE_FLAGS = "commercial"
            

Here is a slightly more complicated example that contains both an explicit recipe name and version (after variable expansion):

     LICENSE_FLAGS = "license_${PN}_${PV}"
            

In order for a component restricted by a LICENSE_FLAGS definition to be enabled and included in an image, it needs to have a matching entry in the global LICENSE_FLAGS_WHITELIST variable, which is a variable typically defined in your local.conf file. For example, to enable the poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly package, you could add either the string "commercial_gst-plugins-ugly" or the more general string "commercial" to LICENSE_FLAGS_WHITELIST. See the "License Flag Matching" section for a full explanation of how LICENSE_FLAGS matching works. Here is the example:

     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
            

Likewise, to additionally enable the package built from the recipe containing LICENSE_FLAGS = "license_${PN}_${PV}", and assuming that the actual recipe name was emgd_1.10.bb, the following string would enable that package as well as the original gst-plugins-ugly package:

     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
            

As a convenience, you do not need to specify the complete license string in the whitelist for every package. You can use an abbreviated form, which consists of just the first portion or portions of the license string before the initial underscore character or characters. A partial string will match any license that contains the given string as the first portion of its license. For example, the following whitelist string will also match both of the packages previously mentioned as well as any other packages that have licenses starting with "commercial" or "license".

     LICENSE_FLAGS_WHITELIST = "commercial license"
            

     LICENSE_FLAGS = "commercial"
                

     COMMERCIAL_AUDIO_PLUGINS ?= ""
     COMMERCIAL_VIDEO_PLUGINS ?= ""
                

     COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
        gst-plugins-ugly-mpegaudioparse"
     COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
        gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
     LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
                

     LICENSE_FLAGS_WHITELIST = "commercial"
                

Differences include changes for SSTATE_MIRRORS and bblayers.conf.

     SSTATE_MIRRORS = "file://.* http://someserver.tld/share/sstate/PATH"
                

Differences include changes for the following:

The naming scheme for kernel output binaries has been changed to now include PE as part of the filename:

     KERNEL_IMAGE_BASE_NAME ?= "${KERNEL_IMAGETYPE}-${PE}-${PV}-${PR}-${MACHINE}-${DATETIME}"
            

Because the PE variable is not set by default, these binary files could result with names that include two dash characters. Here is an example:

     bzImage--3.10.9+git0+cd502a8814_7144bcc4b8-r0-qemux86-64-20130830085431.bin
            

Differences include the following:

Differences include the following:

A new oe-git-proxy script has been added to replace previous methods of handling proxies and fetching source from Git. See the meta-yocto/conf/site.conf.sample file for information on how to use this script.

If you have created your own custom etc/network/interfaces file by creating an append file for the netbase recipe, you now need to create an append file for the init-ifupdown recipe instead, which you can find in the Source Directory at meta/recipes-core/init-ifupdown. For information on how to use append files, see the "Using .bbappend Files" in the Yocto Project Development Manual.

Support for remote debugging with the Eclipse IDE is now separated into an image feature (eclipse-debug) that corresponds to the packagegroup-core-eclipse-debug package group. Previously, the debugging feature was included through the tools-debug image feature, which corresponds to the packagegroup-core-tools-debug package group.

The following variables have changed:

If runtime package management is enabled and the RPM backend is selected, Smart is now installed for package download, dependency resolution, and upgrades instead of Zypper. For more information on how to use Smart, run the following command on the target:

     smart --help
            

The following recipes were moved from their previous locations because they are no longer used by anything in the OpenEmbedded-Core:

The following list shows what has been removed or renamed:

The OpenEmbedded build system now has some additional requirements on the host system:

If the Linux distribution you are using on your build host does not provide packages for these, you can install and use the Buildtools tarball, which provides an SDK-like environment containing them.

For more information on this requirement, see the "Required Git, tar, and Python Versions" section.

The atom-pc hardware reference BSP has been replaced by a genericx86 BSP. This BSP is not necessarily guaranteed to work on all x86 hardware, but it will run on a wider range of systems than the atom-pc did.

The following changes have been made that relate to BitBake:

The following changes have been made to the package QA checks:

The following directory changes exist:

BitBake will now shorten revisions from Git repositories from the normal 40 characters down to 10 characters within SRCPV for improved usability in path and file names. This change should be safe within contexts where these revisions are used because the chances of spatially close collisions is very low. Distant collisions are not a major issue in the way the values are used.

The following changes have been made that relate to IMAGE_FEATURES:

The /run directory from the Filesystem Hierarchy Standard 3.0 has been introduced. You can find some of the implications for this change here. The change also means that recipes that install files to /var/run must be changed. You can find a guide on how to make these changes here.

The image core-image-minimal no longer adds remove_packaging_data_files to ROOTFS_POSTPROCESS_COMMAND. This addition is now handled automatically when "package-management" is not in IMAGE_FEATURES. If you have custom image recipes that make this addition, you should remove the lines, as they are not needed and might interfere with correct operation of postinstall scripts.

The do_rootfs and other related image construction tasks are no longer marked as "nostamp". Consequently, they will only be re-executed when their inputs have changed. Previous versions of the OpenEmbedded build system always rebuilt the image when requested rather when necessary.

The previously deprecated task.bbclass has now been dropped. For recipes that previously inherited from this class, you should rename them from task-* to packagegroup-* and inherit packagegroup instead.

For more information, see the "packagegroup.bbclass" section.

By default, we now split BusyBox into two binaries: one that is suid root for those components that need it, and another for the rest of the components. Splitting BusyBox allows for optimization that eliminates the tinylogin recipe as recommended by upstream. You can disable this split by setting BUSYBOX_SPLIT_SUID to "0".

A new automated image testing framework has been added through the testimage.bbclass class. This framework replaces the older imagetest-qemu framework.

You can learn more about performing automated image tests in the "Performing Automated Runtime Testing" section.

Following are changes to Build History:

For more information on Build History, see the "Maintaining Build Output Quality" section.

Following are changes to udev:

Following is a list of short entries describing other changes:

The archiver class has been rewritten and its configuration has been simplified. For more details on the source archiver, see the "Maintaining Open Source License Compliance During Your Product's Lifecycle" section in the Yocto Project Development Manual.

The following packaging changes have been made:

The following changes have been made to BitBake.

     SRC_URI = "git://server.name/repository;branch=branchname"
                

The following variables have changed. For information on the OpenEmbedded build system variables, see the "Variables Glossary" Chapter.

     ROOTFS_PREPROCESS_COMMAND
     ROOTFS_POSTPROCESS_COMMAND
     SDK_POSTPROCESS_COMMAND
     POPULATE_SDK_POST_TARGET_COMMAND
     POPULATE_SDK_POST_HOST_COMMAND
     IMAGE_POSTPROCESS_COMMAND
     IMAGE_PREPROCESS_COMMAND
     ROOTFS_POSTUNINSTALL_COMMAND
     ROOTFS_POSTINSTALL_COMMAND
                

     my_postprocess_function() {
        echo "hello" > ${IMAGE_ROOTFS}/hello.txt
     }
     ROOTFS_POSTPROCESS_COMMAND += "my_postprocess_function; "
                

Package Tests (ptest) are built but not installed by default. For information on using Package Tests, see the "Setting up and running package test (ptest)" section in the Yocto Project Development Manual. For information on the ptest class, see the "ptest.bbclass" section.

Separate build and source directories have been enabled by default for selected recipes where it is known to work (a whitelist) and for all recipes that inherit the cmake class. In future releases the autotools class will enable a separate build directory by default as well. Recipes building Autotools-based software that fails to build with a separate build directory should be changed to inherit from the autotools-brokensep class instead of the autotools or autotools_stageclasses.

qemu-native now builds without SDL-based graphical output support by default. The following additional lines are needed in your local.conf to enable it:

     PACKAGECONFIG_pn-qemu-native = "sdl"
     ASSUME_PROVIDED += "libsdl-native"
            

core-image-basic has been renamed to core-image-full-cmdline.

In addition to core-image-basic being renamed, packagegroup-core-basic has been renamed to packagegroup-core-full-cmdline to match.

The top-level LICENSE file has been changed to better describe the license of the various components of OE-Core. However, the licensing itself remains unchanged.

Normally, this change would not cause any side-effects. However, some recipes point to this file within LIC_FILES_CHKSUM (as ${COREBASE}/LICENSE) and thus the accompanying checksum must be changed from 3f40d7994397109285ec7b81fdeb3b58 to 4d92cd373abda3937c2bc47fbc49d690. A better alternative is to have LIC_FILES_CHKSUM point to a file describing the license that is distributed with the source that the recipe is building, if possible, rather than pointing to ${COREBASE}/LICENSE.

The "-fpermissive" option has been removed from the default CFLAGS value. You need to take action on individual recipes that fail when building with this option. You need to either patch the recipes to fix the issues reported by the compiler, or you need to add "-fpermissive" to CFLAGS in the recipes.

Custom image output types, as selected using IMAGE_FSTYPES, must declare their dependencies on other image types (if any) using a new IMAGE_TYPEDEP variable.

The do_package_write task has been removed. The task is no longer needed.