Yocto Project Overview and Concepts Manual

Widely Adopted Across the Industry: Semiconductor, operating system, software, and service vendors exist whose products and services adopt and support the Yocto Project. For a look at the Yocto Project community and the companies involved with the Yocto Project, see the "COMMUNITY" and "ECOSYSTEM" tabs on the Yocto Project home page.

Architecture Agnostic: Yocto Project supports Intel, ARM, MIPS, AMD, PPC and other architectures. Most ODMs, OSVs, and chip vendors create and supply BSPs that support their hardware. If you have custom silicon, you can create a BSP that supports that architecture.

Aside from lots of architecture support, the Yocto Project fully supports a wide range of device emulation through the Quick EMUlator (QEMU).

Images and Code Transfer Easily: Yocto Project output can easily move between architectures without moving to new development environments. Additionally, if you have used the Yocto Project to create an image or application and you find yourself not able to support it, commercial Linux vendors such as Wind River, Mentor Graphics, Timesys, and ENEA could take it and provide ongoing support. These vendors have offerings that are built using the Yocto Project.

Flexibility: Corporations use the Yocto Project many different ways. One example is to create an internal Linux distribution as a code base the corporation can use across multiple product groups. Through customization and layering, a project group can leverage the base Linux distribution to create a distribution that works for their product needs.

Ideal for Constrained Embedded and IoT devices: Unlike a full Linux distribution, you can use the Yocto Project to create exactly what you need for embedded devices. You only add the feature support or packages that you absolutely need for the device. For devices that have display hardware, you can use available system components such as X11, GTK+, Qt, Clutter, and SDL (among others) to create a rich user experience. For devices that do not have a display or where you want to use alternative UI frameworks, you can choose to not install these components.

Comprehensive Toolchain Capabilities: Toolchains for supported architectures satisfy most use cases. However, if your hardware supports features that are not part of a standard toolchain, you can easily customize that toolchain through specification of platform-specific tuning parameters. And, should you need to use a third-party toolchain, mechanisms built into the Yocto Project allow for that.

Mechanism Rules Over Policy: Focusing on mechanism rather than policy ensures that you are free to set policies based on the needs of your design instead of adopting decisions enforced by some system software provider.

Uses a Layer Model: The Yocto Project layer infrastructure groups related functionality into separate bundles. You can incrementally add these grouped functionalities to your project as needed. Using layers to isolate and group functionality reduces project complexity and redundancy, allows you to easily extend the system, make customizations, and keep functionality organized.

Supports Partial Builds: You can build and rebuild individual packages as needed. Yocto Project accomplishes this through its shared-state cache (sstate) scheme. Being able to build and debug components individually eases project development.

Releases According to a Strict Schedule: Major releases occur on a six-month cycle predictably in October and April. The most recent two releases support point releases to address common vulnerabilities and exposures. This predictability is crucial for projects based on the Yocto Project and allows development teams to plan activities.

Rich Ecosystem of Individuals and Organizations: For open source projects, the value of community is very important. Support forums, expertise, and active developers who continue to push the Yocto Project forward are readily available.

Binary Reproducibility: The Yocto Project allows you to be very specific about dependencies and achieves very high percentages of binary reproducibility (e.g. 99.8% for core-image-minimal). When distributions are not specific about which packages are pulled in and in what order to support dependencies, other build systems can arbitrarily include packages.

License Manifest: The Yocto Project provides a license manifest for review by people who need to track the use of open source licenses (e.g.legal teams).

Steep Learning Curve: The Yocto Project has a steep learning curve and has many different ways to accomplish similar tasks. It can be difficult to choose how to proceed when varying methods exist by which to accomplish a given task.

Understanding What Changes You Need to Make For Your Design Requires Some Research: Beyond the simple tutorial stage, understanding what changes need to be made for your particular design can require a significant amount of research and investigation. For information that helps you transition from trying out the Yocto Project to using it for your project, see the "What I wish I'd Known" and "Transitioning to a Custom Environment for Systems Development" documents on the Yocto Project website.

Project Workflow Could Be Confusing: The Yocto Project workflow could be confusing if you are used to traditional desktop and server software development. In a desktop development environment, mechanisms exist to easily pull and install new packages, which are typically pre-compiled binaries from servers accessible over the Internet. Using the Yocto Project, you must modify your configuration and rebuild to add additional packages.

Working in a Cross-Build Environment Can Feel Unfamiliar: When developing code to run on a target, compilation, execution, and testing done on the actual target can be faster than running a BitBake build on a development host and then deploying binaries to the target for test. While the Yocto Project does support development tools on the target, the additional step of integrating your changes back into the Yocto Project build environment would be required. Yocto Project supports an intermediate approach that involves making changes on the development system within the BitBake environment and then deploying only the updated packages to the target.

The Yocto Project OpenEmbedded build system produces packages in standard formats (i.e. RPM, DEB, IPK, and TAR). You can deploy these packages into the running system on the target by using utilities on the target such as rpm or ipk.

Initial Build Times Can be Significant: Long initial build times are unfortunately unavoidable due to the large number of packages initially built from scratch for a fully functioning Linux system. Once that initial build is completed, however, the shared-state (sstate) cache mechanism Yocto Project uses keeps the system from rebuilding packages that have not been "touched" since the last build. The sstate mechanism significantly reduces times for successive builds.

CROPS: CROPS is an open source, cross-platform development framework that leverages Docker Containers. CROPS provides an easily managed, extensible environment that allows you to build binaries for a variety of architectures on Windows, Linux and Mac OS X hosts.

devtool: This command-line tool is available as part of the extensible SDK (eSDK) and is its cornerstone. You can use devtool to help build, test, and package software within the eSDK. You can use the tool to optionally integrate what you build into an image built by the OpenEmbedded build system.

The devtool command employs a number of sub-commands that allow you to add, modify, and upgrade recipes. As with the OpenEmbedded build system, “recipes” represent software packages within devtool. When you use devtool add, a recipe is automatically created. When you use devtool modify, the specified existing recipe is used in order to determine where to get the source code and how to patch it. In both cases, an environment is set up so that when you build the recipe a source tree that is under your control is used in order to allow you to make changes to the source as desired. By default, both new recipes and the source go into a “workspace” directory under the eSDK. The devtool upgrade command updates an existing recipe so that you can build it for an updated set of source files.

You can read about the devtool workflow in the Yocto Project Application Development and Extensible Software Development Kit (eSDK) Manual in the "Using devtool in Your SDK Workflow'" section.

Extensible Software Development Kit (eSDK): The eSDK provides a cross-development toolchain and libraries tailored to the contents of a specific image. The eSDK makes it easy to add new applications and libraries to an image, modify the source for an existing component, test changes on the target hardware, and integrate into the rest of the OpenEmbedded build system. The eSDK gives you a toolchain experience supplemented with the powerful set of devtool commands tailored for the Yocto Project environment.

For information on the eSDK, see the Yocto Project Application Development and the Extensible Software Development Kit (eSDK) Manual.

Toaster: Toaster is a web interface to the Yocto Project OpenEmbedded build system. Toaster allows you to configure, run, and view information about builds. For information on Toaster, see the Toaster User Manual.

Auto Upgrade Helper: This utility when used in conjunction with the OpenEmbedded build system (BitBake and OE-Core) automatically generates upgrades for recipes that are based on new versions of the recipes published upstream.

Recipe Reporting System: The Recipe Reporting System tracks recipe versions available for Yocto Project. The main purpose of the system is to help you manage the recipes you maintain and to offer a dynamic overview of the project. The Recipe Reporting System is built on top of the OpenEmbedded Layer Index, which is a website that indexes OpenEmbedded-Core layers.

Patchwork: Patchwork is a fork of a project originally started by OzLabs. The project is a web-based tracking system designed to streamline the process of bringing contributions into a project. The Yocto Project uses Patchwork as an organizational tool to handle patches, which number in the thousands for every release.

AutoBuilder: AutoBuilder is a project that automates build tests and quality assurance (QA). By using the public AutoBuilder, anyone can determine the status of the current "master" branch of Poky.

A goal of the Yocto Project is to lead the open source industry with a project that automates testing and QA procedures. In doing so, the project encourages a development community that publishes QA and test plans, publicly demonstrates QA and test plans, and encourages development of tools that automate and test and QA procedures for the benefit of the development community.

You can learn more about the AutoBuilder used by the Yocto Project here.

Cross-Prelink: Prelinking is the process of pre-computing the load addresses and link tables generated by the dynamic linker as compared to doing this at runtime. Doing this ahead of time results in performance improvements when the application is launched and reduced memory usage for libraries shared by many applications.

Historically, cross-prelink is a variant of prelink, which was conceived by Jakub Jelínek a number of years ago. Both prelink and cross-prelink are maintained in the same repository albeit on separate branches. By providing an emulated runtime dynamic linker (i.e. glibc-derived ld.so emulation), the cross-prelink project extends the prelink software’s ability to prelink a sysroot environment. Additionally, the cross-prelink software enables the ability to work in sysroot style environments.

The dynamic linker determines standard load address calculations based on a variety of factors such as mapping addresses, library usage, and library function conflicts. The prelink tool uses this information, from the dynamic linker, to determine unique load addresses for executable and linkable format (ELF) binaries that are shared libraries and dynamically linked. The prelink tool modifies these ELF binaries with the pre-computed information. The result is faster loading and often lower memory consumption because more of the library code can be re-used from shared Copy-On-Write (COW) pages.

The original upstream prelink project only supports running prelink on the end target device due to the reliance on the target device’s dynamic linker. This restriction causes issues when developing a cross-compiled system. The cross-prelink adds a synthesized dynamic loader that runs on the host, thus permitting cross-prelinking without ever having to run on a read-write target filesystem.

Pseudo: Pseudo is the Yocto Project implementation of fakeroot, which is used to run commands in an environment that seemingly has root privileges.

During a build, it can be necessary to perform operations that require system administrator privileges. For example, file ownership or permissions might need definition. Pseudo is a tool that you can either use directly or through the environment variable LD_PRELOAD. Either method allows these operations to succeed as if system administrator privileges exist even when they do not.

You can read more about Pseudo in the "Fakeroot and Pseudo" section.

BitBake: BitBake is a core component of the Yocto Project and is used by the OpenEmbedded build system to build images. While BitBake is key to the build system, BitBake is maintained separately from the Yocto Project.

BitBake is a generic task execution engine that allows shell and Python tasks to be run efficiently and in parallel while working within complex inter-task dependency constraints. In short, BitBake is a build engine that works through recipes written in a specific format in order to perform sets of tasks.

You can learn more about BitBake in the BitBake User Manual.

OpenEmbedded-Core: OpenEmbedded-Core (OE-Core) is a common layer of metadata (i.e. recipes, classes, and associated files) used by OpenEmbedded-derived systems, which includes the Yocto Project. The Yocto Project and the OpenEmbedded Project both maintain the OpenEmbedded-Core. You can find the OE-Core metadata in the Yocto Project Source Repositories.

Historically, the Yocto Project integrated the OE-Core metadata throughout the Yocto Project source repository reference system (Poky). After Yocto Project Version 1.0, the Yocto Project and OpenEmbedded agreed to work together and share a common core set of metadata (OE-Core), which contained much of the functionality previously found in Poky. This collaboration achieved a long-standing OpenEmbedded objective for having a more tightly controlled and quality-assured core. The results also fit well with the Yocto Project objective of achieving a smaller number of fully featured tools as compared to many different ones.

Sharing a core set of metadata results in Poky as an integration layer on top of OE-Core. You can see that in this figure. The Yocto Project combines various components such as BitBake, OE-Core, script “glue”, and documentation for its build system.

Matchbox: Matchbox is an Open Source, base environment for the X Window System running on non-desktop, embedded platforms such as handhelds, set-top boxes, kiosks, and anything else for which screen space, input mechanisms, or system resources are limited.

Matchbox consists of a number of interchangeable and optional applications that you can tailor to a specific, non-desktop platform to enhance usability in constrained environments.

You can find the Matchbox source in the Yocto Project Source Repositories.

Opkg Open PacKaGe management (opkg) is a lightweight package management system based on the itsy package (ipkg) management system. Opkg is written in C and resembles Advanced Package Tool (APT) and Debian Package (dpkg) in operation.

Opkg is intended for use on embedded Linux devices and is used in this capacity in the OpenEmbedded and OpenWrt projects, as well as the Yocto Project.

git init: Initializes an empty Git repository. You cannot use Git commands unless you have a .git repository.

git clone: Creates a local clone of a Git repository that is on equal footing with a fellow developer’s Git repository or an upstream repository.

git add: Locally stages updated file contents to the index that Git uses to track changes. You must stage all files that have changed before you can commit them.

git commit: Creates a local "commit" that documents the changes you made. Only changes that have been staged can be committed. Commits are used for historical purposes, for determining if a maintainer of a project will allow the change, and for ultimately pushing the change from your local Git repository into the project’s upstream repository.

git status: Reports any modified files that possibly need to be staged and gives you a status of where you stand regarding local commits as compared to the upstream repository.

git checkout branch-name: Changes your local working branch and in this form assumes the local branch already exists. This command is analogous to "cd".

git checkout –b working-branch upstream-branch: Creates and checks out a working branch on your local machine. The local branch tracks the upstream branch. You can use your local branch to isolate your work. It is a good idea to use local branches when adding specific features or changes. Using isolated branches facilitates easy removal of changes if they do not work out.

git branch: Displays the existing local branches associated with your local repository. The branch that you have currently checked out is noted with an asterisk character.

git branch -D branch-name: Deletes an existing local branch. You need to be in a local branch other than the one you are deleting in order to delete branch-name.

git pull --rebase: Retrieves information from an upstream Git repository and places it in your local Git repository. You use this command to make sure you are synchronized with the repository from which you are basing changes (.e.g. the "master" branch). The "--rebase" option ensures that any local commits you have in your branch are preserved at the top of your local branch.

git push repo-name local-branch:upstream-branch: Sends all your committed local changes to the upstream Git repository that your local repository is tracking (e.g. a contribution repository). The maintainer of the project draws from these repositories to merge changes (commits) into the appropriate branch of project's upstream repository.

git merge: Combines or adds changes from one local branch of your repository with another branch. When you create a local Git repository, the default branch is named "master". A typical workflow is to create a temporary branch that is based off "master" that you would use for isolated work. You would make your changes in that isolated branch, stage and commit them locally, switch to the "master" branch, and then use the git merge command to apply the changes from your isolated branch into the currently checked out branch (e.g. "master"). After the merge is complete and if you are done with working in that isolated branch, you can safely delete the isolated branch.

git cherry-pick commits: Choose and apply specific commits from one branch into another branch. There are times when you might not be able to merge all the changes in one branch with another but need to pick out certain ones.

gitk: Provides a GUI view of the branches and changes in your local Git repository. This command is a good way to graphically see where things have diverged in your local repository.

git log: Reports a history of your commits to the repository. This report lists all commits regardless of whether you have pushed them upstream or not.

git diff: Displays line-by-line differences between a local working file and the same file as understood by Git. This command is useful to see what you have changed in any given file.

Target Machine Selection: Controlled by the MACHINE variable.

Download Directory: Controlled by the DL_DIR variable.

Shared State Directory: Controlled by the SSTATE_DIR variable.

Build Output: Controlled by the TMPDIR variable.

Distribution Policy: Controlled by the DISTRO variable.

Packaging Format: Controlled by the PACKAGE_CLASSES variable.

SDK Target Architecture: Controlled by the SDKMACHINE variable.

Extra Image Packages: Controlled by the EXTRA_IMAGE_FEATURES variable.

site.conf: You can use the conf/site.conf configuration file to configure multiple build directories. For example, suppose you had several build environments and they shared some common features. You can set these default build properties here. A good example is perhaps the packaging format to use through the PACKAGE_CLASSES variable.

One useful scenario for using the conf/site.conf file is to extend your BBPATH variable to include the path to a conf/site.conf. Then, when BitBake looks for Metadata using BBPATH, it finds the conf/site.conf file and applies your common configurations found in the file. To override configurations in a particular build directory, alter the similar configurations within that build directory's conf/local.conf file.

auto.conf: The file is usually created and written to by an autobuilder. The settings put into the file are typically the same as you would find in the conf/local.conf or the conf/site.conf files.

Metadata (.bb + Patches): Software layers containing user-supplied recipe files, patches, and append files. A good example of a software layer might be the meta-qt5 layer from the OpenEmbedded Layer Index. This layer is for version 5.0 of the popular Qt cross-platform application development framework for desktop, embedded and mobile.

Machine BSP Configuration: Board Support Package (BSP) layers (i.e. "BSP Layer" in the following figure) providing machine-specific configurations. This type of information is specific to a particular target architecture. A good example of a BSP layer from the Poky Reference Distribution is the meta-yocto-bsp layer.

Policy Configuration: Distribution Layers (i.e. "Distro Layer" in the following figure) providing top-level or general policies for the images or SDKs being built for a particular distribution. For example, in the Poky Reference Distribution the distro layer is the meta-poky layer. Within the distro layer is a conf/distro directory that contains distro configuration files (e.g. poky.conf that contain many policy configurations for the Poky distribution.

DEPLOY_DIR: Defined as tmp/deploy in the Build Directory.

DEPLOY_DIR_*: Depending on the package manager used, the package type sub-folder. Given RPM, IPK, or DEB packaging and tarball creation, the DEPLOY_DIR_RPM, DEPLOY_DIR_IPK, DEPLOY_DIR_DEB, or DEPLOY_DIR_TAR, variables are used, respectively.

PACKAGE_ARCH: Defines architecture-specific sub-folders. For example, packages could exist for the i586 or qemux86 architectures.

kernel-image: A kernel binary file. The KERNEL_IMAGETYPE variable determines the naming scheme for the kernel image file. Depending on this variable, the file could begin with a variety of naming strings. The deploy/images/machine directory can contain multiple image files for the machine.

root-filesystem-image: Root filesystems for the target device (e.g. *.ext3 or *.bz2 files). The IMAGE_FSTYPES variable determines the root filesystem image type. The deploy/images/machine directory can contain multiple root filesystems for the machine.

kernel-modules: Tarballs that contain all the modules built for the kernel. Kernel module tarballs exist for legacy purposes and can be suppressed by setting the MODULE_TARBALL_DEPLOY variable to "0". The deploy/images/machine directory can contain multiple kernel module tarballs for the machine.

bootloaders: If applicable to the target machine, bootloaders supporting the image. The deploy/images/machine directory can contain multiple bootloaders for the machine.

symlinks: The deploy/images/machine folder contains a symbolic link that points to the most recently built file for each machine. These links might be useful for external scripts that need to obtain the latest version of each file.

DEPLOY_DIR: Points to the deploy directory.

SDK_EXT_TYPE: Controls whether or not shared state artifacts are copied into the extensible SDK. By default, all required shared state artifacts are copied into the SDK.

SDK_INCLUDE_PKGDATA: Specifies whether or not packagedata is included in the extensible SDK for all recipes in the "world" target.

SDK_INCLUDE_TOOLCHAIN: Specifies whether or not the toolchain is included when building the extensible SDK.

SDK_LOCAL_CONF_WHITELIST: A list of variables allowed through from the build system configuration into the extensible SDK configuration.

SDK_LOCAL_CONF_BLACKLIST: A list of variables not allowed through from the build system configuration into the extensible SDK configuration.

SDK_INHERIT_BLACKLIST: A list of classes to remove from the INHERIT value globally within the extensible SDK configuration.

DEPLOY_DIR: Points to the deploy directory.

SDKMACHINE: Specifies the architecture of the machine on which the cross-development tools are run to create packages for the target hardware.

SDKIMAGE_FEATURES: Lists the features to include in the "target" part of the SDK.

TOOLCHAIN_HOST_TASK: Lists packages that make up the host part of the SDK (i.e. the part that runs on the SDKMACHINE). When you use bitbake -c populate_sdk imagename to create the SDK, a set of default packages apply. This variable allows you to add more packages.

TOOLCHAIN_TARGET_TASK: Lists packages that make up the target part of the SDK (i.e. the part built for the target hardware).

SDKPATH: Defines the default SDK installation path offered by the installation script.

SDK_HOST_MANIFEST: Lists all the installed packages that make up the host part of the SDK. This variable also plays a minor role for extensible SDK development as well. However, it is mainly used for the standard SDK.

SDK_TARGET_MANIFEST: Lists all the installed packages that make up the target part of the SDK. This variable also plays a minor role for extensible SDK development as well. However, it is mainly used for the standard SDK.

BB_BASEHASH_task-taskname: The base hashes for each task in the recipe.

BB_BASEHASH_filename:taskname: The base hashes for each dependent task.

BBHASHDEPS_filename:taskname: The task dependencies for each task.

BB_TASKHASH: The hash of the currently running task.

Adding "do_deploy" to SSTATETASKS adds some required sstate-related processing, which is implemented in the sstate class, to before and after the do_deploy task.

The do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" declares that do_deploy places its output in ${DEPLOYDIR} when run normally (i.e. when not using the sstate cache). This output becomes the input to the shared state cache.

The do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" line causes the contents of the shared state cache to be copied to ${DEPLOY_DIR_IMAGE}.

The following task definition is glue logic needed to make the previous settings effective:

     python do_deploy_setscene () {
         sstate_setscene(d)
     }
     addtask do_deploy_setscene
                        

sstate_setscene() takes the flags above as input and accelerates the do_deploy task through the shared state cache if possible. If the task was accelerated, sstate_setscene() returns True. Otherwise, it returns False, and the normal do_deploy task runs. For more information, see the "setscene" section in the BitBake User Manual.

The do_deploy[dirs] = "${DEPLOYDIR} ${B}" line creates ${DEPLOYDIR} and ${B} before the do_deploy task runs, and also sets the current working directory of do_deploy to ${B}. For more information, see the "Variable Flags" section in the BitBake User Manual.

     do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
                            

The do_deploy[stamp-extra-info] = "${MACHINE_ARCH}" line appends extra metadata to the stamp file. In this case, the metadata makes the task specific to a machine's architecture. See "The Task List" section in the BitBake User Manual for more information on the stamp-extra-info flag.

sstate-inputdirs and sstate-outputdirs can also be used with multiple directories. For example, the following declares PKGDESTWORK and SHLIBWORK as shared state input directories, which populates the shared state cache, and PKGDATA_DIR and SHLIBSDIR as the corresponding shared state output directories:

     do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
     do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
                        

These methods also include the ability to take a lockfile when manipulating shared state directory structures, for cases where file additions or removals are sensitive:

     do_package[sstate-lockfile] = "${PACKAGELOCK}"