44 Setting up a Hash Equivalence Server

A Hash Equivalence server can help reduce build times by using the mechanism described in the Hash Equivalence section of the Yocto Project Overview and Concepts Manual.

This document will guide you through the steps required to set up the reference Hash Equivalence server provided by the bitbake-hashserv script in BitBake.

This guide will explain how to setup a local Hash Equivalence server and does not explain how to setup the surrounding infrastructure to secure this server.

44.1 Hash Equivalence Server Setup

From this point onwards, the commands displayed below are assumed to be run from the BitBake repository, which can be found here.

To start a basic Hash Equivalence server, one could simply run:

./bin/bitbake-hashserv

This will take all of the default options of the script, which are already sufficient to start a local server.

Run ./bin/bitbake-hashserv --help to see what options the script can take. Some of the important ones are:

--database controls the location of the hash server database (default: ./hashserver.db).
--bind controls the bind address of the server (default: unix://./hashserver.sock).

You can specify three types of addresses:
- unix://PATH: will bind to a Unix socket at PATH.
- wss://ADDRESS:PORT: will bind to a Websocket at ADDRESS:PORT.
- ADDRESS:PORT: will bind to a raw TCP socket at ADDRESS:PORT.
--log can be used to control the logging level of the server (e.g. INFO will print information about clients connection to the server).
--upstream can be used to specify an upstream server to pull hashes from. This has no default value, meaning no upstream server is used.
--db-username and --db-password can be used to control the access to the database.
--read-only will disable hash reports from clients.

These variables can also be set from the environment from where it is being run. Run ./bin/bitbake-hashserv --help to get the variable names that you can export.

Warning

The shared Hash Equivalence server needs to be maintained together with the Shared State cache. Otherwise, the server could report Shared State hashes that only exist on specific clients.

We therefore recommend that one Hash Equivalence server be set up to correspond with a given Shared State cache, and to start this server in read-only mode (--read-only option), so that it doesn’t store equivalences for Shared State caches that are local to clients.

If there is no pre-existing Shared State Cache, the server should allow hashes to be reported (no --read-only option) to create the initial Hash Equivalence database.

44.2 Yocto Project Build Setup

To use the server started in the previous section, set the following variables in a configuration file:

BB_HASHSERVE = "<bind address>"
BB_SIGNATURE_HANDLER = "OEEquivHash"

The <bind address> value should be replaced to point to the server started in the previous section.

See the documentation of BB_SIGNATURE_HANDLER for more details on this variable.

You can optionally specify an upstream server with BB_HASHSERVE_UPSTREAM variable. For example:

BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws"

This will make the local server pull hashes from the upstream server. The BB_HASHSERVE_UPSTREAM only works when a server is running (BB_HASHSERVE is set).

To output debugging information on what is happening with Hash Equivalence when builds are started, you can configure BitBake logging as follows from a configuration file:

BB_LOGCONFIG = "hashequiv.json"

With hashequiv.json containing the following content:

{
   "version": 1,
   "loggers": {
      "BitBake.SigGen.HashEquiv": {
         "level": "VERBOSE",
         "handlers": ["BitBake.verbconsole"]
      },
      "BitBake.RunQueue.HashEquiv": {
         "level": "VERBOSE",
         "handlers": ["BitBake.verbconsole"]
      }
   }
}

This will make Hash Equivalence related changes be printed on the console, such as:

NOTE: Task <recipe.bb>:do_<task> unihash changed to dc0da29c62a2d78d8d853fbb9c173778fe7d6fa4a68c67494b17afffe8ca1894