[yocto] Discussion: Package testing

[Björn Stenberg]

Hi.

Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.

The actions that need to be performed for a package test are roughly:

1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the results

Each action can be done in several ways, and there are different considerations for each solution.

1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.

Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.

We can solve this in different ways:

a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.

b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.


2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:

a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.

b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network+nfs support. Plus it blends the build env and runtime env in an unpleasant way.

c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.


3) Run the test suite
---------------------
Depending on how the test files are presented to the target, the way we run them can take different shapes:

a) (scp) A top-level run-all-tests.sh runs on the build host, copies all test files from build dir to target, logs in and runs each test.

b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on target and build-runs each test in its work dir.

c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for example). Make a package "ptest-runner" that has a script /opt/ptest/run-all-tests to iterate over all installed tests and run them. This is our suggestion.


4) Parse the test results
-------------------------
Just running the tests doesn't give us much. We have to be able to look at the results and make them meaningful to us on a system-global level. Packages present their test results in very different ways, and we need to convert that to a generic format:

a) Patch each package test to produce a generic ptest output format. This is likely difficult to get accepted upstream.

b) Patch the test code minimally and instead use a simple per-packet translate script that converts test suite output from the package-specific format to a generic ptest format. This is our suggestion.


Opinions?

Note that this mail is about the test suites for/in/by specific packages. Standalone test suites such as LTP are a slightly different topic, since they are separate packages (${BPN} == ${PN}) rather than package companion suites.

-- 
Björn