[intel-iot-refkit] speeding up CI builds

[Kartau, Olev]

On Wed, 2017-05-24 at 18:09 +0200, Patrick Ohly wrote:
> Hello!
> 
> Olev recently noticed that compressing images can be quite slow (several
> minutes) and happens to be on the critical path. I looked into disabling
> the compressed image variants (like wic.xz) for the oe-selftest. But at
> least for the installer image tests that forces a rebuild of the
> installer image, which defeats the original goal of using the image that
> was already built earlier during the main build.
> 
> We don't publish images for every pull request test build, do we?
Builders store all images in internal server (iot-ref-kit.ostc.intel.com,
same one that hosts buildhistory and caches), where HW-testers
can fetch images for testing.

>  So I
> think it would be best if we disabled the
> 
>         REFKIT_VM_IMAGE_TYPES = "wic.xz wic.zip wic.bmap
>         wic.xz.sha256sum"
> 
in current HW-tester script, that would cause failure as it tries
to download either wic.xz or wic.zip,
but that's easy to fix by adding 3rd try for wic.


> from refkit-ci.inc completely for pull request tests. Is it possible to
> export some variable similar to BUILD_ID which exports the type of the
> build to refkit-ci.inc? 
We added that in ostro CI, but it's not currently present in refkit CI.
Easy to add, CI_BUILD_TYPE could get value "pull-request" or "product"


> Then we can limit the change above to master
> builds.
> 
> This leads me to another observation. We are starting to add
> oe-selftests that by themselves take much longer than the actual build.
> We have to, because we need these tests to ensure that our features
> don't regress.
> 
> Build time with PR #147 merged will be around 2:15. This may go down
> again when updating OE-core (which has a patch that avoids unnecessary
> image rebuilding), but eventually we are going to have to deal with
> multi-hour build tests again.
> 
> I see some potential solutions:
>      1. parallelize some of our oe-selftests, like OE-core is currently
>         doing
Making post-build tests parallel would be cool, if that's safely doable.
Now these mostly utilize one core on a builder.
But is it so these tests need clean state of workspace,
and parallel run risks to affect same workspace from other tests?

>      2. run only fast and/or selected (*) selftests for PRs, merge them
>         into a master-next branch, and only do full testing of
>         master-next before promoting that to master
> 
> I'm not sure how much parallelization will really help us.
> 
> (*) with "selected" I meant "selected by the author of the PR". If I add
> a new feature and have new tests for that, I need a way to trigger those
> tests for that PR. The mechanism for that would have to be defined and
> implemented. Perhaps some annotations in the PR description which gets
> exported to bitbake and thus refkit-ci.inc?
> 
CI build sees PR description text in it's env, so this seems to be doable,
but adds need to invent some format and parsing of it.
Another channel is Github PR labels, also passed into build.
Managing labels set for that will be needed then as extra effort.

Olev

> Any other opinions?
> 
---------------------------------------------------------------------
Intel Finland Oy
Registered Address: PL 281, 00181 Helsinki 
Business Identity Code: 0357606 - 4 
Domiciled in Helsinki 

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.