[Toaster] Jenkins interviews summary

[Barros Pena, Belen]

Between May 5th and May 11th we ran 4 interviews with engineers who have
experience using Jenkins as their CI solution. They complemented an
additional conversation that happened back in March. None of the people
interviewed has experience with Toaster, although at least two of them saw
the demo of Toaster at Intel's Yocto Summit in April.

The contents and conclusions of those 5 conversations are summarised
below. I will start by the conclusions: if you want to understand those a
bit better, you can read the interviews summary provided after them.

CONCLUSIONS

* Toaster could provide a way to handle the 'artifacts' problem: it could
become the build artifact access point instead of a custom application or
Jenkins itself. For this to work well, we would need to provide an easy
way to specify which artifacts should be collected, and where they should
be stored, in a way that avoids the need for plugins or custom scripts,
and that includes the option of selecting sstate objects as artifacts to
be kept. We would also need to improve how Toaster presents rootfs files
and other artifacts for download, which is not great at the moment.

* The fact that people recommend keeping Jenkins 'configuration agnostic'
as much as possible means that generating configuration in Toaster and
passing it onto Jenkins is not a crazy idea. This is effectively what the
python and shell scripts mentioned to us seem to be doing. Instead of
writing a script, you would generate configuration instructions using
Toaster. 

* Toaster could be particularly useful in contexts where developers must
run builds in remote infrastructure. An existing Toaster project with a
custom layer added could make the flow as easy as: push my changes to the
layer, log in to Toaster, press 'run again' on a previous build for the
desired image. 

* Toaster will provide the added incentive of collecting and displaying
extensive information about the builds ran through Jenkins. None of the
interviewees were familiar with that Toaster feature, but we have
anecdotal evidence from previous interviews that suggests it is useful and
valued by Yocto Project users.

INTERVIEWS SUMMARY

1. Storing build artifacts

Everybody must somehow solve the issue of collecting and saving build
artifacts generated by Jenkins. Here is a good explanation of the problem
taken from 
https://wiki.jenkins-ci.org/display/JENKINS/ArtifactDeployer+Plugin

"By default, Jenkins archives artifacts generated by the build. These
artifacts are stored in the JENKINS_HOME directory with all other elements
such as job configuration files. There is no separation between
infrastructure elements, project elements and outputs. It is often
considered to be a bad practice and it doesn't help to manage JENKINS_HOME
directory from an administrator point of view (backup issues, etc). And
for now, we can't change the 'Archived artifacts' location."

Or, as someone put it to us, "Jenkins by itself does not create a share
drive where you can go and grab stuff."

People handle the artifacts problem in different ways:

* 2 of the people we spoke to tell Jenkins to upload artifacts to a custom
tool, which provides its own web interface that users can access and
download the artifacts from. One person repurposed the error reporting
tool for that. 

* Another person saves the artifacts to the master Jenkins server with the
help of a Jenkins plugin. Developers access the Jenkins UI and download
artifacts from it (Jenkins can be tied to a LDAP server, so authentication
becomes straightforward). This is a bit more convoluted than it sounds,
however. Here is the explanation of the process:

"Our shell scripts will make a directory called 'artifacts' at the top
level of the Jenkins workspace. Then we move all the different files that
BitBake generates and that we care about into that 'artifacts' directory.
Then, at the Jenkins job level, we have the artifact archiver plugin set
up so that everything under the artifacts directory gets stored with the
job as artifacts, and that goes into the Jenkins master server and the
lifetime of those is tied to the lifetime of that build in Jenkins, and
there is a lot of policy you can set around that. You can say keep the
last 10, or keep 2 weeks, and so on. But if you delete the job through the
Jenkins interface the artifacts also get deleted."

Artifact collection might involve more than rootfs files: people also need
to collect logs and in one case, sstate files as well, which are uploaded
to an http server so that they can be used by anybody running builds.

Using plugins, however, comes with a maintenance overhead: there can be
compatibility issues between versions of Jenkins, and they must be tested
to make sure they work together and with Jenkins when upgrading to a new
version. 

2. Keeping Jenkins simple

3 out of the 5 people we spoke to recommended keeping the configuration in
Jenkins itself as simple as possible. One of them warned against the huge
number of plugins, and the ability to create very complex Jenkins projects
(he showed us how you can nest Jenkins projects within other Jenkins
projects). Another recommended including the build configuration required
in the Git repo Jenkins is building (in this case, the build scripts were
stored in a git submodule in a main git "super module"), which makes the
maintenance of Jenkins jobs much easier (Jenkins itself is not great at
facilitating configuration management for a large number of jobs).

Most people used some kind of script to provide to Jenkins the required
build configuration: one used Python scripts for a relatively simple
Android build; people doing Yocto Project builds used shell scripts.

To pass configuration from Toaster to Jenkins people recommended:

* The Jenkins REST API
* The Jenkins CLI
* Use some kind of plugin architecture in Toaster, so that people can
write support for their own CI solutions
* Export some kind of configuration file that Jenkins can read and apply
* Somehow expose the Jenkins UI configuration options within Toaster

3. Software developers run builds

People brought up the fact that, in certain contexts, software developers
must build images reproducing the configuration used by their distro team
(for example, when the hardware they work with doesn't have a writable
file system). In such cases, building locally becomes difficult due to the
build size, hence giving them access to remote build servers via Jenkins.

Making the building workflow as easy as possible for their developers was
an important consideration, and 2 of the people we spoke to used Jenkins
to facilitate the build process for developers: for example, developers
would push their changes to a Git repository, and then would access
Jenkins and tell it to build that repository.

The nice web UI and the fact that Jenkins provides visibility on the state
of the builds make developers' lives easier.

4. Should we integrate Toaster with Jenkins?

When asked what they thought about us integrating Toaster with Jenkins,
most people didn't see the point. Some thought that if they wanted to
interact remotely with Jenkins, they could just use curl. Others that you
can just use a script to do anything you want.

Only one person thought they should be integrated, mainly because of the
configuration ease provided by Toaster. In his own words: "configuring
your builds with Toaster looks much easier than doing it in command line".

If you have any questions about the above, please let me know.

Cheers

Belén