[yocto] RFC: automated runtime testing on real hardware

[Darren Hart]

On Tue, 2013-08-27 at 12:11 +0100, Paul Eggleton wrote:
> Hi Darren,
> 
> On Friday 23 August 2013 20:52:41 Darren Hart wrote:
> > On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote:
> > > Automated testing on real hardware is something that a lot of us do
> > > already, but the likelihood is that we've all been coming up with our own
> > > solutions for this, so it would be really great if we could have a single
> > > working solution to cover the generally applicable use cases. As a start
> > > though it would be interesting to hear from people who have existing test
> > > setups or who are generally interested in this area. I'd like to hear
> > > feedback on the following:
> > >  * What hardware setup do you use for automated testing?
> > 
> > I am starting to put something together using a customized 19" rack
> > along the lines of those used by OSADL for their real-time farm:
> > 
> > https://www.osadl.org/Test-Rack.test-rack.0.html
> > 
> > This is a poor man's BladeCenter for embedded boards in that it provides
> > a common interface to each board for the rack infrastructure. The board
> > specific stuff sits on the trays themselves. The trays are easily
> > removable to take a board to a desk for hands-on work.
> > 
> > A smart-switch, web-power strip, and serial concentrator feed into a
> > "gateway server" where services like conmux glue it all together.
> 
> All makes sense.
> 
> > >  * Do you use any software to manage the hardware? Does this include any
> > > existing open source tools?
> > 
> > conmux
> > dnsmasq
> > udev (for static usb uart naming)
> > 
> > The testrack has some early-stage tools to support it, but I'm still
> > sorting those out and they aren't particularly robust for initial setup
> > quite yet.
> 
> Having a standard way to set up the software side for this kind of 
> infrastructure would certainly be worthwhile I think. One tricky part would 
> perhaps be supporting variations of infrastructure hardware; doing some 
> searches I haven't found a completely definitive open source project for 
> controlling remotely accessible power strips for example, although there are a 
> few scattered projects for doing that with varying levels of recent 
> maintenance.
> 
> > >  * What would you like to see in a common framework to enable this kind of
> > > testing?
> > 
> > To simplify the board management you allude to above, I believe a pull
> > model, rather than a push, is a better architecture.
> > 
> > Your builders can complete builds and inject test records into a
> > database with a priority. The boards sit in a
> > boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and
> > push the results back to the database. They select tests based on
> > priority, and the builder doesn't have to do anything at all to manage
> > the boards. This also decouples the builder from the test
> > infrastructure.
> 
> Interesting. This would present some challenges with the way the automated 
> tests are currently defined; there's nothing unsolvable that precludes them 
> being run outside of the build environment but they will have to be run 
> somewhere that has python and any required modules installed, presumably an 
> intermediate machine rather than the target machine itself (the latter would 
> not be impossible; but if we did do it that way you couldn't test an image 
> that didn't have python in it.)

A new test-image will need to be created regardless, I think including
python is reasonable for the general case. For things like tiny, I
suspect we would have a customized test-suite anyway, possibly written
in C.

> 
> We'd also need to come up with a standard definition for the results database 
> and tools to generate reports out of it, but that's something that will likely 
> be needed regardless of the model used for running tests, especially when it 
> comes to recording the results from ptest.

Agreed.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel