[Toaster] announcing the toaster test framework

[Damian, Alexandru]

On Wed, Apr 29, 2015 at 10:27 AM, Barros Pena, Belen <
belen.barros.pena at intel.com> wrote:

>
>
> On 28/04/2015 20:54, "Michael Wood" <michael.g.wood at intel.com> wrote:
>
> >On 27/04/15 18:22, Damian, Alexandru wrote:
> >> Hi,
> >>
> >> I put together a couple of scripts that will monitor the toaster
> >> mailing list, and provide smoke testing on each patchset submitted to
> >> review request.
> >>
> >> A very rough form of these scripts existed for some time, but I've
> >> taken time to clean that up, and enough code to be able to add tests
> >> very easily, and make everything a bit more reliable.
> >>
> >> It lives outside of the poky/ or bitbake/ trees, and you can check out
> >> the code here, until moved to a proper repo.
> >>
> >>
> >>
> >>
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=yocto-tts/m
> >>aster
> >>
> >>
> >> Currently, my installation is emailing tests results only to me, but
> >> once I've been satisfied that it doesn't crash or burn the mailing
> >> list, I will update the setup to mail results directly to the mailing
> >> list.
> >>
> >>
> >> You can also use it to manually run tests on your branch prior to
> >> submission.
> >>
> >
> >I know this wasn't an RFC but I do have one question... you're
> >encoraging people to write tests for this new framework, but when are we
> >saying a test be sent to the "toaster test framework" and when should it
> >be a unit test in Toaster?
> >
> >...My thoughts are that if it's possible to write the unit test in
> >toaster with django[1] (i.e no external tools) then that should be the
> >primary place for tests, we should be in the habbit of running
> >./manage.py test pre-submission of patches. Those tests are broken or
> >missing at the moment, so I hope we can fix those first.
>
> So, we have unit tests in django that developers can run before submitting
> their patches. We also have the stuff from Alex, which will run tests on
> patches submitted to the mailing list. Both things seem useful, but I
> wonder: is there any way we could integrate them into a coherent workflow?
> Could we centralise all tests in django, for example, and run them on the
> patches landing on the mailing list just in case developers didn't or
> didn't catch a failed result?
>
>
​It's not as bad as you describe, it's a bit worse. There are already two
other places containing Toaster testing: one developed under the
oe-selftest framework, targeted at verifying data collection, etc. and the
UI automated testing using Selenium.

What I'm intending to do is make sure that TTS runs its own tests, Django
unit tests, oe-selftest tests , and the still-pending UI automated tests in
an automated manner just to make sure we have these tests run. It is an
addition to the developer's own testing and to the QA flow, and not a
replacement.



> I am not sure this would make sense, but just in case it does ...
>
> Cheers
>
> Belén
>
>
> >
> >What do you think?
> >
> >[1] e.g. some I wrote for the error-report-web:
> >http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/tree/Post/test
> .
> >py
> >
> >
> >
> >> Cheers,
> >> Alex
> >>
> >> --
> >> Alex Damian
> >> Yocto Project
> >> SSG / OTC
> >>
> >>
> >
> >--
> >_______________________________________________
> >toaster mailing list
> >toaster at yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/toaster
>
> --
> _______________________________________________
> toaster mailing list
> toaster at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20150501/4dfa7a02/attachment.html>