[yocto] QA cycle report for 2.7 M3 RC1

[richard.purdie at linuxfoundation.org]

Hi Sangeeta/Ee Peng,

On Tue, 2019-04-02 at 05:02 +0000, Jain, Sangeeta wrote:
> QA cycle report for 2.7 M3 RC1:
>  
> No high milestone defects. 
> This is completion of first QA cycle after adoption of new QA
> process. Test results are available at following location:
> ·        For results of all automated tests, please refer to results
> at public AB [1].
> ·        For other test results, refer to attachment [2].
> ·        For test report for test cases run by Intel and WR team,
> refer attachment [3]
> ·        For full test report, refer attachment [4]
> 2 new defects are found in this cycle, oe-selftest [5] and qemu-
> shutdown[6].
> Number of existing issues observed in this release is 3- toaster [7],
> Build-appliance [8] and Beaglebone [9]
> For ptest, 3 package tests are failing in this release which were
> passing in previous release: python [10], python3[11] and strace[12].
> 3 packages are facing timeout issues: lttng-tools [13], openssh [14]
> and python3 [15].

Thanks for this. I have some questions/observations.

Looking at the Intel/WR report, I see tests which start oeselftest_XXX
for sdk-extras and qemu-shutdown. I think this means you have some QA
test infrastructure running "oe-selftest -r runqemu,meta_ide",
iterating over MACHINE=qemuppc, qemumips, qemuarm, qemux86?

Is this some automation we're missing in the autobuilder configuration?
We all other tests covered and this is the only remaining automated
piece we haven't covered? What configuration are we missing exactly for
the missing tests?

I'm partly wondering how you found the shutdown issue but this would
explain that.

I think going forward we need to add some kind of "who" identifier to
the test results so we can say where/who they came from (main
autobuilder, Intel, WR, anywhere else). We could add a resulttool
command to "stamp" a set of test results with this, maybe as a
parameter to the store command?

Of the bugs, I didn't see any which I think should block M3, I think we
probably should release that and move forward to M4. I do want to see
some of these fixed before we build M4 though. I've made comments on
them below.
 
> New Bugs :
>  
> [5] Bug 13237 - [2.7 M3 RC1][OE-selftest][Opensuseleap 15.0][Public
> AB]:"test_wic_cp_ext" failing for Opensuseleap 15.0.
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13237

Ross: Are you looking into that one?

> [6] Bug 13234 - [2.7 M3 RC1] qemumips & qemumips64: failed to
> shutdown
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13234

This needs looking into and fixing, as well as figuring out why the
autobuilder doesn't see this (see above for my hypothesis).

> Previous Bugs observed in this release:
>  
> [7] Bug 13191 – [Test Case 1439] Build Failure after adding or
> removing packages with and without dependencies
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13191
>  
> [8] Bug 12991 - [Bug] [2.6 M4 RC1][Build-Appliance] Bitbake build-
> appliance-image getting failed during building image due to webkitgtk
> package.
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991

We need to do something about this. I believe its related to resource
starvation in the VM.

> [9] Bug 13153 – [2.7 M2 RC1] Systemtap doesn't work on beaglebone
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13153

I'm hoping a SRCREV bump for beaglebone-yocto fixes this.

> ptest Bugs:
>  
> [10] Bug 13249 - [2.7 M3 RC1] python ptest failure
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13249

I'm hoping a patch from Ross in master fixes this, need to retest ptest
with this applied.
 
> [11] Bug 13253 - [2.7 M3 RC1] python3 ptest failure
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13253

I'm hoping a patch from Ross in master fixes this, need to retest ptest
with this applied.

> [12] Bug 13254 - [2.7 M3 RC1] strace ptest failure
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13254

No ideas on this one yet.
 
> [13] Bug 13255 - [2.7 M3 rc1] lttng-tools ptest facing timeout issue
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13255

There are patches which are due to arrive soon which should address
this.

> [14] Bug 13256 - [2.7 M3 rc1] openssh ptest facing timeout issue
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13256

No ideas on this one yet.

> [15] Bug 13257 - [2.7 M3 rc1] python3 ptest facing timeout issue
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13257

I'm hoping a patch from Ross in master fixes this, need to retest ptest
with this applied.

Cheers,

Richard