[yocto] 1.3 M3 Full Pass test results

[Serban, Laurentiu]

Hello guys,

I am sorry I did not notice the last line in the e-mail.

When we ran the performance testing we pre-downloaded all the sources, because last week had a lot of connectivity issues. I mentioned in the e-mail that the build time refers to an image built after all the sources were downloaded.
The machine on which the test was ran is the same as for M1.

I will ask Stefan to re-run the performance tests for the different milestones so far, M1 and M2, in the same conditions as for M3 so we would have a clear view.

I add Jiajun in the loop so he can help us if he can with a test to see if this is related to an artifact as Richard said, but also there were some improvements made by Beth on filesystem generation.

Br,
Laurentiu

-----Original Message-----
From: Liu, Song 
Sent: Monday, August 27, 2012 8:50 PM
To: Serban, Laurentiu; Purdie, Richard
Cc: yocto at yoctoproject.org; Stewart, David C; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Hi Laurentiu, 

Do you have any comment on the performance question Richard asked? I know your team is using a machine with different configuration from the one used by the Shanghai team. The performance figure from the Shanghai team has been hovering around 110 mins. That's the case for 1.2 release and 1.3 M2 milestone report. 1.3 M1 milestone report has the build time as 95 minutes, which I believe is from your team. So my question is whether you used the same machine for M3 performance testing as for M1. Another factor that might have caused the difference (between 95 and 83 minutes) is your testing procedure/environment such as other processes running at the same time, memory usage, sstate cache, etc. If you used the same machine and same testing procedure/environment, then we have some improvement in M3 compared with M1. Please let me know your thoughts.

Thanks,
Song

-----Original Message-----
From: Serban, Laurentiu
Sent: Friday, August 24, 2012 6:34 AM
To: Purdie, Richard
Cc: yocto at yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: RE: 1.3 M3 Full Pass test results

Hello Richard,

Even if the installer is used in the default mode, issues still occur (see comment 7). I think the root cause for these is the same, so I did not submit a new bug.

Thank you,
Laurentiu 

-----Original Message-----
From: Purdie, Richard
Sent: Friday, August 24, 2012 1:22 PM
To: Serban, Laurentiu
Cc: yocto at yoctoproject.org; Liu, Song; Stewart, David C; Wold, Saul
Subject: Re: 1.3 M3 Full Pass test results

On Thu, 2012-08-23 at 20:13 +0100, Serban, Laurentiu wrote:

> Here are the results for the full pas tests on 1.3 M3 RC2. The commit 
> used for testing is 8b8748c8f963900b83dc0fdd7757556f917fe4fd.
>
> Some details about the encountered issues below:
>  
> BSP – Sudoku-savant project build issue (2878)
>
> ADT – the relocatable sdk issue (2980) causes 13 test cases to be on 
> faile/blocked state

I thought it worked as long as you didn't have to relocate it so no tests should have been blocked, we just have the relocation issue?

>  , also the Clutter C template issue is unsolved (2577)
>
> Core Build System – x32 is still an issue (2888), cleaning sstate 
> issue is still not solved (2897), incremental RPM image generation 
> (2969), source archiving (2619), the kvm issue was reproduced by 
> another colleague (2790) Yocto BSP creation via JSON (2693) or for 
> qemu (2991) fails, multilib issue (2918 – this requires a little more 
> investigation from QA),
>
> HOB - all seems ok for RC2
>
> Self-hosted-image  - cannot start on Virtual Box (X issue), it is very 
> slow on qemu and it has a m4 package build (3005) issue on VMWare. If 
> the self-hosted-image is used on machine with internet connectivity 
> via proxy there will be an initial sanity check failure, but this is 
> not a blocking issue.
>  
> A mention for the performance testing: on a Ubbuntu 12.04  i7 machine 
> using 8 threads the build time was 83 minutes (with prior fetching).

How does this compare with our other performance numbers. From what I remember, we used to hover around the 105-115 minute mark. Did we have some significant speed gains or is this just an artefact of changing the test machine?

Cheers,

Richard