[poky] SSTATE builds more broken than I thought

[Tian, Kevin]

>From: Richard Purdie
>Sent: Monday, November 15, 2010 1:58 AM
>To: Gary Thomas
>Cc: Poky
>Subject: Re: [poky] SSTATE builds more broken than I thought
>
>On Fri, 2010-11-12 at 16:55 -0700, Gary Thomas wrote:
>> On 11/12/2010 04:31 PM, Richard Purdie wrote:
>> > On Fri, 2010-11-12 at 01:35 -0700, Gary Thomas wrote:
>> >> On 11/12/2010 01:30 AM, Richard Purdie wrote:
>> >>> On Fri, 2010-11-12 at 01:18 -0700, Gary Thomas wrote:
>> >>>>> Are these all files or are they symlinks? Are they just present for
>> >>>>> deploy-ipk or are there others?
>> >>>>
>> >>>> They are files.
>> >>>>
>> >>>> Sorry, but I don't understand your other question.  These are the only
>> >>>> extra files polluting the tree, except for Python *.py? object files
>> >>>> which are also always created.
>> >>>
>> >>> Are there any files related to other sstate tasks such as
>> >>> populate-sysroot*.tgz or package*.tgz or is there just *deploy-ipk*.tgz?
>> >>>
>> >>> In other words I'm wondering if this is something specific to the
>> >>> deploy-ipk sstate task...
>> >>
>> >> The only extra files created are *deploy-ipk*.tgz
>> >> This most certainly is restricted to the deploy-ipk sstate task and only
>> >> happens if SSTATE_MIRRORS is defined.
>> >
>> > This should be fixed in ae98f7eacb9e61fe086d88dc694b4c651af9fee3, it
>> > turns out to be an issue with the local fetcher not searching DL_DIR.
>>
>> Trying the same process as in bug #526, it failed with this error:
>[...]
>
>Right, I messed up those commits. I should know better than make commits
>when travelling and jetlagged.
>
>I ended up reverting the first fixes and pushing a set of different
>fixes.
>
>Previously you (and I think Kevin Tian) reporting bitbake spending a
>huge amount of time thinking about shared state. When I was debugging
>this I noticed a potential cause of that, my local test setups were
>evidently just too fast to notice it. I added another commit:

how did your local test exactly setup?

>
>http://git.pokylinux.org/cgit.cgi/poky/commit/?id=89929e1f283c8508c505c9731ad93388
>0abf22a1
>
>which should make things faster (n^2 faster where n is the number of
>tasks).

I did a quick test just now, the result is not that promising. The total time
of sstate setscene took ~8min this time, however there're also less tasks
executed:

Previous run: ~25min, sstate-setscene starts from 133 of 1673 tasks
Current run: ~8min, sstate_setscene starts from 1283 of 1673 tasks

>From the build process, there did have more recipes rebuilt from scratch
instead of taking prebuilt this time, such as gcc, etc. The build even didn't
finish, with C preprocessor sanity check failure on gcc-runtime.

This may not be a good test though, as I'm still using the prebuilts generated
from last week (at d9ff2f897aa271e6b2d7e4dcaf8b8d19de513b50), while
the 2nd build uses latest ddbf5e9c48afdeefeaec120a02d43536f5fd1ce1 plus
my siggen fix.

I'll redo a slow test by having both builds from the same commit, while keeping
original env for further analysis later.

Thanks
Kevin