[poky] update about SSTATE status

[Tian, Kevin]

>From: Richard Purdie [mailto:rpurdie at linux.intel.com]
>Sent: Monday, November 22, 2010 2:17 AM
>
>Hi Kevin,
>
>On Sun, 2010-11-21 at 21:02 +0800, Tian, Kevin wrote:
>> Recently I took some time to try out SSTATE prebuilt feature, and now I think it's time
>> to summarize issues I've seen and also some enhancements I've made. All the changes
>> are currently in poky-contrib:tk/sstate branch, which are still under test and on a slightly
>> old commit (at Nov.13).
>> http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=tk/sstate
>>
>> I use the sstate_cache generated from the 1st build as the SSTATE_DIR for the 2nd
>> build, and then check the SSTATE use status in 2nd build. Most issues below I think
>> apply to a MIRROR mode too.
>
>This is great analysis, thanks!

I'd like to suggest "bitbake -S" which is really great to figure out the very cause for the
signature difference. It did take me quite some time to dump debug info before using it.
Thanks Richard to create this neat option from the start. :-)

>
>> I've observed SSTATE broken in below scenarios:
>>
>> a) ~31 recipes fail to reuse sstate packages when the 1st build and the 2nd build are
>> continuously build, when using same source tree
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000174.html
>>
>> b) more recipes fail if the 2nd build was started in next day after when the 1st build
>> was generated. This is related to DATE/TIME variables referenced by some tasks.
>
>I'd be interested to see which recipes are doing this...

It's referred by patch_do_patch, which is one example:
                        if srcdate == "now":
                                srcdate = bb.data.getVar('DATE', d, 1)

Other tasks like do_distro_check refer to DATETIME, which is however not counted in
normal task dependency and thus should be OK.

Frankly speaking I think this DATE issue should cause wider problem than what I have
observed, since do_patch task is invoked for many recipes. Perhaps it's due to my
messed build directories when tackling multiple issues in parallel, and thus my memory 
is not accurate. But anyway the DATE variable does need filtered out. :-)

>
>> c) My build machine suffered from a power down, and then all STATE packages can't
>> be reused after reboot. Comparison between two builds show that some shell
>> variables change, e.g: GNOME_KEYRING_SOCKET, SSH_AUTH_SOCK, etc.
>
>We should whitelist the ones we don't mind if they change.

I'm just afraid that this set is volatile to various host systems, and thus maintain such a
list would be a headache. That's why I'm trying to adopt a way to define all environmental
variables coming from shell instead and then whitelist all of them out.

>
>> d) no STATE package from the 1st build could be reused when the two builds use
>> different source trees, though two source trees are identical with only difference on
>> the path. This is related to OEROOT
>
>I was sure I tested this and the checksums matched :/. Which variables
>are changing to cause this - we really need to track them down...

Sorry that I don't have such info from existing debug log. There I just recorded dep
variables used in checksum calculation and also the task data itself. However I didn't 
see direct invocation on OEROOT. I'll figure it out later.

>
>> e) If there're changes on libc/gcc packages between the 1st build and the 2nd build,
>> the 2nd build will always fail, either on gcc-runtime or eglibc, or other libc/gcc
>> bootstrap packages. The reason is that sstate_installation skips -initial/-intermediate
>> packages which breaks bootstrap process.
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
>>
>> The major problem for above issues is that many undesired variables which are
>> bound to build environment are counted into signature generation. Bitbake takes
>> a list of environmental variables from the shell and then convert into internal
>> variables. We need to separate those from bitbake internal variables. So I split
>> BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE into two groups:
>>
>> 	BB_ENV_WHITELIST_BB/BB_ENV_EXTRAWHITE_BB
>> 	BB_ENV_WHITELIST_SHELL/BB_ENV_EXTRAWHITE_SHELL
>>
>> The 1st group covers variables defined internally in bitbake, while the other group
>> points to the true external environmental variables. This way we can always wipe
>> the _SHELL group from the dependency chain, while use BB_BASEHASH_WHITELIST
>> to control bitbake variables:
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=8e1b6a292b3
>c796c5c0273de76d2e83271c83e26
>>
>> Above looks somewhat ugly, but it should improve current situation a lot. We
>> do need a better way to mark such variable attribute efficiently though.
>
>I'm afraid I don't like this solution, its too complicated. Why don't we
>just whitelist the variables we are happy shouldn't influence the
>checksums even if they are present in the environment?

All variables coming from shell (but not bitbake defined variables) belong to that category,
imho, which is why I want to mark those variables automatically.

If you think the simple whitelist is acceptable, I can do that. It leaves two issues though:
   o We need sync any change on preserved shell variables into this whitelist manually
   o User is allowed to use BB_ENV_EXTRAWHITE to preserve more shell variables, which
     are dynamic

But perhaps it's fine to use simple whitelist as a quick workaround for now.

>
>To be honest there are several of these we should only export to
>"interactive" tasks anyway so perhaps marking some as unexport and then
>exporting them in devshell and the interactive patch handler would be a
>better approach?

Do you have an example list of them? I'm still unclear when those shell variables
are used, and thus need time to understand them.

>
>> In the meantime, there're other two enhancements which are also helpful:
>>
>>   - update-alternative expands an absolute path when appending to do_install,
>> which impacts signature:
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=a09df20442b
>cfc033b955992948e2dc2f6cbb0b4
>
>Yes, this looks correct, nice catch!
>
>>   - BBCLASSEXTEND is not covered by BB_TASKHASH_WHITELIST, i.e
>> "virtual:native:" and "virtual:nativesdk:":
>>
>http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/commit/?h=tk/sstate&id=07697ff94ff1
>829ef16effb48c73a2312af6a2ff
>
>Again, good catch!
>
>I've merged these two.
>
>> The remaining thing in the list:
>>
>> - more tests on my branch
>> - rebase to latest master
>> - address incremental build failure when libc/gcc recipes are changed. Tom Rini
>> gave a suggestion to use a similar 'deptask' as in previous stage. I need more
>> thought on that and not sure whether it can solve it completely. If there's no clean
>> way, we may wait until libc/gcc bootstrap process is changed (sysroot to different
>> places) which is a 1.0 task, and then suggest people to always wipe out related
>> sstate packages when seeing those errors
>
>Poky's sstate code workd very differently to packaged staging, I don't
>think dependencies can help us in the sstate case. We perhaps just need
>to improve the logic somehow. I need to think about the problem in more
>detail to come up with better direction suggestions.
>

OK. 

Thanks
Kevin