[poky] SSTATE builds more broken than I thought

[Tian, Kevin]

well, a typo in siggen.py causes below confusion, which I sent a pull request at:
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=tk/master&id=e0df54e800a811c355a748308218edb33f963e9a

After clearing that out, the symptom of 31 cases could be refined as:

- all 4 sstate tasks vary (populate_sysroot, deploy_ipk, deploy_rpm, and package), and
all vary on do_install taskhash. do_install taskhash is same for 4 tasks in same build

- populate_sysroot has one more difference on basehash

- both basehash and taskhash differences don't list explicit variable differences

- it's always reproducible. I use bzip2 as the example. In original build providing sstate
prebuilts, bzip2 clean/rebuild always takes the desired prebuilts. In 2nd build using
prebuilts, bzip2 clean/rebuild always start from scratch with same checksum cross
the builds (by manually removing new prebuilts generated by itself)

- their native equivalents DO reuse prebuilts, such as bzip2-native

I may need have more debug into the execution process to catch the difference, but
I only have time back on this tomorrow. So if anyone else could reproduce it
and take some look, it'd be great. If we could make rest 31 recipes working, that
would really boost build performance, since several big time-consuming recipes fall
into this remaining group like kernel, webkit-gtk, etc.

Thanks
Kevin

>From: Tian, Kevin
>Sent: Monday, November 15, 2010 9:36 AM
>
>>From: Tian, Kevin
>>Sent: Monday, November 15, 2010 9:14 AM
>>
>>I haven't got time looking into all of them. For random 3 (bzip2, gnome-vfs, webkit-gtk),
>>they have the same symptom on populate_sysroot, e.g:
>>
>>basehash changed from c7c1d0c7ea052eef6f6665b8959b44fc to
>>3eb4a010cbfb3b15df49d433f6593b0c
>>Variable BB_TASKHASH value changed from 9e077b25e1e2c50d4f8a09a9356a6a6c to
>>0d6608b425d8a59b21f5874b4914b412
>>Hash for dependent task
>>/home/poky/tasks/poky/meta/recipes-extended/bzip2/bzip2_1.0.5.bb.do_install
>changed
>>from c4cba89fa491ecb0916a56f1e75c5f17 to c4e7f58c385e384749cc20ca1bd8fccf
>>
>>* basehash varies while no variable is explicitly listed by bitbake-diffsigs. I expected
>>to see the variable causing difference to be marked explicitly here.
>>* taskhash varies due to do_install, however bitbake-diffsigs doesn't list how do_install
>>checksum is calculated. I made a quick comparison between run.do_install scripts, with
>>major difference still on expanded variables. There may still have some whitelist work
>>to be done.
>>
>>I used same source tree for two builds, which share same sstate-cache directory.
>>
>
>Another interesting thing. When I compare other prebuilt tasks, all the rest tasks
>show the difference on task dependencies of do_install, which then results in the
>taskhash difference. Take 'package' for example:
>
>---
>
>Tasks this task depends on changed from set
>
>(['GLIBCTARGETOS', 'TARGET_VENDOR', 'FILE', 'PACKAGES', 'PR', 'PV',
>'package_do_package', 'DEPLOY_DIR', 'PF', 'PE', 'PN', 'package_rpm_do_package',
>'MULTIMACH_TARGET_SYS', 'D', 'TARGET_ARCH', 'TARGET_OS', 'TMPDIR', 'PKGD',
>'WORKDIR', 'PACKAGEFUNCS', 'EXTENDPE', 'MULTIMACH_ARCH',
>'PACKAGE_PREPROCESS_FUNCS'])
>
>to set
>
>(['GLIBCTARGETOS', 'TARGET_VENDOR', 'FILE', 'PACKAGES', 'PR', 'PV',
>'package_do_package', 'DEPLOY_DIR', 'PF', 'PE', 'PN', 'package_rpm_do_package',
>'MULTIMACH_TARGET_SYS', 'D', 'TARGET_ARCH', 'TARGET_OS', 'TMPDIR', 'PKGD',
>'WORKDIR', 'PACKAGEFUNCS', 'EXTENDPE', 'MULTIMACH_ARCH',
>'PACKAGE_PREPROCESS_FUNCS'])
>
>Hash for dependent task
>/home/poky/tasks/poky/meta/recipes-extended/bzip2/bzip2_1.0.5.bb.do_install changed
>from c4cba89fa491ecb0916a56f1e75c5f17 to c4e7f58c385e384749cc20ca1bd8fccf
>
>---
>
>First, I carefully checked above dependency list, which show NO difference at all.
>
>Second, you can see the taskhash values actually match my earlier post about
>populate_sysroot prebuilts. However at that place no dep info is shown except
>the hashvalue change.
>
>I'll try more to see whether it's consistent.
>
>Thanks
>Kevin