[poky] [PATCH 0/1][RFC] Optimize file parsing speed

[Xu, Dongxiao]

Richard Purdie wrote:
> On Mon, 2010-11-29 at 13:45 +0800, Xu, Dongxiao wrote:
>> Richard Purdie wrote:
>>> I'm going to take the patch but I'd like to be clear where the speed
>>> gains come from with this change. I suspect some are due to a
>>> smaller number of keys but I also suspect the smaller number of
>>> tasks involved helps too!
>> 
>> I did a profiling for the parsing time w/ and w/o the patch.
>> 
>> Here a piece of the profiling result:
>> 
>> W/ PATCH:
>> 
>> Mon Nov 29 13:23:42 2010    profile.log
>> 
>>          32756805 function calls (31268348 primitive calls) in 60.195
>> CPU seconds
>> 
>>    Ordered by: internal time
>> 
>>    ncalls  tottime  percall  cumtime  percall
>>   filename:lineno(function) 3572737    4.950    0.000   12.849   
>>   0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:285(getVarFlag)
>>   3917977    4.464    0.000    4.464    0.000
>>   
>>   
>>      
>>   
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:191(_findVar)
>>      873129/431998    3.040    0.000   16.824    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:86(expandWithRefs)
>> 1745340    3.007    0.000    4.555    0.000
>> /usr/lib/python2.6/copy.py:65(copy) 286785    2.570    0.000  
>> 20.051    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:271(build_dependencies)
>> 565710    2.532    0.000    5.015    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/COW.py:97(__getitem__) 917   
>> 2.394    0.003   27.726    0.030
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:299(generate_dependencies)
>> 288439    2.350    0.000   11.607    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:212(setVar)
>> 1428066/1144003    1.882    0.000   16.762    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:246(getVar) ...
>> 2389    0.004    0.000    2.528    0.001
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:267(update_data)     
>> 
>> 
>> W/O PATCH:
>> 
>> Mon Nov 29 13:28:36 2010    profile.log
>> 
>>          49110091 function calls (47618793 primitive calls) in 75.338
>> CPU seconds
>> 
>>    Ordered by: internal time
>> 
>>    ncalls  tottime  percall  cumtime  percall
>>   filename:lineno(function) 7547331    8.230    0.000   19.467   
>>   0.000
>>      
>>  
>>    
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:285(getVarFlag)
>>    7908236    8.113    0.000    8.113    0.000
>>   /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:191(_findVar)
>>    917    3.987    0.004   43.045    0.047
>>     
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:299(generate_dependencies)
>> 5061763    3.812    0.000    5.633    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:301(<genexpr>) 84656 
>> 3.318    0.000   13.475    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:302(<genexpr>)
>> 897804/450300    2.838    0.000   15.801    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:86(expandWithRefs)
>> 294083    2.777    0.000   18.854    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:271(build_dependencies)
>> 1783941    2.740    0.000    4.154    0.000
>> /usr/lib/python2.6/copy.py:65(copy) 592115    2.432    0.000   
>> 5.179    0.000
>> /sda1/yocto/scripts/..//bitbake/lib/bb/COW.py:97(__getitem__) ...
>> 2389    0.004    0.000    2.656    0.001
>> /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:267(update_data)     
>> 
>> From the profiling result we can see that, generate_dependencies()
>> time reduces from 43 seconds to 27 seconds, while
>> build_dependencies() 
>> mostly keeps unchanged (From 20 seconds to 18.8 seconds). Therefore
>> the biggest overhead reduced by the patch should be the two lines of
>> code to parsing keys in generate_dependencies() function.
>> 
>>> 
>>> Also, a lot of those keys are override keys so perhaps its speeding
>>> up update_data() calls and some of the gain is from there too?
>> 
>> Update_data() has some gains but not much, see the above profile
>> result. 
> 
> Looks good, thanks.
> 
> Interestingly looking at the profile overall, we dropped from 49
> million function calls to 32 million function calls which is always a
> good way to speed things up.  
> 
> getVarFlag and _findVar each dropped by 3 million calls each (which
> is from the construction of the keys() list). 
> 
> So a very valid speedup :). I still think we might be able to speed
> this area up further though such as if we directly keep an index of
> exported variables.  

Hi Richard,

Like the whole d.keys(), exported variables are also different among
recipes, since recipe may export or unset certain variables for itself.
So to implement this, we need to keep a common list, and then handle
recipe specific export/unset variables.

Does my understanding correct? 

Thanks,
Dongxiao



> 
> Cheers,
> 
> Richard