[poky] [PATCH 0/1][RFC] Optimize file parsing speed

[Xu, Dongxiao]

Richard Purdie wrote:
> Hi Dongxiao,
> 
> On Mon, 2010-11-22 at 14:04 +0800, Dongxiao Xu wrote:
>> I found that when parsing bitbake files, most of the variables in
>> d.keys() are in distro_tracking_fields.inc, and they are not used in
>> normal build. 
>> 
>> This pull request moves some distro related functions in
>> utility-tasks.bbclass into distrodata.bbclass, and remove the
>> inclusion of distro_tracking_fields.inc from poky.conf. This could
>> gain about 25% parsing time saving.
> 
> I'm going to take the patch but I'd like to be clear where the speed
> gains come from with this change. I suspect some are due to a smaller
> number of keys but I also suspect the smaller number of tasks
> involved helps too!   

Hi Richard,

I did a profiling for the parsing time w/ and w/o the patch.

Here a piece of the profiling result:


W/ PATCH:

Mon Nov 29 13:23:42 2010    profile.log

         32756805 function calls (31268348 primitive calls) in 60.195 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  3572737    4.950    0.000   12.849    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:285(getVarFlag)
  3917977    4.464    0.000    4.464    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:191(_findVar)
873129/431998    3.040    0.000   16.824    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:86(expandWithRefs)
  1745340    3.007    0.000    4.555    0.000 /usr/lib/python2.6/copy.py:65(copy)
   286785    2.570    0.000   20.051    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:271(build_dependencies)
   565710    2.532    0.000    5.015    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/COW.py:97(__getitem__)
      917    2.394    0.003   27.726    0.030 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:299(generate_dependencies)
   288439    2.350    0.000   11.607    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:212(setVar)
1428066/1144003    1.882    0.000   16.762    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:246(getVar)
...
     2389    0.004    0.000    2.528    0.001 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:267(update_data)


W/O PATCH:

Mon Nov 29 13:28:36 2010    profile.log

         49110091 function calls (47618793 primitive calls) in 75.338 CPU seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  7547331    8.230    0.000   19.467    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:285(getVarFlag)
  7908236    8.113    0.000    8.113    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:191(_findVar)
      917    3.987    0.004   43.045    0.047 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:299(generate_dependencies)
  5061763    3.812    0.000    5.633    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:301(<genexpr>)
    84656    3.318    0.000   13.475    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:302(<genexpr>)
897804/450300    2.838    0.000   15.801    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data_smart.py:86(expandWithRefs)
   294083    2.777    0.000   18.854    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:271(build_dependencies)
  1783941    2.740    0.000    4.154    0.000 /usr/lib/python2.6/copy.py:65(copy)
   592115    2.432    0.000    5.179    0.000 /sda1/yocto/scripts/..//bitbake/lib/bb/COW.py:97(__getitem__)
...
     2389    0.004    0.000    2.656    0.001 /sda1/yocto/scripts/..//bitbake/lib/bb/data.py:267(update_data)



>From the profiling result we can see that, generate_dependencies() time reduces from 43 seconds to 27 seconds, while build_dependencies() mostly keeps unchanged (From 20 seconds to 18.8 seconds). Therefore the biggest overhead reduced by the patch should be the two lines of code to parsing keys in generate_dependencies() function.

> 
> Also, a lot of those keys are override keys so perhaps its speeding up
> update_data() calls and some of the gain is from there too?

Update_data() has some gains but not much, see the above profile result.

Thanks,
Dongxiao

> 
> Anyhow, its a good move :)
> 
> Cheers,
> 
> Richard