[poky] [PATCH 0/1] [PULL] bitbake: optimize the file parsing speed

[Dongxiao Xu]

Hi Richard and Saul,

This patch is to optimize the file parsing speed. Please help to review
and pull. Thanks!

>From the profiling result, we found that generate_dependencies()
occupies a lot of time where the following code will be run when
parsing each bb file.

keys = set(key for key in d.keys() if not key.startswith("__"))
shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))

To optimize the code, main thought is to build the data cache for
generate_dependencies() on hand, and later each time when parsing the
bb file, we do not need to build them again and again.

Here I used "bitbake -p -P" to measure the file parsing time.

Each time before testing, we will do "touch conf/local.conf" to ensure
that all the 846 files are re-parsed.

Here are the test results:

Before optimization: 
Round 1: 74.980s  (deprecated)
Round 2: 60.281s
Round 3: 59.824s
Round 4: 60.771s
--------------------
Average: 60.292s


After optimization:
Round 1: 45.003s  (deprecated)
Round 2: 33.063s
Round 3: 32.991s
Round 4: 32.043s
--------------------
Average: 32.699s

For both cases, the first time result is a big higher than later, I think
it is due to cold cache, so here I deprecated it.

For the rest three rounds, I calculated the average time, and we can see
the optimization could gain ~50% parsing time.


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dxu4/perf
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf

Thanks,
    Dongxiao Xu <dongxiao.xu at intel.com>
---


Dongxiao Xu (1):
  bitbake: optimize file parsing speed

 bitbake/lib/bb/cooker.py |    2 ++
 bitbake/lib/bb/data.py   |   11 ++++++-----
 2 files changed, 8 insertions(+), 5 deletions(-)