[Toaster] [PATCH 0/5] Fix task buildstats gathering

[Barros Pena, Belen]

On 19/02/2016 15:49, "toaster-bounces at yoctoproject.org on behalf of Elliot
Smith" <toaster-bounces at yoctoproject.org on behalf of
elliot.smith at intel.com> wrote:

>Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken,
>partly because of changes to how the data is organised in the buildstats
>files, and partly because tasks aren't being updated correctly anyway.
>
>Clean up the code which collects the buildstats (it had lots of
>duplication), collect the stats from the new stat file format (some line
>leaders had changed), and correctly identify the Task objects to update
>in BuildInfoHelper.
>
>To test:
>
>1. Remove the content of the directories which cache tasks to see the
>best effects (I usually delete buildstats, tmp, sstate-cache, and cache,
>just to be safe).
>
>2. Run a build against a local Yocto (as one patch is for
>toaster.bbclass).
>
>3. When the build finishes, view the "Tasks" page for the build and
>enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage
>percentages are > 100, as discussed in bug 8842.
>
>Changes since c0fe43b (toaster-next) are in
>git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842
>http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/bu
>ild_data-8842

So I ran a build following the steps above, and had a look at the data.
Information now shows, which is definitely an improvement :)

* Time and Disk I/O show for all executed tasks, which is what's supposed
to happen.
* CPU usage is missing from some executed tasks. This used to happen
before as well, but we never actually worked out why.

Regarding the numbers, I am not sure how useful is the Disk I/O figure
expressed in bytes. Should we convert it to something else? And then, our
big problem is definitely the CPU usage, which shows pretty crazy numbers.
The highest one in my build was for linux-yocto do_compile_kernelmodules
at a whopping 2455.15%

So, Richard Purdie pointed out to us that % over 100 are related to task
parallelism. And in fact, if I divide the CPU usage value of the compile
tasks by the value of PARALLEL_MAKE (36), I do get percentages below 100
for all of them. In the example of linux-yocto do_compile_kernelmodules,
we get 68.20%

If I divide the CPU usage value of all the install tasks by the value of
PARALLEL_MAKEINST (36), the same happens: % below 100.

However, we do get % over 100 for tasks that we have been told have no
parallelism at all. I see such values for unpack, patch, configure,
package and populate_sysroot tasks. So, the question is, why are those
happening? Because for tasks that we know have parallelism we might be
able to divide the value by the parallelism set, as I did for compile and
install tasks. But for the others, I genuinely have no answer, other that
there must be some kind of bug in the data collection.

Thoughts?

Cheers

Belén

>
>Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842
>
>NOTE: The first commit in this patch series is for oe-core.
>
>Elliot Smith (5):
>  toaster.bbclass: improve how we gather buildstats for Toaster
>  toaster: elapsed time for a task is calculated in toaster.bbclass
>  toaster: when updating task stats, task must exist
>  toaster: use a more straightforward query to find tasks to update
>  toaster: clarify the units used for buildstats
>
> bitbake/lib/bb/ui/buildinfohelper.py    | 49 +++++++--------------
> bitbake/lib/toaster/orm/models.py       |  7 +++
> bitbake/lib/toaster/toastergui/views.py |  4 +-
> meta/classes/toaster.bbclass            | 76
>++++++++++++++++++++++-----------
> 4 files changed, 77 insertions(+), 59 deletions(-)
>
>--
>Elliot Smith
>Software Engineer
>Intel OTC
>
>---------------------------------------------------------------------
>Intel Corporation (UK) Limited
>Registered No. 1134945 (England)
>Registered Office: Pipers Way, Swindon SN3 1RJ
>VAT No: 860 2173 47
>
>This e-mail and any attachments may contain confidential material for
>the sole use of the intended recipient(s). Any review or distribution
>by others is strictly prohibited. If you are not the intended
>recipient, please contact the sender and delete all copies.
>
>-- 
>_______________________________________________
>toaster mailing list
>toaster at yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster