[Toaster] [PATCH 0/2][v8] Fix task buildstats gathering

[Elliot Smith]

v8:

Remove tests (defer until we have Selenium UI framework sorted).

Fix issues found during review.

v7:

Changes by others necessitated a rebase.

v6:

Fix comments on function which converts started and ended time to a datetime
(values are seconds, not milliseconds).

Modify the task page:

- Show the split CPU times (User and System).
- Format Disk IO as done on the tasks page (with commas).
- Disk IO value is now in bytes, so modify heading.

v5:

Fixed an error where the started and ended times for a task were treated as being
in milliseconds, when they are in seconds.

v4:

I messed up the patch submission, so I'm sending the correct patches this time.

v3:

Only show a single navigation item in the left-hand nav panel for CPU times,
but show both System and User time columns when it is selected.

Add a test for CPU time subpage.

v2:

Store "started" and "ended" times for each task. (This will enable other
improvements later, such as showing when tasks overlap each other.)
Compute elapsed_time from start and end times, rather than in
toaster.bbclass.

Store disk IO in two extra fields, one for read and the other for write.
Compute total disk_io from these two values.

(elapsed_time and disk_io are still stored, as these are used for sorting
and we can't easily sort on a computed field. This can be amended when we
get round to putting the tasks table into ToasterTable; see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.)

Modify the UI to show the CPU times split between system and user.

v1:

Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway.

Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper.

To test:

1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe).

2. Run a build against a local Yocto (as one patch is for toaster.bbclass).

3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842.

Note: one patch in this set needs to go to oe-core.

Changes since c5194851573c91c0aab8fe4c549086037daf2370 (toaster-next) are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842

Related bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842

Elliot Smith (2):
  toaster.bbclass: improve how we gather buildstats for Toaster
  toaster: rework task buildstats storage and display

 bitbake/lib/bb/ui/buildinfohelper.py               | 72 ++++++++++------------
 .../orm/migrations/0005_task_field_separation.py   | 48 +++++++++++++++
 bitbake/lib/toaster/orm/models.py                  | 18 +++++-
 .../toastergui/templates/basebuildpage.html        |  4 +-
 bitbake/lib/toaster/toastergui/templates/task.html | 23 ++++---
 .../lib/toaster/toastergui/templates/tasks.html    | 20 +++---
 bitbake/lib/toaster/toastergui/urls.py             |  2 +-
 bitbake/lib/toaster/toastergui/views.py            | 55 +++++++++++------
 meta/classes/toaster.bbclass                       | 70 +++++++++++++--------
 9 files changed, 208 insertions(+), 104 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.