[Yocto-builds] buildbot failure in The Yocto Autobuilder on nightly-qa-systemd

[Stanacar, StefanX]

On Wed, 2013-07-31 at 08:36 +0000, yocto-builds at yoctoproject.org wrote:

>         
>       * DEST : /srv/www/vhosts/autobuilder.yoctoproject.org/pub/nightly/20130731-1
>       * Contents of auto.conf (local.conf for autobuilders)
>         DISTRO = "poky"
>         PACKAGE_CLASSES = "package_rpm package_deb package_ipk"
>         BB_NUMBER_THREADS = "16"
>         PARALLEL_MAKE = "-j 16"
>         SDKMACHINE ?= "i686"
>         DL_DIR =
>         "/srv/www/vhosts/autobuilder.yoctoproject.org/current_sources"
>         SSTATE_DIR ?= "/srv/www/vhosts/sstate.yoctoproject.org//" 
>         MACHINE = "qemux86-64"
>         PREMIRRORS = ""
>         BB_GENERATE_MIRROR_TARBALLS = "1"
>         ADTREPO = "http://adtrepo.yoctoproject.org//1.4+snapshot"
>         DISTRO_FEATURES_append = " systemd"
>         VIRTUAL-RUNTIME_init_manager = "systemd"
>         DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
>         

Just as a note: the nightly-qa-target has 3 BuildImages steps.
We are building systemd+sysvinit (meaning systemd is the init-manager),
sysvinit+systemd and systemd only images, and the above conf is valid
only for the  last (systemd only) image built.


> Detailed log of last build step:
> http://autobuilder.yoctoproject.org:8011/builders/nightly-qa-systemd/builds/108/steps/Running Sanity Tests/logs/stdio 


> 
> | NOTE: rootfs
> file: /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/core-image-sato-qemux86-64-testimage.ext3
> | NOTE: Qemu
> logfile: /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130731081922
> | NOTE: runqemu started, pid is 14557
> | NOTE: waiting at most 60 seconds for qemu pid
> | No children found matching 14557
> | NOTE: qemu started - qemu procces pid is 14612
> | NOTE: Waiting at most 500 seconds for login banner
> | NOTE: Reached login banner
> | NOTE: Couldn't determine ip, all I got was:
> | ip addr show eth0 | sed -n '3p' | awk '{ print $2 }' | cut -f 
> | ip: can't find device 'eth0'
> | root at qemux86-64:~#
> | DEBUG: Python function do_testimage finished

There is patch on ML for this one - systemd likes to rename interfaces.
Until recently it couldn't do that (dmesg test failed before with -
"systemd-udevd[79]: error changing net interface name eth0 to enp0s3:
Device or resource busy"). It seems something changed recently and it
can.


> Detailed log of last build step:
> http://autobuilder.yoctoproject.org:8011/builders/nightly-qa-systemd/builds/108/steps/Building Images_2/logs/stdio 
> 
> Last 60 lines of "Building Images_2.stdio" Error log:
> | Note: configuring RPM cross-install scriptlet_wrapper
> | Note: to be installed: packagegroup-core-ssh-dropbear at all
> packagegroup-core-x11-sato-games at qemux86_64
> packagegroup-core-x11-base at all smartpm at x86_64 psplash at x86_64
> packagegroup-core-boot at qemux86_64
> packagegroup-base-extended at qemux86_64 rpm at x86_64
> packagegroup-core-x11-sato at qemux86_64 locale-base-en-us at x86_64
> locale-base-en-gb at x86_64
> | Loading cache...
> | Updating cache... ######################################## [100%]
> | 
> | Computing transaction...error: Can't install
> packagegroup-core-boot-1.0-r11 at qemux86_64: no package provides
> busybox-hwclock
> | 
> | Saving cache...
> | 
> | ERROR: Function failed: do_rootfs (log file is located
> at /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp/log.do_rootfs.28362)
> ERROR: Task 7
> (/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/meta/recipes-sato/images/core-image-sato.bb, do_rootfs) failed with exit code '1'


This is the systemd only image (and the fail isn't related to
testimage :)), the last one built and because it failed to build we get
the next fail:


> Detailed log of last build step:
> http://autobuilder.yoctoproject.org:8011/builders/nightly-qa-systemd/builds/108/steps/Running Sanity Tests_2/logs/stdio 
> 
> Last 60 lines of "Running Sanity Tests_2.stdio" Error log:
> | NOTE: DISPLAY value: :1
> | NOTE: rootfs
> file: /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/core-image-sato-qemux86-64-testimage.ext3
> | NOTE: Qemu
> logfile: /srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/build/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/testimage/qemu_boot_log.20130731083413
> | NOTE: runqemu started, pid is 21609
> | NOTE: waiting at most 60 seconds for qemu pid
> | No children found matching 21609
> | NOTE: qemu started - qemu procces pid is 21670
> | NOTE: Waiting at most 500 seconds for login banner
> | NOTE: Reached login banner
> | NOTE: Ip found: 192.168.7.2
> | NOTE: Test modules ['oeqa.runtime.ping', 'oeqa.runtime.ssh',
> 'oeqa.runtime.connman', 'oeqa.runtime.df', 'oeqa.runtime.rpm',
> 'oeqa.runtime.smart', 'oeqa.runtime.xorg', 'oeqa.runtime.syslog',
> 'oeqa.runtime.dmesg', 'oeqa.runtime.systemd']
> | NOTE: Found 19 tests
> | test_ping (oeqa.runtime.ping.PingTest) ... ok
> | test_ssh (oeqa.runtime.ssh.SshTest) ... ok
> | test_connmand_help (oeqa.runtime.connman.ConnmanTest) ... ok
> | test_connmand_running (oeqa.runtime.connman.ConnmanTest) ... ok
> | test_connmand_unique (oeqa.runtime.connman.ConnmanTest) ... ok
> | test_df (oeqa.runtime.df.DfTest) ... ok
> | test_rpm_help (oeqa.runtime.rpm.RpmHelpTest) ... ok
> | test_rpm_query (oeqa.runtime.rpm.RpmQueryTest) ... ok
> | test_smart_help (oeqa.runtime.smart.SmartHelpTest) ... ok
> | test_smart_info (oeqa.runtime.smart.SmartQueryTest) ... ok
> | test_smart_query (oeqa.runtime.smart.SmartQueryTest) ... ok
> | test_xorg_error (oeqa.runtime.xorg.XorgTest) ... ok
> | test_xorg_running (oeqa.runtime.xorg.XorgTest) ... ok
> | test_syslog_help (oeqa.runtime.syslog.SyslogTest) ... ok
> | test_syslogd_running (oeqa.runtime.syslog.SyslogTest) ... ok
> | test_syslogd_configurable (oeqa.runtime.syslog.SyslogTestConfig) ...
> FAIL
> | test_dmesg (oeqa.runtime.dmesg.DmesgTest) ... ok
> | test_systemd_version (oeqa.runtime.systemd.SystemdBasicTest) ...
> FAIL
> | test_systemd_failed (oeqa.runtime.systemd.SystemdTests) ... skipped
> 'Testcase dependency not met: test_systemd_version'
> | 
> |
> ======================================================================
> | FAIL: test_syslogd_configurable
> (oeqa.runtime.syslog.SyslogTestConfig)
> |
> ----------------------------------------------------------------------
> | Traceback (most recent call last):
> | File
> "/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/meta/lib/oeqa/utils/decorators.py", line 48, in wrapped_f
> | f(*args)
> | File
> "/srv/home/pokybuild/yocto-autobuilder-new/yocto-slave/nightly-qa-systemd/build/meta/lib/oeqa/runtime/syslog.py", line 32, in test_syslogd_configurable
> | self.assertEqual(status, 0, msg="Could not restart syslog service.
> Status and output: %s and %s" % (status,output))
> | AssertionError: Could not restart syslog service. Status and output:
> 127 and sh: systemctl: command not found
> | 
> |

And here I have a problem and I don't know what to do about it.
The test themselves are correct in trying to use systemctl (remember
this is the last image, the systemd one) but because the image failed
to build it actually found in the deploy dir the previous image built
(which was sysvinit+systemd)... because the task looks for
IMAGE_LINK_NAME. It can't use IMAGE_NAME (the full image name with the
timestamp) because the timestamp will be different by the time the task
starts. 
I can see two approaches in solving it:
  - quick fix on AB side: do a -c clean target (or perhaps clean the
entire build dir?) before every build (the artifacts aren't published
for this target so it can't hurt)
  - on the task side: - depend on do_rootfs but that means we'll have to
steps failing on the same thing on the AB (and see the same do rootfs
error twice...)
I don't like either. Ideas? Help? :)

Cheers,
Stefan