[Toaster] Need help with Production toaster setup

Damian, Alexandru alexandru.damian at intel.com
Wed Jul 9 05:07:53 PDT 2014


Oh, I think I have something - it happens on mysql, but not on sqlite. This
is important :)

I'll dive into it.

Alex


On Wed, Jul 9, 2014 at 12:54 PM, Damian, Alexandru <
alexandru.damian at intel.com> wrote:

> Hi Craig - I'd like to have a chat with you, are you on IRC ?
>
> Unfortunately, I am unable to reproduce the errors you're seeing.
> I created the database at commit:
> *a5531a2b8983318b99c119a87b78a9**2cf84160b8*
>
> and started the bitbake server and toasterui at commit:
> *5306aaab07a7a9b3b0c7e8008dabbd**2e89daec28*
>
> and the build gets recorded without error.
>
> I'm thinking you have *.pyc files remaining on disk between git checkouts,
> python not picking up changes, and getting old code running - but it may be
> any cause.
>
> I'd like to debug this live with you online, do you have time for that ?
>
> Alex
>
>
>
>
>
>
> On Tue, Jul 8, 2014 at 3:40 PM, Neth, Craig Allen <craig.neth at verizon.com>
> wrote:
>
>> Hi Alex,
>>
>> Ok, thanks.   Bug 6512 was filed.   Do you have any speculation about
>> when the problem might have been introduced?   I don’t have any particular
>> need to run the bleeding edge toaster server; if there is an earlier branch
>> that might work I’m happy to switch branches on the server end of things.
>>
>> The weird thing is that I tried reverting the server back to the Daisy
>> branch but that also failed with a different set of problems, I received
>> errors when I ran the ‘migrate orm’ step.   I figured that I had just fat
>> fingered some of the setup commands but I can’t seem to find the problem.
>>
>> Craig
>>
>> From: <Damian>, Alexandru <alexandru.damian at intel.com>
>> Date: Tuesday, July 8, 2014 at 7:09 AM
>> To: "Neth, Craig A" <craig.neth at one.verizon.com>
>> Cc: "Lerner, Dave" <dave.lerner at windriver.com>, "toaster at yoctoproject.org"
>> <toaster at yoctoproject.org>, IonutX Chisanovici <
>> ionutx.chisanovici at intel.com>
>>
>> Subject: Re: [Toaster] Need help with Production toaster setup
>>
>> Hi,
>>
>> Thanks for the info -
>>
>> No, the setup seems as it is designed to be.
>>
>> I guess somewhere in origin/master things broke. We don't test this setup
>> on our suite, but maybe we should start testing this, thus I CC'ed Ionut.
>>
>> Meanwhile, I will start looking into fixing this. Can you please open a
>> bug in Bugzilla so I can track it ?
>>
>> Cheers,
>> Alex
>>
>>
>>
>> On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <craig.neth at verizon.com
>> > wrote:
>>
>>> Hi Alex,
>>>
>>> Thanks for the response!
>>>
>>> I just did a git pull on the master, and then did the following steps on
>>> the master machine:
>>>
>>> mysql> drop database toaster;
>>> mysql> create database toaster;
>>>
>>> python bitbake/lib/toaster/manage.py syncdb
>>> python bitbake/lib/toaster/manage.py migrate orm
>>> python bitbake/lib/toaster/manage.py migrate bldcontrol
>>>
>>> (The wiki page does not mention this last one but the ‘syncdb' command
>>> mentions that this one is also 'not synced’.)
>>>
>>> To start the master, I have written a short upstart script that does the
>>> following:
>>>
>>> DATABASE_URL='mysql://user:pass@node:3306/toaster' python
>>> $BASEDIR/bitbake/lib/toaster/manage.py runserver
>>>
>>> The server is up and running (I see the "This is Toaster" page).
>>>
>>> On the clients, I normally use a modified version of the ‘toaster’ script
>>> (I called it toaster-remote) that avoids starting the local webserver but
>>> doing the commands listed on the wiki page doesn't change the outcome,  I
>>> still see the same Assertion errors I mentioned in my initial message.
>>>
>>> As to your questions:  We are indeed using mysql on the master.   Git
>>> commit ids:
>>>
>>> master:a5531a2b8983318b99c119a87b78a92cf84160b8
>>> client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28
>>>
>>> I mentioned the commands I used above.
>>>
>>> Thanks for any help.   As I mentioned, we did have all this working
>>> previously before I updated the master, so I don’t think my fundamental
>>> setup is wrong...
>>>
>>> Craig
>>>
>>>
>>>
>>>
>>> From:  <Damian>, Alexandru <alexandru.damian at intel.com>
>>> Date:  Friday, July 4, 2014 at 8:06 AM
>>> To:  "Lerner, Dave" <dave.lerner at windriver.com>
>>> Cc:  "Neth, Craig A" <craig.neth at one.verizon.com>,
>>> "toaster at yoctoproject.org" <toaster at yoctoproject.org>
>>> Subject:  Re: [Toaster] Need help with Production toaster setup
>>>
>>>
>>> >Hi Craig,
>>> >
>>> >
>>> >The system should work with the server running master, and the clients
>>> >running daisy. The production database MUST be set up with the master,
>>> >and the clients MUST NOT be started with the toaster script - the script
>>> >will get confused as to the current revision of the database. The flow
>>> is:
>>> >
>>> >
>>> >- start toaster on the server, and let it create / update the database
>>> >
>>> >- on the clients, you need to just start the bitbake server and the
>>> >toasterui client before issuing command to the bitbake server - here
>>> >there is the race condition that Dave already mentioned
>>> >
>>> >
>>> >
>>> >If you need to drop and recreate the database, please always use the
>>> >latest master - the clients, even if using an older version, should
>>> >accomodate, we try keep the database schema changes backward compatible.
>>> >
>>> >
>>> >
>>> >
>>> >For a short while, the "master" was broken due to a bad commit - you may
>>> >have landed there with your master, can you please git pull lastest
>>> >master ?
>>> >
>>> >
>>> >
>>> >
>>> >About the specific bug you see - can you please let me know more about
>>> >your setup ?
>>> >
>>> >
>>> >- db engine you're using is mysql, right ?
>>> >
>>> >
>>> >- commit ids for master and daisy that you're using ?
>>> >
>>> >- which commands do you use to start the server and the clients ?
>>> >
>>> >
>>> >
>>> >Cheers,
>>> >Alex
>>> >
>>> >
>>> >
>>> >
>>> >On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <
>>> dave.lerner at windriver.com>
>>> >wrote:
>>> >
>>> >Hi Craig,
>>> >
>>> >Did you delete your old database and recreate it using the current clone
>>> >of poky as shown on
>>> >
>>> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
>>> >that is, running the database creation/migration commands that execute
>>> in
>>> >        'source toaster start'?
>>> >
>>> >-dave
>>> >
>>> >> -----Original Message-----
>>> >> From: Neth, Craig Allen [mailto:craig.neth at verizon.com]
>>> >> Sent: Thursday, July 03, 2014 4:05 PM
>>> >> To: Lerner, Dave
>>> >> Cc: toaster at yoctoproject.org
>>> >> Subject: Re: Need help with Production toaster setup
>>> >>
>>> >> Hi Dave,
>>> >>
>>> >> Thanks for the quick response!
>>> >>
>>> >> I had previous experienced the problem you mentioned and worked around
>>> >>it
>>> >> by adding a sleep between the —observe-only and the bitbake command.
>>> >> Without the sleep, as you mention the events were just getting
>>> dropped.
>>> >> It’s nice to see a proper fix for that, however.
>>> >>
>>> >> But I don’t think that’s what’s happening now; from the tracebacks it
>>> >> looks like the clients are sending events to the database but it’s not
>>> >> setup the way they expect...
>>> >>
>>> >> My suspicion is that the schemas have changed but I don’t know how to
>>> >>get
>>> >> my database back into a state where it matches the schema the client
>>> >> expects.
>>> >>
>>> >> Craig
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com> wrote:
>>> >>
>>> >> >Hi Craig,
>>> >> >
>>> >> >A patch submitted 15 minutes ago may solve your problem, it sounds
>>> >> >similar to the problem that appeared after daisy in my script that
>>> did:
>>> >> >
>>> >> >bitbake --server-only ...
>>> >> >bitbake --observe-only ... ToasterUI
>>> >> >bitbake <some-image>
>>> >> >
>>> >> >With recent releases the 'bitbake <some-image>' fires before the
>>> >> >ToasterUI is ready (2nd command) to read events.  The patch in
>>> >> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a
>>> message
>>> >> >to the log file.  You can change your script to block until that log
>>> >> >message appears,  before launching a build - if that's your problem.
>>> >> >
>>> >> >Hope this helps,
>>> >> >Dave Lerner
>>> >> >
>>> >> >> -----Original Message-----
>>> >> >> From: toaster-bounces at yoctoproject.org
>>> >> >>[mailto:toaster-bounces at yoctoproject.org] On
>>> >> >> Behalf Of Neth, Craig Allen
>>> >> >> Sent: Thursday, July 03, 2014 3:34 PM
>>> >> >> To: toaster at yoctoproject.org
>>> >> >> Subject: [Toaster] Need help with Production toaster setup
>>> >> >>
>>> >> >> A month or so ago, I set up a production toaster instance using the
>>> >>poky
>>> >> >> master branch.
>>> >> >> Our projects were building with Daisy.   Everything was working
>>> fine.
>>> >> >>
>>> >> >> For various reasons, I needed to redo the production instance and
>>> >>that
>>> >> >> involved recloning the poky master repo, and now I¹m having issuesŠ
>>> >> >>
>>> >> >> After recloning poky and restarting the server, all my builds with
>>> >>the
>>> >> >> build logging turned on started throwing weird errors complaining
>>> >>like
>>> >> >> this:
>>> >> >>
>>> >> >> ERROR: Field 'project_id' doesn't have a default value
>>> >> >> <traceback omitted>
>>> >> >>
>>> >> >>
>>> >> >> And :
>>> >> >>
>>> >> >> ERROR:
>>> >> >> Traceback (most recent call last):
>>> >> >>   File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
>>> >> >>     buildinfohelper.store_started_task(event)
>>> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
>>> >> >> store_started_task
>>> >> >>     recipe_information =
>>> >> >> self._get_recipe_information_from_taskfile(event.taskfile)
>>> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
>>> >> >> _get_recipe_information_from_taskfile
>>> >> >>     layer_version_obj =
>>> >>self._get_layer_version_for_path(localfilepath)
>>> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
>>> >> >> _get_layer_version_for_path
>>> >> >>     assert 'build' in self.internal_state
>>> >> >> AssertionError
>>> >> >>
>>> >> >> So I figured that there was probably some difference of opinion
>>> >>between
>>> >> >> the client and the server about the schema; I tried running the
>>> >>Œsyncdb¹
>>> >> >> and Œmigrate orm¹ commands again on the server but that didn¹t
>>> help.
>>> >> >>
>>> >> >>
>>> >> >> ---
>>> >> >>
>>> >> >> So next I figured I would just delete the Œtoaster¹ database and
>>> >>start
>>> >> >> over.   I tried going into mysql and dropping the toaster database,
>>> >> >> recreating it, and then redoing the two python commands, but that
>>> >> >>results
>>> >> >> in this error from the migrate orm command:
>>> >> >>
>>> >> >> python bitbake/lib/toaster/manage.py migrate orm
>>> >> >> Running migrations for orm:
>>> >> >>  - Migrating forwards to 0007_auto__add_helptext.
>>> >> >>  > orm:0001_initial
>>> >> >>  > orm:0002_auto__add_field_build_timespent
>>> >> >>  > orm:0003_timespent
>>> >> >>  - Migration 'orm:0003_timespent' is marked for no-dry-run.
>>> >> >>  > orm:0004_auto__add_field_package_installed_name
>>> >> >>  >
>>> >> >>
>>> >>
>>>
>>> >>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
>>> >>>>le
>>> >> >>hi
>>> >> >> stor
>>> >> >> FATAL ERROR - The following SQL query failed: ALTER TABLE
>>> `orm_build`
>>> >> >>ADD
>>> >> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
>>> >> >> The error was: (1060, "Duplicate column name 'timespent'")
>>> >> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and
>>> >>its
>>> >> >> values cannot be restored.
>>> >> >>
>>> >> >> This was all done while working on the daisy branch.
>>> >> >>
>>> >> >> So, then I tried the delete/recreate database trick again and
>>> changed
>>> >> >>back
>>> >> >> to the master branch.  Now I can run the two python commands ok,
>>> but
>>> >>my
>>> >> >> daisy clients are back to throwing the assert errors...
>>> >> >>
>>> >> >> So, a few questions:
>>> >> >>
>>> >> >>
>>> >> >> - Does it still work to run the production server on the master and
>>> >> >> clients running daisy, or do I have to use the daisy version of the
>>> >> >> production server, or do I need something newer than daisy but not
>>> as
>>> >> >>new
>>> >> >> as master?
>>> >> >>
>>> >> >> - If for whatever reason I need to recreate the database, what are
>>> >>the
>>> >> >> exact steps that I need to do to delete/recreate?
>>> >> >> It seems like just dropping the Œtoaster¹ database and recreating
>>> it
>>> >>is
>>> >> >> insufficient, but I am not sure what else to do.
>>> >> >>
>>> >> >> The wiki pages are a bit vague on all this stuff, thanks for any
>>> >>help.
>>> >> >>
>>> >> >> Craig
>>> >> >>
>>> >> >> --
>>> >> >> _______________________________________________
>>> >> >> toaster mailing list
>>> >> >> toaster at yoctoproject.org
>>> >> >> https://lists.yoctoproject.org/listinfo/toaster
>>> >
>>> >--
>>> >_______________________________________________
>>> >toaster mailing list
>>> >toaster at yoctoproject.org
>>> >https://lists.yoctoproject.org/listinfo/toaster
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >--
>>> >Alex DamianYocto Project
>>> >
>>> >SSG / OTC
>>>
>>>
>>
>>
>> --
>> Alex Damian
>> Yocto Project
>> SSG / OTC
>>
>>
>
>
> --
> Alex Damian
> Yocto Project
> SSG / OTC
>



-- 
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140709/813c8ae1/attachment-0001.html>


More information about the toaster mailing list