[Toaster] Need help with Production toaster setup
Neth, Craig Allen
craig.neth at verizon.com
Tue Jul 8 07:40:57 PDT 2014
Hi Alex,
Ok, thanks. Bug 6512 was filed. Do you have any speculation about when the problem might have been introduced? I don’t have any particular need to run the bleeding edge toaster server; if there is an earlier branch that might work I’m happy to switch branches on the server end of things.
The weird thing is that I tried reverting the server back to the Daisy branch but that also failed with a different set of problems, I received errors when I ran the ‘migrate orm’ step. I figured that I had just fat fingered some of the setup commands but I can’t seem to find the problem.
Craig
From: <Damian>, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>>
Date: Tuesday, July 8, 2014 at 7:09 AM
To: "Neth, Craig A" <craig.neth at one.verizon.com<mailto:craig.neth at one.verizon.com>>
Cc: "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>, "toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>" <toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>>, IonutX Chisanovici <ionutx.chisanovici at intel.com<mailto:ionutx.chisanovici at intel.com>>
Subject: Re: [Toaster] Need help with Production toaster setup
Hi,
Thanks for the info -
No, the setup seems as it is designed to be.
I guess somewhere in origin/master things broke. We don't test this setup on our suite, but maybe we should start testing this, thus I CC'ed Ionut.
Meanwhile, I will start looking into fixing this. Can you please open a bug in Bugzilla so I can track it ?
Cheers,
Alex
On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <craig.neth at verizon.com<mailto:craig.neth at verizon.com>> wrote:
Hi Alex,
Thanks for the response!
I just did a git pull on the master, and then did the following steps on
the master machine:
mysql> drop database toaster;
mysql> create database toaster;
python bitbake/lib/toaster/manage.py syncdb
python bitbake/lib/toaster/manage.py migrate orm
python bitbake/lib/toaster/manage.py migrate bldcontrol
(The wiki page does not mention this last one but the ‘syncdb' command
mentions that this one is also 'not synced’.)
To start the master, I have written a short upstart script that does the
following:
DATABASE_URL='mysql://user:pass@node:3306/toaster' python
$BASEDIR/bitbake/lib/toaster/manage.py runserver
The server is up and running (I see the "This is Toaster" page).
On the clients, I normally use a modified version of the ‘toaster’ script
(I called it toaster-remote) that avoids starting the local webserver but
doing the commands listed on the wiki page doesn't change the outcome, I
still see the same Assertion errors I mentioned in my initial message.
As to your questions: We are indeed using mysql on the master. Git
commit ids:
master:a5531a2b8983318b99c119a87b78a92cf84160b8
client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28
I mentioned the commands I used above.
Thanks for any help. As I mentioned, we did have all this working
previously before I updated the master, so I don’t think my fundamental
setup is wrong...
Craig
From: <Damian>, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>>
Date: Friday, July 4, 2014 at 8:06 AM
To: "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>
Cc: "Neth, Craig A" <craig.neth at one.verizon.com<mailto:craig.neth at one.verizon.com>>,
"toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>" <toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>>
Subject: Re: [Toaster] Need help with Production toaster setup
>Hi Craig,
>
>
>The system should work with the server running master, and the clients
>running daisy. The production database MUST be set up with the master,
>and the clients MUST NOT be started with the toaster script - the script
>will get confused as to the current revision of the database. The flow is:
>
>
>- start toaster on the server, and let it create / update the database
>
>- on the clients, you need to just start the bitbake server and the
>toasterui client before issuing command to the bitbake server - here
>there is the race condition that Dave already mentioned
>
>
>
>If you need to drop and recreate the database, please always use the
>latest master - the clients, even if using an older version, should
>accomodate, we try keep the database schema changes backward compatible.
>
>
>
>
>For a short while, the "master" was broken due to a bad commit - you may
>have landed there with your master, can you please git pull lastest
>master ?
>
>
>
>
>About the specific bug you see - can you please let me know more about
>your setup ?
>
>
>- db engine you're using is mysql, right ?
>
>
>- commit ids for master and daisy that you're using ?
>
>- which commands do you use to start the server and the clients ?
>
>
>
>Cheers,
>Alex
>
>
>
>
>On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>
>wrote:
>
>Hi Craig,
>
>Did you delete your old database and recreate it using the current clone
>of poky as shown on
> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
>that is, running the database creation/migration commands that execute in
> 'source toaster start'?
>
>-dave
>
>> -----Original Message-----
>> From: Neth, Craig Allen [mailto:craig.neth at verizon.com<mailto:craig.neth at verizon.com>]
>> Sent: Thursday, July 03, 2014 4:05 PM
>> To: Lerner, Dave
>> Cc: toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> Subject: Re: Need help with Production toaster setup
>>
>> Hi Dave,
>>
>> Thanks for the quick response!
>>
>> I had previous experienced the problem you mentioned and worked around
>>it
>> by adding a sleep between the —observe-only and the bitbake command.
>> Without the sleep, as you mention the events were just getting dropped.
>> It’s nice to see a proper fix for that, however.
>>
>> But I don’t think that’s what’s happening now; from the tracebacks it
>> looks like the clients are sending events to the database but it’s not
>> setup the way they expect...
>>
>> My suspicion is that the schemas have changed but I don’t know how to
>>get
>> my database back into a state where it matches the schema the client
>> expects.
>>
>> Craig
>>
>>
>>
>>
>> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>> wrote:
>>
>> >Hi Craig,
>> >
>> >A patch submitted 15 minutes ago may solve your problem, it sounds
>> >similar to the problem that appeared after daisy in my script that did:
>> >
>> >bitbake --server-only ...
>> >bitbake --observe-only ... ToasterUI
>> >bitbake <some-image>
>> >
>> >With recent releases the 'bitbake <some-image>' fires before the
>> >ToasterUI is ready (2nd command) to read events. The patch in
>> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a message
>> >to the log file. You can change your script to block until that log
>> >message appears, before launching a build - if that's your problem.
>> >
>> >Hope this helps,
>> >Dave Lerner
>> >
>> >> -----Original Message-----
>> >> From: toaster-bounces at yoctoproject.org<mailto:toaster-bounces at yoctoproject.org>
>> >>[mailto:toaster-bounces at yoctoproject.org<mailto:toaster-bounces at yoctoproject.org>] On
>> >> Behalf Of Neth, Craig Allen
>> >> Sent: Thursday, July 03, 2014 3:34 PM
>> >> To: toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> >> Subject: [Toaster] Need help with Production toaster setup
>> >>
>> >> A month or so ago, I set up a production toaster instance using the
>>poky
>> >> master branch.
>> >> Our projects were building with Daisy. Everything was working fine.
>> >>
>> >> For various reasons, I needed to redo the production instance and
>>that
>> >> involved recloning the poky master repo, and now I¹m having issuesŠ
>> >>
>> >> After recloning poky and restarting the server, all my builds with
>>the
>> >> build logging turned on started throwing weird errors complaining
>>like
>> >> this:
>> >>
>> >> ERROR: Field 'project_id' doesn't have a default value
>> >> <traceback omitted>
>> >>
>> >>
>> >> And :
>> >>
>> >> ERROR:
>> >> Traceback (most recent call last):
>> >> File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
>> >> buildinfohelper.store_started_task(event)
>> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
>> >> store_started_task
>> >> recipe_information =
>> >> self._get_recipe_information_from_taskfile(event.taskfile)
>> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
>> >> _get_recipe_information_from_taskfile
>> >> layer_version_obj =
>>self._get_layer_version_for_path(localfilepath)
>> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
>> >> _get_layer_version_for_path
>> >> assert 'build' in self.internal_state
>> >> AssertionError
>> >>
>> >> So I figured that there was probably some difference of opinion
>>between
>> >> the client and the server about the schema; I tried running the
>>Œsyncdb¹
>> >> and Œmigrate orm¹ commands again on the server but that didn¹t help.
>> >>
>> >>
>> >> ---
>> >>
>> >> So next I figured I would just delete the Œtoaster¹ database and
>>start
>> >> over. I tried going into mysql and dropping the toaster database,
>> >> recreating it, and then redoing the two python commands, but that
>> >>results
>> >> in this error from the migrate orm command:
>> >>
>> >> python bitbake/lib/toaster/manage.py migrate orm
>> >> Running migrations for orm:
>> >> - Migrating forwards to 0007_auto__add_helptext.
>> >> > orm:0001_initial
>> >> > orm:0002_auto__add_field_build_timespent
>> >> > orm:0003_timespent
>> >> - Migration 'orm:0003_timespent' is marked for no-dry-run.
>> >> > orm:0004_auto__add_field_package_installed_name
>> >> >
>> >>
>>
>>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
>>>>le
>> >>hi
>> >> stor
>> >> FATAL ERROR - The following SQL query failed: ALTER TABLE `orm_build`
>> >>ADD
>> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
>> >> The error was: (1060, "Duplicate column name 'timespent'")
>> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and
>>its
>> >> values cannot be restored.
>> >>
>> >> This was all done while working on the daisy branch.
>> >>
>> >> So, then I tried the delete/recreate database trick again and changed
>> >>back
>> >> to the master branch. Now I can run the two python commands ok, but
>>my
>> >> daisy clients are back to throwing the assert errors...
>> >>
>> >> So, a few questions:
>> >>
>> >>
>> >> - Does it still work to run the production server on the master and
>> >> clients running daisy, or do I have to use the daisy version of the
>> >> production server, or do I need something newer than daisy but not as
>> >>new
>> >> as master?
>> >>
>> >> - If for whatever reason I need to recreate the database, what are
>>the
>> >> exact steps that I need to do to delete/recreate?
>> >> It seems like just dropping the Œtoaster¹ database and recreating it
>>is
>> >> insufficient, but I am not sure what else to do.
>> >>
>> >> The wiki pages are a bit vague on all this stuff, thanks for any
>>help.
>> >>
>> >> Craig
>> >>
>> >> --
>> >> _______________________________________________
>> >> toaster mailing list
>> >> toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> >> https://lists.yoctoproject.org/listinfo/toaster
>
>--
>_______________________________________________
>toaster mailing list
>toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>
>--
>Alex DamianYocto Project
>
>SSG / OTC
--
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140708/04630a6f/attachment-0001.html>
More information about the toaster
mailing list