[Toaster] Need help with Production toaster setup

Neth, Craig Allen craig.neth at verizon.com
Fri Jul 11 06:33:34 PDT 2014


Hi Alex,

Thanks for all the hard work.   It does indeed sound a mess.

And I am able to verify that your workaround works for us.   Thank you!

Craig

From: <Damian>, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>>
Date: Friday, July 11, 2014 at 6:49 AM
To: "Neth, Craig A" <craig.neth at one.verizon.com<mailto:craig.neth at one.verizon.com>>
Cc: "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>, "toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>" <toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>>, IonutX Chisanovici <ionutx.chisanovici at intel.com<mailto:ionutx.chisanovici at intel.com>>
Subject: Re: [Toaster] Need help with Production toaster setup

Hi Craig,

This is a bit of a mess - multiple problems throught Django and MySQL layers happen here.

The gist of it is that Django aborts a transaction when Mysqldb link layer raises an exception when a warning happens in the database itself; this is a needless escalation of importance for an exception. Furthermore, the warning from MySQL is incorrect, because we already instruct MySQL that the field that it reports missing may be missing, so that's that.

I have two workarounds for this:
- use a different database engine, like postgreSQL
- on builders (yocto-1.6 branch), modify in toastermain/settings.py DEBUG from True to False

There is nothing in the code that I can do to solve this.
I will document the needed setup on the wiki.

Cheers,
Alex


On Wed, Jul 9, 2014 at 1:07 PM, Damian, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>> wrote:
Oh, I think I have something - it happens on mysql, but not on sqlite. This is important :)

I'll dive into it.

Alex


On Wed, Jul 9, 2014 at 12:54 PM, Damian, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>> wrote:
Hi Craig - I'd like to have a chat with you, are you on IRC ?

Unfortunately, I am unable to reproduce the errors you're seeing.
I created the database at commit:
a5531a2b8983318b99c119a87b78a92cf84160b8

and started the bitbake server and toasterui at commit:
5306aaab07a7a9b3b0c7e8008dabbd2e89daec28

and the build gets recorded without error.

I'm thinking you have *.pyc files remaining on disk between git checkouts, python not picking up changes, and getting old code running - but it may be any cause.

I'd like to debug this live with you online, do you have time for that ?

Alex






On Tue, Jul 8, 2014 at 3:40 PM, Neth, Craig Allen <craig.neth at verizon.com<mailto:craig.neth at verizon.com>> wrote:
Hi Alex,

Ok, thanks.   Bug 6512 was filed.   Do you have any speculation about when the problem might have been introduced?   I don’t have any particular need to run the bleeding edge toaster server; if there is an earlier branch that might work I’m happy to switch branches on the server end of things.

The weird thing is that I tried reverting the server back to the Daisy branch but that also failed with a different set of problems, I received errors when I ran the ‘migrate orm’ step.   I figured that I had just fat fingered some of the setup commands but I can’t seem to find the problem.

Craig

From: <Damian>, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>>
Date: Tuesday, July 8, 2014 at 7:09 AM
To: "Neth, Craig A" <craig.neth at one.verizon.com<mailto:craig.neth at one.verizon.com>>
Cc: "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>, "toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>" <toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>>, IonutX Chisanovici <ionutx.chisanovici at intel.com<mailto:ionutx.chisanovici at intel.com>>

Subject: Re: [Toaster] Need help with Production toaster setup

Hi,

Thanks for the info -

No, the setup seems as it is designed to be.

I guess somewhere in origin/master things broke. We don't test this setup on our suite, but maybe we should start testing this, thus I CC'ed Ionut.

Meanwhile, I will start looking into fixing this. Can you please open a bug in Bugzilla so I can track it ?

Cheers,
Alex



On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <craig.neth at verizon.com<mailto:craig.neth at verizon.com>> wrote:
Hi Alex,

Thanks for the response!

I just did a git pull on the master, and then did the following steps on
the master machine:

mysql> drop database toaster;
mysql> create database toaster;

python bitbake/lib/toaster/manage.py syncdb
python bitbake/lib/toaster/manage.py migrate orm
python bitbake/lib/toaster/manage.py migrate bldcontrol

(The wiki page does not mention this last one but the ‘syncdb' command
mentions that this one is also 'not synced’.)

To start the master, I have written a short upstart script that does the
following:

DATABASE_URL='mysql://user:pass@node:3306/toaster' python
$BASEDIR/bitbake/lib/toaster/manage.py runserver

The server is up and running (I see the "This is Toaster" page).

On the clients, I normally use a modified version of the ‘toaster’ script
(I called it toaster-remote) that avoids starting the local webserver but
doing the commands listed on the wiki page doesn't change the outcome,  I
still see the same Assertion errors I mentioned in my initial message.

As to your questions:  We are indeed using mysql on the master.   Git
commit ids:

master:a5531a2b8983318b99c119a87b78a92cf84160b8
client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28

I mentioned the commands I used above.

Thanks for any help.   As I mentioned, we did have all this working
previously before I updated the master, so I don’t think my fundamental
setup is wrong...

Craig




From:  <Damian>, Alexandru <alexandru.damian at intel.com<mailto:alexandru.damian at intel.com>>
Date:  Friday, July 4, 2014 at 8:06 AM
To:  "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>
Cc:  "Neth, Craig A" <craig.neth at one.verizon.com<mailto:craig.neth at one.verizon.com>>,
"toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>" <toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>>
Subject:  Re: [Toaster] Need help with Production toaster setup


>Hi Craig,
>
>
>The system should work with the server running master, and the clients
>running daisy. The production database MUST be set up with the master,
>and the clients MUST NOT be started with the toaster script - the script
>will get confused as to the current revision of the database. The flow is:
>
>
>- start toaster on the server, and let it create / update the database
>
>- on the clients, you need to just start the bitbake server and the
>toasterui client before issuing command to the bitbake server - here
>there is the race condition that Dave already mentioned
>
>
>
>If you need to drop and recreate the database, please always use the
>latest master - the clients, even if using an older version, should
>accomodate, we try keep the database schema changes backward compatible.
>
>
>
>
>For a short while, the "master" was broken due to a bad commit - you may
>have landed there with your master, can you please git pull lastest
>master ?
>
>
>
>
>About the specific bug you see - can you please let me know more about
>your setup ?
>
>
>- db engine you're using is mysql, right ?
>
>
>- commit ids for master and daisy that you're using ?
>
>- which commands do you use to start the server and the clients ?
>
>
>
>Cheers,
>Alex
>
>
>
>
>On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>>
>wrote:
>
>Hi Craig,
>
>Did you delete your old database and recreate it using the current clone
>of poky as shown on
> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
>that is, running the database creation/migration commands that execute in
>        'source toaster start'?
>
>-dave
>
>> -----Original Message-----
>> From: Neth, Craig Allen [mailto:craig.neth at verizon.com<mailto:craig.neth at verizon.com>]
>> Sent: Thursday, July 03, 2014 4:05 PM
>> To: Lerner, Dave
>> Cc: toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> Subject: Re: Need help with Production toaster setup
>>
>> Hi Dave,
>>
>> Thanks for the quick response!
>>
>> I had previous experienced the problem you mentioned and worked around
>>it
>> by adding a sleep between the —observe-only and the bitbake command.
>> Without the sleep, as you mention the events were just getting dropped.
>> It’s nice to see a proper fix for that, however.
>>
>> But I don’t think that’s what’s happening now; from the tracebacks it
>> looks like the clients are sending events to the database but it’s not
>> setup the way they expect...
>>
>> My suspicion is that the schemas have changed but I don’t know how to
>>get
>> my database back into a state where it matches the schema the client
>> expects.
>>
>> Craig
>>
>>
>>
>>
>> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com<mailto:dave.lerner at windriver.com>> wrote:
>>
>> >Hi Craig,
>> >
>> >A patch submitted 15 minutes ago may solve your problem, it sounds
>> >similar to the problem that appeared after daisy in my script that did:
>> >
>> >bitbake --server-only ...
>> >bitbake --observe-only ... ToasterUI
>> >bitbake <some-image>
>> >
>> >With recent releases the 'bitbake <some-image>' fires before the
>> >ToasterUI is ready (2nd command) to read events.  The patch in
>> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a message
>> >to the log file.  You can change your script to block until that log
>> >message appears,  before launching a build - if that's your problem.
>> >
>> >Hope this helps,
>> >Dave Lerner
>> >
>> >> -----Original Message-----
>> >> From: toaster-bounces at yoctoproject.org<mailto:toaster-bounces at yoctoproject.org>
>> >>[mailto:toaster-bounces at yoctoproject.org<mailto:toaster-bounces at yoctoproject.org>] On
>> >> Behalf Of Neth, Craig Allen
>> >> Sent: Thursday, July 03, 2014 3:34 PM
>> >> To: toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> >> Subject: [Toaster] Need help with Production toaster setup
>> >>
>> >> A month or so ago, I set up a production toaster instance using the
>>poky
>> >> master branch.
>> >> Our projects were building with Daisy.   Everything was working fine.
>> >>
>> >> For various reasons, I needed to redo the production instance and
>>that
>> >> involved recloning the poky master repo, and now I¹m having issuesŠ
>> >>
>> >> After recloning poky and restarting the server, all my builds with
>>the
>> >> build logging turned on started throwing weird errors complaining
>>like
>> >> this:
>> >>
>> >> ERROR: Field 'project_id' doesn't have a default value
>> >> <traceback omitted>
>> >>
>> >>
>> >> And :
>> >>
>> >> ERROR:
>> >> Traceback (most recent call last):
>> >>   File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
>> >>     buildinfohelper.store_started_task(event)
>> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
>> >> store_started_task
>> >>     recipe_information =
>> >> self._get_recipe_information_from_taskfile(event.taskfile)
>> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
>> >> _get_recipe_information_from_taskfile
>> >>     layer_version_obj =
>>self._get_layer_version_for_path(localfilepath)
>> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
>> >> _get_layer_version_for_path
>> >>     assert 'build' in self.internal_state
>> >> AssertionError
>> >>
>> >> So I figured that there was probably some difference of opinion
>>between
>> >> the client and the server about the schema; I tried running the
>>Œsyncdb¹
>> >> and Œmigrate orm¹ commands again on the server but that didn¹t help.
>> >>
>> >>
>> >> ---
>> >>
>> >> So next I figured I would just delete the Œtoaster¹ database and
>>start
>> >> over.   I tried going into mysql and dropping the toaster database,
>> >> recreating it, and then redoing the two python commands, but that
>> >>results
>> >> in this error from the migrate orm command:
>> >>
>> >> python bitbake/lib/toaster/manage.py migrate orm
>> >> Running migrations for orm:
>> >>  - Migrating forwards to 0007_auto__add_helptext.
>> >>  > orm:0001_initial
>> >>  > orm:0002_auto__add_field_build_timespent
>> >>  > orm:0003_timespent
>> >>  - Migration 'orm:0003_timespent' is marked for no-dry-run.
>> >>  > orm:0004_auto__add_field_package_installed_name
>> >>  >
>> >>
>>
>>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
>>>>le
>> >>hi
>> >> stor
>> >> FATAL ERROR - The following SQL query failed: ALTER TABLE `orm_build`
>> >>ADD
>> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
>> >> The error was: (1060, "Duplicate column name 'timespent'")
>> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and
>>its
>> >> values cannot be restored.
>> >>
>> >> This was all done while working on the daisy branch.
>> >>
>> >> So, then I tried the delete/recreate database trick again and changed
>> >>back
>> >> to the master branch.  Now I can run the two python commands ok, but
>>my
>> >> daisy clients are back to throwing the assert errors...
>> >>
>> >> So, a few questions:
>> >>
>> >>
>> >> - Does it still work to run the production server on the master and
>> >> clients running daisy, or do I have to use the daisy version of the
>> >> production server, or do I need something newer than daisy but not as
>> >>new
>> >> as master?
>> >>
>> >> - If for whatever reason I need to recreate the database, what are
>>the
>> >> exact steps that I need to do to delete/recreate?
>> >> It seems like just dropping the Œtoaster¹ database and recreating it
>>is
>> >> insufficient, but I am not sure what else to do.
>> >>
>> >> The wiki pages are a bit vague on all this stuff, thanks for any
>>help.
>> >>
>> >> Craig
>> >>
>> >> --
>> >> _______________________________________________
>> >> toaster mailing list
>> >> toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>> >> https://lists.yoctoproject.org/listinfo/toaster
>
>--
>_______________________________________________
>toaster mailing list
>toaster at yoctoproject.org<mailto:toaster at yoctoproject.org>
>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>
>--
>Alex DamianYocto Project
>
>SSG / OTC




--
Alex Damian
Yocto Project
SSG / OTC



--
Alex Damian
Yocto Project
SSG / OTC



--
Alex Damian
Yocto Project
SSG / OTC



--
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140711/aa8d49c6/attachment-0001.html>


More information about the toaster mailing list