[Toaster] Need help with Production toaster setup

Damian, Alexandru alexandru.damian at intel.com
Tue Jul 8 04:09:04 PDT 2014


Hi,

Thanks for the info -

No, the setup seems as it is designed to be.

I guess somewhere in origin/master things broke. We don't test this setup
on our suite, but maybe we should start testing this, thus I CC'ed Ionut.

Meanwhile, I will start looking into fixing this. Can you please open a bug
in Bugzilla so I can track it ?

Cheers,
Alex



On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <craig.neth at verizon.com>
wrote:

> Hi Alex,
>
> Thanks for the response!
>
> I just did a git pull on the master, and then did the following steps on
> the master machine:
>
> mysql> drop database toaster;
> mysql> create database toaster;
>
> python bitbake/lib/toaster/manage.py syncdb
> python bitbake/lib/toaster/manage.py migrate orm
> python bitbake/lib/toaster/manage.py migrate bldcontrol
>
> (The wiki page does not mention this last one but the ‘syncdb' command
> mentions that this one is also 'not synced’.)
>
> To start the master, I have written a short upstart script that does the
> following:
>
> DATABASE_URL='mysql://user:pass@node:3306/toaster' python
> $BASEDIR/bitbake/lib/toaster/manage.py runserver
>
> The server is up and running (I see the "This is Toaster" page).
>
> On the clients, I normally use a modified version of the ‘toaster’ script
> (I called it toaster-remote) that avoids starting the local webserver but
> doing the commands listed on the wiki page doesn't change the outcome,  I
> still see the same Assertion errors I mentioned in my initial message.
>
> As to your questions:  We are indeed using mysql on the master.   Git
> commit ids:
>
> master:a5531a2b8983318b99c119a87b78a92cf84160b8
> client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28
>
> I mentioned the commands I used above.
>
> Thanks for any help.   As I mentioned, we did have all this working
> previously before I updated the master, so I don’t think my fundamental
> setup is wrong...
>
> Craig
>
>
>
>
> From:  <Damian>, Alexandru <alexandru.damian at intel.com>
> Date:  Friday, July 4, 2014 at 8:06 AM
> To:  "Lerner, Dave" <dave.lerner at windriver.com>
> Cc:  "Neth, Craig A" <craig.neth at one.verizon.com>,
> "toaster at yoctoproject.org" <toaster at yoctoproject.org>
> Subject:  Re: [Toaster] Need help with Production toaster setup
>
>
> >Hi Craig,
> >
> >
> >The system should work with the server running master, and the clients
> >running daisy. The production database MUST be set up with the master,
> >and the clients MUST NOT be started with the toaster script - the script
> >will get confused as to the current revision of the database. The flow is:
> >
> >
> >- start toaster on the server, and let it create / update the database
> >
> >- on the clients, you need to just start the bitbake server and the
> >toasterui client before issuing command to the bitbake server - here
> >there is the race condition that Dave already mentioned
> >
> >
> >
> >If you need to drop and recreate the database, please always use the
> >latest master - the clients, even if using an older version, should
> >accomodate, we try keep the database schema changes backward compatible.
> >
> >
> >
> >
> >For a short while, the "master" was broken due to a bad commit - you may
> >have landed there with your master, can you please git pull lastest
> >master ?
> >
> >
> >
> >
> >About the specific bug you see - can you please let me know more about
> >your setup ?
> >
> >
> >- db engine you're using is mysql, right ?
> >
> >
> >- commit ids for master and daisy that you're using ?
> >
> >- which commands do you use to start the server and the clients ?
> >
> >
> >
> >Cheers,
> >Alex
> >
> >
> >
> >
> >On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <dave.lerner at windriver.com>
> >wrote:
> >
> >Hi Craig,
> >
> >Did you delete your old database and recreate it using the current clone
> >of poky as shown on
> >
> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
> >that is, running the database creation/migration commands that execute in
> >        'source toaster start'?
> >
> >-dave
> >
> >> -----Original Message-----
> >> From: Neth, Craig Allen [mailto:craig.neth at verizon.com]
> >> Sent: Thursday, July 03, 2014 4:05 PM
> >> To: Lerner, Dave
> >> Cc: toaster at yoctoproject.org
> >> Subject: Re: Need help with Production toaster setup
> >>
> >> Hi Dave,
> >>
> >> Thanks for the quick response!
> >>
> >> I had previous experienced the problem you mentioned and worked around
> >>it
> >> by adding a sleep between the —observe-only and the bitbake command.
> >> Without the sleep, as you mention the events were just getting dropped.
> >> It’s nice to see a proper fix for that, however.
> >>
> >> But I don’t think that’s what’s happening now; from the tracebacks it
> >> looks like the clients are sending events to the database but it’s not
> >> setup the way they expect...
> >>
> >> My suspicion is that the schemas have changed but I don’t know how to
> >>get
> >> my database back into a state where it matches the schema the client
> >> expects.
> >>
> >> Craig
> >>
> >>
> >>
> >>
> >> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com> wrote:
> >>
> >> >Hi Craig,
> >> >
> >> >A patch submitted 15 minutes ago may solve your problem, it sounds
> >> >similar to the problem that appeared after daisy in my script that did:
> >> >
> >> >bitbake --server-only ...
> >> >bitbake --observe-only ... ToasterUI
> >> >bitbake <some-image>
> >> >
> >> >With recent releases the 'bitbake <some-image>' fires before the
> >> >ToasterUI is ready (2nd command) to read events.  The patch in
> >> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a
> message
> >> >to the log file.  You can change your script to block until that log
> >> >message appears,  before launching a build - if that's your problem.
> >> >
> >> >Hope this helps,
> >> >Dave Lerner
> >> >
> >> >> -----Original Message-----
> >> >> From: toaster-bounces at yoctoproject.org
> >> >>[mailto:toaster-bounces at yoctoproject.org] On
> >> >> Behalf Of Neth, Craig Allen
> >> >> Sent: Thursday, July 03, 2014 3:34 PM
> >> >> To: toaster at yoctoproject.org
> >> >> Subject: [Toaster] Need help with Production toaster setup
> >> >>
> >> >> A month or so ago, I set up a production toaster instance using the
> >>poky
> >> >> master branch.
> >> >> Our projects were building with Daisy.   Everything was working fine.
> >> >>
> >> >> For various reasons, I needed to redo the production instance and
> >>that
> >> >> involved recloning the poky master repo, and now I¹m having issuesŠ
> >> >>
> >> >> After recloning poky and restarting the server, all my builds with
> >>the
> >> >> build logging turned on started throwing weird errors complaining
> >>like
> >> >> this:
> >> >>
> >> >> ERROR: Field 'project_id' doesn't have a default value
> >> >> <traceback omitted>
> >> >>
> >> >>
> >> >> And :
> >> >>
> >> >> ERROR:
> >> >> Traceback (most recent call last):
> >> >>   File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
> >> >>     buildinfohelper.store_started_task(event)
> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
> >> >> store_started_task
> >> >>     recipe_information =
> >> >> self._get_recipe_information_from_taskfile(event.taskfile)
> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
> >> >> _get_recipe_information_from_taskfile
> >> >>     layer_version_obj =
> >>self._get_layer_version_for_path(localfilepath)
> >> >>   File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
> >> >> _get_layer_version_for_path
> >> >>     assert 'build' in self.internal_state
> >> >> AssertionError
> >> >>
> >> >> So I figured that there was probably some difference of opinion
> >>between
> >> >> the client and the server about the schema; I tried running the
> >>Œsyncdb¹
> >> >> and Œmigrate orm¹ commands again on the server but that didn¹t help.
> >> >>
> >> >>
> >> >> ---
> >> >>
> >> >> So next I figured I would just delete the Œtoaster¹ database and
> >>start
> >> >> over.   I tried going into mysql and dropping the toaster database,
> >> >> recreating it, and then redoing the two python commands, but that
> >> >>results
> >> >> in this error from the migrate orm command:
> >> >>
> >> >> python bitbake/lib/toaster/manage.py migrate orm
> >> >> Running migrations for orm:
> >> >>  - Migrating forwards to 0007_auto__add_helptext.
> >> >>  > orm:0001_initial
> >> >>  > orm:0002_auto__add_field_build_timespent
> >> >>  > orm:0003_timespent
> >> >>  - Migration 'orm:0003_timespent' is marked for no-dry-run.
> >> >>  > orm:0004_auto__add_field_package_installed_name
> >> >>  >
> >> >>
> >>
> >>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
> >>>>le
> >> >>hi
> >> >> stor
> >> >> FATAL ERROR - The following SQL query failed: ALTER TABLE `orm_build`
> >> >>ADD
> >> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
> >> >> The error was: (1060, "Duplicate column name 'timespent'")
> >> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and
> >>its
> >> >> values cannot be restored.
> >> >>
> >> >> This was all done while working on the daisy branch.
> >> >>
> >> >> So, then I tried the delete/recreate database trick again and changed
> >> >>back
> >> >> to the master branch.  Now I can run the two python commands ok, but
> >>my
> >> >> daisy clients are back to throwing the assert errors...
> >> >>
> >> >> So, a few questions:
> >> >>
> >> >>
> >> >> - Does it still work to run the production server on the master and
> >> >> clients running daisy, or do I have to use the daisy version of the
> >> >> production server, or do I need something newer than daisy but not as
> >> >>new
> >> >> as master?
> >> >>
> >> >> - If for whatever reason I need to recreate the database, what are
> >>the
> >> >> exact steps that I need to do to delete/recreate?
> >> >> It seems like just dropping the Œtoaster¹ database and recreating it
> >>is
> >> >> insufficient, but I am not sure what else to do.
> >> >>
> >> >> The wiki pages are a bit vague on all this stuff, thanks for any
> >>help.
> >> >>
> >> >> Craig
> >> >>
> >> >> --
> >> >> _______________________________________________
> >> >> toaster mailing list
> >> >> toaster at yoctoproject.org
> >> >> https://lists.yoctoproject.org/listinfo/toaster
> >
> >--
> >_______________________________________________
> >toaster mailing list
> >toaster at yoctoproject.org
> >https://lists.yoctoproject.org/listinfo/toaster
> >
> >
> >
> >
> >
> >
> >
> >
> >--
> >Alex DamianYocto Project
> >
> >SSG / OTC
>
>


-- 
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140708/dff2f5e0/attachment.html>


More information about the toaster mailing list