[Toaster] Need help with Production toaster setup
Damian, Alexandru
alexandru.damian at intel.com
Wed Jul 9 04:54:51 PDT 2014
Hi Craig - I'd like to have a chat with you, are you on IRC ?
Unfortunately, I am unable to reproduce the errors you're seeing.
I created the database at commit:
*a5531a2b8983318b99c119a87b78a9**2cf84160b8*
and started the bitbake server and toasterui at commit:
*5306aaab07a7a9b3b0c7e8008dabbd**2e89daec28*
and the build gets recorded without error.
I'm thinking you have *.pyc files remaining on disk between git checkouts,
python not picking up changes, and getting old code running - but it may be
any cause.
I'd like to debug this live with you online, do you have time for that ?
Alex
On Tue, Jul 8, 2014 at 3:40 PM, Neth, Craig Allen <craig.neth at verizon.com>
wrote:
> Hi Alex,
>
> Ok, thanks. Bug 6512 was filed. Do you have any speculation about when
> the problem might have been introduced? I don’t have any particular need
> to run the bleeding edge toaster server; if there is an earlier branch that
> might work I’m happy to switch branches on the server end of things.
>
> The weird thing is that I tried reverting the server back to the Daisy
> branch but that also failed with a different set of problems, I received
> errors when I ran the ‘migrate orm’ step. I figured that I had just fat
> fingered some of the setup commands but I can’t seem to find the problem.
>
> Craig
>
> From: <Damian>, Alexandru <alexandru.damian at intel.com>
> Date: Tuesday, July 8, 2014 at 7:09 AM
> To: "Neth, Craig A" <craig.neth at one.verizon.com>
> Cc: "Lerner, Dave" <dave.lerner at windriver.com>, "toaster at yoctoproject.org"
> <toaster at yoctoproject.org>, IonutX Chisanovici <
> ionutx.chisanovici at intel.com>
>
> Subject: Re: [Toaster] Need help with Production toaster setup
>
> Hi,
>
> Thanks for the info -
>
> No, the setup seems as it is designed to be.
>
> I guess somewhere in origin/master things broke. We don't test this setup
> on our suite, but maybe we should start testing this, thus I CC'ed Ionut.
>
> Meanwhile, I will start looking into fixing this. Can you please open a
> bug in Bugzilla so I can track it ?
>
> Cheers,
> Alex
>
>
>
> On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <craig.neth at verizon.com>
> wrote:
>
>> Hi Alex,
>>
>> Thanks for the response!
>>
>> I just did a git pull on the master, and then did the following steps on
>> the master machine:
>>
>> mysql> drop database toaster;
>> mysql> create database toaster;
>>
>> python bitbake/lib/toaster/manage.py syncdb
>> python bitbake/lib/toaster/manage.py migrate orm
>> python bitbake/lib/toaster/manage.py migrate bldcontrol
>>
>> (The wiki page does not mention this last one but the ‘syncdb' command
>> mentions that this one is also 'not synced’.)
>>
>> To start the master, I have written a short upstart script that does the
>> following:
>>
>> DATABASE_URL='mysql://user:pass@node:3306/toaster' python
>> $BASEDIR/bitbake/lib/toaster/manage.py runserver
>>
>> The server is up and running (I see the "This is Toaster" page).
>>
>> On the clients, I normally use a modified version of the ‘toaster’ script
>> (I called it toaster-remote) that avoids starting the local webserver but
>> doing the commands listed on the wiki page doesn't change the outcome, I
>> still see the same Assertion errors I mentioned in my initial message.
>>
>> As to your questions: We are indeed using mysql on the master. Git
>> commit ids:
>>
>> master:a5531a2b8983318b99c119a87b78a92cf84160b8
>> client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28
>>
>> I mentioned the commands I used above.
>>
>> Thanks for any help. As I mentioned, we did have all this working
>> previously before I updated the master, so I don’t think my fundamental
>> setup is wrong...
>>
>> Craig
>>
>>
>>
>>
>> From: <Damian>, Alexandru <alexandru.damian at intel.com>
>> Date: Friday, July 4, 2014 at 8:06 AM
>> To: "Lerner, Dave" <dave.lerner at windriver.com>
>> Cc: "Neth, Craig A" <craig.neth at one.verizon.com>,
>> "toaster at yoctoproject.org" <toaster at yoctoproject.org>
>> Subject: Re: [Toaster] Need help with Production toaster setup
>>
>>
>> >Hi Craig,
>> >
>> >
>> >The system should work with the server running master, and the clients
>> >running daisy. The production database MUST be set up with the master,
>> >and the clients MUST NOT be started with the toaster script - the script
>> >will get confused as to the current revision of the database. The flow
>> is:
>> >
>> >
>> >- start toaster on the server, and let it create / update the database
>> >
>> >- on the clients, you need to just start the bitbake server and the
>> >toasterui client before issuing command to the bitbake server - here
>> >there is the race condition that Dave already mentioned
>> >
>> >
>> >
>> >If you need to drop and recreate the database, please always use the
>> >latest master - the clients, even if using an older version, should
>> >accomodate, we try keep the database schema changes backward compatible.
>> >
>> >
>> >
>> >
>> >For a short while, the "master" was broken due to a bad commit - you may
>> >have landed there with your master, can you please git pull lastest
>> >master ?
>> >
>> >
>> >
>> >
>> >About the specific bug you see - can you please let me know more about
>> >your setup ?
>> >
>> >
>> >- db engine you're using is mysql, right ?
>> >
>> >
>> >- commit ids for master and daisy that you're using ?
>> >
>> >- which commands do you use to start the server and the clients ?
>> >
>> >
>> >
>> >Cheers,
>> >Alex
>> >
>> >
>> >
>> >
>> >On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <dave.lerner at windriver.com
>> >
>> >wrote:
>> >
>> >Hi Craig,
>> >
>> >Did you delete your old database and recreate it using the current clone
>> >of poky as shown on
>> >
>> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
>> >that is, running the database creation/migration commands that execute in
>> > 'source toaster start'?
>> >
>> >-dave
>> >
>> >> -----Original Message-----
>> >> From: Neth, Craig Allen [mailto:craig.neth at verizon.com]
>> >> Sent: Thursday, July 03, 2014 4:05 PM
>> >> To: Lerner, Dave
>> >> Cc: toaster at yoctoproject.org
>> >> Subject: Re: Need help with Production toaster setup
>> >>
>> >> Hi Dave,
>> >>
>> >> Thanks for the quick response!
>> >>
>> >> I had previous experienced the problem you mentioned and worked around
>> >>it
>> >> by adding a sleep between the —observe-only and the bitbake command.
>> >> Without the sleep, as you mention the events were just getting dropped.
>> >> It’s nice to see a proper fix for that, however.
>> >>
>> >> But I don’t think that’s what’s happening now; from the tracebacks it
>> >> looks like the clients are sending events to the database but it’s not
>> >> setup the way they expect...
>> >>
>> >> My suspicion is that the schemas have changed but I don’t know how to
>> >>get
>> >> my database back into a state where it matches the schema the client
>> >> expects.
>> >>
>> >> Craig
>> >>
>> >>
>> >>
>> >>
>> >> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com> wrote:
>> >>
>> >> >Hi Craig,
>> >> >
>> >> >A patch submitted 15 minutes ago may solve your problem, it sounds
>> >> >similar to the problem that appeared after daisy in my script that
>> did:
>> >> >
>> >> >bitbake --server-only ...
>> >> >bitbake --observe-only ... ToasterUI
>> >> >bitbake <some-image>
>> >> >
>> >> >With recent releases the 'bitbake <some-image>' fires before the
>> >> >ToasterUI is ready (2nd command) to read events. The patch in
>> >> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a
>> message
>> >> >to the log file. You can change your script to block until that log
>> >> >message appears, before launching a build - if that's your problem.
>> >> >
>> >> >Hope this helps,
>> >> >Dave Lerner
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: toaster-bounces at yoctoproject.org
>> >> >>[mailto:toaster-bounces at yoctoproject.org] On
>> >> >> Behalf Of Neth, Craig Allen
>> >> >> Sent: Thursday, July 03, 2014 3:34 PM
>> >> >> To: toaster at yoctoproject.org
>> >> >> Subject: [Toaster] Need help with Production toaster setup
>> >> >>
>> >> >> A month or so ago, I set up a production toaster instance using the
>> >>poky
>> >> >> master branch.
>> >> >> Our projects were building with Daisy. Everything was working
>> fine.
>> >> >>
>> >> >> For various reasons, I needed to redo the production instance and
>> >>that
>> >> >> involved recloning the poky master repo, and now I¹m having issuesŠ
>> >> >>
>> >> >> After recloning poky and restarting the server, all my builds with
>> >>the
>> >> >> build logging turned on started throwing weird errors complaining
>> >>like
>> >> >> this:
>> >> >>
>> >> >> ERROR: Field 'project_id' doesn't have a default value
>> >> >> <traceback omitted>
>> >> >>
>> >> >>
>> >> >> And :
>> >> >>
>> >> >> ERROR:
>> >> >> Traceback (most recent call last):
>> >> >> File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
>> >> >> buildinfohelper.store_started_task(event)
>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
>> >> >> store_started_task
>> >> >> recipe_information =
>> >> >> self._get_recipe_information_from_taskfile(event.taskfile)
>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
>> >> >> _get_recipe_information_from_taskfile
>> >> >> layer_version_obj =
>> >>self._get_layer_version_for_path(localfilepath)
>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
>> >> >> _get_layer_version_for_path
>> >> >> assert 'build' in self.internal_state
>> >> >> AssertionError
>> >> >>
>> >> >> So I figured that there was probably some difference of opinion
>> >>between
>> >> >> the client and the server about the schema; I tried running the
>> >>Œsyncdb¹
>> >> >> and Œmigrate orm¹ commands again on the server but that didn¹t help.
>> >> >>
>> >> >>
>> >> >> ---
>> >> >>
>> >> >> So next I figured I would just delete the Œtoaster¹ database and
>> >>start
>> >> >> over. I tried going into mysql and dropping the toaster database,
>> >> >> recreating it, and then redoing the two python commands, but that
>> >> >>results
>> >> >> in this error from the migrate orm command:
>> >> >>
>> >> >> python bitbake/lib/toaster/manage.py migrate orm
>> >> >> Running migrations for orm:
>> >> >> - Migrating forwards to 0007_auto__add_helptext.
>> >> >> > orm:0001_initial
>> >> >> > orm:0002_auto__add_field_build_timespent
>> >> >> > orm:0003_timespent
>> >> >> - Migration 'orm:0003_timespent' is marked for no-dry-run.
>> >> >> > orm:0004_auto__add_field_package_installed_name
>> >> >> >
>> >> >>
>> >>
>>
>> >>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
>> >>>>le
>> >> >>hi
>> >> >> stor
>> >> >> FATAL ERROR - The following SQL query failed: ALTER TABLE
>> `orm_build`
>> >> >>ADD
>> >> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
>> >> >> The error was: (1060, "Duplicate column name 'timespent'")
>> >> >> RuntimeError: Cannot reverse this migration. 'Target.file_name' and
>> >>its
>> >> >> values cannot be restored.
>> >> >>
>> >> >> This was all done while working on the daisy branch.
>> >> >>
>> >> >> So, then I tried the delete/recreate database trick again and
>> changed
>> >> >>back
>> >> >> to the master branch. Now I can run the two python commands ok, but
>> >>my
>> >> >> daisy clients are back to throwing the assert errors...
>> >> >>
>> >> >> So, a few questions:
>> >> >>
>> >> >>
>> >> >> - Does it still work to run the production server on the master and
>> >> >> clients running daisy, or do I have to use the daisy version of the
>> >> >> production server, or do I need something newer than daisy but not
>> as
>> >> >>new
>> >> >> as master?
>> >> >>
>> >> >> - If for whatever reason I need to recreate the database, what are
>> >>the
>> >> >> exact steps that I need to do to delete/recreate?
>> >> >> It seems like just dropping the Œtoaster¹ database and recreating it
>> >>is
>> >> >> insufficient, but I am not sure what else to do.
>> >> >>
>> >> >> The wiki pages are a bit vague on all this stuff, thanks for any
>> >>help.
>> >> >>
>> >> >> Craig
>> >> >>
>> >> >> --
>> >> >> _______________________________________________
>> >> >> toaster mailing list
>> >> >> toaster at yoctoproject.org
>> >> >> https://lists.yoctoproject.org/listinfo/toaster
>> >
>> >--
>> >_______________________________________________
>> >toaster mailing list
>> >toaster at yoctoproject.org
>> >https://lists.yoctoproject.org/listinfo/toaster
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >--
>> >Alex DamianYocto Project
>> >
>> >SSG / OTC
>>
>>
>
>
> --
> Alex Damian
> Yocto Project
> SSG / OTC
>
>
--
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140709/d685573e/attachment-0001.html>
More information about the toaster
mailing list