[Toaster] Need help with Production toaster setup
Damian, Alexandru
alexandru.damian at intel.com
Fri Jul 11 03:49:26 PDT 2014
Hi Craig,
This is a bit of a mess - multiple problems throught Django and MySQL
layers happen here.
The gist of it is that Django aborts a transaction when Mysqldb link layer
raises an exception when a warning happens in the database itself; this is
a needless escalation of importance for an exception. Furthermore, the
warning from MySQL is incorrect, because we already instruct MySQL that the
field that it reports missing may be missing, so that's that.
I have two workarounds for this:
- use a different database engine, like postgreSQL
- on builders (yocto-1.6 branch), modify in toastermain/settings.py DEBUG
from True to False
There is nothing in the code that I can do to solve this.
I will document the needed setup on the wiki.
Cheers,
Alex
On Wed, Jul 9, 2014 at 1:07 PM, Damian, Alexandru <
alexandru.damian at intel.com> wrote:
> Oh, I think I have something - it happens on mysql, but not on sqlite.
> This is important :)
>
> I'll dive into it.
>
> Alex
>
>
> On Wed, Jul 9, 2014 at 12:54 PM, Damian, Alexandru <
> alexandru.damian at intel.com> wrote:
>
>> Hi Craig - I'd like to have a chat with you, are you on IRC ?
>>
>> Unfortunately, I am unable to reproduce the errors you're seeing.
>> I created the database at commit:
>> *a5531a2b8983318b99c119a87b78a9**2cf84160b8*
>>
>> and started the bitbake server and toasterui at commit:
>> *5306aaab07a7a9b3b0c7e8008dabbd**2e89daec28*
>>
>> and the build gets recorded without error.
>>
>> I'm thinking you have *.pyc files remaining on disk between git
>> checkouts, python not picking up changes, and getting old code running -
>> but it may be any cause.
>>
>> I'd like to debug this live with you online, do you have time for that ?
>>
>> Alex
>>
>>
>>
>>
>>
>>
>> On Tue, Jul 8, 2014 at 3:40 PM, Neth, Craig Allen <craig.neth at verizon.com
>> > wrote:
>>
>>> Hi Alex,
>>>
>>> Ok, thanks. Bug 6512 was filed. Do you have any speculation about
>>> when the problem might have been introduced? I don’t have any particular
>>> need to run the bleeding edge toaster server; if there is an earlier branch
>>> that might work I’m happy to switch branches on the server end of things.
>>>
>>> The weird thing is that I tried reverting the server back to the Daisy
>>> branch but that also failed with a different set of problems, I received
>>> errors when I ran the ‘migrate orm’ step. I figured that I had just fat
>>> fingered some of the setup commands but I can’t seem to find the problem.
>>>
>>> Craig
>>>
>>> From: <Damian>, Alexandru <alexandru.damian at intel.com>
>>> Date: Tuesday, July 8, 2014 at 7:09 AM
>>> To: "Neth, Craig A" <craig.neth at one.verizon.com>
>>> Cc: "Lerner, Dave" <dave.lerner at windriver.com>, "
>>> toaster at yoctoproject.org" <toaster at yoctoproject.org>, IonutX
>>> Chisanovici <ionutx.chisanovici at intel.com>
>>>
>>> Subject: Re: [Toaster] Need help with Production toaster setup
>>>
>>> Hi,
>>>
>>> Thanks for the info -
>>>
>>> No, the setup seems as it is designed to be.
>>>
>>> I guess somewhere in origin/master things broke. We don't test this
>>> setup on our suite, but maybe we should start testing this, thus I CC'ed
>>> Ionut.
>>>
>>> Meanwhile, I will start looking into fixing this. Can you please open a
>>> bug in Bugzilla so I can track it ?
>>>
>>> Cheers,
>>> Alex
>>>
>>>
>>>
>>> On Mon, Jul 7, 2014 at 3:45 PM, Neth, Craig Allen <
>>> craig.neth at verizon.com> wrote:
>>>
>>>> Hi Alex,
>>>>
>>>> Thanks for the response!
>>>>
>>>> I just did a git pull on the master, and then did the following steps on
>>>> the master machine:
>>>>
>>>> mysql> drop database toaster;
>>>> mysql> create database toaster;
>>>>
>>>> python bitbake/lib/toaster/manage.py syncdb
>>>> python bitbake/lib/toaster/manage.py migrate orm
>>>> python bitbake/lib/toaster/manage.py migrate bldcontrol
>>>>
>>>> (The wiki page does not mention this last one but the ‘syncdb' command
>>>> mentions that this one is also 'not synced’.)
>>>>
>>>> To start the master, I have written a short upstart script that does the
>>>> following:
>>>>
>>>> DATABASE_URL='mysql://user:pass@node:3306/toaster' python
>>>> $BASEDIR/bitbake/lib/toaster/manage.py runserver
>>>>
>>>> The server is up and running (I see the "This is Toaster" page).
>>>>
>>>> On the clients, I normally use a modified version of the ‘toaster’
>>>> script
>>>> (I called it toaster-remote) that avoids starting the local webserver
>>>> but
>>>> doing the commands listed on the wiki page doesn't change the outcome,
>>>> I
>>>> still see the same Assertion errors I mentioned in my initial message.
>>>>
>>>> As to your questions: We are indeed using mysql on the master. Git
>>>> commit ids:
>>>>
>>>> master:a5531a2b8983318b99c119a87b78a92cf84160b8
>>>> client:5306aaab07a7a9b3b0c7e8008dabbd2e89daec28
>>>>
>>>> I mentioned the commands I used above.
>>>>
>>>> Thanks for any help. As I mentioned, we did have all this working
>>>> previously before I updated the master, so I don’t think my fundamental
>>>> setup is wrong...
>>>>
>>>> Craig
>>>>
>>>>
>>>>
>>>>
>>>> From: <Damian>, Alexandru <alexandru.damian at intel.com>
>>>> Date: Friday, July 4, 2014 at 8:06 AM
>>>> To: "Lerner, Dave" <dave.lerner at windriver.com>
>>>> Cc: "Neth, Craig A" <craig.neth at one.verizon.com>,
>>>> "toaster at yoctoproject.org" <toaster at yoctoproject.org>
>>>> Subject: Re: [Toaster] Need help with Production toaster setup
>>>>
>>>>
>>>> >Hi Craig,
>>>> >
>>>> >
>>>> >The system should work with the server running master, and the clients
>>>> >running daisy. The production database MUST be set up with the master,
>>>> >and the clients MUST NOT be started with the toaster script - the
>>>> script
>>>> >will get confused as to the current revision of the database. The flow
>>>> is:
>>>> >
>>>> >
>>>> >- start toaster on the server, and let it create / update the database
>>>> >
>>>> >- on the clients, you need to just start the bitbake server and the
>>>> >toasterui client before issuing command to the bitbake server - here
>>>> >there is the race condition that Dave already mentioned
>>>> >
>>>> >
>>>> >
>>>> >If you need to drop and recreate the database, please always use the
>>>> >latest master - the clients, even if using an older version, should
>>>> >accomodate, we try keep the database schema changes backward
>>>> compatible.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >For a short while, the "master" was broken due to a bad commit - you
>>>> may
>>>> >have landed there with your master, can you please git pull lastest
>>>> >master ?
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >About the specific bug you see - can you please let me know more about
>>>> >your setup ?
>>>> >
>>>> >
>>>> >- db engine you're using is mysql, right ?
>>>> >
>>>> >
>>>> >- commit ids for master and daisy that you're using ?
>>>> >
>>>> >- which commands do you use to start the server and the clients ?
>>>> >
>>>> >
>>>> >
>>>> >Cheers,
>>>> >Alex
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >On Thu, Jul 3, 2014 at 10:17 PM, Lerner, Dave <
>>>> dave.lerner at windriver.com>
>>>> >wrote:
>>>> >
>>>> >Hi Craig,
>>>> >
>>>> >Did you delete your old database and recreate it using the current
>>>> clone
>>>> >of poky as shown on
>>>> >
>>>> https://wiki.yoctoproject.org/wiki/Setting_up_a_local_instance_of_Toaster
>>>> >that is, running the database creation/migration commands that execute
>>>> in
>>>> > 'source toaster start'?
>>>> >
>>>> >-dave
>>>> >
>>>> >> -----Original Message-----
>>>> >> From: Neth, Craig Allen [mailto:craig.neth at verizon.com]
>>>> >> Sent: Thursday, July 03, 2014 4:05 PM
>>>> >> To: Lerner, Dave
>>>> >> Cc: toaster at yoctoproject.org
>>>> >> Subject: Re: Need help with Production toaster setup
>>>> >>
>>>> >> Hi Dave,
>>>> >>
>>>> >> Thanks for the quick response!
>>>> >>
>>>> >> I had previous experienced the problem you mentioned and worked
>>>> around
>>>> >>it
>>>> >> by adding a sleep between the —observe-only and the bitbake command.
>>>> >> Without the sleep, as you mention the events were just getting
>>>> dropped.
>>>> >> It’s nice to see a proper fix for that, however.
>>>> >>
>>>> >> But I don’t think that’s what’s happening now; from the tracebacks it
>>>> >> looks like the clients are sending events to the database but it’s
>>>> not
>>>> >> setup the way they expect...
>>>> >>
>>>> >> My suspicion is that the schemas have changed but I don’t know how to
>>>> >>get
>>>> >> my database back into a state where it matches the schema the client
>>>> >> expects.
>>>> >>
>>>> >> Craig
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> On 7/3/14, 4:52 PM, "Lerner, Dave" <dave.lerner at windriver.com>
>>>> wrote:
>>>> >>
>>>> >> >Hi Craig,
>>>> >> >
>>>> >> >A patch submitted 15 minutes ago may solve your problem, it sounds
>>>> >> >similar to the problem that appeared after daisy in my script that
>>>> did:
>>>> >> >
>>>> >> >bitbake --server-only ...
>>>> >> >bitbake --observe-only ... ToasterUI
>>>> >> >bitbake <some-image>
>>>> >> >
>>>> >> >With recent releases the 'bitbake <some-image>' fires before the
>>>> >> >ToasterUI is ready (2nd command) to read events. The patch in
>>>> >> >https://bugzilla.yoctoproject.org/show_bug.cgi?id=6504 writes a
>>>> message
>>>> >> >to the log file. You can change your script to block until that log
>>>> >> >message appears, before launching a build - if that's your problem.
>>>> >> >
>>>> >> >Hope this helps,
>>>> >> >Dave Lerner
>>>> >> >
>>>> >> >> -----Original Message-----
>>>> >> >> From: toaster-bounces at yoctoproject.org
>>>> >> >>[mailto:toaster-bounces at yoctoproject.org] On
>>>> >> >> Behalf Of Neth, Craig Allen
>>>> >> >> Sent: Thursday, July 03, 2014 3:34 PM
>>>> >> >> To: toaster at yoctoproject.org
>>>> >> >> Subject: [Toaster] Need help with Production toaster setup
>>>> >> >>
>>>> >> >> A month or so ago, I set up a production toaster instance using
>>>> the
>>>> >>poky
>>>> >> >> master branch.
>>>> >> >> Our projects were building with Daisy. Everything was working
>>>> fine.
>>>> >> >>
>>>> >> >> For various reasons, I needed to redo the production instance and
>>>> >>that
>>>> >> >> involved recloning the poky master repo, and now I¹m having
>>>> issuesŠ
>>>> >> >>
>>>> >> >> After recloning poky and restarting the server, all my builds with
>>>> >>the
>>>> >> >> build logging turned on started throwing weird errors complaining
>>>> >>like
>>>> >> >> this:
>>>> >> >>
>>>> >> >> ERROR: Field 'project_id' doesn't have a default value
>>>> >> >> <traceback omitted>
>>>> >> >>
>>>> >> >>
>>>> >> >> And :
>>>> >> >>
>>>> >> >> ERROR:
>>>> >> >> Traceback (most recent call last):
>>>> >> >> File "/poky/bitbake/lib/bb/ui/toasterui.py", line 191, in main
>>>> >> >> buildinfohelper.store_started_task(event)
>>>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 653, in
>>>> >> >> store_started_task
>>>> >> >> recipe_information =
>>>> >> >> self._get_recipe_information_from_taskfile(event.taskfile)
>>>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 551, in
>>>> >> >> _get_recipe_information_from_taskfile
>>>> >> >> layer_version_obj =
>>>> >>self._get_layer_version_for_path(localfilepath)
>>>> >> >> File "/poky/bitbake/lib/bb/ui/buildinfohelper.py", line 533, in
>>>> >> >> _get_layer_version_for_path
>>>> >> >> assert 'build' in self.internal_state
>>>> >> >> AssertionError
>>>> >> >>
>>>> >> >> So I figured that there was probably some difference of opinion
>>>> >>between
>>>> >> >> the client and the server about the schema; I tried running the
>>>> >>Œsyncdb¹
>>>> >> >> and Œmigrate orm¹ commands again on the server but that didn¹t
>>>> help.
>>>> >> >>
>>>> >> >>
>>>> >> >> ---
>>>> >> >>
>>>> >> >> So next I figured I would just delete the Œtoaster¹ database and
>>>> >>start
>>>> >> >> over. I tried going into mysql and dropping the toaster
>>>> database,
>>>> >> >> recreating it, and then redoing the two python commands, but that
>>>> >> >>results
>>>> >> >> in this error from the migrate orm command:
>>>> >> >>
>>>> >> >> python bitbake/lib/toaster/manage.py migrate orm
>>>> >> >> Running migrations for orm:
>>>> >> >> - Migrating forwards to 0007_auto__add_helptext.
>>>> >> >> > orm:0001_initial
>>>> >> >> > orm:0002_auto__add_field_build_timespent
>>>> >> >> > orm:0003_timespent
>>>> >> >> - Migration 'orm:0003_timespent' is marked for no-dry-run.
>>>> >> >> > orm:0004_auto__add_field_package_installed_name
>>>> >> >> >
>>>> >> >>
>>>> >>
>>>>
>>>> >>>>orm:0005_auto__add_target_image_file__add_target_file__add_field_variab
>>>> >>>>le
>>>> >> >>hi
>>>> >> >> stor
>>>> >> >> FATAL ERROR - The following SQL query failed: ALTER TABLE
>>>> `orm_build`
>>>> >> >>ADD
>>>> >> >> COLUMN `timespent` integer NOT NULL DEFAULT 0;
>>>> >> >> The error was: (1060, "Duplicate column name 'timespent'")
>>>> >> >> RuntimeError: Cannot reverse this migration. 'Target.file_name'
>>>> and
>>>> >>its
>>>> >> >> values cannot be restored.
>>>> >> >>
>>>> >> >> This was all done while working on the daisy branch.
>>>> >> >>
>>>> >> >> So, then I tried the delete/recreate database trick again and
>>>> changed
>>>> >> >>back
>>>> >> >> to the master branch. Now I can run the two python commands ok,
>>>> but
>>>> >>my
>>>> >> >> daisy clients are back to throwing the assert errors...
>>>> >> >>
>>>> >> >> So, a few questions:
>>>> >> >>
>>>> >> >>
>>>> >> >> - Does it still work to run the production server on the master
>>>> and
>>>> >> >> clients running daisy, or do I have to use the daisy version of
>>>> the
>>>> >> >> production server, or do I need something newer than daisy but
>>>> not as
>>>> >> >>new
>>>> >> >> as master?
>>>> >> >>
>>>> >> >> - If for whatever reason I need to recreate the database, what are
>>>> >>the
>>>> >> >> exact steps that I need to do to delete/recreate?
>>>> >> >> It seems like just dropping the Œtoaster¹ database and recreating
>>>> it
>>>> >>is
>>>> >> >> insufficient, but I am not sure what else to do.
>>>> >> >>
>>>> >> >> The wiki pages are a bit vague on all this stuff, thanks for any
>>>> >>help.
>>>> >> >>
>>>> >> >> Craig
>>>> >> >>
>>>> >> >> --
>>>> >> >> _______________________________________________
>>>> >> >> toaster mailing list
>>>> >> >> toaster at yoctoproject.org
>>>> >> >> https://lists.yoctoproject.org/listinfo/toaster
>>>> >
>>>> >--
>>>> >_______________________________________________
>>>> >toaster mailing list
>>>> >toaster at yoctoproject.org
>>>> >https://lists.yoctoproject.org/listinfo/toaster
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >--
>>>> >Alex DamianYocto Project
>>>> >
>>>> >SSG / OTC
>>>>
>>>>
>>>
>>>
>>> --
>>> Alex Damian
>>> Yocto Project
>>> SSG / OTC
>>>
>>>
>>
>>
>> --
>> Alex Damian
>> Yocto Project
>> SSG / OTC
>>
>
>
>
> --
> Alex Damian
> Yocto Project
> SSG / OTC
>
--
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140711/89c74268/attachment-0001.html>
More information about the toaster
mailing list