[Toaster] any tricks when modifying "buildinfohelper.py"?

Mon Mar 31 03:46:04 PDT 2014

Hello,

This is a very good write-up :). The main dev workflow is exactly as you
outlined. I have some tips & tricks below, though.

Alex

On Sun, Mar 30, 2014 at 4:43 AM, Reyna, David <david.reyna at windriver.com>wrote:

>  Hi Alex,
>
> I was able to get a process for "buildinfohelper.py" to work and resolve
> #6050. Here are my notes in case that may be useful to anyone else who may
> start working with the backend, plus I am sure you have additional
> techniques and tricks of the trade.
>
> I would be curious to learn how one would use pydev to debug the backend.
> I know that you passed some information to Dave (which I quote below) but
> those instructions appears to be for custom modules and not the mainline
> code base. No hurry, but it would be very helpful for Toaster-1.7
> development.
>
> Here was my devel/debug cycle:
>
>   1) Stop Toaster
>   2) Edit the backend files like "buildinfohelper.py” and “models.py”.
>   3) Restart Toaster
>   4) Build a trivial package to execute the new code (e.g. “bitbake
> base-files”)
>   5) Test the changes in your browser
>
>
[Alex] To avoid stopping and starting the whole system, I use two
terminals on which I invoked souce oe-init-build-env first.
In the first terminal I start toaster without the toasterui/buildinfohelper:
> source toaster start noui

In the second terminal, I start just the toasteui:
> bitbake --observe-only -u toasterui

This allows me to monitor the output of toasterui on the terminal, and not
in the toaster_ui.log; and also to restart to load changes without
affecting the whole system.

Another trick, I have this patch here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=adamian/dev
called "debug settings" that adds extensive logging of the events that come
from the server.

>  Here are the error states and workflow issues that I encountered:
>
>   * Unlike the GUI files when you can make changes midstream with Toaster
> still running, you have to stop and restart Toaster in order for your
> backend changes to become active (I think that when Toaster starts is when
> the updated code gets compiled into the “.pyc” runtime code). Some files
> (like “models.py”) are in between (where schema changes need a restart but
> method changes may not), but it is easy to get into an intermediate state
> and have unexpected failures doing this on the fly.
>

[Alex] Toaster is loaded into memory, and will need to be stopped and
started when any changes occur. For the web development, the files are read
from the disk for each request, this is why you see changes immediately.
This only happens on the Django web server, if you use a production set-up
with a frontend (Apache/nginx) and a FCGI Django process, you'll also need
to stop and start the server to see changes.

>
>   * If there is a syntax error in your backend code, the Toaster will
> start and then very quickly halt. You will most likely see many messages of
> the type “*ERROR: Could not connect to server at localhost:8200 ([Errno
> 111] Connection refused)*”, and a similar error when you try to connect
> with the browser. You will also know it failed if the “*.pyc” file did not
> get updated.
>
[Alex] Use the two-terminal trick above, and you'll see the syntax error
message whenever you actually try to start toaster ui again.

>
> My resolution was to stop the Toaster to insure that the full state was
> cleaned up, re-examine the code, and then re-start Toaster. When I was
> lucky the actual error was captured in the “toaster_ui.log” file. I found
> that some of the background tasks took some 5 to 15 seconds before they
> fully stopped, so I had to either retry a few times or just be patient.
>
>   * If there was a runtime error, it would most likely also be captured in
> “toaster_ui.log”. One may not notice the failure state while using the
> browser until you get unexpected page errors, like indexes out of range
> (for when the respective data object never got populated because your code
> failed).
>
>   * If you put “print” statements into your code for simple tracing
> purposes, that output will again appear in “toaster_ui.log”. (I myself
> typically insert and use a “_log” helper function to append the messages to
> a log file under “/tmp/toaster.log” for easy persistence and management so
> that I can track and study it at length).
>
> Thanks,
> David
>
> vvvvvvvvvvvvvvvvvvvvvvvvvvv
>
> Dave Lerner wrote: “I found the best way to debug python with the database
> is (with help from Alex)”
>
> 1) To get into python and have access to the database class models,
> execute this script which will prompt you to get to the right spot, then
> prompt you as to how to setup for the models.
>
> + if [ ! -e manage.py ] ; then
> +    echo "Please run from the lib/toaster directory"
> +    exit 1
> + fi
> + echo
> + echo do...
> + echo from orm.models import Build, Package, Target
> + echo
> + MANAGE=$PWD
> + python $MANAGE/manage.py shell
>
> Import any python model that you want from orm.models. Then you can do
> things like (>>> is the prompt)
> >>> target_name = Target.objects.get(pk=3).target
>
>
> 2) To run code in a module that you put into toaster/toastergui/mytest.py
>
> * import toastergui.mytest
> * This will run any code that isn't inside a class or def.
> * In another session, if you want to change the module, edit it, then
>   >>>  reload(toastergui.mytest)
>   and it will rerun that module
>
> 3) to debug or use a function in that module
>
> * edit functions in your module, and rename the functions with a '_' or
> '__' prefix (at first) so that a simple import works without hiding it
> during import which is what the prefixes do
> - def _my_ func(build_id, target_id)
> + def my_func(build_id, target_id)
> ...
>
> * insert 'import pdb ; pdb.set_trace()' where you want debugging to start
>         import toastergui.mytest
> * invoke the full path to the function
>          toastergui.mytest.my_func(1, 2)
>   and the code will break where you put your breakpoint
> * iterate development by ...
> *** editing the module
> *** exiting debug mode with
>         q
> *** reload the module with
>         reload(toastergui.mytest)
> *** rerun the function with
>         toastergui.mytest.my_func(1, 2)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^
> _____________________________________________
> *From:* Reyna, David
> *Sent:* Friday, March 28, 2014 11:20 AM
> *To:* belen.barros.pena at intel.com; Damian, Alexandru (
> alexandru.damian at intel.com)
> *Subject:* any tricks when modifying "buildinfohelper.py"?
>
>
> Hi,
>
> I am trying to implement adding the task descriptions to the Variable
> table.
>
> It appears that the changes I need to make are in “def
> save_build_variables” in the "buildinfohelper.py” file, but my changes do
> not appear to be making any affect, including writing to external log files
> if/when this procedure is called.
>
> Is there a trick I need to do to have file changes registered, like
> restarting Toaster or removing the "buildinfohelper.pyc”?
>
> - David
>
>
>

-- 
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140331/81d4813b/attachment-0001.html>