[Toaster] Toaster testing

[Ed Bartosh]

Hi,

Recently I've been fixing toaster testing and discovered some info which
I'd like to share with you.

Toaster has several types of tests. I'll be talking here about Django
testing. You can read about it in details in Django documentation:
https://docs.djangoproject.com/en/1.8/topics/testing/overview/

Here is how it works for Toaster:

$ cd bitbake/lib/toaster/
$ ./manage.py test toastergui
Creating test database for alias 'default'...
.......
----------------------------------------------------------------------
Ran 7 tests in 0.256s

OK

toastergui is one of 3 Toaster modules with tests.
The rest two are orm and bldcontrol.

orm and toastergui tests are in working state and bldcontrol is still
broken.

It's possible to run both working tests. It can be done this way:
$ ./manage.py test toastergui orm toastergui

To see more verbose output you can use -v2 or -v3 commandline options:

$ ./manage.py test -v2 toastergui
Creating test database for alias 'default' (':memory:')...
Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
...
Creating table orm_logmessage
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
test_get_base_call_returns_html (toaster.toastergui.tests.ViewTests)
Basic test for all-projects view. ... ok
test_get_json_call_returns_json (toaster.toastergui.tests.ViewTests)
Test for all projects output in json format. ... ok
test_xhr_createcustomrecipe_absent_parameter
(toaster.toastergui.tests.ViewTests)
Test not passing parameter to xhr_createcustomrecipe. ... ok
test_xhr_createcustomrecipe_invalid_base_recipe
(toaster.toastergui.tests.ViewTests)
Test passing wrong base recipe id. ... ok
test_xhr_createcustomrecipe_invalid_project
(toaster.toastergui.tests.ViewTests)
Test passing wrong project id. ... ok
test_xhr_createcustomrecipe_ok (toaster.toastergui.tests.ViewTests)
Test successful return from ReST API xhr_createcustomrecipe. ... ok
test_xhr_datatypeahead_layer (toaster.toastergui.tests.ViewTests)
Test ReST API xhr_datatypeahead. ... ok
----------------------------------------------------------------------
Ran 7 tests in 0.253s

OK


It's also possible to generate code coverage report using coverage.py
tool. You can install in with pip or package manager of your Linux
distro. I'm using OpenSUSE 13.2 and installed it this way:
$ zypper install python-coverage

To  information tests should be run the way described in this document:
https://docs.djangoproject.com/en/1.8/topics/testing/advanced/#integration-with-coverage-py

For example, coverage report for toastergui tests can be generated the
folowing way:

$ coverage run --source ./toastergui/ ./manage.py test toastergui
...
$ coverage report
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
toastergui/__init__                       0      0   100%
toastergui/tables                       192    143    26%
toastergui/templatetags/__init__          0      0   100%
toastergui/templatetags/projecttags     153    108    29%
toastergui/tests                         77      0   100%
toastergui/urls                           5      0   100%
toastergui/views                       1399   1100    21%
toastergui/widgets                      169    134    21%
---------------------------------------------------------
TOTAL                                  1995   1485    26%


coverage report -m will include information about code lines not covered
by tests. This is very useful option for test writers.


I'd like to encaurage Toaster developers and contributors to run tests
before sending patches for review to this mailing list and to upstream.

It would be also great to cover more code with tests. I'd propose to
concentrate on writing tests for toastergui module as this module is
closer to end user UI than 2 others.


PS: My fixes are not yet upstreamed. You can find them in poky-contrib
branches ed/toaster/fix-orm-tests and ed/toaster/fix-toastergui-tests

--
Regards,
Ed