[Toaster] Toaster master breakages

[Ed Bartosh]

Hi Alex,

On Mon, Jul 06, 2015 at 12:37:12PM +0100, Damian, Alexandru wrote:
> Hello all,
> 
> I've posted a patchset, on Friday 26, that broke the Toaster master in a
> number of ways. As usual in these situations, the breakage was not caused
> by a single fault, but by a chain of mis-happenings that show policy
> failures in our processes.

Where can I see the current policy/processes/tests described?

> 
> Paul was kind enough to help analyze the situation, and we've identified
> some of the gaps:
> - I've posted upstream a patch that I didn't actually intend to upstream in
> this release, and which wasn't properly reviewed. Probably happened because
> I hurried on manually submitting.
> - While, thanks to latest efforts, we've got better in terms of unit
> testing, we still have a significant gap in functional and integration
> tests, especially automated tests.​
> 
> 
> ​Michael was very proactive and helpful in fixing the master, and I want to
> thank him for that - but we shouldn't continue on this path. While we
> continue to have the policy of not breaking master, we should take steps to
> set out proper policies in place enforce correct submissions, and to
> prevent us from chasing fires all the time.
> 
> 
> ​This is a call for ​proposals and suggestions. Please help me figure out
> solutions to have better policies around submission and testing, and proper
> automation against these tasks.
> 
> Please reply to this thread with:
> - observations on any process/policy gaps that you might notice, and
> proposals for fixing them
> - suggestions around automating the patch review process
> - suggestions around possibly changing the review requirements
> - suggestions around automating the patch upstreaming process
> - suggestions on improving the functional and integration testing coverage

My suggestions are quite obvious and may be too generic. As I don't know
what the current policy is I can't come up with something more specific
for Toaster.

1. Review of changes:

- Developers must run tests manually before submitting them
  for review. Changes that don't pass tests should not be submitted.
  If testing environment is not easy to replicate then
  it should be possible to upload changes to testing system and run
  tests remotely.

- It would be great to have automatic testing triggered and results posted
  on the mailing list or emailed to maintainers and submitter for every review
  request.

2. Accepting changes to target branches:

- Changes should be always tested either manually or automatically *before*
  they merged into target branch.

- Changes must be rebased on top of target branch before testing.
  It would allow us to skip testing after the merge.

- Pending review requests should be rebased and retested after target branch
  is updated.

- Target branch must *always* pass *all* test cases. This would make it
  easy to track test regressions as if any test case doesn't pass it
  would mean regression. This also means that new test cases must not
  introduce regressions.

- If set of changes causes regression it should be bisected to identify
  guilty change(s) and remove them from the set. After that all test cases
  should be run again.

- If target branch is broken fixing it should have highest priority.

3. Testing coverage

- Fresh installation up to successful build of core-minimal image
  should be covered by functional tests.

- The more tests we have the better. Functional and integration tests
  are more important to have then unit or code style tests so it's
  better to start from them if we have limited resources.

- If full testing takes long we should separate most important test
  cases into 'smoke test'.

- Full test run should be continuosly checking target branches to
  identify test regressions as soon as they appear.

--
Regards,
Ed