[linux-yocto] KMETA branch/directory redux (yocto #5090)

[Bruce Ashfield]

On 14-08-29 03:31 PM, Peter A. Bigot wrote:
> On 08/29/2014 02:17 PM, Bruce Ashfield wrote:
>> On 14-08-29 03:09 PM, Peter A. Bigot wrote:
>>> On 08/07/2014 11:16 AM, Bruce Ashfield wrote:
>>>> On Thu, Aug 7, 2014 at 10:27 AM, Peter A. Bigot <pab at pabigot.com>
>>>> wrote:
>>>>> First, some yak shaving:
>>>>>
>>>>> If something goes wrong with linux-yocto's do_patch, it prints:
>>>>>
>>>>> | ERROR. Could not apply patches for beaglebone.
>>>>> |        Patch failures can be resolved in the devshell (bitbake -c
>>>>> devshell
>>>>> linux-yocto)
>>>>>
>>>>> In fact, you can't, because meta/classes/devshell.bbclass has:
>>>>>
>>>>> addtask devshell after do_patch
>>>>>
>>>>> so there's a chicken/egg thing here because bitbake never gets to
>>>>> devshell.
>>>>> No idea how to fix that.
>>>>>
>>>>> Now: What goes wrong with do_patch for me ultimately turns out to be
>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5090 biting me
>>>>> again a
>>>>> year after I filed and forgot it, because I set KMETA=yocto-3.14/meta
>>>>> because that's the branch name in my local kernel directory.  Various
>>>>> kgit-*
>>>>> tools still want to use KMETA to name the directory as well as the
>>>>> branch,
>>>>> and they are not happy when KMETA isn't meta.
>>>>>
>>>>> I'm going to try to do a patch for that, if it's agreed that KMETA
>>>>> really
>>>>> does only name the branch and something else needs to propagate the
>>>>> directory name among all the scripts.
>>>> I have an entire series that addresses issues like this. The work with
>>>> 3.16
>>>> delayed sending anything out. When I'm back from holidays, I'll be
>>>> sending
>>>> out those changes. So by about the M3 milestone of 1.7, most everything
>>>> should be sorted.
>>>>
>>>> It definitely isn't trivial to break the branch == directory rule that
>>>> has been
>>>> around for 7 years, but I think I've sorted through most of the
>>>> wreckage
>>>> now.
>>>
>>> I see there are some patches that have already been merged into
>>
>> Yep. Some work in progress that I didn't want to lose, rebasing some
>> old patches into a larger series.
>>
>>> yocto-kernel-tools for this.  At a glance, there's still a construct
>>> that breaks my use case: I have branch names like yocto-3.14/meta and
>>
>> directory names or branch names ?
>
> Branch names.  I don't have a separate repository for each kernel
> release, so I encode that into the branch name.  In the version
> currently in oe-core, the directory name was taken from the branch name
> with no ability to override it.

gotcha .. and that's a valid use case.

The new code should definitely handle that, it did in my tests .. so
I'll run some more on my remaining commits (my queue is larger than
what I've staged).

>
>>
>>> the pattern:
>>>
>>>      if [ ! -d ".$checkpoint_dir" ]; then
>>>         mv $checkpoint_dir .$checkpoint_dir
>>>      fi
>>>
>>> does not work in this situation because there is no directory
>>> .yocto-3.14 into which yocto-3.14/meta can be moved.
>>
>> I'm not looking for completely flexibility, since my statement of
>> not introducing any new variables to broadcast the directory name
>> holds. But within that bound, I'm willing to tweak things more.
>>
>> The construct is simply to hide the top level meta directory from
>> the content branches, git status, etc. So it take a directory name
>> of "foo", which should be where the meta data is stored on the
>> meta branch, and renames it ".foo". What exactly are you using
>> as a directory to hold the meta-data ?
>
> I don't particularly care what the directory name is; "meta" would be
> fine regardless of the branch name, if there's some way to communicate
> that to the infrastructure.

Great. So they can definitely all be 'meta' as the directory name,
or something else. The tools will figure it out, and use whatever
name it happens to be.

>
> The point is that issuing a mv like:
>
>         mv $checkpoint_dir .$checkpoint_dir
>
> without ensuring that the value of the variable doesn't contain path
> components is fragile.  I'm not really clear on the goals of the

It can use some extra sanity checking, but since it is searching out
directories already on the disk to use, we do know that the contents
of the variable are safe for on disk representation.

> construct, especially as the dot-qualified name didn't appear to be
> referenced again when I solved this locally.  If that's true, why not
> just delete it?  If there's some reason it needs to continue to exist in
> the filesystem, why not move it to ".checkpoint_dir" (as a bare name,
> not as the value of a variable)?

The dot directory isn't used by the checkpoint scripts after that point,
but it is used by most of the tools/scripts that follow. So it does
need to stay around.

There can technically be multiple meta data directories in a single
tree, hence why the name of the directory is reused in the .<name> format.

The point of the checkpoint restored directory is to make it available
to each and every branch in the tree during processing, since the
update, patching and configuration steps use that meta data to
manipulate the tree while changing branches, etc.

To make it globally available .. without needing to have it be part of
those branches (i.e. committed and merged to the branches), making it
untracked content solves the problem. And to make it so it doesn't show
up in 'git status', and so that we can continue to check out and work
with the meta branch itself, making the directory a ".<meta>" name
solves the problem.

Bruce

>
>> I'll run some extra tests to make sure I hit those cases, but
>> I'll send the bulk of the changes regardless, since small incremental
>> changes aren't issues post feature freeze.
>
> OK, thanks.
>
> Peter
>
>>
>> Bruce
>>
>>>
>>> I'd hoped to have a chance to try out your solution before it got
>>> merged, but please do announce once the full set of changes is available
>>> so if there are problems we can get them resolved before the feature
>>> freeze.
>>>
>>> Thanks.
>>>
>>> Peter
>>>
>>>
>>>>
>>>>> Before I do that, is that the intended behavior?  I was going to add
>>>>> KMETA_DIR as the companion variable, and probably have all the
>>>>> meta_dir()
>>>>> shell functions in the *me scripts export that.
>>>> It is the intended behaviour. Adding another variable isn't the right
>>>> fix either,
>>>> I'm deleting linux-yocto specific variables, not adding more. Logic to
>>>> determine
>>>> the right directory name has been added to the scripts, when the
>>>> directory and
>>>> branch don't match.
>>>>
>>>>> Are patches to yocto-kernel-tools to be sent to this list?
>>>>>
>>>> To the linux-yocto mailing list.
>>>>
>>>>> Also, is there a reason all the support scripts are present in the
>>>>> meta/scripts directory of the KMETA branches, even though when you run
>>>>> bitbake the versions that come from kern-tools are the ones that are
>>>>> actually used?  Their presence tricked me into thinking that the ones
>>>>> in the
>>>>> meta branch were what I should be patching....
>>>> They are there as a snapshot / reference. They can be used in a pinch
>>>> to regenerate tree, based only on a clone of the tree. But the
>>>> default is
>>>> to use the ones from the tools .. since that is where changes get
>>>> pushed first.
>>>>
>>>> Bruce
>>>>
>>>>> Peter
>>>>> --
>>>>> _______________________________________________
>>>>> linux-yocto mailing list
>>>>> linux-yocto at yoctoproject.org
>>>>> https://lists.yoctoproject.org/listinfo/linux-yocto
>>>>
>>>
>>
>