[Toaster] [RFC] Configuration variables in Toaster
Damian, Alexandru
alexandru.damian at intel.com
Mon Jul 14 04:00:40 PDT 2014
Hi,
Thanks for the input,
I have comments inline, and I hope this will shed a bit more light on how I
think about builds,
Cheers,
Alex
On Fri, Jul 11, 2014 at 8:16 AM, Reyna, David L (Wind River) <
david.reyna at windriver.com> wrote:
> Hi Alex,
>
>
>
> > In this respect, I propose that we blacklist from Toaster interface any
> variable in category b), at least for 1.7 release.
>
> While I am reluctant, I can accept this if we are just trying to solve
> 1.7, and come back to this in 1.8.
>
>
>
> I still stand by my point that the buildbot (or other infrastructure)
> will _*always*_ have to dynamically compute and "sanitize" the (b)
> variables to do its job, so I am still unclear how you are saving work for
> the buildbot with your proposal. Perhaps you are hoping to use buildbot
> unmodified and/or without an intermediary agent, but I do not see how that
> will work.
>
This is exactly the point - the building infrastructure will have to
sanitize the variables set. Since these variables will ALWAYS be
overridden, wouldn't it be confusing for the users if they could change
these variables, but the changes would have no effect ?
>
> I would also like to understand your OOB scenario for Toaster users, since
> you are counter-proposing my suggestions on that point.
>
Not sure what OOB :) is ?
Out Of Band ?
>
>
> For example:
>
> * Will Toaster create a default admin setup on-the-fly for first time
> users, just as today it will create a default database on-the-fly?
>
When launched in managed mode (i.e. outside a build environment), there
are additional tables created on first startup. During this extra step, an
extra step is taken to create a default admin user. We don't currently use
this admin user, nor we enable the admin interface, but at lease we have
the base to allow us to extend related functionality in the future.
* Will Toaster aggressively detect and reject if the user should attempt
> to manually add the (b) variables in the custom configure var page?
>
I would suggest a big red warning that these variables could be
overridden without further warnings by the build system.
>
> I will mention that my immediate issue is that I am positioning Toaster to
> replace the Wind River Linux 7.0 project configuration tools, and for that
> I _*do*_ have to consider the local case. I can accept some extra work on
> my team above and beyond your Toaster-1.7 design, as long as the
> Toaster-1.7 does not prevent what I need to do.
>
I understand this, and I'm trying to support your model of usage for
Toaster.
The thing is, unless I understand in detail how you intend to use Toaster,
I will probably favour decisions that will create some hardship for you.
I'm trying to avoid this situation.
What variables are set influence how the build control system is designed,
actually. There are two ways to go about it:
1). postconf - start bitbake server, and connect to it and push changed
variables; this will create problems for variables in category b), but we
can be sure that the variables that we set are not overridden by the
infrastructure; in order to do this, we need to make sure that we don't set
any variables that could create problems for the infrastructure, so the
sanitize is on toaster side.
2). preconf - create a specific config file (let's say user-toaster.conf)
that can be pushed to bitbake to be used when the server starts. This
allows us to configure any kind of variable, because a proper "local.conf"
deployed by the infrastructure can override dangerous variables. The
downside is that we can't guarantee that the variables set by the user will
not be affected or changed by the variables pushed by infrastructure.
In between choices 1). and 2)., I would favour 1)., because, while limiting
the user in what he can do, it will not lie to the user by setting
variables that will not be reflected in the build.
As for the *local* and *remote* build, I would suggest avoiding thinking
in these terms - what I think we should concerned about is the
*repeatability* of the build, i.e. regardless of where the build executes,
we must make sure that on the same project configuration, we get the same
output.
The use case I'm using to evaluate the repeatability is : if a user
configures a project to his liking, working only on his machine, and then
if he exports the project settings to a file, and then imports that file to
a hosted Toaster infrastructure: Can we certify that the output of the
build on the infrastructure is identical to the build that the developer
had on his machine ?
I am not sure how compatible is this view with WR's intended usage for
Toaster, so I'm appreciating all your input :)
>
> Thanks,
>
> David
>
>
>
> *From:* Damian, Alexandru [mailto:alexandru.damian at intel.com]
> *Sent:* Thursday, July 10, 2014 8:48 AM
> *To:* Reyna, David
> *Cc:* BARROS PENA, BELEN; toaster at yoctoproject.org; Wymore, Farrell; Paul
> Eggleton; Richard Purdie
> *Subject:* Re: [Toaster] [RFC] Configuration variables in Toaster
>
>
>
> Hello,
>
> I've been reviewing this proposal, and I think it's good to have this
> discussion now, and the points raised are excellent.
>
> I'm seeing a different angle to the variable story. CCing Paul and Richard
> so they can track this discussion.
>
> I think that we can separate the bitbake variables in two different
> categories, even if bitbake itself makes no such separation, -
>
> a). variables that influence WHAT will be built - e.g
> EXTRA_IMAGE_FEATURES, or IMAGE_INSTALL_append
>
>
> b). variables that influence HOW it will be build - e.g. PARALLEL_MAKE, or
> SSTATE_MIRRORS for that matter
>
>
>
> When you configure bitbake, you have to put two different hats, a
> developer hat for a). and a sysadmin hat for b).
>
> I think that, at the current point, Toaster should focus at being only a
> developer tool, and it should concern itself only with the a). hat.
>
> In this respect, I propose that we blacklist from Toaster interface any
> variable in category b), at least for 1.7 release.
>
> In this way, we reduce the chance that Toaster settings will screw
> something up on the infrastructure setup, and we'll also solve the problem
> of having the buildbot (or other infrastructure) "sanitize" the variables
> set by Toaster.
>
>
>
> In respect to the a). variables, I think it's a good idea to provide a set
> of commonly-used variables with sane defaults, and offer help for the user
> when he adds other variables, e.g. link to manual description or help
> searching variable name.
>
> What do you think ?
>
> Cheers,
> Alex
>
>
>
> On Wed, Jul 9, 2014 at 11:28 PM, Reyna, David <david.reyna at windriver.com>
> wrote:
>
> Hi Belén,
>
> You write: "In a Toaster instance connected to a build farm, a user of
> Toaster should not be able to override any variables that impact how
> hardware runs builds and where
> artifacts are stored."
>
> I would think of this in the opposite manner.
>
> Proposed wording: "In a Toaster instance connected to a build farm, the
> remote build instance will _always_ override any variables that impact how
> hardware runs builds and where artifacts are stored, regardless of any
> default bitbake settings or explicit user settings."
>
> Here is my reasoning:
>
> * The remote build instance's manager/configuration will need to
> dynamically set these values regardless and in all cases, based on its
> current resources and environment, whether they are previously defined or
> not.
>
> * We cannot prevent the user from manually setting the values no matter
> what we do, as per your mention of the free form "add variables" option.
>
> * We can think of this class of variables as "preferences". If it is a
> local build then these values will be used (not overridden) and user will
> get what they expect. If it is a remote build then the values will be
> expected to be overridden by design and again the user will get what they
> expect.
>
> * With this model there are no surprises, both workflows are easily
> covered, and everyone is happy.
>
> - David
>
>
>
>
> > -----Original Message-----
> > From: Barros Pena, Belen [mailto:belen.barros.pena at intel.com]
> > Sent: Wednesday, July 09, 2014 9:42 AM
> > To: toaster at yoctoproject.org
> > Cc: Wymore, Farrell; Reyna, David; Lerner, Dave
> > Subject: [RFC] Configuration variables in Toaster
> >
> > Some discussion happened today about the list of variables that Toaster
> > should expose (see below from previous thread). It was observed that, for
> > cases when you are in control of the build hardware, it would be handy to
> > be able to set variables like PARALLEL_MAKE, BB_NUMBER_THREADS and
> > SSTATE_DIR. I took out those variables from the list because they impact
> > how hardware runs builds and where build artifacts are stored, which are
> > things that a user of a Toaster instance connected to a build farm should
> > probably not touch.
> >
> > It was suggested that maybe the list of variables could change if Toaster
> > is running on the same machine that runs the builds. There is also the
> > "add variables" option, which is free form and that will allow you to
> type
> > any variable and any value you want. So that option would in theory allow
> > you to set PARALLEL_MAKE or anything else you might want to set.
> >
> > Now: that brings up a question about variable handling. In a Toaster
> > instance connected to a build farm, a user of Toaster should not be able
> > to override any variables that impact how hardware runs builds and where
> > artifacts are stored. If the user adds PARALLEL_MAKE = "8" to a Toaster
> > project configuration, that should have no effect on the PARALLEL_MAKE
> > value that a build server in a farm is supposed to use, and that should
> > have been set by whomever administers that farm. Same goes for
> SSTATE_DIR,
> > SSTATE_MIRRORS, DL_DIR, etc. Maybe I am just not looking at this in the
> > right way, or there is something I don't fully understand, but it seems
> to
> > me that Toaster could become a liability otherwise.
> >
> > Any thoughts on how this should be handled would be much appreciated.
> >
> > Thanks!
> >
> > Belén
> >
> > On 08/07/2014 15:40, "Barros Pena, Belen" <belen.barros.pena at intel.com>
> > wrote:
> >
> > >That's the Toaster page that allows you to edit the project local.conf
> > >file. I am trying to come up with a final list of the variables that
> will
> > >be listed in that page.
> > >
> > >Looking at the default local.conf file, and excluding stuff that should
> be
> > >defined by the Toaster instance (things like SSTATE_MIRRORS or DL_DIR),
> > >this is what I'm left with:
> > >
> > >PACKAGE_CLASSES
> > >SDKMACHINE
> > >EXTRA_IMAGE_FEATURES
> > >USER_CLASSES
> > >TEST_IMAGE
> > >OE_TERMINAL
> > >PATCHRESOLVE
> > >PACKAGECONFIG_pn-qemu-native (for Qemu configuration)
> > >ASSUME_PROVIDED (for Qemu configuration)
> > >CONF_VERSION
> > >
> > >Of the above, I am not sure we should expose the Qemu configuration
> stuff
> > >or CONF_VERSION, but I could be wrong.
> > >
> > >
> > >Hob also allows you to set:
> > >
> > >INCOMPATIBLE_LICENSE
> > >IMAGE_FSTYPES
> > >IMAGE_EXTRA_SPACE
> > >TOOLCHAIN_BUILD
> > >LINGUAS_INSTALL
> > >plus proxy configuration variables (http_proxy, https_proxy, ftp_proxy,
> > >all_proxy, CVS_PROXY_HOST, CVS_PROXY_PORT)
> > >
> > >Should we list this Hob stuff as well (except the proxy variables, I
> > >guess)?
> > >
> > >Apart from the above, there will be a way to set IMAGE_INSTALL_append,
> and
> > >also an option to add any other variable (free text kind of thing).
> > >
> > >Any thoughts on this list?
> > >
> > >Thanks!
> > >
> > >Belén
> > >
> > >--
> > >_______________________________________________
> > >toaster mailing list
> > >toaster at yoctoproject.org
> > >https://lists.yoctoproject.org/listinfo/toaster
> >
>
> --
> _______________________________________________
> toaster mailing list
> toaster at yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
> --
>
> Alex Damian
>
> Yocto Project
>
> SSG / OTC
>
--
Alex Damian
Yocto Project
SSG / OTC
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.yoctoproject.org/pipermail/toaster/attachments/20140714/66e32bf5/attachment-0001.html>
More information about the toaster
mailing list