[yocto-security] Yocto security responses.

[Paul Eggleton]

On Friday, 13 October 2017 2:08:52 PM NZDT Jim Gettys wrote:
> On Thu, Oct 12, 2017 at 5:32 PM, Paul Eggleton <
> paul.eggleton at linux.intel.com> wrote:
> > On Friday, 13 October 2017 8:56:50 AM NZDT Jim Gettys wrote:
> > > I'm trying to understand the Yocto response to security issues.
> > >
> > > The OE https://patchwork.openembedded.org/project/oe/patches/ page shows
> > > you added a patch to fix the recent dnsmasq problems on October 3, by
> > > updating to dnsmasq 2.78.
> > >
> > > But being new to Yocto, I don't know how long it takes to percolate
> > > through the Yocto system.
> > >
> > > https://layers.openembedded.org/layerindex/recipe/4473/
> > >
> > > Still says version 2.76, whereas the fixes are in 2.78.
> > >
> > > If one built Yocto today from current head of tree, what version would
> > > be picked up?
> > >
> > > Any insight on how Yocto handles security issues would be helpful.
> >
> > To explain a little background - OpenEmbedded recipes are split up into
> > various different layers. The layer in which dnsmasq is contained,
> > meta-networking, isn't actually maintained by the Yocto Project - it's not
> > part of our release/test cycle that we do for the core. I just happened to
> > send a patch because I noticed the security issue announcement and it was
> > a straightforward upgrade from 2.76.
> >
> > Patch testing for meta-networking and other layers in the meta-
> > openembedded repository in which meta-networking is contained is\
> > graciously carried out for the community by Martin (on CC) which involves
> > world builds for every architecture we support. Usually when patches get
> > sent it takes a few days for them to get through those tests. I would like
> > to see security patches like this one have some kind of fast track - I'd
> > have to agree that 10 days is longer than we would like. We do have a
> > number of volunteers who work on security issues particularly within OE-
> > Core (which is officially maintained by the Yocto Project) however not so
> > much for recipes outside of that - we are largely reliant upon vendors or
> > other community members responding to these issues and sending back their
> > patches. We also unfortunately do not have advance embargoed notice of
> > security issues (apart from employees of some vendors who are themselves
> > bound not to discuss those embargoed issues) so we
> > can only react when they are publicly announced.
> 
> ​There are common network facing packages, dnsmasq being a good example,
> but web servers, ssh servers, snmp servers, and the like, are security
> critical to essentially everyone who uses them.
> 
> Unfortunately, many/most embedded vendors have shipped/continue to ship kit
> with ​everything running as root, so if you own the process, you own the
> device, with whatever consequence you care to name.

Right, it's certainly a serious problem. I would love for us to be able to 
take on more of the maintenance burden and throw resources at the issue, but 
unfortunately I don't have much influence over that side of things.
 
> That this is the case of much embedded software, I feel some personal
> responsibility, as much of it descended from the Compaq/HP iPAQ project I
> worked on (opkg is descended from ipkg Carl Worth wrote for iPAQ, etc).

Indeed, I was around for the latter part of the handhelds.org days as well 
(albeit as a minor contributor). In fact were it not for handhelds.org I 
wouldn't be doing what I do now, so thank you for your part in that :)
 
> The author of the first squash file system for Linux had thought it a good
> idea to save two bytes per inode for Linux, so there was only root, and we
> didn't go fix that when we had the opportunity. That pattern has continued
> in large part in many people's systems.  Sigh.

Well, even were it not for that, running all services as root is much older 
practice and on systems that didn't have filesystem-based restrictions. In any 
event hindsight is 20/20.

> Getting the behavior of the vendors to change is really hard at this date.
> 
> I am sure someone at LF is privy to the vendor security list.

Hmm, it seems like vendor-sec no longer exists[1] - I'd completely missed that 
myself. I suppose in any case getting the notification is a small part of the 
problem, the more important part is what you do after that.
 
> > I realise this is not a great answer. However until we have someone who is
> > willing to step up and institute a more rigorous process for handling
> > security updates, particularly for layers outside of the core that the
> > Yocto Project maintains, that is where things stand.
> 
> ​I don't think Yocto can afford to say "this is OE's responsibility", for
> any common network facing service, any of which can be security critical. I
> would hope the LF could canvas the vendors for enough additional money to
> put security in Yocto on a more sound footing.

We can certainly ask, but I believe it has been raised before. (If you feel 
like you could make noise through different channels than we could then that 
may help.)
 
> > FYI there is an alternative if you are working on something where you need
> > updated recipes faster - it's pretty easy to add the updated recipes to
> > your own layer on top of the OE/Yocto provided ones (or add a bbappend to
> > apply the patch), and a little less messy than for example forking the
> > original layer and applying the patch there. The only issue is you have to
> > remember to remove it later, otherwise when the time comes that we move
> > beyond that update you will still be building that same version (less of
> > an issue with the bbappend, because that.
> 
> ​Thanks for the explanation.
> 
> FWIW: I installed an updated dnsmasq package for my home router running
> LEDE/OpenWrt by October 4.

Right, they are clearly doing a better job of this than we are. Having said 
that, it would be a bit more eyebrow-raising for a network-centric 
distribution to not have a rapid fix for something like this, so the situation 
is a little different, but I agree we should still strive to do better.

> Installing it was a one liner on the device: "opkg update; opkg upgrade
> dnsmasq"; it was not a "reload the firmware and reboot" event, nor did I
> have to build the firmware/dnsmasq myself.

This is where the differences between their system and ours is a little more 
apparent. Aside from testing we don't produce package feeds - people use our 
tools to produce their own, so we are an extra step away from the systems 
built with our tools.

> If the router firmware had been built to support the new LEDE/OpenWrt
> container system (some are, most aren't at this instant; it's a one line
> configuration option but hasn't percolated through all the devices), it
> would not have even been able to be owned in the first place.
> 
> I'd like to see the container system integrated into Yocto (and I'd like to
> see OpenWrt fix its own shortcomings in the area; but that's not your
> problem).

Container management is certainly being worked on, FWIW, and some folks in the 
wider community have already implemented more complete container-based 
solutions (ResinOS is one example). I've not looked at OpenWrt's container 
system and hadn't heard about it, I will take a look - thanks for the pointer.

> Defense in depth is necessary. It's gotten really bad out there.

Indeed - and we are a big part of that process whether we like it or not.

Cheers,
Paul

[1] http://www.openwall.com/lists/oss-security/2011/03/03/3

-- 

Paul Eggleton
Intel Open Source Technology Centre