[meta-xilinx] Xilinx, why are you breaking the meta-xilinx-bsp layer?

Tue May 22 14:03:05 PDT 2018

Hi Martin,

> -----Original Message-----
> From: meta-xilinx-bounces at yoctoproject.org [mailto:meta-xilinx-
> bounces at yoctoproject.org] On Behalf Of Martin Lund
> Sent: Wednesday, May 16, 2018 3:50 AM
> To: meta-xilinx at yoctoproject.org
> Subject: Re: [meta-xilinx] Xilinx, why are you breaking the meta-xilinx-bsp layer?
> 
> Hi Manju,
> 
> 
> 
> Thank you for your answers - please see my responses below.
> 
> >> We are trying to put together a system built on top of the
> >> meta-xilinx-bsp layer
> 
> >> and we have done so successfully up until the v2017.4 release with a
> >> few helpful patches from this excellent community. That is, we have
> >> been able to create our system without using the meta-xilinx-tools
> >> and meta-petalinux layers. Meaning meta-xilinx-bsp has offered a
> >> fairly independent and functional BSP layer, in our case, for the zcu102 dev
> board.
> >
> >I am not sure how you achieved to build pmu firmware in OSS/OSL flow, using a
> rel-v2017.4 branch.
> >https://github.com/Xilinx/meta-xilinx/tree/rel-v2017.4/recipes-bsp
> ><https://github.com/Xilinx/meta-xilinx/tree/rel-v2017.4/recipes-bsp>
> >There was no recipe available to build one nor support to build SPL.
> 
> I stand corrected. We actually did use the rocko branch of meta-xilinx. I meant to
> write v2017.3 because we see a lot of v2017.3 footprints in the rocko branch.
> 
> 
> >> However, with the introduction of Xilinx release v2018.1 it seems
> >> some basic BSP things are broken or split out and moved to the
> >> meta-xilinx-tools or meta- petalinux for no apparent good reason -
> >> basically breaking the independence of the meta-xilinx-bsp layer. Up
> >> until this point we are still struggling to get meta- xilinx-bsp booting on our
> zcu102 board.
> >>
> >
> >Release branches in github is very specific for Xilinx tool release. Meaning all the
> rel-v2018.x are supposed to work with the layer stack (meta-petalinux, meta-
> xilinx-tools, meta-xilinx-bsp and meta-xilinx-contrib). This is the same practice
> which has >been done since 2017. If you see older releases like rel-v2017.1, SPL
> support, certain board support code is not present in release branches.
> >
> >Please see
> >https://lists.yoctoproject.org/pipermail/meta-xilinx/2016-October/00217
> >4.html
> 
> Thank you for clarifying - we didn't realize the substantial difference between
> Xilinx release branches and Yocto release branches in meta-xilinx.
> 
> It explains why we are running into trouble trying to use rel-v2018.1 with YP
> rocko.
> 
> That being said, I'm not sure I understand why Xilinx insists on maintaining it's
> own branch for what seems an effort to enforce the Xilinx toolchain. I mean, it
> seems clear to me that not many developers appreciate the way the Xilinx
> toolchain is put together. In fact most professionals I talk to seem to prefer the
> open source standard tools offered by e.g. the YP toolchain. I understand there is a
> lot history to Xilinx using its own tools and maybe a hint of NIH but perhaps it
> would be easier for everyone if Xilinx adapted the OSS tools when operating in the
> domain of modern Linux BSP solutions. Also, wouldn't it be in the interest of Xilinx
> to have one less branch to maintain?
> 
> I realize I might be stepping on some toes stating the above but I'm just relaying
> what many developers here are thinking.
> 

We understand the deviation and are working on to resolve it.  
Thud release is supposed to clean-up some the efforts on baremetal, multiconfig etc.

> 
> >Saying that, 2018.2 release branch will be fixed to include u-boot-mkimage, and
> fw-utils.
> >This will be pushed mid-june.
> 
> Ok.
> 
> 
> >One more thing to add, Xilinx release branches works on top of a previous
> version of the available Yocto release.
> >Meaning rel-v2018.1 (and rest of rel-v2018.x ) is based on rocko branch of meta-
> xilinx-bsp.
> >rel-v2019.1 will be based on thud (v2.6) branch of meta-xilinx-bsp etc
> >
> >We don't change any of the OSS/OSL flow related recipes in the Xilinx tool
> related releases.
> 
> Hopefully we don't have to wait for 2019 for the next Xilinx YP branch update of
> meta-xilinx-bsp.
> 

Show me an instance where YP branch has not been cut?
Nathan and others have helped immensely in the past to make release branches consistently with YP releases. 

We have been sending patches to mailing list for review for quite some weeks now to upgrade and prepare for Sumo release.

> 
> >> Another more important example is the omission of an updated
> >> pmu-firmware in meta-xilinx-bsp for the v2018.1 release. A simple and
> >> straightforward way of building the pmu-firmware used to reside in
> >> meta-xilinx-bsp (see
> >> https://github.com/Xilinx/meta-xilinx/tree/rel-v2018.1/meta-xilinx-bs
> >> p/recipes-
> >> <https://github.com/Xilinx/meta-xilinx/tree/rel-v2018.1/meta-xilinx-b
> >> sp/recipes-> bsp/pmu-firmware
> >> <https://github.com/Xilinx/meta-xilinx/tree/rel-v2018.1/meta-
> >> xilinx-bsp/recipes-bsp/pmu-firmware> ) but now a new way of building
> >> the pmu- firmware has been introduced/forced in meta-xilinx-tools
> >> (see
> >> https://github.com/Xilinx/meta-xilinx-tools/tree/rel-v2018.1/recipes-
> >> bsp/pmu-
> >> <https://github.com/Xilinx/meta-xilinx-tools/tree/rel-v2018.1/recipes
> >> -bsp/pmu->
> >> firmware).
> >
> >Using xsct to build pmu-firmware is not new, this layer and recipes has been
> introduced probably in 2016 timeframe.
> >
> 
> Ok - is there a particular good reason for using xsct to build the PMU firmware?
> 

xsct is a tool that we officially support, and users can add additional customization to PMU firmware if required.

> I mean, isn't the microblaze gcc/newlib toolchain good enough for Xilinx?
> 

We are working on this to clean up certain things wrt newlib, libglos, inclusion of this in OE-Core if possible and also enable to build using multiconfig approach.

> 
> >> As mentioned in https://lists.yoctoproject.org/pipermail/meta-
> >> <https://lists.yoctoproject.org/pipermail/meta->
> >> xilinx/2018-May/003809.html, this new approach uses all the spokes
> >> and wheels of the proprietary xilinx toolchain involving all sorts of
> >> cumbersome tools and components (xsdk, X/gtk3, xsct, tcl, yaml, etc.)
> >> which makes it quite troublesome to deploy on common build servers. I
> >> can't understand why Xilinx thinks it is a good idea to go down this
> >> route. They take something simple and make it complicated, much
> >> slower, and storage demanding. Additionally, what makes matters even
> >> worse is that the build definition for is pmu-firmware is actually
> >> placed in a bbclass (see
> >> https://github.com/Xilinx/meta-xilinx-tools/blob/rel-
> >> <https://github.com/Xilinx/meta-xilinx-tools/blob/rel->
> >> v2018.1/classes/xsctapp.bbclass
> >> <https://github.com/Xilinx/meta-xilinx-
> >> tools/blob/rel-v2018.1/classes/xsctapp.bbclass> ) but it really
> >> belongs in the pmu- firmware_*.bb file. Either way, the end result is
> >> that if one is using only the
> >> v2018.1 meta-xilinx-bsp layer then the old v2017.3 pmu-firmware is
> >> built which does not seem to be compatible with v2018.1 atf, u-boot, kernel
> etc.
> >
> >
> >Same as the above, recipes are tied together with the above mentioned layers
> for release branches.
> >
> 
> Understood. The Xilinx release branches ties the four layers together and the
> effect is various inter-dependencies between the layers - in other words, the are
> tightly coupled - use em all or nothing.
> 

Yes

> 
> >>
> >> Seen from the outside, it all seems like a bit of a mess.
> >
> >Xilinx tools release cannot be independent, the workflow is different.
> >If you continue to use rel-v2018 (release branches), then the recommendation is
> to use all layers.
> 
> Got it.
> 
> Though, seen from the outside it is not very clear what the responsibility / feature
> split is between the xilinx layers but I guess that does not matter as long as Xilinx
> can keep track.
> Either way, my main concern is the meta-xilinx-bsp layer and how to use it the
> OSS YP way independently from the other xilinx layers.
> 
> 
> >If you use yocto based release branches (rocko, pyro, etc), meta-xilinx-bsp should
> fairly work.
> >
> >Look out for patches that are being posted for sumo branch.
> 
> Ok. We see a handful of commits on the meta-xilinx master since the rocko
> release. I assume master will eventually be branched out into a sumo branch?

Yes, patches are being sent on mailing list and under review. 
You can review them, test them and ACK

> Can we expect the current Xilinx rocko branch in meta-xilinx to be updated to use
> updated (v2018.1?) versions of linux-xlnx, u-boot-xlnx, arm-trusted-firmware, and
> pmu-firmware?

No, we are not going to upgrade kernel versions for Rocko release.

> Or do we have to wait for a sumo branch? If so, when would such a branch be
> available for a test run?

Look for patches on mailing list. Sumo was available from upstream on 05/14, we usually take about 3-4 weeks after the upstream release.

> It would be nice if Xilinx would maintain a wiki documenting the schedule for
> Xilinx YP branches. Xilinx wiki is good stuff!
> 

Will try to document the process on wiki

> While we now realize trying to bring up Xilinx rel-v2018.1 with YP rocko is a
> mistake we do feel we are very close to having a bootable zcu102 system with our
> effort - it's just that we are running into a hard showstopper issue with the arm-
> trusted-firmware/pmu-firmware combination and we are not sure what the issue
> is except something goes wrong with the communication between ATF and PMU
> when the ATF requests the chip ID from the PMU during boot. See my findings
> here: https://lists.yoctoproject.org/pipermail/meta-xilinx/2018-May/003809.html
> <https://lists.yoctoproject.org/pipermail/meta-xilinx/2018-May/003809.html>
> 
> I don't know how far Xilinx is preparing for the next YP rocko/sumo update but I
> assume you will have to fix the same issue - perhaps it would possible to get some
> support from Xilinx to resolve this issue now?

Check the patches for pmu-firmware upgrade. 
https://lists.yoctoproject.org/pipermail/meta-xilinx/2018-May/003841.html

You will still need Mike's workaround to patch PMUFW for SDBoot to be ported. I tested on 2018.1 branch and it works. I did not use his second patch 0002-pm_cfg_obj.c, just the first one which embeds the config inside PMUFW

https://github.com/topic-embedded-products/meta-topic/blob/master/recipes-bsp/pmu-firmware/pmu-firmware_2017.1.bbappend

> I mean, if Xilinx helps fix the issue now it can go directly into the next rocko/sumo
> update for everyone to benefit now and when next release is due. I suspect this is
> an issue which can be quickly resolved with the help of Xilinx ATF/PMU experts.
> Also, we have backtracked through the mailing list archive but couldn't find any
> patches that seems related to the ATF/PMU issue we see on the zcu102.
> 

SPL has a gap compared to FSBL for loading pmu configuration. 
I don't think this is resolved in SPL or if there is a way to fix in SPL. 
You will need the above patch for SD-boot

> 
> > We have been sending patches to mailing list to update the
> > meta-xilinx-bsp master for Sumo release, this will include newer
> > kernel etc
> 
> We are perhaps a front runner here but the very reason we tried to bring up the
> rel-v2018.1 with YP rocko is that we wanted to use the new v2018.1 components
> to benefit from the various bug fixes and improvements. In particular we wanted
> to try out the much never kernel (4.9 vs 4.14) because we see some instability
> issues with the Xilinx qspi-nor driver when running some UBI stress tests. First we
> tried back porting most of the qspi-nor changes from 4.14 to 4.9 but that didn't fix
> the instability. There are some deeper infrastructure changes in the qspi-nor driver
> (e.g. change from mmio to ioctl API calls) that we coulnd't easily backport so that
> put us on the path of trying to upgrade everything to the new rel-v2081.1 to get
> to the new kernel.
> 

Can we start a new thread on this issue please? 
We would like to understand the failure and try to provide a fix 
Not sure if this helps:
https://github.com/Xilinx/linux-xlnx/commit/1859edf811e7960ddc73a08b2e8bac93be463a87

> 
> >> I understand that Xilinx only tests meta-petalinux in combination
> >> with the rest but I really want to make a plea for Xilinx to also
> >> start testing meta-xilinx-bsp in the aim to keep it independent and functional.
> >
> >Xilinx tools release cannot be independent, the workflow is different and needs
> all the layers.
> >If you continue to use rel-v2018 (release branches), then the recommendation is
> to use all layers.
> >If you use yocto based release branches, meta-xilinx-bsp should fairly
> >work ( except master as changes to OE-Core master might cause
> >breakages)
> 
> Understood.
> 
> 
> >The plan to collapse all layers under meta-xilinx is proposed, and the first step of
> splitting has happened.
> >We will continue to make all the necessary changes to have all layers under
> meta-xilinx later this year.
> >There are certain dependencies which needs to be addressed before having
> meta-petalinux in meta-xilinx umbrella.
> >We are working with OE-core to resolve some of these depedencies
> >https://lists.yoctoproject.org/pipermail/meta-xilinx/2017-November/0033
> >09.html
> 
> Great - as long as Xilinx is committed to continue putting out Xilinx YP branch
> releases so that we can use the clean cut YP / meta-xilinx-bsp combination
> without any involvement of meta-xilinx-tools and meta-xilinx-petalinux, then I
> think both the community and many customers will be happy.
> 

Patches are always welcome!

Thanks,
Manju