[yocto] How to maintain a global DL_DIR?

[Gary Thomas]

On 2016-08-16 18:38, Robert P. J. Day wrote:
> On Tue, 16 Aug 2016, Khem Raj wrote:
>
>>> On Aug 16, 2016, at 8:37 AM, Robert P. J. Day <rpjday at crashcourse.ca> wrote:
>>>
>>> On Tue, 16 Aug 2016, Patrick Doyle wrote:
>>>
>>>> As I am playing with several different Yocto/OE projects, I thought
>>>> it would be nice to set up a global downloads directory.
>>>>
>>>> Also, being a somewhat forgetful sort, I thought it would be nice if
>>>> I could set DL_DIR in my .bashrc and not have to remember to edit
>>>> conf/local.conf each time I follow the directions from a third party
>>>> site to set up their Yocto based distribution.
>>>>
>>>> Can I do this?  If so, how?
>>>
>>>  if you're talking about creating a local source mirror for tarballs
>>> to be shared across multiple projects, i taught an OE/YP course last
>>> week based on a digi ConnectCore 6 dev kit, and i wrote some wiki
>>> pages for it, including this one:
>>>
>>> http://www.crashcourse.ca/wiki/index.php/CC6_local_source_mirror
>>
>> a small nit, perhaps its better to use rsync instead of cp.
>
>   good point, i will adjust that later. anything else i overlooked?

I use the attached script to manage my download mirror.  There are
two differences with the one Robert proposed:

* It copies all files, regardless of name.  This is important if you
   want your download mirror to be useful in a no-network environment
   (which seems to be very important to some of my larger customers)
   I normally run my builds with BB_NO_NETWORK="1" which lets me know
   immediately if my download mirror needs updating.  I disable BB_NO_NETWORK
   to download the new images, then update the mirror and turn it back
   on again.
* Once uploaded to the mirror, files are replaced by symbolic links
   just as if the file had been present in the first place.  This has
   two benefits - the downloads directory stays as small as possible
   and the next time the 'save_downloads' script runs, it won't have
   to bother with files already uploaded.  I use this process on a
   number of build machines, some with Poky/Yocto build trees that have
   been around for a long time (sometimes years) and keeping all those
   old files in 'downloads' can really add up.

However you manage it, using 'own-mirrors' with a local mirror really
improves the speed of builds - think about having to re-download the
git tree for the RaspberryPi firmware (>3.5GB) for every new build!

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
-------------- next part --------------
#! /bin/bash

MIRROR=/work/misc/Poky/sources

cd downloads
find . -maxdepth 1 -type f | sed -e 's;^\./;;' | grep -v '.lock$' | grep -v '.done$' >/tmp/files.$$
LEN=`ls -l /tmp/files.$$ | awk '{print $5}'`
if [ "${LEN}" == "0" ]; then
    exit 0
fi
rsync -auv --files-from=/tmp/files.$$ . ${MIRROR}
if [ $? == 0 ]; then
    read -p "Upload successful - purge files(y/n)? " PURGE
    if [ "${PURGE}" == "y" ]; then
	xargs -n1 -t rm </tmp/files.$$
	xargs -n1 -t -I\{} ln -s ${MIRROR}/\{} . </tmp/files.$$
    fi
fi