[yocto] how to set up PREMIRRORS_prepend to avoid constant git clones?
Gary Thomas
gary at mlbassoc.com
Sat Mar 10 02:52:34 PST 2012
On 2012-03-10 03:35, Robert P. J. Day wrote:
>
> over the years, i've built up a sizable collection of source
> tarballs for use in various projects -- top-level directory called
> "~/dl", about 5G in size with lots and lots of .tgz and .bz2 files and
> so on, and i'd like to confirm the best way to take advantage of all
> that source whenever i configure and build a yocto project.
>
> based on my reading, whenever i create a new project, i immediately
> add the following to the bottom of its local.conf file:
>
> PREMIRRORS_prepend = "\
> git://.*/.* file:///home/rpjday/dl/ \n \
> svn://.*/.* file:///home/rpjday/dl/ \n \
> cvs://.*/.* file:///home/rpjday/dl/ \n \
> ftp://.*/.* file:///home/rpjday/dl/ \n \
> http://.*/.* file:///home/rpjday/dl/ \n \
> https://.*/.* file:///home/rpjday/dl/ \n"
>
> and that certainly makes a massive difference -- whenever the fetching
> starts, matching tarballs will be located in my "dl" directory, and
> the project's downloads directory will be populated with symlinks to
> the actual tarballs, as in:
>
> vala-0.14.2.tar.xz -> /home/rpjday/dl/vala-0.14.2.tar.xz
>
> and after the download, i sneak a peek at whatever newer tarballs
> needed to be downloaded, and quietly copy them over to "dl" to avoid
> ever downloading them again, and that's worked well so far. so, the
> questions:
>
> 1) does this sound reasonable? i know about using DL_DIR and perhaps
> sharing that across projects, but i rather like the idea of an
> application-agnostic repository of straight tarballs and zip files
> that any build system is allowed to pilfer from (especially since i
> can make it read-only).
>
> 2) am i missing any fetch protocols from that list?
>
> 3) how do i set it up to avoid repeated git clones?
>
> it seems that, while i can avoid fetching simple tarballs, i still
> end up doing the same "git clone" operations over and over. am i just
> missing something obvious?
All you really need are these lines:
SOURCE_MIRROR_URL ?= "file://${COREBASE}/sources/"
INHERIT += "own-mirrors"
Adjust the SOURCE_MIRROR_URL to wherever you want the files. I also always add
BB_GENERATE_MIRROR_TARBALLS = "1"
This will let the fetcher make a git2_XXX tarball, once the repo has been cloned.
My understanding is that the git (and other SCM) fetcher works like this:
* If there is a git2_XXX tar file in the DL_DIR (or mirror), unpack it
into ${BUILD}/downloads/git2
* Check to see if the desired ${SRCREV} is present. If not, do a 'git pull'
to update the local clone and then rebuild the git2_XXX tarball.
This way, you only have to clone once - just copy any generated tarballs back
to your source mirror.
Finally, if you set
BB_NO_NETWORK = "1"
the build will fail if bitbake *ever* has to go to the network, i.e. not find
the desired files in the source mirror or DL_DIR. This is a great way to insure
that you've got what you need locally. I use this all the time to make sure I
have a fully populated local cache of sources.
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
More information about the yocto
mailing list