[yocto] liblzma: memory allocation failed

[Manjukumar Harthikote Matha]

Hi Peter,

> -----Original Message-----
> From: yocto-bounces at yoctoproject.org [mailto:yocto-bounces at yoctoproject.org]
> On Behalf Of Peter Bergin
> Sent: Sunday, September 16, 2018 1:41 PM
> To: yocto at yoctoproject.org
> Subject: [yocto] liblzma: memory allocation failed
> 
> Hi,
> 
> during the task do_package_write_rpm I get the error "liblzma: Memory
> allocation failed". It happens during packaging of binary RPM packages.
> The root cause seems to be the host environment that is used in our
> project. We run our builds on a big server with 32 cores and 256GB of
> physical RAM but each user has a limit of virtual memory usage to 32GB
> (ulimit -v). The packaging in rpm-native has been parallelized in the
> commit
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-
> devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3.
> What seems to happen is that rpm-native put up 32 parallel tasks with
> '#pragma omp', each task is using liblzma that also put up 32 tasks for
> the compression work. The memory calculations in liblzma is based on the
> amount of physical RAM but as the user is limited by 'ulimit -v' we get
> into a OOM situation in liblzma.
> 
> Here is the code snippet from rpm-native/build/pack.c where it happens:
> 
>      #pragma omp parallel
>      #pragma omp single
>      // re-declaring task variable is necessary, or older gcc versions will produce code
> that segfaults
>      for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) {
>          if (task != tasks)
>          #pragma omp task
>          {
>              task->result = packageBinary(spec, task->pkg, cookie, cheating, &(task-
> >filename), buildTime, buildHost);
>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, filename
> %s\n"), task->result, task->filename);
>          }
>      }
> 
> 
> Steps to reproduce is to set 'ulimit -v' in your shell to, for example,
> 1/8 of the amount of physical RAM and then build for example
> glibc-locale. I have tested this with rocko. If the '#pragma omp'
> statements in code snippet above is removed the problem is solved. But
> that not good as the parallel processing speed up the process.
> 
> Is the host environment used here with restricted virtual memory
> supported by Yocto? If it is, someone that have any suggestion for a
> solution on this issue?
> 

We had seen this issue as well and concluded that some settings in the server was causing this.
See http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146705.html

Thanks,
Manju