[yocto] liblzma: memory allocation failed

[Randy MacLeod]

On 09/16/2018 04:40 PM, Peter Bergin wrote:
> Hi,
> 
> during the task do_package_write_rpm I get the error "liblzma: Memory 
> allocation failed". It happens during packaging of binary RPM packages. 
> The root cause seems to be the host environment that is used in our 
> project. We run our builds on a big server with 32 cores and 256GB of 
> physical RAM but each user has a limit of virtual memory usage to 32GB 
> (ulimit -v). The packaging in rpm-native has been parallelized in the 
> commit 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
> What seems to happen is that rpm-native put up 32 parallel tasks with 
> '#pragma omp', each task is using liblzma that also put up 32 tasks for 

#pragma omp

Tha'ts OpenMP, right? I haven't played with that at all but
it looks like you can limit the number of threads using an
environment variable:
    OMP_NUM_THREADS num
https://www.openmp.org/wp-content/uploads/OpenMP3.0-SummarySpec.pdf

Doing that would be a little ugly but for now at least, there doesn't
seem to be that many packages using such a pragma.

Does that work for your case?

> the compression work. The memory calculations in liblzma is based on the 
> amount of physical RAM but as the user is limited by 'ulimit -v' we get 
> into a OOM situation in liblzma.
> 
> Here is the code snippet from rpm-native/build/pack.c where it happens:
> 
>      #pragma omp parallel
>      #pragma omp single
>      // re-declaring task variable is necessary, or older gcc versions 
> will produce code that segfaults
>      for (struct binaryPackageTaskData *task = tasks; task != NULL; task 
> = task->next) {
>          if (task != tasks)
>          #pragma omp task
>          {
>              task->result = packageBinary(spec, task->pkg, cookie, 
> cheating, &(task->filename), buildTime, buildHost);
>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, 
> result %d, filename %s\n"), task->result, task->filename);
>          }
>      }
> 
> 
> Steps to reproduce is to set 'ulimit -v' in your shell to, for example, 
> 1/8 of the amount of physical RAM and then build for example 
> glibc-locale. I have tested this with rocko. If the '#pragma omp' 
> statements in code snippet above is removed the problem is solved. But 
> that not good as the parallel processing speed up the process.
> 
> Is the host environment used here with restricted virtual memory 
> supported by Yocto? If it is, someone that have any suggestion for a 
> solution on this issue?


This is a little tricky.

 From bitbake's point of view, it's almost like you are building
on a 32 core, 32 GB box and runing out of RAM/swap.
Clearly we would not fix a build that OOMs in that case
(it does seem odd that 32 GB isn't enough ...)

Are you sure that there isn't something else going on?
I have a 24 core machine with 64 GB RAM that never comes
close to such a problem (so I haven't paid attention to RAM usage).


../Randy

> 
> Best regards,
> /Peter
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux