[yocto] liblzma: memory allocation failed

[Peter Bergin]

Hi,

during the task do_package_write_rpm I get the error "liblzma: Memory 
allocation failed". It happens during packaging of binary RPM packages. 
The root cause seems to be the host environment that is used in our 
project. We run our builds on a big server with 32 cores and 256GB of 
physical RAM but each user has a limit of virtual memory usage to 32GB 
(ulimit -v). The packaging in rpm-native has been parallelized in the 
commit 
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
What seems to happen is that rpm-native put up 32 parallel tasks with 
'#pragma omp', each task is using liblzma that also put up 32 tasks for 
the compression work. The memory calculations in liblzma is based on the 
amount of physical RAM but as the user is limited by 'ulimit -v' we get 
into a OOM situation in liblzma.

Here is the code snippet from rpm-native/build/pack.c where it happens:

     #pragma omp parallel
     #pragma omp single
     // re-declaring task variable is necessary, or older gcc versions will produce code that segfaults
     for (struct binaryPackageTaskData *task = tasks; task != NULL; task = task->next) {
         if (task != tasks)
         #pragma omp task
         {
             task->result = packageBinary(spec, task->pkg, cookie, cheating, &(task->filename), buildTime, buildHost);
             rpmlog(RPMLOG_NOTICE, _("Finished binary package job, result %d, filename %s\n"), task->result, task->filename);
         }
     }


Steps to reproduce is to set 'ulimit -v' in your shell to, for example, 
1/8 of the amount of physical RAM and then build for example 
glibc-locale. I have tested this with rocko. If the '#pragma omp' 
statements in code snippet above is removed the problem is solved. But 
that not good as the parallel processing speed up the process.

Is the host environment used here with restricted virtual memory 
supported by Yocto? If it is, someone that have any suggestion for a 
solution on this issue?

Best regards,
/Peter