[yocto] liblzma: memory allocation failed

[Peter Bergin]

Hi Randy,

On 2018-09-17 06:25, Randy MacLeod wrote:
> On 09/16/2018 04:40 PM, Peter Bergin wrote:
>> Hi,
>>
>> during the task do_package_write_rpm I get the error "liblzma: Memory 
>> allocation failed". It happens during packaging of binary RPM 
>> packages. The root cause seems to be the host environment that is 
>> used in our project. We run our builds on a big server with 32 cores 
>> and 256GB of physical RAM but each user has a limit of virtual memory 
>> usage to 32GB (ulimit -v). The packaging in rpm-native has been 
>> parallelized in the commit 
>> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/recipes-devtools/rpm?id=84e0bb8d936f1b9094c9d5a92825e9d22e1bc7e3. 
>> What seems to happen is that rpm-native put up 32 parallel tasks with 
>> '#pragma omp', each task is using liblzma that also put up 32 tasks for 
>
> #pragma omp
>
> Tha'ts OpenMP, right? I haven't played with that at all but
> it looks like you can limit the number of threads using an
> environment variable:
>    OMP_NUM_THREADS num
> https://www.openmp.org/wp-content/uploads/OpenMP3.0-SummarySpec.pdf
>
> Doing that would be a little ugly but for now at least, there doesn't
> seem to be that many packages using such a pragma.
>
> Does that work for your case?
>
Yes, it's OpenMP. I tried '#pragma omp parallel num_thread(4)' and it 
worked as a workaround. On the failing server the build succeeded. The 
problem is to get this as a generic solution based on the host settings 
because the #pragma is a compiler directive. But for sure we can make a 
bbappend on this to get it working on our host.

>> the compression work. The memory calculations in liblzma is based on 
>> the amount of physical RAM but as the user is limited by 'ulimit -v' 
>> we get into a OOM situation in liblzma.
>>
>> Here is the code snippet from rpm-native/build/pack.c where it happens:
>>
>>      #pragma omp parallel
>>      #pragma omp single
>>      // re-declaring task variable is necessary, or older gcc 
>> versions will produce code that segfaults
>>      for (struct binaryPackageTaskData *task = tasks; task != NULL; 
>> task = task->next) {
>>          if (task != tasks)
>>          #pragma omp task
>>          {
>>              task->result = packageBinary(spec, task->pkg, cookie, 
>> cheating, &(task->filename), buildTime, buildHost);
>>              rpmlog(RPMLOG_NOTICE, _("Finished binary package job, 
>> result %d, filename %s\n"), task->result, task->filename);
>>          }
>>      }
>>
>>
>> Steps to reproduce is to set 'ulimit -v' in your shell to, for 
>> example, 1/8 of the amount of physical RAM and then build for example 
>> glibc-locale. I have tested this with rocko. If the '#pragma omp' 
>> statements in code snippet above is removed the problem is solved. 
>> But that not good as the parallel processing speed up the process.
>>
>> Is the host environment used here with restricted virtual memory 
>> supported by Yocto? If it is, someone that have any suggestion for a 
>> solution on this issue?
>
>
> This is a little tricky.
>
> From bitbake's point of view, it's almost like you are building
> on a 32 core, 32 GB box and runing out of RAM/swap.
> Clearly we would not fix a build that OOMs in that case
> (it does seem odd that 32 GB isn't enough ...)
>
> Are you sure that there isn't something else going on?
> I have a 24 core machine with 64 GB RAM that never comes
> close to such a problem (so I haven't paid attention to RAM usage).
>
I'm pretty sure I have narrowed down the root cause to the restriction 
of virtual memory and that liblzma base its memory calculations on 
physical RAM.

To prove this I added a printout in rpm-native/rpmio/rpmio.c and the 
function lzopen_internal.

         uint64_t memory_usage = lzma_stream_encoder_mt_memusage(&mt_options);
	rpmlog(RPMLOG_NOTICE, "DBG: memory_usage %lu\n", memory_usage);


The value of memory_usage is the same regardless of which 'ulimit -v' 
value I set. On the host with 256GB of physical RAM and 32GB of virtual 
memory, memory_usage is ~5.1GB. On another host with 16GB of physical 
RAM I get memory_usage of ~660MB.

I guess you have not seen this kind of failure if you not have 
restricted virutal memory on your host. If you want to try to reproduce 
this set 'ulimit -v 8388608' (8GB) in your shell and then 'bitbake 
glibc-locale -c package_write_rpm -f'.

Best regards,
/Peter

>
> ../Randy
>
>>
>> Best regards,
>> /Peter
>>
>>
>>
>>
>
>