[OE-core] [PATCH 2/2] rpm: Assume a max limit of 1024 open files
Peter Kjellerstedt
peter.kjellerstedt at axis.com
Fri May 11 10:39:51 UTC 2018
> -----Original Message-----
> From: Alexander Kanavin [mailto:alexander.kanavin at linux.intel.com]
> Sent: den 11 maj 2018 07:44
> To: Peter Kjellerstedt <peter.kjellerstedt at axis.com>; openembedded-
> core at lists.openembedded.org
> Subject: Re: [OE-core] [PATCH 2/2] rpm: Assume a max limit of 1024 open
> files
>
> On 05/11/2018 01:20 AM, Peter Kjellerstedt wrote:
> > If sysconf(_SC_OPEN_MAX) is much greater than the usual 1024 (for
> > example inside a Docker container), the performance drops sharply.
>
> Please do not drop the link to the upstream bug when replacing one
> patch with another.
Right, forgot about that.
> You can also simply amend my patch and add your SOB.
Well, I thought it was more clear this way, since your patch removed
the functionality that closed the files, whereas mine modifies it.
> Also, why assume specifically 1024? We either need to close all the
> open files, or none, and your patch creates a situation where only
> some of the files may be closed. I'd say we should drop those two
> code snippets altogether instead of hardcoding 1024 into them.
Well, my expectation was that there is a reason that they want to make
sure that all open file descriptors have FD_CLOEXEC set so that they
are closed when forking. At the same time I don't assume rpm to actually
have more than 1024 files open at one time, so even if using
sysconf(_SC_OPEN_MAX) is the right thing to do, using 1024 should work
in practice.
However, I gave this some more thought, and maybe it would be better to
let bitbake set the soft limit for max open files to, e.g., 1024. That
way we would not need to modify the code in rpm as it would adapt itself
automatically. I tried adding the following to bitbake_main and it worked:
import resource
# Set the maximum number of open files as there are performance problems
# with, e.g., rpm if this is unlimited (which it may be if running inside
# a Docker container).
(soft, hard) = resource.getrlimit(resource.RLIMIT_NOFILE)
soft = int(os.environ.get("BB_LIMIT_NOFILE", "1024"))
resource.setrlimit(resource.RLIMIT_NOFILE, (min(soft, hard), hard))
I do not know if bitbake_main is the right place for this, or if there is
some more appropriate location for it. Richard?
I can send a new patch to add this, once I know where it should be added.
> Oh, and: please do comment in the upstream bug (e.g. with your
> benchmarks from the cover letter), otherwise upstream may do nothing
> because they're not using Docker.
Will do. I think suggesting that rpm should set the limit itself, as per
my suggestion for bitbake above, might be an appropriate solution that
they can accept.
> Alex
//Peter
More information about the Openembedded-core
mailing list