[OE-core] Very large size of libraries in core-image-minimal rootfs

[Adrian Bunk]

On Tue, Nov 26, 2019 at 10:17:09AM -0800, Khem Raj wrote:
> On Mon, 2019-11-25 at 23:10 +0200, Adrian Bunk wrote:
> > On Mon, Nov 25, 2019 at 07:26:58PM +0000, Ross Burton wrote:
> > > On 25/11/2019 18:37, Ankur Tyagi wrote:
> > > > Hi,
> > > > 
> > > > Based upon "thud" branch, I created core-image-minimal for
> > > > am335x-evm
> > > > board and resulting image size is very big(71M)
> > > 
> > > If disk size is important then a good first step is to use musl
> > > instead of
> > > glibc:
> > > 
> > > TCLIBC = "musl"
> > > ...
> > 
> > musl is mainly useful for tiny systems, a 40 MB partition is such a
> > huge
> > amount of space that the size of the C library no longer matters
> > much.
> > 
> > The size benefit of replacing glibc with musl is around 2 MB.
> 
> I would refrain from labeling as useful for tiny systems. since
> static size is one thing, there is DRAM behaviour which is equally
> important, how malloc and other resource oriented algorithms behave is
> important as well.

It is not a clear picture once you ignore "glibc is old and bloated" 
claims, and I am not aware of any reliable objective general overview in 
which cases which C library (or a 3rd party memory allocator) is better.

> secondly, if I have 40MB partition, one might rather want to use extra
> space for implementing features in their product than spend it for
> enabling the system if they can.

If it ain't broken don't fix it.

For the case in question there was already the point that two copies
of libstdc++ were installed, which is a size penalty on the order of 
magnitude of the glibc/musl difference.

And for such huge partitions -Os should bring more benefits than musl,
so that's a more reasonable first choice for size optimization.

> there are many other design considerations where musl might make more
> sense

Which of these are not covered by linking only your application
with musl?

It is not clear that all this outweights the constant pains of forcing 
people to fix the steady stream of new musl issues for all recipes.

> > A partition with an -Os build of Yocto with glibc+systemd (sic)
> > is around 9 MB, together with bootloader and kernel this all
> > fits in a 16 MB flash. Use compression for kernel and root
> > filesystem and everything fits in a 8 MB flash.
> 
> -Os is not as well tested option as -O2 is. So we should always
> recommend it with grain of salt, speaking from experience there are
> bugs that will show up with Os which you wont see with O2 and leave a
> bad taste for user

Where is the grain of salt when recommending musl?

I would consider -Os less risky than musl, especially considering some
of the OE-only patching for software noone else tries to use with musl.
But neither is a huge risk as long as you stay away from known-broken 
musl cases like systemd or elfutils.

On a more general note it is weird how much effort is spent on getting 
things to work with musl, but noone seems to care that this builds and 
works with -Os.

The combinations that are most relevant in practice are:
- normal: -O2 + glibc + systemd
- tiny: -Os + musl + busybox init

Using musl with -O2 or with systemd is far from the primary usecase
of using musl for size benefits.

The other advantage of -Os is that it is also useful for non-tiny 
systems since the benefits scale with the size of the filesystem.
It is a nice knob to fix kernel or userspace space problems that
does not change the API or ABI.

cu
Adrian