[OE-core] [PATCH] wic: add --reserved-size wks option

Tue Oct 18 11:05:57 UTC 2016

On Tue, Oct 18, 2016 at 01:07:48PM +0200, Maciej Borzęcki wrote:
> On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> > On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
> >> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> >> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
> >> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> >> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
> >> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> >> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
> >> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> >> >> >> >> > Hi Maciej,
> >> >> >> >> >
> >> >> >> >> > There is already --size and --extra-space options.
> >> >> >> >> > Can we get the same or similar result by just using them? Do we really
> >> >> >> >> > need new option for similar purpose?
> >> >> >> >>
> >> >> >> >> --reserved-size serves a different purpose, it establishes an upper
> >> >> >> >> bound on the size of a partition during layout. Unlike
> >> >> >> >> --size/--extra-space does not depend on the size of the filesystem
> >> >> >> >> image.
> >> >> >> >>
> >> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
> >> >> >> >> fixed partition layout (something simple: boot partition, primary &
> >> >> >> >> secondary rootfs partitions, some data partition). Because future
> >> >> >> >> system updates are delivered as filesystem image, I want to make sure
> >> >> >> >> that there is exactly xxx MBs for my current and future rootfs images
> >> >> >> >> (regardless of current image size). Neither --size nor --extra-space
> >> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
> >> >> >> >> this will needlessly create a 200MB rootfs image and if I happen to
> >> >> >> >> cross the 200MB boundary I will not get an error.
> >> >> >> >>
> >> >> >> >> I had a private patch that added --fixed-size to enforce --size, but
> >> >> >> >> that would still end up creating filesystem image to fill the whole
> >> >> >> >> space.
> >> >> >> > I didn't get the difference between enforcing partition size and below
> >> >> >> > implementation. Can you elaborate a bit?
> >> >> >>
> >> >> >> `--fixed-size` was something that I had added to my fork back in 2014,
> >> >> >> even before `--overhead-factor` came in. The problem is that depending
> >> >> >> on a project you might want to have more control over how partitions
> >> >> >> are laid out, or even need to have a fixed layout. Adding
> >> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
> >> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
> >> >> >> say, 200MB large. The downside was that wic would actually create a
> >> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
> >> >> >> wanted to have 200MB gap so that I have some spare space for future
> >> >> >> updates (where update is just a rootfs image you dd to the partition).
> >> >> >>
> >> >> > Thanks for the explanations. Now I got it - reserved size is not a part
> >> >> > of partition, it's a gap between partitions.
> >> >>
> >> >> I might have not been clear enough when explaining. It's not a gap,
> >> >> it's just a container of size --reserved-size listed in MBR/GPT.
> >> >> There's probably a filesystem inside but not necessarily.
> >> >> Graphically it looks as like this:
> >> >>
> >> >>                          --reserved-size
> >> >>                       |----------------------|
> >> >>                       v                      v
> >> >>     +-----------------+----------------------+---------------------+
> >> >>     |..... stuff .....|xxxxxxxxxx            |..... stuff .........|
> >> >>     +-----------------+----------------------+---------------------+
> >> >>                       ^         ^            ^
> >> >>                       |---------|------------|
> >> >>                        --size    --extra-space
> >> >>
> >> >>
> >> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
> >> > formatted, right? It's only formatted if size == reserved_size.
> >> >
> >> >> >
> >> >> > What's the advantage of creating unusable gap over creating partition of
> >> >> > the same size that can be used?
> >> >>
> >> >> Just convenience.
> >> >>
> >> > What's the convenience of having extra space on partition that can't be
> >> > used for data over having it formatted and used?
> >> >
> >> >> > Even if that space is not needed it doesn't harm to have it, does it?
> >> >>
> >> >> I have not seen any negative side effects.
> >> >>
> >> > I do. If user needs that reserved space it's impossible to get it without
> >> > reformatting partition. The free space exists, but can't be used.
> >>
> >> That's not the point and is not aligned with use case I'm trying to solve.
> >>
> >> My case is rather simple, I'm creating an image for SD card that is
> >> deployed in the field. In that particular case, there's a primary and
> >> a secondary (aka. active and inactive) rootfs partitions that are
> >> switched whenever a system update comes in. The update is a file
> >> system image that is copied over to the inactive partition, followed
> >> by a system reboot.
> >>
> >> What I need is the ability to set a certain size of a partition (say
> >> 100MB), regardless of current rootfs size (which may be, say 70MB).
> >> The remaining unused space sets an upper limit on how much the rootfs
> >> may grow in the future (in this example case, it's 30MB). RIght now
> >> the best I can do is to describe a partition like this: `part /
> >> --source rootfs --size 100MB --overhead-factor 1`, hoping that if
> >> rootfs grows beyond 100MB I will somehow still be able to catch that
> >> and that the future images remain size compatible.
> >>
> >> The resulting filesystem inside the partition is larger than what
> >> IMAGE_CMD (ex. IMAGE_CMD_ext4) would give me, because of explicit
> >> --size in kickstart. I would prefer to have something comparable in
> >> size just to avoid later surprises, what implies using defaults.
> >> However, using defaults, means that I cannot control the layout
> >> because it will likely change each time rootfs size gets changed.
> >> There is no `--fixed-size` or other option to enforce specific size.
> >>
> >> Summing up, a simple use case that cannot be currently solved using wic.
> >>
> >> BTW. actually we're missing an ability to enforce --size (i.e.
> >> --fixed-size?) and a method passing an explicit partition offset
> >> inside the disk image (something useful for `--source rawcopy
> >> --no-table` partitions, currently solved with `--align`).
> >>
> > I undertood the problem and I agree that wic doesn't provide a solution.
> >
> > However, instead of making unformatted space I'd propose to format it,
> > i.e. to have --max-size option that would confict with --size and
> > specify upper size limit for the partition. All partition will be
> > formatted and available for data. This is identical to --fixed-size option
> > you've described. This approach would solve the problem you're
> > addressing and it would also make additional space usable.
> >
> > I'd also suggest to rename --size to --min-size and make --size deprecated.
> >
> > Does this make sense to you?
> 
> No strong opinions here, just that deprecating --size might current
> users uneasy.
> 
By deprecting --size I didn't mean removing it completely. We can just
print a warning suggesting usage of other options.

> Perhaps --max-size could be a boolean switch? We could just name it
> --fixed-size (bool, defaults to False), with semantics that if
> --fixed-size is provided, the partition will have size --size,
> occurrence of --overhead-factor or --extra-space will raise an error.
> 
That would work too, but it looks a bit confusing to me to have 2 different
types of size-related options.

I agree on raising error on --overhead-factor and --extra-space.

Frankly I'd make --overhead-factor deprecated too as it's too implicit
from my point of view. May be there are some use cases for it, but I can't
imagine why would someone want free space to be n times bigger than used space.
Anyway, it's totally different story. Let's not talk about it now.

--
Regards,
Ed