[OE-core] [PATCH] wic: add --reserved-size wks option

Tue Oct 18 11:07:48 UTC 2016

On Tue, Oct 18, 2016 at 12:17 PM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
> On Tue, Oct 18, 2016 at 12:24:55PM +0200, Maciej Borzęcki wrote:
>> On Tue, Oct 18, 2016 at 11:27 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
>> > On Tue, Oct 18, 2016 at 11:24:14AM +0200, Maciej Borzęcki wrote:
>> >> On Tue, Oct 18, 2016 at 10:38 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
>> >> > On Tue, Oct 18, 2016 at 10:37:22AM +0200, Maciej Borzęcki wrote:
>> >> >> On Tue, Oct 18, 2016 at 9:31 AM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
>> >> >> > On Mon, Oct 17, 2016 at 04:46:00PM +0200, Maciej Borzęcki wrote:
>> >> >> >> On Mon, Oct 17, 2016 at 3:22 PM, Ed Bartosh <ed.bartosh at linux.intel.com> wrote:
>> >> >> >> > Hi Maciej,
>> >> >> >> >
>> >> >> >> > There is already --size and --extra-space options.
>> >> >> >> > Can we get the same or similar result by just using them? Do we really
>> >> >> >> > need new option for similar purpose?
>> >> >> >>
>> >> >> >> --reserved-size serves a different purpose, it establishes an upper
>> >> >> >> bound on the size of a partition during layout. Unlike
>> >> >> >> --size/--extra-space does not depend on the size of the filesystem
>> >> >> >> image.
>> >> >> >>
>> >> >> >> For instance, assume I'm creating an image for SD card/eMMC with a
>> >> >> >> fixed partition layout (something simple: boot partition, primary &
>> >> >> >> secondary rootfs partitions, some data partition). Because future
>> >> >> >> system updates are delivered as filesystem image, I want to make sure
>> >> >> >> that there is exactly xxx MBs for my current and future rootfs images
>> >> >> >> (regardless of current image size). Neither --size nor --extra-space
>> >> >> >> can do that. I could use, say `--size 200 --overhead-factor 1`, but
>> >> >> >> this will needlessly create a 200MB rootfs image and if I happen to
>> >> >> >> cross the 200MB boundary I will not get an error.
>> >> >> >>
>> >> >> >> I had a private patch that added --fixed-size to enforce --size, but
>> >> >> >> that would still end up creating filesystem image to fill the whole
>> >> >> >> space.
>> >> >> > I didn't get the difference between enforcing partition size and below
>> >> >> > implementation. Can you elaborate a bit?
>> >> >>
>> >> >> `--fixed-size` was something that I had added to my fork back in 2014,
>> >> >> even before `--overhead-factor` came in. The problem is that depending
>> >> >> on a project you might want to have more control over how partitions
>> >> >> are laid out, or even need to have a fixed layout. Adding
>> >> >> `--fixed-size` would had a similar effect to what `--overhead-factor
>> >> >> 1` does right now. Combined with `--size` would ensure that rootfs is
>> >> >> say, 200MB large. The downside was that wic would actually create a
>> >> >> 200MB rootfs, something that is not really necessary. In fact, I only
>> >> >> wanted to have 200MB gap so that I have some spare space for future
>> >> >> updates (where update is just a rootfs image you dd to the partition).
>> >> >>
>> >> > Thanks for the explanations. Now I got it - reserved size is not a part
>> >> > of partition, it's a gap between partitions.
>> >>
>> >> I might have not been clear enough when explaining. It's not a gap,
>> >> it's just a container of size --reserved-size listed in MBR/GPT.
>> >> There's probably a filesystem inside but not necessarily.
>> >> Graphically it looks as like this:
>> >>
>> >>                          --reserved-size
>> >>                       |----------------------|
>> >>                       v                      v
>> >>     +-----------------+----------------------+---------------------+
>> >>     |..... stuff .....|xxxxxxxxxx            |..... stuff .........|
>> >>     +-----------------+----------------------+---------------------+
>> >>                       ^         ^            ^
>> >>                       |---------|------------|
>> >>                        --size    --extra-space
>> >>
>> >>
>> > Ah, I'm wrong again. It's a partition size limit, but it's not necessary
>> > formatted, right? It's only formatted if size == reserved_size.
>> >
>> >> >
>> >> > What's the advantage of creating unusable gap over creating partition of
>> >> > the same size that can be used?
>> >>
>> >> Just convenience.
>> >>
>> > What's the convenience of having extra space on partition that can't be
>> > used for data over having it formatted and used?
>> >
>> >> > Even if that space is not needed it doesn't harm to have it, does it?
>> >>
>> >> I have not seen any negative side effects.
>> >>
>> > I do. If user needs that reserved space it's impossible to get it without
>> > reformatting partition. The free space exists, but can't be used.
>>
>> That's not the point and is not aligned with use case I'm trying to solve.
>>
>> My case is rather simple, I'm creating an image for SD card that is
>> deployed in the field. In that particular case, there's a primary and
>> a secondary (aka. active and inactive) rootfs partitions that are
>> switched whenever a system update comes in. The update is a file
>> system image that is copied over to the inactive partition, followed
>> by a system reboot.
>>
>> What I need is the ability to set a certain size of a partition (say
>> 100MB), regardless of current rootfs size (which may be, say 70MB).
>> The remaining unused space sets an upper limit on how much the rootfs
>> may grow in the future (in this example case, it's 30MB). RIght now
>> the best I can do is to describe a partition like this: `part /
>> --source rootfs --size 100MB --overhead-factor 1`, hoping that if
>> rootfs grows beyond 100MB I will somehow still be able to catch that
>> and that the future images remain size compatible.
>>
>> The resulting filesystem inside the partition is larger than what
>> IMAGE_CMD (ex. IMAGE_CMD_ext4) would give me, because of explicit
>> --size in kickstart. I would prefer to have something comparable in
>> size just to avoid later surprises, what implies using defaults.
>> However, using defaults, means that I cannot control the layout
>> because it will likely change each time rootfs size gets changed.
>> There is no `--fixed-size` or other option to enforce specific size.
>>
>> Summing up, a simple use case that cannot be currently solved using wic.
>>
>> BTW. actually we're missing an ability to enforce --size (i.e.
>> --fixed-size?) and a method passing an explicit partition offset
>> inside the disk image (something useful for `--source rawcopy
>> --no-table` partitions, currently solved with `--align`).
>>
> I undertood the problem and I agree that wic doesn't provide a solution.
>
> However, instead of making unformatted space I'd propose to format it,
> i.e. to have --max-size option that would confict with --size and
> specify upper size limit for the partition. All partition will be
> formatted and available for data. This is identical to --fixed-size option
> you've described. This approach would solve the problem you're
> addressing and it would also make additional space usable.
>
> I'd also suggest to rename --size to --min-size and make --size deprecated.
>
> Does this make sense to you?

No strong opinions here, just that deprecating --size might current
users uneasy.

Perhaps --max-size could be a boolean switch? We could just name it
--fixed-size (bool, defaults to False), with semantics that if
--fixed-size is provided, the partition will have size --size,
occurrence of --overhead-factor or --extra-space will raise an error.

Cheers,
-- 
Maciej Borzecki
RnDity