[OE-core] [PATCHv2] wic: fix images build in parallel

[Paul Barker]

On Mon, 13 Jan 2020 at 14:12, Maxim Uvarov <maxim.uvarov at linaro.org> wrote:
>
> On Mon, 13 Jan 2020 at 17:00, Paul Barker <pbarker at konsulko.com> wrote:
> >
> > On Mon, 13 Jan 2020 at 13:57, Maxim Uvarov <maxim.uvarov at linaro.org> wrote:
> > >
> > > On Mon, 13 Jan 2020 at 16:31, Paul Barker <pbarker at konsulko.com> wrote:
> > > >
> > > > On Mon, 13 Jan 2020 at 13:08, Maxim Uvarov <maxim.uvarov at linaro.org> wrote:
> > > > >
> > > > > OE wic plugins create temporary file with the index of the line
> > > > > tmp file name. This causes race in case several builds run in time.
> > > > > Add more entropy as timestamp to remove this race.
> > > >
> > > > How would two wic images to be built in parallel with the same work
> > > > directory? To my understanding an image recipe only supports building
> > > > a single wic image.
> > > >
> > > > Thanks,
> > > > Paul
> > >
> > > bitbake image1 image2 image3
> > > all images build .wics and use about the same files, like firmware.
> > > Issue is similar to that:
> > > https://www.yoctoproject.org/pipermail/yocto/2018-June/041373.html
> >
> > Each image has its own work directory though.
> >
> > I'll take a look in more detail later today or tomorrow, if wic is
> > writing temporary files outside of the work directory then that's a
> > bug and should be fixed.
>
> Thanks. I saw bug in rawcopy plugin. All other places were patched due
> to the same code.  I guess then for rawcopy temp files are in
> DEST_IMAGE_DIR (which is common) instead of WORKDIR.

I can't see how these files can possibly collide across parallel image
builds. In the lines you changed in your patch, cr_workdir passed in
as an argument set to imager.workdir (in
scripts/lib/wic/plugins/imager/direct.py) which is a temporary
directory created under options.outdir. image_types_wic.bbclass passes
the -o option to wic to set the outdir to something under ${WORKDIR}.
And different images have different WORKDIR paths. So things should be
safe.

Which branch & version of poky or oe-core are you using?

DEST_IMAGE_DIR is not defined in openembedded-core. Are you sure
that's the right variable name?

Could you provide full bitbake output showing the exact error message
when this collision happens?

And lastly if you want to look into this in more detail maybe add some
print statements just after each of those lines you patched to print
the full paths used so we can see if they step outside WORKDIR. I
think doing this on your side with your exact wks files that are
causing errors will give the most useful output.

Thanks,
Paul