[OE-core] broken sstate archives

[André Draszik]

On Tue, 2018-10-30 at 16:49 +0000, Richard Purdie wrote:
> On Tue, 2018-10-30 at 15:45 +0000, André Draszik wrote:
> > Having updated poky from 028a292001f64ad86c6b960a05ba1f6fd72199de
> > (end-of
> > July) to 3b77e7b7852549dcfbc426d4ce258e6e857c0acd (mid October), at
> > least
> > two broken sstate archives have been created:
> > 
> > * the sstate archive for update-rc.d package_write_ipk contains a
> > broken
> >   main package update-rc.d_0.8-r0_all.ipk
> > * zlib populate_sysroot has a broken sysroot-
> > destdir/lib/libz.so.1.2.11
> > 
> > For the update-rc.d case:
> > * tar tzvvf displays a reasonable size for all files inside the
> > sstate
> >   archive
> > * tar xzf extracts all files and sets a size on update-rc.d_0.8-
> > r0_all.ipk,
> >   but it's all NULs, and hence is broken
> > * for those who know midnight commander, it's 'open' displays a size
> > of
> >   0 bytes for update-rc.d_0.8-r0_all.ipk in the first place
> > 
> > 
> > For the broken zlib sstate archive, things are similar, additionally:
> > * the zlib ipk packages (and their contents) contained inside
> >   sstate_zlib_*_package_write_ipk.tgz are actually not broken
> > 
> > 
> > The original (first) build resulting from my poky.git update had
> > actually
> > completed successfully. It is only subsequent builds trying to use
> > the
> > generated sstate artefacts that now don't work.
> > 
> > I can't say for sure whether or not other sstate artefacts are
> > broken, too.
> > 
> > 
> > Any ideas how this could have happened? Have similar issues been seen
> > before?
> 
> I've not seen/heard of any reports of that before and it is worrying.
> The question is can you reproduce it? As things stand that is a little
> bit of a hard one to replicate/debug :(
> 
> Which host OS and filesystem was it?

I now have continuous rebuilds of above described scenario on two different
physical machines and am getting broken sstate archives in a similar way to
the description above every now and then for varying recipes and/or varying
files inside a given sstate archive on one of the two machines (the faster
one) so far.

Unfortunately I don't know which of the two physical machines was involved
in my original description.

The faster machine is running Debian 9.5 and kernel 4.9.110-3+deb9u5 the
other Debian unstable and kernel 4.18.8-1 - builds happen inside a Debian 9
docker container in any case. Both machines use btrfs.


Nevertheless I'm tempted to rule out hardware issues, though because:
* The sstate archives are either just archiving the output of
  do_populate_sysroot (which itself is just hard-linking the output of
  do_install), and that same do_populate_sysroot output is also used during
  compilation of dependent recipes
* Or the sstate is created by archiving the output of do_package_write_ipk,
  where again the same IPKs (hard-linked) are used to build the final image
* The defect seems to be inside the .tar, but an sstate archive is a
  .tar.gz, and no temporary uncompressed .tar is stored in the file-system
  to create the compressed .tar.gz

Please correct me if this reasoning is flawed.


I'll play with a few bits and variations to try to narrow things down, but
given the nature it'll likely take a while to get more insights.


Cheers,
Andre'