[OE-core] broken sstate archives

[richard.purdie at linuxfoundation.org]

On Thu, 2018-11-01 at 12:22 +0000, André Draszik wrote:
> On Tue, 2018-10-30 at 16:49 +0000, Richard Purdie wrote:
> > I've not seen/heard of any reports of that before and it is
> > worrying.
> > The question is can you reproduce it? As things stand that is a
> > little
> > bit of a hard one to replicate/debug :(
> > 
> > Which host OS and filesystem was it?
> 
> I now have continuous rebuilds of above described scenario on two
> different
> physical machines and am getting broken sstate archives in a similar
> way to
> the description above every now and then for varying recipes and/or
> varying
> files inside a given sstate archive on one of the two machines (the
> faster
> one) so far.
> 
> Unfortunately I don't know which of the two physical machines was
> involved
> in my original description.
> 
> The faster machine is running Debian 9.5 and kernel 4.9.110-3+deb9u5
> the
> other Debian unstable and kernel 4.18.8-1 - builds happen inside a
> Debian 9
> docker container in any case. Both machines use btrfs.
> 
> 
> Nevertheless I'm tempted to rule out hardware issues, though because:
> * The sstate archives are either just archiving the output of
>   do_populate_sysroot (which itself is just hard-linking the output
> of
>   do_install), and that same do_populate_sysroot output is also used
> during
>   compilation of dependent recipes
> * Or the sstate is created by archiving the output of
> do_package_write_ipk,
>   where again the same IPKs (hard-linked) are used to build the final
> image
> * The defect seems to be inside the .tar, but an sstate archive is a
>   .tar.gz, and no temporary uncompressed .tar is stored in the file-
> system
>   to create the compressed .tar.gz
> 
> Please correct me if this reasoning is flawed.

I think the reasoning is reasonable. My prime suspect would probably be
btrfs and if you have the space/capability, I'd be tempted to try
builds on ext4 partitions on those machines...

> I'll play with a few bits and variations to try to narrow things
> down, but given the nature it'll likely take a while to get more
> insights.

Yes, problems like this tend to be "fun" to track down :(

Cheers,

Richard