[OE-core] Automatic prebuilt management

[Richard Purdie]

On Fri, 2019-08-30 at 18:27 +0200, Loic Poulain wrote:
> HI Andre,
> 
> On Fri, 23 Aug 2019 at 20:11, Andre McCurdy <armccurdy at gmail.com>
> wrote:
> > On Mon, Aug 5, 2019 at 9:06 AM Loic Poulain <
> > loic.poulain at linaro.org> wrote:
> > > Say a company works with an OE internal tree, with open-source
> > and closed-source packages. The company wants to release the tree
> > so that its customers can fully customize the images/distro. The
> > closed sources being only available internally, the company has to
> > generate and handle prebuilt binaries for proprietary packages.
> > Ideally, with a unified public/internal OE tree (same recipes for
> > internal/public build).
> > >
> > > This question is actually similar to an older thread:
> > > https://marc.info/?l=openembedded-core&m=146779329804683
> > 
> > That very old thread was also referenced recently in a follow up
> > where
> > I shared a later solution, see:
> > 
> >   
> > http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284896.html
> 
> Thanks, I missed that one.
> 
> > 
> > > I started to work on this and added a 'generate-prebuilt' class
> > which generates a tarball of ${D} in deploy/prebuilts after
> > do_install task. Symmetrically, It's also possible to create an
> > 'install-prebuilt' class which bypasses do_fetch, do_unpack, ...,
> > do_install with noexec flag and instead uncompresses a previously
> > generated prebuilt tarball into ${D} and continue normally. But I
> > would like something smarter, e.g. first trying to check if the
> > SRC_URI is available, if not switching on using the prebuilt
> > package if available (e.g. in a DIR_PREBUILT)...
> > >
> > > Before going further is there already an existing solution for
> > that? do you have any recommendation on the easier/best way to
> > achieve this?
> > 
> > I did initially try the approach of having a single recipe which
> > can
> > automatically support both building from source and extracting a
> > prebuilts tar file, but that (for me anyway) turned out to be a
> > dead
> > end. Building from source requires build dependencies and config
> > options but extracting a prebuilt tar file does not, so the two end
> > up
> > sharing very little... so 90% of the recipe ends up being
> > conditional
> > on which mode it's running in. The solution I ended up with (see
> > link
> > above) was for the class which creates the prebuilt tar file to
> > also
> > create a dedicated mini recipe to extract it. 
> > From my experience however an equally hard part of the problem is
> > the
> > distribution of prebuilts. It starts off easy (you share a tar file
> > via email or an ftp site and the receiver manually copies to their
> > downloads directory...) but that doesn't scale if you need to make
> > regular updates.
> 
> Very good point. maybe having binaries also in a repo could help a
> bit (e.g. using git LFS).
>  
> > My solution is discussed a little more in the thread
> > linked to above.
> 
> Your solution is interesting and I'll probably jump on the thread
> with more questions.
> Having an upstream solution part of oe-core would be nice though.

We'd be happy to have one, it will take someone to propose something
suitably general purpose that others can use along with tests.

Cheers,

Richard