[OE-core] Fwd: [oe] Source Archiver Class

[Saul Wold]

Xiaofeng,

I replied last night to your IRC, but was not sure if you saw it, I am 
adding it to this email.

(11:05:36 PM) sgw: xiaofeng, you around?
(11:08:22 PM) sgw: zlib is a bad example, since there are no patches!
(11:08:38 PM) sgw: Let's use "at" since it is a simple recipe and has 
patches
(11:09:27 PM) sgw: if you start with bitbake -c cleanall at, so we have 
a fresh start.
(11:12:26 PM) sgw: bitbake -c unpack at, will give you the upacked 
source in at-3.1.12-r7/at-3.1.12/..., this will give you a base to build 
from
(11:15:12 PM) sgw: "bitbake -c patch at" will then patch the above 
sources and create the patches directory with the series file.  So given 
that info, you be able to add a do_patch[prefunc] to first tar the 
unpatched directory and then a do_patch[postfunc] to tar the patches 
directory into the tar file from the first tar.
(11:16:32 PM) sgw: Does this make sense?
(11:17:49 PM) sgw: Something else we will want to consider is creating 
one tarball of all the gnu source and patches in one large tarball.


On 02/14/2012 08:51 AM, Saul Wold wrote:
> On 02/13/2012 02:41 AM, Xiaofeng Yan wrote:
>> Hi Saul,
>>
>> I have some issues when I writing design document. The following
>> description is my understanding. I take package "zlib" for example .
>>>
>>>
>>> This is a progression list of what the source archives should include:
>>>
>>> 1 - Original Upstream Archive & Patches
>>> - 2 archives (tarballs)
>> $ls zlib-orgsource
>> zlib-1.2.5.tar.bz2(after do_fetch)
>> zlib-patches.tar.bz2( the patches come from
>> meta/recipes-core/zlib/zlib-1.2.5)
>
> Almost correct, I believe that the work Bruce and Chris have done
> recently will allow you to get the patches.
>
> I am not sure we want to just tar up the
> meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other
> file or patches that we don't actaully use. We need to get the patches
> that are actually applied.
>
>>> 2 - Original Source code & Patches
>>> - could include additional code
>> Please give me more detailed information
> One tarball with the patches extracted in the patches directory with the
> series file (possibly a script to apply the patches).
>
>>> - post unpack
>>> - 1 archive
>> $ls zlib-orgsource
>> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this
>> package.)
>>
>> No patch in this directory.
> Need to add the patches via patch mechanism to the source tarball, but
> without actually applying the patches.
>
> So, this is just one tarball.
> zlib-1.2.5-prepatch.tar.bz2
>>>
>>> 3 - Original Source code & Patches & temp (scripts & logs)
>>> - Could possibly include the .bb and .inc files
>>> - 2 archives (from #2 & temp tarball)
>> $ls zlib-orgsource
>> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this
>> package.)
>> zlib-patches.tar.bz2( the patches come from
>> meta/recipes-core/zlib/zlib-1.2.5)
>
> Again see above, we need to get only the patches that will actually be
> applied.
>
>> zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)
>>
>> So there are 3 tarballs in this directory.
> Yes
>
>>> 4 - Patched source code
>>> - original source code could be removed
>>> - post do_patch
>> $ls zlib-source
>> zlib-1.2.5.tar.bz2(after do_patch)
> Correct
> Maybe better zlib-1.2.5-patched.tar.bz2
>
>>> 5 - Patched source code & temp
>>> - 2 archives (from #4 & temp tarball)
>> $ls zlib-source
>> zlib-1.2.5.tar.bz2(after do_patch)
>> zlib-logs.tar.bz2(include tmp/log_*)
> Correct
> Same as #4 change the tarball name
>
>>> 6 - Configured Source
>>> - post do_configure
>> $ls zlib-source
>> zlib-1.2.5.tar.bz2(after do_configure)
> zlib-1.2.5-configured.tar.bz2
>
>>> 7 - SRPM format of Original Source & Patches
>>> - rpm will apply the patches
>>> - internally contains #2 above
>> $ls zlib-srpm
>> zlib-1.2.5.src.rpm
>>
>> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches
>> are not included in this package.) and zlib-patches.tar.bz2
>>> 8 - SRPM various of #3 above
>> $ls zlib-srpm
>> zlib-1.2.5.src.rpm
>>
>> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches
>> are not included in this package.) ,zlib-patches.tar.bz2 and
>> zlib-scripts-logs.tar.bz2
>>
>> Do you have any suggestion about the above description?
>>
> Just the naming and where the patches come from we need to provide the
> mechanism (series file) for how to apply the patches also.
>
> Sau!
>
>> Thanks
>> Yan
>>
>>
>>> ...
>>> 100 - Buildable SRPM
>>> - can actually build, this is way future!
>>>
>>> (Patches = patch files & series list)
>>>
>>> Each of these build on the previous in some way, the key being that we
>>> generate a tarball for the source from the existing state of the
>>> WORKDIR, a challenge maybe to create the source snapshot after the build
>>> has already occurred.
>>>
>>> The SPDX License info could also be included in any of these
>>>
>>> After talking with Richard, we think we have a novel approach to make
>>> this work. It would entail using the postfuncs feature similar to the
>>> way that Shared State does it's work. The existing copyleft_compliance
>>> class functionality can be folded into this as a filter. Additional
>>> flags could be passed from individual classes to a core set of methods
>>> in an archiver class defining the type of data (source, patches, temp,
>>> env, recipe info, ...) and format (tar, sprm, ...)
>>>
>>> I noted that recipe type code might be better suited as generic code, as
>>> I believe there are other places that could benefit from that code.
>>>
>>> The sourcepkg class seems to be a basic archiver and differ that
>>> includes the metadata/environment (as dumpdata), this could be replaced
>>> by the new approach. While the src_distribute class copies the
>>> downloaded archive and then creates a link, into LICENSE directories
>>> along with the patches. The src_distribute by default seems to move
>>> files and create links (incorrectly btw!). This work can be done by the
>>> archiver class.
>>>
>>> The Nugget: Create a new core "archiver" class that implements a general
>>> functions that can archive the original tarball or workdir at various
>>> states along task list with additional metadata (recipe info, temp dir,
>>> environment). This class would be inherited by a set of classes that use
>>> the postfunc (similar to sstate) that setup what level of archive is
>>> needed (based on the list above).
>>>
>>> Thoughts, Comments?
>>>
>>> Thanks
>>>
>>>
>>>
>>
>>
>