[OE-core] Fwd: [oe] Source Archiver Class

[Saul Wold]

Xiaofeng & Community:

We had a chat with Chris Larson and Mark Hatle here at ELC.

We focused this issue down to a fewer number of options that meet the 
needs of the licenses.  Please review this and let us know if there are 
any issues or concerns.


Best Practices:
  - Archive during build, we do not support nor recommend post archiving
  - Original tarball and patches w/ordering file (maybe comment the
    series), grab non-patch files also.
  - Unless requesting configured sources, which will just be the
    configured source tarball
  - Grab temp as postfunc of do_package - latest logs via links & pid

For sstate Builds (LATER):
  - need to verify that fetch/patch/configure will get re-run for archiving
  - Add temp dir to sstate capture (without links)

3 Implementations classes
  - Filter in oe/lib/license.py
  - source_archive.bb
    - archives tarballs to ${BP}/...
    - reuse copyleft_compliance for patch handling
  - srpm
    - take output from above and create SRPM - (LATER)

2 Configuration classes
  - uses prefunc/postfunc at correct place
  - Original Tarball / Patches
  - Post Configuration

SOURCE_ARCHIVE_PACKAGE_TYPE = {tar, srpm}
SOURCE_ARCHIVE_LOG = {True, False}

Please let us know if you have any questions.

Sau!

On 02/15/2012 05:19 PM, Xiaofeng Yan wrote:
> On 2012年02月15日 23:23, Saul Wold wrote:
>>
>> Xiaofeng,
>>
>> I replied last night to your IRC, but was not sure if you saw it, I am
>> adding it to this email.
>>
>> (11:05:36 PM) sgw: xiaofeng, you around?
>> (11:08:22 PM) sgw: zlib is a bad example, since there are no patches!
>> (11:08:38 PM) sgw: Let's use "at" since it is a simple recipe and has
>> patches
>> (11:09:27 PM) sgw: if you start with bitbake -c cleanall at, so we
>> have a fresh start.
>> (11:12:26 PM) sgw: bitbake -c unpack at, will give you the upacked
>> source in at-3.1.12-r7/at-3.1.12/..., this will give you a base to
>> build from
>> (11:15:12 PM) sgw: "bitbake -c patch at" will then patch the above
>> sources and create the patches directory with the series file. So
>> given that info, you be able to add a do_patch[prefunc] to first tar
>> the unpatched directory and then a do_patch[postfunc] to tar the
>> patches directory into the tar file from the first tar.
>> (11:16:32 PM) sgw: Does this make sense?
>> (11:17:49 PM) sgw: Something else we will want to consider is creating
>> one tarball of all the gnu source and patches in one large tarball.
>>
>>
> Hi Saul,
>
> I saw the reply in IRC yesterday.
> Thanks very much. I will also realize a function to include all of gnu
> sources and patches in one tarball in archiver.bbclass.


>> On 02/14/2012 08:51 AM, Saul Wold wrote:
>>> On 02/13/2012 02:41 AM, Xiaofeng Yan wrote:
>>>> Hi Saul,
>>>>
>>>> I have some issues when I writing design document. The following
>>>> description is my understanding. I take package "zlib" for example .
>>>>>
>>>>>
>>>>> This is a progression list of what the source archives should include:
>>>>>
>>>>> 1 - Original Upstream Archive & Patches
>>>>> - 2 archives (tarballs)
>>>> $ls zlib-orgsource
>>>> zlib-1.2.5.tar.bz2(after do_fetch)
>>>> zlib-patches.tar.bz2( the patches come from
>>>> meta/recipes-core/zlib/zlib-1.2.5)
>>>
>>> Almost correct, I believe that the work Bruce and Chris have done
>>> recently will allow you to get the patches.
>>>
>>> I am not sure we want to just tar up the
>>> meta/recipes-core/zlib/lib-1.2.5 directory since it might contain other
>>> file or patches that we don't actaully use. We need to get the patches
>>> that are actually applied.
>>>
>>>>> 2 - Original Source code & Patches
>>>>> - could include additional code
>>>> Please give me more detailed information
>>> One tarball with the patches extracted in the patches directory with the
>>> series file (possibly a script to apply the patches).
>>>
>>>>> - post unpack
>>>>> - 1 archive
>>>> $ls zlib-orgsource
>>>> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this
>>>> package.)
>>>>
>>>> No patch in this directory.
>>> Need to add the patches via patch mechanism to the source tarball, but
>>> without actually applying the patches.
>>>
>>> So, this is just one tarball.
>>> zlib-1.2.5-prepatch.tar.bz2
>>>>>
>>>>> 3 - Original Source code & Patches & temp (scripts & logs)
>>>>> - Could possibly include the .bb and .inc files
>>>>> - 2 archives (from #2 & temp tarball)
>>>> $ls zlib-orgsource
>>>> zlib-1.2.5.tar.bz2(after do_unpack, patches are not included in this
>>>> package.)
>>>> zlib-patches.tar.bz2( the patches come from
>>>> meta/recipes-core/zlib/zlib-1.2.5)
>>>
>>> Again see above, we need to get only the patches that will actually be
>>> applied.
>>>
>>>> zlib-scripts-logs.tar.bz2(include zlib_1.2.5.bb,zlib.inc and tmp/log_*)
>>>>
>>>> So there are 3 tarballs in this directory.
>>> Yes
>>>
>>>>> 4 - Patched source code
>>>>> - original source code could be removed
>>>>> - post do_patch
>>>> $ls zlib-source
>>>> zlib-1.2.5.tar.bz2(after do_patch)
>>> Correct
>>> Maybe better zlib-1.2.5-patched.tar.bz2
>>>
>>>>> 5 - Patched source code & temp
>>>>> - 2 archives (from #4 & temp tarball)
>>>> $ls zlib-source
>>>> zlib-1.2.5.tar.bz2(after do_patch)
>>>> zlib-logs.tar.bz2(include tmp/log_*)
>>> Correct
>>> Same as #4 change the tarball name
>>>
>>>>> 6 - Configured Source
>>>>> - post do_configure
>>>> $ls zlib-source
>>>> zlib-1.2.5.tar.bz2(after do_configure)
>>> zlib-1.2.5-configured.tar.bz2
>>>
>>>>> 7 - SRPM format of Original Source & Patches
>>>>> - rpm will apply the patches
>>>>> - internally contains #2 above
>>>> $ls zlib-srpm
>>>> zlib-1.2.5.src.rpm
>>>>
>>>> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches
>>>> are not included in this package.) and zlib-patches.tar.bz2
>>>>> 8 - SRPM various of #3 above
>>>> $ls zlib-srpm
>>>> zlib-1.2.5.src.rpm
>>>>
>>>> zlib-1.2.5.src.rpm includes zlib-1.2.5.tar.bz2(after do_unpack, patches
>>>> are not included in this package.) ,zlib-patches.tar.bz2 and
>>>> zlib-scripts-logs.tar.bz2
>>>>
>>>> Do you have any suggestion about the above description?
>>>>
>>> Just the naming and where the patches come from we need to provide the
>>> mechanism (series file) for how to apply the patches also.
>>>
>>> Sau!
>>>
>>>> Thanks
>>>> Yan
>>>>
>>>>
>>>>> ...
>>>>> 100 - Buildable SRPM
>>>>> - can actually build, this is way future!
>>>>>
>>>>> (Patches = patch files & series list)
>>>>>
>>>>> Each of these build on the previous in some way, the key being that we
>>>>> generate a tarball for the source from the existing state of the
>>>>> WORKDIR, a challenge maybe to create the source snapshot after the
>>>>> build
>>>>> has already occurred.
>>>>>
>>>>> The SPDX License info could also be included in any of these
>>>>>
>>>>> After talking with Richard, we think we have a novel approach to make
>>>>> this work. It would entail using the postfuncs feature similar to the
>>>>> way that Shared State does it's work. The existing copyleft_compliance
>>>>> class functionality can be folded into this as a filter. Additional
>>>>> flags could be passed from individual classes to a core set of methods
>>>>> in an archiver class defining the type of data (source, patches, temp,
>>>>> env, recipe info, ...) and format (tar, sprm, ...)
>>>>>
>>>>> I noted that recipe type code might be better suited as generic
>>>>> code, as
>>>>> I believe there are other places that could benefit from that code.
>>>>>
>>>>> The sourcepkg class seems to be a basic archiver and differ that
>>>>> includes the metadata/environment (as dumpdata), this could be
>>>>> replaced
>>>>> by the new approach. While the src_distribute class copies the
>>>>> downloaded archive and then creates a link, into LICENSE directories
>>>>> along with the patches. The src_distribute by default seems to move
>>>>> files and create links (incorrectly btw!). This work can be done by
>>>>> the
>>>>> archiver class.
>>>>>
>>>>> The Nugget: Create a new core "archiver" class that implements a
>>>>> general
>>>>> functions that can archive the original tarball or workdir at various
>>>>> states along task list with additional metadata (recipe info, temp
>>>>> dir,
>>>>> environment). This class would be inherited by a set of classes
>>>>> that use
>>>>> the postfunc (similar to sstate) that setup what level of archive is
>>>>> needed (based on the list above).
>>>>>
>>>>> Thoughts, Comments?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>