[OE-core] [PATCH v2 1/3] export: new plugin to export the devtool workspace

[Paul Eggleton]

Hi Leo,

This looks OK. However I just noticed - the output tarfile isn't compressed. Can we enable gzip compression? In practice that means mode should be 'w:gz' when writing out and also .tar.gz instead of .tar extension.

A couple of minor message notes below as well.


On Friday, 2 June 2017 7:12:59 PM CEST leonardo.sandoval.gonzalez at linux.intel.com wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez at linux.intel.com>
> 
> By default, exports the whole workspace (all recipes) including the source code.
> User can also limit what is exported with --included/--excluded flags. As
> a result of this operation, a tar archive containing only workspace metadata
> and its corresponding source code is created, which can be properly imported
> with 'devtool import'.
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=10510
> 
> [YOCTO #10510]
> 
> Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez at linux.intel.com>
> ---
>  scripts/lib/devtool/export.py | 120 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 120 insertions(+)
>  create mode 100644 scripts/lib/devtool/export.py
> 
> diff --git a/scripts/lib/devtool/export.py b/scripts/lib/devtool/export.py
> new file mode 100644
> index 0000000000..64ac85162b
> --- /dev/null
> +++ b/scripts/lib/devtool/export.py
> @@ -0,0 +1,120 @@
> +# Development tool - export command plugin
> +#
> +# Copyright (C) 2014-2017 Intel Corporation
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License version 2 as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License along
> +# with this program; if not, write to the Free Software Foundation, Inc.,
> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> +"""Devtool export plugin"""
> +
> +import os
> +import argparse
> +import tarfile
> +import logging
> +import datetime
> +import json
> +
> +logger = logging.getLogger('devtool')
> +
> +# output files
> +default_arcname_prefix = "workspace-export"
> +default_arcname_link = default_arcname_prefix + '.tar'
> +metadata = '.export_metadata'
> +
> +def export(args, config, basepath, workspace):
> +    """Entry point for the devtool 'export' subcommand"""
> +
> +    def add_metadata(tar):
> +        """Archive the workspace object"""
> +        # finally store the workspace metadata
> +        with open(metadata, 'w') as fd:
> +            fd.write(json.dumps((config.workspace_path, workspace)))
> +        tar.add(metadata)
> +        os.unlink(metadata)
> +
> +    def add_recipe(tar, recipe, data):
> +        """Archive recipe with proper arcname"""
> +        # Create a map of name/arcnames
> +        arcnames = []
> +        for key, name in data.items():
> +            if name:
> +                if key == 'srctree':
> +                    # all sources, no matter where are located, goes into the sources directory
> +                    arcname = 'sources/%s' % recipe
> +                else:
> +                    arcname = name.replace(config.workspace_path, '')
> +                arcnames.append((name, arcname))
> +
> +        for name, arcname in arcnames:
> +            tar.add(name, arcname=arcname)
> +
> +
> +    # Make sure workspace is non-empty and possible listed include/excluded recipes are in workspace
> +    if not workspace:
> +        logger.info('Workspace has not recipes, nothing to export')

"Workspace has not recipes" -> "Workspace contains no recipes"


> +        return 0
> +    else:
> +        for param, recipes in {'include':args.include,'exclude':args.exclude}.items():
> +            for recipe in recipes:
> +                if recipe not in workspace:
> +                    logger.error('Recipe (%s) on %s argument not in the current workspace' % (recipe, param))
> +                    return 1
> +
> +    name = args.file
> +
> +    default_name = "%s-%s.tar" % (default_arcname_prefix, datetime.datetime.now().strftime('%Y%m%d%H%M%S'))
> +    if not name:
> +        name = default_name
> +    else:
> +        # if name is a directory, append the default name
> +        if os.path.isdir(name):
> +            name = os.path.join(name, default_name)
> +
> +    if os.path.exists(name) and not args.overwrite:
> +        logger.error('Tar archive %s exists. Use --overwrite/-o to overwrite it')
> +        return 1
> +
> +    # if all workspace is excluded, quit
> +    if not len(set(workspace.keys()).difference(set(args.exclude))):
> +        logger.warn('All recipes in workspace excluded, no tar archive created')

"no tar archive created" -> "nothing to export"


> +        return 0
> +
> +    exported = []
> +    with tarfile.open(name, 'w') as tar:
> +        if args.include:
> +            for recipe in args.include:
> +                add_recipe(tar, recipe, workspace[recipe])
> +                exported.append(recipe)
> +        else:
> +            for recipe, data in workspace.items():
> +                if recipe not in args.exclude:
> +                    add_recipe(tar, recipe, data)
> +                    exported.append(recipe)
> +
> +        add_metadata(tar)
> +
> +    logger.info('Tar archive created at %s with the following recipes: %s' % (name, ' '.join(exported)))

', '.join() makes for a slightly more readable output.

> +    return 0
> +
> +def register_commands(subparsers, context):
> +    """Register devtool export subcommands"""
> +    parser = subparsers.add_parser('export',
> +                                   help='Export workspace into a tar archive',
> +                                   description='Export one or more recipes from current workspace into a tar archive',
> +                                   group='advanced')
> +
> +    parser.add_argument('--file', '-f', help='Use archive file or device')

For the -f help text I think we ought to just say "Output archive file name".

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre