Difference between revisions of "Migrating metadata to OE-Core"

From Openembedded.org
Jump to: navigation, search
(Link to OE-core page)
(Recommend useradd.bbclass)
(20 intermediate revisions by one user not shown)
Line 1: Line 1:
 
The classes and BitBake configuration used in [[OpenEmbedded-Core]] require a few changes for recipes brought over from OE-Classic (and for new recipes written by developers used to working with OE-Classic).
 
The classes and BitBake configuration used in [[OpenEmbedded-Core]] require a few changes for recipes brought over from OE-Classic (and for new recipes written by developers used to working with OE-Classic).
  
== Mandatory changes ==
+
== Recipes ==
 +
 
 +
The following section lists changes applicable to recipes (.bb files).
 +
 
 +
=== Mandatory changes ===
  
 
For recipes the following changes are required in order to build successfully:
 
For recipes the following changes are required in order to build successfully:
  
* LIC_FILES_CHKSUM is mandatory, and will be checked at the end of do_configure. It is used in order to specify one or more files or sections of files from the source that can be used to detect if the license of the software has been changed in a newer version. See [http://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#usingpoky-specifying-LIC_FILES_CHKSUM Specifying LIC_FILES_CHKSUM] for more details.
+
* '''Add LIC_FILES_CHKSUM''': this specifies one or more files or sections of files from the source that can be used to detect if the license of the software has been changed in a newer version. It is mandatory and checked at the end of do_configure. See [http://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#usingpoky-specifying-LIC_FILES_CHKSUM Specifying LIC_FILES_CHKSUM] for more details.
* Legacy staging (do_stage) is no longer supported and BitBake will error out on parse of recipes that use it. Items previously in do_stage should be moved to do_install as appropriate; however they should actually install the files to somewhere under ${D} and ''not'' copy them into the staging directory directly or things will break later on.
+
* '''Remove legacy staging''': legacy staging (do_stage) is no longer supported and BitBake will error out on parse of recipes that use it. Items previously in do_stage should be moved to do_install as appropriate; however they should actually install the files to somewhere under ${D} and ''not'' copy them into the staging directory directly or things will break later on. See [[Legacy staging]] for more information.
* The LICENSE field, whilst usually included in most recipes in the past anyway, is now mandatory. It should be as specific and as correct as possible (e.g. "GPLv2", not just "GPL") and try to be consistent with other recipes.
+
* '''Ensure LICENSE is present''': the LICENSE field, whilst usually included in most recipes in the past anyway, is now mandatory. It should be as specific and as correct as possible (e.g. "GPLv2", not just "GPL") and try to be consistent with other recipes.
* Any calls to oenote, oewarn, oefatal etc. need to be replaced with the "bb" equivalent, i.e. bbnote, bbwarn, bbfatal etc.
+
* '''Ensure SRC_URI checksums are present''': for entries in SRC_URI pointing to sources other than SCMs or local files, i.e. http, ftp, etc., at least one SRC_URI checksum (md5sum/sha256sum) must be provided. If these are not present for an entry where it is required you will get an error which gives you the appropriate values to be added to the recipe. You can use the "name=" parameter within SRC_URI to distinguish between multiple entries. (This used to be optional, now it is mandatory.)
* Recent versions of BitBake no longer allow comments in multi-line strings, so the tradition of being able to comment out lines within a multi-line SRC_URI or when setting configure flags etc. is no longer allowed. Remove the commented out lines or move them to beyond where the string terminates.
+
* '''Remove comments in string values''': recent versions of BitBake no longer allow comments in multi-line strings, so the tradition of being able to comment out lines within a multi-line SRC_URI or when setting configure flags etc. is no longer allowed. Remove the commented out lines or move them to beyond where the string terminates.
* Modifications to FILESPATHPKG and/or FILESPATH should be replaced by prepending to FILESEXTRAPATHS; in addition, you no longer need to define THISDIR (and should not do so). For example, to add a directory to be searched for patches and other files which is named with PN, do the following (note the immediate evaluation operator := and trailing colon, these are important):
+
* '''Ensure values are quoted''': current versions of BitBake require that values are quoted; this was optional in the past. Single or double quotes are allowed but all values must now be quoted. (This does not affect how numeric values are interpreted, these have always been treated as strings.)
 +
* '''Use package name override with packaging variables''': the runtime package specific variables (RDEPENDS, RRECOMMENDS, RSUGGESTS, RPROVIDES, RCONFLICTS, RREPLACES, FILES, ALLOW_EMPTY, and the pre/post install/uninstall script functions pkg_preinst, pkg_postinst, pkg_prerm, and pkg_postrm) should always have a package name override. For example, to set a runtime dependency for the main package use RDEPENDS_${PN}.
 +
* '''Replace oe* logging commands''': Any calls to oenote, oewarn, oefatal etc. in shell scripts need to be replaced with the "bb" equivalent, i.e. bbnote, bbwarn, bbfatal etc.
 +
* '''Use FILESEXTRAPATHS to extend file search path''': Modifications to FILESPATHPKG and/or FILESPATH (or FILESDIR which is deprecated) should be replaced by prepending to FILESEXTRAPATHS; in addition, you no longer need to define THISDIR (and should not do so). For example, to add a directory to be searched for patches and other files which is named with PN, do the following (note the immediate evaluation operator := and trailing colon, these are important):
 
  FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
 
  FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
* RDEPENDS and RRECOMMENDS should always have a package name override. To set depends/recommends for the main package use RDEPENDS_${PN} / RRECOMMENDS_${PN} respectively.
+
Changes applicable to the "danny" release and later:
 +
* '''Python functions should always use four spaces for indentation''' and not tabs - a warning will be produced if tabs are found. Previously four spaces was the convention but many python functions used tabs. This has been fixed in OE-Core and other layers, and since python uses indentation for statement blocks, is very important that you make the same changes to python functions in your recipes.
 +
* '''Change proto= to protocol= in SRC_URI''' - all fetchers have been changed to be consistent. Previously, some fetchers used proto and some protocol.
 +
* '''Change tasks to package groups''' - for custom task recipes, rename them from task-*.bb to packagegroup-*.bb, inherit packagegroup instead of inheriting task and remove anything that is now handled by meta/classes/packagegroup.bbclass. Also change any references to task-* to packagegroup-*.
 +
* '''Change any assumptions that libexecdir is /usr/libexec''': we now set libexecdir to ${libdir}/${BPN} to match with the Filesystem Hierarchy Standard (FHS).
 +
Changes applicable to the "dylan" release and later:
 +
* '''Fix continuation on last line of a comment''': if a comment ends in a line continuation (\) then the next line must also be a comment; any instance where this is not the case will now trigger a warning. Either remove the continuation character or make the next line a comment as well.
 +
* '''Remove use of BROKEN''' - use EXCLUDE_FROM_WORLD instead; ideally fix up the cause of the recipe being marked as BROKEN.
 +
Changes applicable to the "dora" release and later:
 +
* '''Rename functions/variables that currently contain "_remove_" or end with "_remove"''' - _remove is now a BitBake operator, so referencing it when you aren't intending to remove a value from a list variable will result in unexpected behaviour.
  
== Best practices ==
+
=== Best practices ===
  
 
The following practices are required for patch submission to layers such as OE-Core and meta-oe, and recommended for recipes in your own layers:
 
The following practices are required for patch submission to layers such as OE-Core and meta-oe, and recommended for recipes in your own layers:
  
 
* Machine and distro-specific overrides should be removed from recipes for generic layers (especially meta-oe and OE-Core). These can be moved to bbappends in the appropriate machine / distro layer respectively. (Note however that architecture-specific overrides are allowed if necessary.)
 
* Machine and distro-specific overrides should be removed from recipes for generic layers (especially meta-oe and OE-Core). These can be moved to bbappends in the appropriate machine / distro layer respectively. (Note however that architecture-specific overrides are allowed if necessary.)
* In-tree patches to be applied to the source should have a proper header explaining what the patch does, a valid Signed-off-by, and for OE-Core should have an [http://lists.yoctoproject.org/pipermail/yocto/2011-April/003446.html Upstream-Status] indicator.
+
* Fix any QA issues reported during packaging.
* Initially reset PR to "r0" (or remove it which amounts to the same thing). PR is definitely still used but we can start again from r0.
+
* Avoid hardcoding system paths such as /etc, /usr/bin, /usr/share and so on. Instead the values of the appropriate variables should be used (e.g. ${libdir}, ${bindir}, ${datadir} ...). If there are scripts or other files provided by the application such as initscripts you can fix the paths in them within do_install using sed. See bitbake.conf for a list of standard path variables and their default values (which may be - and often are - overridden by the distro configuration).
 +
* Initscripts should have the standard [http://wiki.debian.org/LSBInitScripts LSB header] (especially if no systemd unit file is being provided, since systemd relies upon information in the header when handling an initscript.)
 +
* In-tree patches to be applied to the source should have a proper header explaining what the patch does, a valid Signed-off-by, an [http://lists.yoctoproject.org/pipermail/yocto/2011-April/003446.html Upstream-Status] indicator.
 +
* Reset PR to "r0" by removing it. PR is still used, but new recipes being migrated should start again from r0 (the default value if not specified), and if you need it to increment on recipe changes you should use the [http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#working-with-a-pr-service PR service].
 +
* Use ${BPN} instead of ${PN} where you just want the recipe name without any prefixes/suffixes added by things like multilib and nativesdk (e.g. installation paths)
 +
* Use the <code>useradd</code> class instead of running useradd/groupadd directly to add users/groups.
 
* NATIVE_INSTALL_WORKS for native recipes is no longer used and should be removed.
 
* NATIVE_INSTALL_WORKS for native recipes is no longer used and should be removed.
 
* PRIORITY is no longer used and should be removed.
 
* PRIORITY is no longer used and should be removed.
 +
* ENTERPRISE_DISTRO is no longer used. We now have LICENSE_FLAGS to support the uses that ENTERPRISE_DISTRO covered - see the [http://www.yoctoproject.org/docs/current/ref-manual/ref-manual.html#enabling-commercially-licensed-recipes Yocto Project Reference Manual].
 +
* Use d.getVar/d.setVar instead of bb.data.getVar/bb.data.setVar in python functions. The latter will still work, but are deprecated.
 
* "[http://mywiki.wooledge.org/Bashism Bashisms]" in shell scripts (particularly those that will be executed on the host) should be avoided. This allows compatibility with alternative shells e.g. Ubuntu's default (dash).
 
* "[http://mywiki.wooledge.org/Bashism Bashisms]" in shell scripts (particularly those that will be executed on the host) should be avoided. This allows compatibility with alternative shells e.g. Ubuntu's default (dash).
 +
* Pre/postinstall scripts (pkg_preinst / pkg_postinst) should ideally be written such that they can run on the host during do_rootfs instead of only on the target. In practice this means that you need to use ${D} before all paths, and anything that is architecture-specific will need to be run under qemu (although the latter is not very common).
 +
* For readability, the variables/functions within a recipe should be ordered in the same order in which they are used - i.e. "meta" information above (DESCRIPTION, SUMMARY, DEPENDS, LICENSE etc.), followed by information needed for do_fetch (SRC_URI, etc.), then anything for do_patch, do_compile, do_install, and then do_package (e.g. PACKAGES, FILES, RDEPENDS, RRECOMMENDS...).
 +
 +
== Machine configuration ==
 +
 +
The following section lists changes applicable to machine configuration files (conf/machine/*).
 +
 +
* '''Ensure you are using the correct tune files''': the tune files in OE-Core have been reorganised and updated; see conf/machine/include/ and check that you are including the appropriate file(s).
 +
* '''Remove variable settings already handled by the tune files''': check if any values you are setting are already set by the tune files; if they are, you don't need to set them again. Of course if you wish to override one of the settings for your machine that is fine.
 +
* '''Update to a newer Linux kernel''': current versions of kernel-sensitive userspace components (e.g. udev) expect a modern kernel; older kernels may not allow the system to boot. If you're not able to do this you may need to provide downgraded versions of these components that are compatible with the kernel you are building. Note that all support for the 2.4 kernel has been removed.
 +
* '''Check MACHINE_FEATURES value''': check the list you are setting against the currently available features (see [http://www.yoctoproject.org/docs/current/poky-ref-manual/poky-ref-manual.html#ref-features-machine here], or you can do "git grep MACHINE_FEATURES" in the layers containing recipes you are using.) In particular, the "kernel26" feature can be removed as we no longer support the 2.4 kernel which necessitated having a flag to say the machine is using a 2.6 kernel instead.
 +
* '''Remove unused variables''': the following variables are no longer used:
 +
** KERNEL
 +
** PCMCIA_MANAGER
 +
** MACHINE_KERNEL_VERSION
 +
** ROOT_FLASH_SIZE
 +
* '''Use =+ to set IMAGE_FSTYPES''': this interacts best when adding/overriding from other configuration files.
 +
* '''Remove MACHINE_KERNEL_PR''': this is no longer needed when BB_SIGNATURE_HANDLER is OEBasicHash (now the default) and the PR service is being used (in the dylan branch and newer); any changes to underlying metadata or dependencies of the kernel will automatically rebuild the kernel so there is no need to force it by hand.
 +
 +
== Distro configuration ==
 +
 +
* '''Build on top of defaultsetup.conf''': OE-Core provides a very basic distro config in meta/conf/distro/defaultsetup.conf which is included automatically by bitbake.conf, thus you only need to include bits of this config file that you want to change.
 +
* '''Replace references to glibc with eglibc''': set TCMODE and TCLIBC to select the toolchain mode and libc variant if you are not using the defaults.
 +
* '''Check DISTRO_FEATURES value''': if you are setting DISTRO_FEATURES, check the list you have against the currently available features (look at the default value in conf/distro/include/default-distrovars.inc in OE-Core; you can also do "git grep DISTRO_FEATURES" in the layers containing recipes you are using.)
 +
* '''Avoid setting machine configuration variables''': machine configuration should almost always be handled in each machine configuration file and not in the distro configuration.
 +
* '''Update blacklisting''': Replace ANGSTROM_BLACKLIST_pn-somerecipe = "message" with BLACKLIST[somerecipe] = "message" (assuming the recipe still exists and the blacklisting is still needed)
 +
* '''Replace references to ONLINE_PACKAGE_MANAGEMENT''': this variable has been replaced by checking for package-management within IMAGE_FEATURES.

Revision as of 18:14, 25 November 2013

The classes and BitBake configuration used in OpenEmbedded-Core require a few changes for recipes brought over from OE-Classic (and for new recipes written by developers used to working with OE-Classic).

Contents

Recipes

The following section lists changes applicable to recipes (.bb files).

Mandatory changes

For recipes the following changes are required in order to build successfully:

  • Add LIC_FILES_CHKSUM: this specifies one or more files or sections of files from the source that can be used to detect if the license of the software has been changed in a newer version. It is mandatory and checked at the end of do_configure. See Specifying LIC_FILES_CHKSUM for more details.
  • Remove legacy staging: legacy staging (do_stage) is no longer supported and BitBake will error out on parse of recipes that use it. Items previously in do_stage should be moved to do_install as appropriate; however they should actually install the files to somewhere under ${D} and not copy them into the staging directory directly or things will break later on. See Legacy staging for more information.
  • Ensure LICENSE is present: the LICENSE field, whilst usually included in most recipes in the past anyway, is now mandatory. It should be as specific and as correct as possible (e.g. "GPLv2", not just "GPL") and try to be consistent with other recipes.
  • Ensure SRC_URI checksums are present: for entries in SRC_URI pointing to sources other than SCMs or local files, i.e. http, ftp, etc., at least one SRC_URI checksum (md5sum/sha256sum) must be provided. If these are not present for an entry where it is required you will get an error which gives you the appropriate values to be added to the recipe. You can use the "name=" parameter within SRC_URI to distinguish between multiple entries. (This used to be optional, now it is mandatory.)
  • Remove comments in string values: recent versions of BitBake no longer allow comments in multi-line strings, so the tradition of being able to comment out lines within a multi-line SRC_URI or when setting configure flags etc. is no longer allowed. Remove the commented out lines or move them to beyond where the string terminates.
  • Ensure values are quoted: current versions of BitBake require that values are quoted; this was optional in the past. Single or double quotes are allowed but all values must now be quoted. (This does not affect how numeric values are interpreted, these have always been treated as strings.)
  • Use package name override with packaging variables: the runtime package specific variables (RDEPENDS, RRECOMMENDS, RSUGGESTS, RPROVIDES, RCONFLICTS, RREPLACES, FILES, ALLOW_EMPTY, and the pre/post install/uninstall script functions pkg_preinst, pkg_postinst, pkg_prerm, and pkg_postrm) should always have a package name override. For example, to set a runtime dependency for the main package use RDEPENDS_${PN}.
  • Replace oe* logging commands: Any calls to oenote, oewarn, oefatal etc. in shell scripts need to be replaced with the "bb" equivalent, i.e. bbnote, bbwarn, bbfatal etc.
  • Use FILESEXTRAPATHS to extend file search path: Modifications to FILESPATHPKG and/or FILESPATH (or FILESDIR which is deprecated) should be replaced by prepending to FILESEXTRAPATHS; in addition, you no longer need to define THISDIR (and should not do so). For example, to add a directory to be searched for patches and other files which is named with PN, do the following (note the immediate evaluation operator := and trailing colon, these are important):
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

Changes applicable to the "danny" release and later:

  • Python functions should always use four spaces for indentation and not tabs - a warning will be produced if tabs are found. Previously four spaces was the convention but many python functions used tabs. This has been fixed in OE-Core and other layers, and since python uses indentation for statement blocks, is very important that you make the same changes to python functions in your recipes.
  • Change proto= to protocol= in SRC_URI - all fetchers have been changed to be consistent. Previously, some fetchers used proto and some protocol.
  • Change tasks to package groups - for custom task recipes, rename them from task-*.bb to packagegroup-*.bb, inherit packagegroup instead of inheriting task and remove anything that is now handled by meta/classes/packagegroup.bbclass. Also change any references to task-* to packagegroup-*.
  • Change any assumptions that libexecdir is /usr/libexec: we now set libexecdir to ${libdir}/${BPN} to match with the Filesystem Hierarchy Standard (FHS).

Changes applicable to the "dylan" release and later:

  • Fix continuation on last line of a comment: if a comment ends in a line continuation (\) then the next line must also be a comment; any instance where this is not the case will now trigger a warning. Either remove the continuation character or make the next line a comment as well.
  • Remove use of BROKEN - use EXCLUDE_FROM_WORLD instead; ideally fix up the cause of the recipe being marked as BROKEN.

Changes applicable to the "dora" release and later:

  • Rename functions/variables that currently contain "_remove_" or end with "_remove" - _remove is now a BitBake operator, so referencing it when you aren't intending to remove a value from a list variable will result in unexpected behaviour.

Best practices

The following practices are required for patch submission to layers such as OE-Core and meta-oe, and recommended for recipes in your own layers:

  • Machine and distro-specific overrides should be removed from recipes for generic layers (especially meta-oe and OE-Core). These can be moved to bbappends in the appropriate machine / distro layer respectively. (Note however that architecture-specific overrides are allowed if necessary.)
  • Fix any QA issues reported during packaging.
  • Avoid hardcoding system paths such as /etc, /usr/bin, /usr/share and so on. Instead the values of the appropriate variables should be used (e.g. ${libdir}, ${bindir}, ${datadir} ...). If there are scripts or other files provided by the application such as initscripts you can fix the paths in them within do_install using sed. See bitbake.conf for a list of standard path variables and their default values (which may be - and often are - overridden by the distro configuration).
  • Initscripts should have the standard LSB header (especially if no systemd unit file is being provided, since systemd relies upon information in the header when handling an initscript.)
  • In-tree patches to be applied to the source should have a proper header explaining what the patch does, a valid Signed-off-by, an Upstream-Status indicator.
  • Reset PR to "r0" by removing it. PR is still used, but new recipes being migrated should start again from r0 (the default value if not specified), and if you need it to increment on recipe changes you should use the PR service.
  • Use ${BPN} instead of ${PN} where you just want the recipe name without any prefixes/suffixes added by things like multilib and nativesdk (e.g. installation paths)
  • Use the useradd class instead of running useradd/groupadd directly to add users/groups.
  • NATIVE_INSTALL_WORKS for native recipes is no longer used and should be removed.
  • PRIORITY is no longer used and should be removed.
  • ENTERPRISE_DISTRO is no longer used. We now have LICENSE_FLAGS to support the uses that ENTERPRISE_DISTRO covered - see the Yocto Project Reference Manual.
  • Use d.getVar/d.setVar instead of bb.data.getVar/bb.data.setVar in python functions. The latter will still work, but are deprecated.
  • "Bashisms" in shell scripts (particularly those that will be executed on the host) should be avoided. This allows compatibility with alternative shells e.g. Ubuntu's default (dash).
  • Pre/postinstall scripts (pkg_preinst / pkg_postinst) should ideally be written such that they can run on the host during do_rootfs instead of only on the target. In practice this means that you need to use ${D} before all paths, and anything that is architecture-specific will need to be run under qemu (although the latter is not very common).
  • For readability, the variables/functions within a recipe should be ordered in the same order in which they are used - i.e. "meta" information above (DESCRIPTION, SUMMARY, DEPENDS, LICENSE etc.), followed by information needed for do_fetch (SRC_URI, etc.), then anything for do_patch, do_compile, do_install, and then do_package (e.g. PACKAGES, FILES, RDEPENDS, RRECOMMENDS...).

Machine configuration

The following section lists changes applicable to machine configuration files (conf/machine/*).

  • Ensure you are using the correct tune files: the tune files in OE-Core have been reorganised and updated; see conf/machine/include/ and check that you are including the appropriate file(s).
  • Remove variable settings already handled by the tune files: check if any values you are setting are already set by the tune files; if they are, you don't need to set them again. Of course if you wish to override one of the settings for your machine that is fine.
  • Update to a newer Linux kernel: current versions of kernel-sensitive userspace components (e.g. udev) expect a modern kernel; older kernels may not allow the system to boot. If you're not able to do this you may need to provide downgraded versions of these components that are compatible with the kernel you are building. Note that all support for the 2.4 kernel has been removed.
  • Check MACHINE_FEATURES value: check the list you are setting against the currently available features (see here, or you can do "git grep MACHINE_FEATURES" in the layers containing recipes you are using.) In particular, the "kernel26" feature can be removed as we no longer support the 2.4 kernel which necessitated having a flag to say the machine is using a 2.6 kernel instead.
  • Remove unused variables: the following variables are no longer used:
    • KERNEL
    • PCMCIA_MANAGER
    • MACHINE_KERNEL_VERSION
    • ROOT_FLASH_SIZE
  • Use =+ to set IMAGE_FSTYPES: this interacts best when adding/overriding from other configuration files.
  • Remove MACHINE_KERNEL_PR: this is no longer needed when BB_SIGNATURE_HANDLER is OEBasicHash (now the default) and the PR service is being used (in the dylan branch and newer); any changes to underlying metadata or dependencies of the kernel will automatically rebuild the kernel so there is no need to force it by hand.

Distro configuration

  • Build on top of defaultsetup.conf: OE-Core provides a very basic distro config in meta/conf/distro/defaultsetup.conf which is included automatically by bitbake.conf, thus you only need to include bits of this config file that you want to change.
  • Replace references to glibc with eglibc: set TCMODE and TCLIBC to select the toolchain mode and libc variant if you are not using the defaults.
  • Check DISTRO_FEATURES value: if you are setting DISTRO_FEATURES, check the list you have against the currently available features (look at the default value in conf/distro/include/default-distrovars.inc in OE-Core; you can also do "git grep DISTRO_FEATURES" in the layers containing recipes you are using.)
  • Avoid setting machine configuration variables: machine configuration should almost always be handled in each machine configuration file and not in the distro configuration.
  • Update blacklisting: Replace ANGSTROM_BLACKLIST_pn-somerecipe = "message" with BLACKLIST[somerecipe] = "message" (assuming the recipe still exists and the blacklisting is still needed)
  • Replace references to ONLINE_PACKAGE_MANAGEMENT: this variable has been replaced by checking for package-management within IMAGE_FEATURES.
Personal tools
Namespaces

Variants
Actions
Navigation
Categories
OE services
Toolbox