[OE-core] [PATCH] glibc-locale: Rewrite do_install using install utility instead of cp
Khem Raj
raj.khem at gmail.com
Wed Feb 13 17:42:02 UTC 2019
Peter did you have a chance to try
https://patchwork.openembedded.org/patch/158661/
On Sun, Feb 10, 2019 at 6:09 PM Khem Raj <raj.khem at gmail.com> wrote:
>
> On Sun, Feb 10, 2019 at 3:32 PM Peter Kjellerstedt
> <peter.kjellerstedt at axis.com> wrote:
> >
> > > -----Original Message-----
> > > From: openembedded-core-bounces at lists.openembedded.org <openembedded-
> > > core-bounces at lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > > Sent: den 10 februari 2019 07:25
> > > To: Richard Purdie <richard.purdie at linuxfoundation.org>; Khem Raj
> > > <raj.khem at gmail.com>; openembedded-core at lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > install utility instead of cp
> > >
> > > > -----Original Message-----
> > > > From: openembedded-core-bounces at lists.openembedded.org <openembedded-
> > > > core-bounces at lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> > > > Sent: den 10 februari 2019 02:29
> > > > To: Richard Purdie <richard.purdie at linuxfoundation.org>; Khem Raj
> > > > <raj.khem at gmail.com>; openembedded-core at lists.openembedded.org
> > > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install using
> > > > install utility instead of cp
> > > >
> > > > > -----Original Message-----
> > > > > From: openembedded-core-bounces at lists.openembedded.org
> > > <openembedded-
> > > > > core-bounces at lists.openembedded.org> On Behalf Of Richard Purdie
> > > > > Sent: den 7 februari 2019 10:01
> > > > > To: Khem Raj <raj.khem at gmail.com>; openembedded-
> > > > > core at lists.openembedded.org
> > > > > Subject: Re: [OE-core] [PATCH] glibc-locale: Rewrite do_install
> > > using
> > > > > install utility instead of cp
> > > > >
> > > > > On Wed, 2019-02-06 at 16:35 -0800, Khem Raj wrote:
> > > > > > This has been a constant source of trouble for build failures due
> > > > to
> > > > > host-user-contaminated QA
> > > > > > errors of sort
> > > > > >
> > > > > > ERROR: QA Issue: glibc-locale: /glibc-binary-localedata-ca-
> > > > > es+valencia/usr/lib/locale/ca_ES at valencia/LC_MONETARY is owned by
> > > uid
> > > > > 3004, which is the same as the user running bitbake. This may be
> > > due
> > > > to
> > > > > host contamination [host-user-contaminated]
> > > > > >
> > > > > > So far we have tried to mould cp command into not carrying the
> > > > build
> > > > > > user permissions into install area but it is never entirely fixed
> > > > > since
> > > > > > the issue keeps popping up in various scenes
> > > > > >
> > > > > > This patch replaces use of cp with install utility and specifies
> > > > > install
> > > > > > mode for files explcitly
> > > > > >
> > > > > > Signed-off-by: Khem Raj <raj.khem at gmail.com>
> > > > > > ---
> > > > > > meta/recipes-core/glibc/glibc-locale.inc | 44 ++++++++++++++----
> > > --
> > > > --
> > > > > --
> > > > > > 1 file changed, 25 insertions(+), 19 deletions(-)
> > > > > >
> > > > > > diff --git a/meta/recipes-core/glibc/glibc-locale.inc
> > > > b/meta/recipes-
> > > > > core/glibc/glibc-locale.inc
> > > > > > index 6384f9cbf1..9b256a5108 100644
> > > > > > --- a/meta/recipes-core/glibc/glibc-locale.inc
> > > > > > +++ b/meta/recipes-core/glibc/glibc-locale.inc
> > > > > > @@ -72,27 +72,33 @@ FILES_localedef = "${bindir}/localedef"
> > > > > > LOCALETREESRC = "${COMPONENTS_DIR}/${PACKAGE_ARCH}/glibc-stash-
> > > > > locale"
> > > > > >
> > > > > > do_install () {
> > > > > > - mkdir -p ${D}${bindir} ${D}${datadir}
> > > > > > - if [ -n "$(ls ${LOCALETREESRC}/${bindir})" ]; then
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${bindir}/* ${D}${bindir}
> > > > > > - fi
> > > > > > - if [ -n "$(ls ${LOCALETREESRC}/${localedir})" ]; then
> > > > > > - mkdir -p ${D}${localedir}
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${localedir}/* ${D}${localedir}
> > > > > > - fi
> > > > > > + install -d ${D}${bindir}
> > > > > > + find "${LOCALETREESRC}/${bindir}" -maxdepth 1 -type f \
> > > > > > + -exec install -m 0755 -t "${D}${bindir}" {} \;
> > > > > > +
> > > > > > + for d in . $(find "${LOCALETREESRC}/${localedir}" -type
> > > d
> > > > -
> > > > > printf '%P ') ; do
> > > > > > + install -d "${D}${localedir}/$d"
> > > > > > + find "${LOCALETREESRC}/${localedir}/$d" -
> > > maxdepth
> > > > 1
> > > > > -type f \
> > > > > > + -exec install -m 0644 -t "${D}${localedir}/$d"
> > > {}
> > > > \;
> > > > > > + done
> > > > > > if [ ${@d.getVar('PACKAGE_NO_GCONV')} -eq 0 ]; then
> > > > > > - mkdir -p ${D}${libdir}
> > > > > > - if [ -e ${LOCALETREESRC}/${libdir}/gconv ]; then
> > > > > > - cp -R --no-dereference --
> > > > > preserve=mode,links ${LOCALETREESRC}/${libdir}/gconv ${D}${libdir}
> > > > > > - fi
> > > > > > - if [ -e ${LOCALETREESRC}/${datadir}/i18n ]; then
> > > > > > - cp -R --no-dereference --
> > > > > preserve=mode,links ${LOCALETREESRC}/${datadir}/i18n ${D}${datadir}
> > > > > > - fi
> > > > > > - fi
> > > > > > - if [ -e ${LOCALETREESRC}/${datadir}/locale ]; then
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/${datadir}/locale ${D}${datadir}
> > > > > > + for d in . $(find
> > > > "${LOCALETREESRC}/${libdir}/gconv"
> > > > > -type d -printf '%P ') ; do
> > > > > > + install -d "${D}${libdir}/gconv/$d"
> > > > > > + find
> > > "${LOCALETREESRC}/${libdir}/gconv/$d"
> > > > -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0755 -t
> > > > > "${D}${libdir}/gconv/$d" {} \;
> > > > > > + done
> > > > > > + for d in . $(find
> > > > "${LOCALETREESRC}/${datadir}/i18n"
> > > > > -type d -printf '%P ') ; do
> > > > > > + install -d "${D}${datadir}/i18n/$d"
> > > > > > + find
> > > "${LOCALETREESRC}/${datadir}/i18n/$d"
> > > > -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0644 -t
> > > > > "${D}${datadir}/i18n/$d" {} \;
> > > > > > + done
> > > > > > fi
> > > > > > - cp -R --no-dereference --preserve=mode,links
> > > > > ${LOCALETREESRC}/SUPPORTED ${WORKDIR}
> > > > > > + for d in . $(find "${LOCALETREESRC}/${datadir}/locale" -
> > > > type
> > > > > d -printf '%P ') ; do
> > > > > > + install -d "${D}${datadir}/locale/$d"
> > > > > > + find "${LOCALETREESRC}/${datadir}/locale/$d" -
> > > > > maxdepth 1 -type f \
> > > > > > + -exec install -m 0644 -t
> > > > "${D}${datadir}/locale/$d"
> > > > > {} \;
> > > > > > + done
> > > > > > + install -m 0644 ${LOCALETREESRC}/SUPPORTED
> > > > > ${WORKDIR}/SUPPORTED
> > > > > > }
> > > > > >
> > > > > > inherit libc-package
> > > > >
> > > > > The trouble is this is a workaround. The cp commands should work
> > > and
> > > > > there is some underlying issue in pseudo causing this. We really
> > > need
> > > > > to figure out what that is since this will likely just mean we see
> > > > the
> > > > > problem somewhere else :(
> > > > >
> > > > > I still suspect some kind of inode number reuse problem which these
> > > > cp
> > > > > commands trigger...
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Richard
> > > >
> > > > For the record, even if I think the patch was the right thing to do,
> > > > it did not help the situation. The first build I did after it was
> > > > integrated, I was bit by the following:
> > > >
> > > > Failed to determine the user name of UID 323 for
> > > > tmp/work/core2-64-poky-linux/glibc-locale/2.29-
> > > r0/package/usr/lib/locale
> > > >
> > > > where UID 323 is my UID. If you do not recognize the error message,
> > > > it is because it comes from a local patch we have for rpm to protect
> > > > us from any incorrect UIDs/GIDs causing incorrect RPMs to be
> > > generated
> > > > (see below). I checked the build directory and the /usr/lib/locale
> > > > directory was the only file or directory with an incorrect owner,
> > > > which is typical of these pseudo related problems with incorrect
> > > > owners.
> > > >
> > > > [rpm will by default silently fall back to use root for any files
> > > > where it cannot determine the name of the user or group based on
> > > > its UID/GID. This can be due to missing information in the
> > > > /etc/passwd or /etc/groups files, typically due to relying on
> > > > another recipe in DEPENDS to create the user without also having
> > > > an RDEPENDS on the package that creates the user, or because of
> > > > pseudo messing up. After this bit us once where a device in /dev
> > > > ended up in an rpm in the sstate cache with root as group instead
> > > > of the intended video group, which in turn caused the video
> > > > application to fail as it was no longer allowed to open its device,
> > > > I created the patch. It is causing a fair bit of grief as it causes
> > > > builds to fail randomly, but at least we don't end up with an
> > > > incorrect sstate anymore, which is worse. I am not sure this patch
> > > > is suitable for integration to OE-Core, but I have attached it if
> > > > anyone is interested.]
> > > >
> > > > //Peter
> > >
> > > Ok, this just got a lot more weird. At the time I wrote the mail
> > > above, I had only tried to rebuild once. Now I have done it a number
> > > of more times, and the result is not random any more. Every build
> > > fails the same way. I've done `bitbake glibc-locale -c cleansstate`
> > > followed by `bitbake glibc-locale` and the result is consistent.
> > > What I have noted now, is that it is _not_ do_install that causes
> > > the problem. The /usr/lib/locale directory created in ${D} has
> > > the correct owner. It is when the data is copied to ${PKGD} that
> > > it somehow ends up with the wrong owner. I cannot tell from a quick
> > > glance what differs /usr/lib/locale from, e.g., /usr/lib/gconv,
> > > where the latter does get the correct owner.
> > >
> > > Unfortunately, I do not have more time to look at this right now,
> > > but I thought I should share this information with you. If anyone
> > > has any ideas of what to look at to determine why
> > > ${PKGD}/usr/lib/locale ends up with the wrong owner, I'm all ears.
> > > It would also be interesting to hear if the result is the same for
> > > anyone else with the patch I provided in my previous mail applied.
> > >
> > > //Peter
> >
> > Further investigation indicates that the problem I'm seeing is not
> > related to pseudo, but an actual problem in the code. I have not dug
> > too far into it, but I'm seeing the problem in the directories used
> > in do_prep_locale_tree() and do_collect_bins_from_locale_tree() in
> > libc-package.bbclass where some of them seems to have been created
> > without being run under pseudo. The following patch works around the
> > problem, though it would probably be better to correct the problem
> > at the source by running whatever part of the code under pseudo that
> > is currently missing it:
> >
> > diff --git a/meta/classes/libc-package.bbclass b/meta/classes/libc-package.bbclass
> > index 0b4c666a74..174465a891 100644
> > --- a/meta/classes/libc-package.bbclass
> > +++ b/meta/classes/libc-package.bbclass
> > @@ -72,6 +72,7 @@ do_prep_locale_tree() {
> > tar -cf - -C ${STAGING_DIR_NATIVE}/${prefix_native}/${base_libdir} -p libgcc_s.* | tar -xf - -C $treedir/${base_libdir}
> > fi
> > install -m 0755 ${LOCALETREESRC}${bindir}/localedef $treedir/${base_bindir}
> > + chown -R root:root $treedir
> > }
> >
> > do_collect_bins_from_locale_tree() {
> >
>
>
> Can you try something on the lines
>
> diff --git a/meta/classes/libc-package.bbclass
> b/meta/classes/libc-package.bbclass
> index 0b4c666a74..34025635ce 100644
> --- a/meta/classes/libc-package.bbclass
> +++ b/meta/classes/libc-package.bbclass
> @@ -61,7 +61,7 @@ LOCALETREESRC ?= "${PKGD}"
> do_prep_locale_tree() {
> treedir=${WORKDIR}/locale-tree
> rm -rf $treedir
> - mkdir -p $treedir/${base_bindir} $treedir/${base_libdir}
> $treedir/${datadir} $treedir/${localedir}
> + install -d $treedir/${base_bindir} $treedir/${base_libdir}
> $treedir/${datadir} $treedir/${localedir}
> tar -cf - -C ${LOCALETREESRC}${datadir} -p i18n | tar -xf - -C
> $treedir/${datadir}
> # unzip to avoid parsing errors
> for i in $treedir/${datadir}/i18n/charmaps/*gz; do
>
>
>
> > //Peter
More information about the Openembedded-core
mailing list