[OE-core] Using users/groups from another recipe than the one creating them

Tue Jun 10 10:04:54 UTC 2014

> -----Original Message-----
> From: openembedded-core-bounces at lists.openembedded.org
> [mailto:openembedded-core-bounces at lists.openembedded.org] On Behalf Of
> Mark Hatle
> Sent: den 9 juni 2014 19:03
> To: Peter Kjellerstedt; OE Core (openembedded-
> core at lists.openembedded.org)
> Subject: Re: [OE-core] Using users/groups from another recipe than the
> one creating them
> 
> On 6/9/14, 11:47 AM, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: openembedded-core-bounces at lists.openembedded.org
> >> [mailto:openembedded-core-bounces at lists.openembedded.org] On Behalf
> >> Of Mark Hatle
> >> Sent: den 9 juni 2014 16:47
> >> To: Peter Kjellerstedt; OE Core (openembedded-
> >> core at lists.openembedded.org)
> >> Subject: Re: [OE-core] Using users/groups from another recipe than
> >> the one creating them
> >>
> >> On 6/9/14, 8:39 AM, Peter Kjellerstedt wrote:
> >>>> -----Original Message-----
> >>>> From: openembedded-core-bounces at lists.openembedded.org
> >>>> [mailto:openembedded-core-bounces at lists.openembedded.org] On
> >>>> Behalf Of Peter Kjellerstedt
> >>>> Sent: den 23 maj 2014 12:38
> >>>> To: OE Core (openembedded-core at lists.openembedded.org)
> >>>> Subject: Re: [OE-core] Using users/groups from another recipe than
> >>>> the one creating them
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: openembedded-core-bounces at lists.openembedded.org
> >>>>> [mailto:openembedded-core-bounces at lists.openembedded.org] On
> >>>>> Behalf Of Peter Kjellerstedt
> >>>>> Sent: den 19 maj 2014 10:15
> >>>>> To: OE Core (openembedded-core at lists.openembedded.org)
> >>>>> Subject: [OE-core] Using users/groups from another recipe than
> >>>>> the one creating them
> >>>>>
> >>>>> Which assumption is correct: "a recipe A that depends on another
> >>>>> recipe B can use users/groups that B creates" or "all recipes
> >>>>> must create the users/groups they require themselves"?
> >>>>>
> >>>>> The problem for us is that we have a lot of recipes that create
> >>>>> users and groups, and subsequently a number of other related
> >>>>> recipes that need to use those users and groups.
> >>>>>
> >>>>> If the first assumption is correct then the useradd.bbclass needs
> >>>>> to be corrected so that it adds a dependency from do_install to
> >>>>> base-passwd:do_populate_sysroot and
> >>>>> base-passwd:do_populate_sysroot_setscene, because if either of
> >>>>> those tasks execute they will overwrite /etc/passwd and
> >>>>> /etc/group, effectively removing any users/groups that were 
> >>>>> created earlier...
> >>>>>
> >>>>> On the other hand, if it is the second assumption that is correct,
> >>>>> then there should be QA tests in place to make sure all recipes
> >>>>> create the users/groups they use.
> >>>>>
> >>>>> //Peter
> >>>>
> >>>> *ping*
> >>>>
> >>>> Doesn't anyone know how users and groups are supposed to work?
> >>>>
> >>>> //Peter
> >>>
> >>> *ping again*
> >>
> >> I never saw the original emails, either of them.
> >>
> >>> Well, this is somewhat discouraging. Three weeks and not a single
> >>> response. Are we really the only ones that care about users and
> >>> groups and how they are created? Doesn't anyone know which of my
> >>> two assumptions above are correct?
> >>>
> >>> Personally I would prefer that all recipes create the users and
> >>> groups they require themselves as this keeps them more self
> >>> contained. I have no idea how to write a QA test to verify this,
> >>> but I assume it should be possible...
> >>
> >> The rule is:
> >>
> >> There is a limited number of base users and groups, if your package
> >> uses a user/group in that base set nothing else is needed.
> >>
> >> If your package needs an additional user/group, then it must either:
> >>
> >> *) Add the user/group itself.  Multiple packages are allowed to add
> >>     the same users/groups, with the understanding that the options
> >>     -must- be identical!
> >
> > I have actually been toying with an extension to the
> > useradd.bbclass that allows one to configure the user and group
> > options in one configuration file, and then just state in the
> > recipes which users are needed. That way there is no risk of using
> > different options for the same user if it is created from different
> > recipes. Haven't finished it though. One thing I have not solved is
> > how to handle requiring the configuration file from all layers as
> > they may all need to add users (or what to do if multiple layers
> > define the same user differently...)
> 
> If you implement this, you should follow the example of the
> "useradd-staticids.bbclass".  This is something that a user can
> optionally add into their environment to better define how things 
> are configured.
> 
> I still contend that these items belong in the recipes and that it's 
> a recipe bug if they're not done properly.  We can help automate this, 
> but I don't know if that is yet a reasonable general solution.
> 
> The staticids class collects together "USERADD_UID_TABLES" values, 
> and then rewrites the USERADD_PARAMS/GROUPADD_PARAMS to match whatever 
> the table(s) say.  This is the same mechanism that we use for the 
> filesystem table to synchronize standard owners/groups/perms in various 
> recipes.
> 
> I would still expect the recipe to say I need the following
> users/groups.. and then to pick up the data from the tables if necessary.

Thanks for the tips. I will definitely look at what 
useradd-staticids.bbclass does if I decide to continue to work 
on this.

> >> *) Must -require- (not recommend) a package that adds the group that
> >>     they require.  It's a good idea in the recipe to comment that
> >>     the 'require' adds the user/group as well.
> >>
> >> This is pretty common where a single recipe provides multiple
> >> packages.  If a common package creates the user/group, then all 
> >> of the other packages [that need that user/group] should then 
> >> require the common package.
> >
> > Ok, then my first assumption was the correct one. In that case I
> > will look into fixing the useradd.bbclass so that the missing
> > dependency is added.
> 
> useradd isn't the right place for the dependency, it really is a 
> recipe level dependency.

useradd.bbclass adds a dependency on base-passwd for each recipe 
that inherits it. In addition it does this:

USERADDSETSCENEDEPS = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
USERADDSETSCENEDEPS_class-cross = ""
USERADDSETSCENEDEPS_class-native = ""
USERADDSETSCENEDEPS_class-nativesdk = ""
do_package_setscene[depends] += "${USERADDSETSCENEDEPS}"
do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}"

However, this does not seem to trigger anything. If I first do 
"bitbake base-passwd -c clean && bitbake base-passwd" (which 
will wipe /etc/passwd and /etc/groups in the sysroot) and then 
try to rebuild a recipe that creates a user, e.g. "bitbake dbus", 
then noting happens and the messagebus user is not added to 
/etc/passwd. Consequently, any other recipes that rely on dbus 
to create the messagebus user will then fail if one tries to 
rebuild them.

I tried to add the following to see if any of it did any 
difference, but it didn't:

USERADDDEPS = "${MLPREFIX}base-passwd:do_populate_sysroot ${MLPREFIX}base-passwd:do_populate_sysroot_setscene"
USERADDDEPS_class-cross = ""
USERADDDEPS_class-native = ""
USERADDDEPS_class-nativesdk = ""
do_install[depends] += "${USERADDDEPS}"
do_package[depends] += "${USERADDDEPS}"
do_populate_sysroot[depends] += "${USERADDDEPS}"

My guess is that since the signature of the
base-passwd:do_populate_sysroot_setscene task has not changed it 
will not trigger the dependent tasks to run even if it has changed 
the sysroot...

Since it does not seem possible to trigger the recipes that 
install users to redo it after base-passwd has trashed /etc/passwd, 
maybe the solution is to use the "database" you describe below. 
That way, when base-passwd does its job, it would only affect the 
users it has installed itself and leave the others alone.

> >> As far as the QA resources go.  There are actually two sides to
> >> the QA. The first is in the do_install step, if the users/groups
> >> are being used there -- and don't exist -- a failure should be
> >> generated.  That exists just by the nature of the system build
> >> processes.
> >>
> >> The second is that once a package has been generated (the package
> >> directory that is), it should be verified that all files are owned
> >> by in the base set of users/groups, the user/group has been added,
> >> or a dependency on the package that added them exist.  This is
> >> difficult to do though.  There is currently no tracking mechanism
> >> in the system to say which recipes added which users/groups to the
> >> system.  A mechanism similar to the sysroot file population would
> >> be required to capture the user and group changes and by which
> >> package(s). This of course would also have to work with the
> >> sstate-cache mechanism.
> >>
> >> The first step in implementing this would be to capture the
> >> user/group changes and record them in a database.  (again, I think
> >> similar to the populate sysroot file).  Then during the QA process,
> >> you can iterate over the files and find out where each user/group
> >> in use comes from.  If the provider is not in the 'required' set of
> >> packages generate a QA -warning- (since this is new and unproven
> >> code).. We can upgrade it to an error once it's stable.
> >
> > Hmm, ok. This all sound like something we want. However, my Python
> > skills are probably not up to the task, unfortunately.
> 
> You should look into how the populate_sysroot function writes out 
> the list of files installed into the sysroot.. and see if you can 
> do something similar for the packages.  It's python, but I don't 
> think it's that heavy of a python code.
> 
> --Mark

I will have to save the database idea to after I am back from my 
summer vacation in August. I will not have time to do any work 
on it before then anyway.

//Peter