[OE-core] Add warning in externalsrc when dir from tasks' cleandirs is removed because EXTERNALSRC is a parent

[Quentin Schulz]

Hi all,

I've struggled to debug one issue that happened to me during the
do_package task of one of my recipes.

The issue was particularly weird since I could build the package or the
image requiring the package, once and only once. After that first time,
I had to do bitbake -c clean `package` to be able to complete a build.

The error was the following:
ERROR: project-1.0-r0 do_package: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:populate_packages(d)
     0003:
File: '/media/qschulz/yocto/build/poky/meta/classes/package.bbclass', lineno: 1230, function: populate_packages
     1226:
     1227:            mkdir_recurse(dvar, root, os.path.dirname(file))
     1228:            fpath = os.path.join(root,file)
     1229:            if not cpath.islink(file):
 *** 1230:                os.link(file, fpath)
     1231:                continue
     1232:            ret = bb.utils.copyfile(file, fpath)
     1233:            if ret is False or ret == 0:
     1234:                bb.fatal("File population failed")
Exception: FileExistsError: [Errno 17] File exists: './usr/share/file' -> '/media/qschulz/yocto/build/.build-yocto/tmp/work/machine-poky-linux/project/1.0-r0/packages-split/project-dbg/./usr/share/file'

ERROR: project-1.0-r0 do_package: Function failed: populate_packages
ERROR: Logfile of failure stored in: /media/qschulz/yocto/build/.build-yocto/tmp/work/machine-poky-linux/project/1.0-r0/temp/log.do_package.26719
ERROR: Task (/media/qschulz/yocto/build/oe/recipes/project/project.bb:do_package) failed with exit code '1'

We have a krogoth branch that still works fine with the same recipe,
while the one on Thud just does not want to build (well, to be fair,
only after the first build).

So, I actually found that the culprit might well be:
https://git.yoctoproject.org/cgit.cgi/poky/commit/?id=5e6ffb2adbf544700470bbdb9428c2e020c9f8a6

The issue is that we have the following tree:
/path/project
/path/project/yocto/meta-*
/path/project/yocto/build

And /path/project is where all the sources for the externalsrc package
`project` is. And with the above patch, we stop removing directories
that are leaves of an externalsrc package.

And because package, packages-split and pkgdata are cleandirs of
do_package and are also indirect leaves of /path/project, they do not
get removed. The issue is that then all the files in packages-split are
still there, and creating a link with os.link while trying to overwrite
a file, throws an Exception.

While I can agree our tree organization is... not perfect, I'd have
liked Yocto to notify me that there might be something unusual going on.
Especially when the behavior changed in between versions. I have to say
that I have absolutely nothing against the aforementioned patch, it
makes a lot of sense and I know we're in a very unusual setup that I'll
try to change. But I think other people might benefit from such a
warning message.

So now, I added a bb.warn() right before removing cleandir from
cleandirs. The issue is that it does not always make sense to warn the
user that a dir has been removed from the list of dirs to be cleaned
before a task. For example, removing dirs from the do_unpack's cleandirs
seems to make sense, so it wouldn't make much sense to warn the user.
However, it is definitely an issue that needs to be raised for, e.g.,
do_package since it'll fail without telling you where the issue
originated from.

What could be done? I don't think we want to keep a list of "safe" or
"unsafe" tasks and warn only when needed.

Let me know your thoughts,
Thanks,
Quentin
-- 
StreamUnlimited Engineering GmbH
High Tech Campus Vienna, Gutheil-Schoder-Gasse 10, 1100 Vienna, Austria
quentin.schulz at streamunlimited.com, www.streamunlimited.com