[OE-core] Deployment for machine X will remove its results from machine Y's deploy dir

[Mike Looijmans]

On 11/27/2014 03:41 PM, Richard Purdie wrote:
> On Thu, 2014-11-27 at 15:22 +0100, Mike Looijmans wrote:
>> On 11/27/2014 02:17 PM, Richard Purdie wrote:
>>> On Thu, 2014-11-27 at 05:02 -0700, Gary Thomas wrote:
>>>> On 2014-11-27 01:35, Mike Looijmans wrote:
>>>>> Here's an example recipe to demonstrate the issue. Save it as "deployme.bb" into a recipe dir. Then build it for two machines. Building it for one machine will remove it from the
>>>>> deployment directory of the other. This problem has been bugging me for months, I had files just "disappear" mysteriously from the deploy directory and seemingly random times, and
>>>>> now I finally figured out what causes it.
>>>>>
>>>>> (cut here)
>>>>>
>>>>> SUMMARY = "Demonstrate a bug in OE deployment"
>>>>> DESCRIPTION = "Build this package for a machine X, then look at the image's \
>>>>>     deploy directory. You'll see a deployme.txt there. Now build it for another \
>>>>>     machine, e.g. "Y". The deployme.txt for machine X will have disappeared \
>>>>>     from the image dir. This appears to be a bug in OE's deployment."
>>>>> LICENSE = "GPLv2"
>>>>> LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690"
>>>>>
>>>>> inherit allarch deploy
>>>>>
>>>>> do_compile () {
>>>>>        echo "Hello world!" > deployme.txt
>>>>> }
>>>>>
>>>>> do_deploy () {
>>>>>        install -d ${DEPLOYDIR}
>>>>>        install -m 644 ${B}/deployme.txt ${DEPLOYDIR}/
>>>>> }
>>>>>
>>>>> addtask deploy before do_build after do_compile
>>>>>
>>>>> (cut here)
>>>>
>>>> Very interesting & verified with the latest master.
>>>>
>>>> Have you filed a bug?  https://bugzilla.yoctoproject.org/
>>>
>>> Well, I'm not convinced this is a bug as such. You've created an
>>> "allarch" deploy task, how would you expect this to behave?
>>>
>>> "allarch" means that the output from this task is universal and can be
>>> used on all targets. It will therefore get run once.
>>>
>>> A "deploy" task is machine specific.
>>>
>>> What ends up happening is therefore the task has a stamp is
>>> "universally" created. When you change machine, the checksum of the task
>>> changes, the previous version is removed, the new version is installed.
>>>
>>> So in many ways the system is doing exactly what I would expect it to do
>>> and it isn't a bug in that sense.
>>
>> It's not a bug in the sense that it doesn't do as it was programmed to do.
>
> Its doing *exactly* what the was designed to do. That doesn't match what
> you want/expect though.
>
>>   I understand what's happening here.
>>
>> I just think that the logic here is wrong. If "deploy" is machine specific,
>> then the implicit "undeploy" should be machine specific too, right?
>
> Well, its more complex than that.
>
> deploy.bbclass defaults to DEPLOY_DIR_IMAGE.
>
> DEPLOY_DIR_IMAGE defaults to ${DEPLOY_DIR}/images/${MACHINE}
>
> I actually put off merging the latter since I knew it would cause
> issues, I just couldn't articulate all of them at the time :(.
>
>>> The real question is how should an "allarch" + "deploy" task behave when
>>> you've specified machine specific paths? Perhaps erroring would be
>>> better?
>>
>> That would mean that roughly all deploy tasks will fail.
>
> I'm not sure we have many deploy+allarch tasks so I think "roughly none"
> would be a better description.
>
> deploy is usually used for bootloaders and kernels, both of which are
> not allarch.
>
>> At best they're tied
>> to MACHINE_ARCH, but never to MACHINE itself.
>
> No, they're tried to MACHINE itself, see above.
>
>> Would be strange to put PACKAGE_ARCH="${MACHINE}" in a recipe that clearly has
>> no dependency on machine specific things. And I wrote "${MACHINE}" here on
>> purpose.
>
> Dropping the "allarch" would be better than that.
>
>> I was thinking along the lines of "DEPLOY_DIR_IMAGE must have the same prefix"
>> or so.
>>
>> If I knew the solution, I'd have posted a patch instead of a question or report.
>
> Well, allarch.bbclass could override DEPLOY_DIR_IMAGE to remove
> the /${MACHINE} suffix. That would appear to fix the issues you're
> seeing, at the risk of having a different group of people upset that you
> don't get a complete directory per machine.
>
> It comes down to which behaviour we want. Changing MACHINE in the
> definition of DEPLOY_DIR_IMAGE to PACKAGE_ARCH might be the better
> solution, then it will deploy based on how specific or not specific the
> resulting code it. That will likely upset certain people even more
> though. The other option is to accept that its machine specific and do
> PACKAGE_ARCH="${MACHINE_ARCH}" in the class. That is suboptimal for the
> reasons you describe but would get the behaviour some people want.

We might be able to reduce the number of upset people drastically by doing the 
deployment in two stages.

Step one is to deploy the binaries to ${DEPLOY_DIR}/${PACKAGE_ARCH} as per 
your suggestion. Having pondered about this for many days now, that is 
actually the only thing that is "right" from a technical point of view.

That will wreak havoc on all scripts that want to take the deployment results 
and copy them to target devices (I have quite a few of those scripts myself), 
because these now suddenly need to become aware of which package is for what 
architecture.

That can be solved in a second step: For the MACHINE, create symbolic links 
from ${DEPLOY_DIR}/${MACHINE} to the files in ${DEPLOY_DIR}/${PACKAGE_ARCH}. 
Most packages (at least, bootloader and kernel do), already create symlinks to 
actual targets, so I'd expect existing scripts to be able to cope with that 
already. This second step is purely MACHINE specific, and only serves to help 
gather everything that is needed for a machine deployment. This second step is 
to be rerun for each MACHINE.

This second step could even be automated, one could iterate through all 
supported ARCH for the machine, and create a symlink for each and every file 
found there in the ${DEPLOY_DIR}/${MACHINE} directory.


The only problem I can see with this approach is that you may end up with 
dangling symlinks for machine B after updating and deploying a shared package 
for machine A, or worse, a version mismatch of the deployed target.



> An opinion from Martin/Koen might be useful at this point since it could
> affect them more than others (as well as things like the yocto project
> autobuilder output and output processing/testing/releasing).

So far, we haven't heard from them :(



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans at topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/