[OE-core] [PATCH] rootfs: always update the opkg index

[Ioan-Adrian Ratiu]

On Tue, 25 Sep 2018, Ioan-Adrian Ratiu <adrian.ratiu at ni.com> wrote:
> On Mon, 24 Sep 2018, richard.purdie at linuxfoundation.org wrote:
>> On Mon, 2018-09-24 at 18:00 +0300, Ioan-Adrian Ratiu wrote:
>>> > With master, a copy of the feed is constructed under WORKDIR which
>>> > is
>>> > then indexed since other image generation or package writes could
>>> > occur
>>> > and the index isn't static. I'm guessing it doesn't do this for
>>> > external feeds and assumes those feeds are static.
>>> 
>>> My description of the problem mirrors my understanding of the feed
>>> construction process which might be limited especially if it changed
>>> on master post-Sumo, but the answer is yes, external feeds are
>>> assumed static.
>>
>> Ok.
>>
>>> > I am wondering what happens if you put a remote (http://) url into
>>> > this
>>> > though? Does that still work or does it break?
>>> 
>>> It still works, we're using http paths all the time in our distro
>>> (public feed server is at http://download.ni.com/ni-linux-rt/)
>>
>> That is good to know and worth mentioning in the commit message as that
>> isn't clear.
>>
>>> > 
>>> > I'm also worried about what happens if the file:// path you point
>>> > at is
>>> > owned by another user and isn't writable.
>>> 
>>> Thanks for pointing this out. Looks like it enters an infinite loop:
>>> the
>>> do_rootfs task is stuck at 3% and a worker process pegs a core to
>>> 100%.
>>> 
>>> Without this change, do_rootfs gets stuck at 12%, worker pegging a
>>> core
>>> if the feed is owned by another user with no write permissions.
>>> 
>>> This was unexpected :) I'll roll up my sleeves and start digging.
>>> 
>>> Do you have any pointers about what might be happening?
>>
>> I was expecting it to error, not hang. If I had to totally guess, I'd
>> guess at the bb.utils.lockfile() hanging.
>
> I can't reproduce the hang anymore (with or without my change)... I
> tried re-using the sstate, clearing it, all combinations of calling
> bitbake packagegroups, package-index, image, etc and still no hang.
>
> I'm thinking the hang was caused by a build race I can't easily
> reproduce.
>
> Now I get "operation not permitted" errors every time like this one:
>
> Exception: subprocess.CalledProcessError: Command 'cd
> (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks; 
> find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/deploy-ipks
> -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xhf - -C (...)/tmp-glibc/deploy/ipk' returned non-zero exit status 2.
>
> Subprocess output:
> tar: ./core2-64: Cannot utime: Operation not permitted
> tar: .: Cannot utime: Operation not permitted
> tar: Exiting with failure status due to previous errors
>
> ERROR: safemode-image-1.0-r0 do_package_write_ipk: Function failed:
> sstate_task_postfunc
> ERROR: Logfile of failure stored in:
> (...)/tmp-glibc/work/core2-64-nilrt-linux/safemode-image/1.0-r0/temp/log.do_package_write_ipk.21668
>
> Seeing that the hang is not related to this change, are you ok with
> integrating v2 in which I'll update the commit message to mention
> http:// URI's still work as expected?

Please ignore this message, I was misled by the bad naming (the
safemode-image is actually a package), the hang reproduces, I'm on it.

>
>>
>> Cheers,
>>
>> Richard