[OE-core] Fix for the APIC hangs in qemux86-64

Thu Dec 14 01:12:14 UTC 2017

On 2017-12-13 7:04 PM, Richard Purdie wrote:
> On Wed, 2017-12-13 at 19:01 -0500, Bruce Ashfield wrote:
>> On 2017-12-13 9:14 AM, Richard Purdie wrote:
>>>
>>> On Wed, 2017-12-13 at 09:07 -0500, Bruce Ashfield wrote:
>>>>
>>>> On 12/13/2017 09:05 AM, Richard Purdie wrote:
>>>>>
>>>>>
>>>>> On Wed, 2017-12-13 at 08:38 -0500, Bruce Ashfield wrote:
>>>>>>
>>>>>>
>>>>>> On 12/13/2017 07:34 AM, Richard Purdie wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Bruce,
>>>>>>>
>>>>>>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12301
>>>>>>>
>>>>>>> We've root caused the hangs we're seeing in qemux86-64 on
>>>>>>> our
>>>>>>> autobuilder workers. We've identified a commit on 4.15-rc3
>>>>>>> which
>>>>>>> fixes
>>>>>>> the issue but backporting it to the autobuilder's host
>>>>>>> kernels
>>>>>>> will
>>>>>>> be
>>>>>>> a pain.
>>>>>>>
>>>>>>> As a workaround I'd like to enable x2apic on qemux86-64.
>>>>>>> Since
>>>>>>> that
>>>>>>> works via MSRs it avoids the problem.
>>>>>>>
>>>>>>> Would it be a big issue to set CONFIG_X86_X2APIC=y for
>>>>>>> qemux86-
>>>>>>> 64?
>>>>>> Nope. No big deal at all. I've used the same config i the
>>>>>> past
>>>>>> when doing work with rt-KVM, so I know it well.
>>>>>>
>>>>>> I'll prep a commit, and send it along as part of a rebased
>>>>>> set
>>>>>> of kernel patches.
>>>>> Thanks, much appreciated.
>>>>>
>>>>> Did we get to the bottom of the 4.12 selftest failures?
>>>>>
>>>> Not yet no. I was distracted for a day, but need to learn how
>>>> to run them myself. I was able to log in via serial in my normal
>>>> boot flow, so that didn't get me anywhere.
>>>>
>>>> Is the selftest HOWO in the yocto docs ? I'll google it up this
>>>> morning.
>>> No HOWTO but I did give you the command. In a poky checkout run:
>>>
>>> oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
>>>
>>> which runs a single test that fails (setup DL_DIR/SSTATE_MIRRORS
>>> first
>>> to speed it up a bit).
>>>
>>> It may tell you to clear SANITY_TESTED_DISTROS, I think it auto
>>> adds
>>> meta-selftest now. You may also need to set DISPLAY to something
>>> (e.g.
>>> a dummy vncserver).
>>>
>> With the cleanup that Ross gave me, I'm now able to see the issue.
>>
>> I'll start debugging on Thursday.
>>
>> One additional question: what's the right way to find the logs of
>> that
>> run ?
>>
>> I'm getting a bit of noise from:
>>
>> ----------------
>>        0155:            else:
>> Exception: FileNotFoundError: [Errno 2] No such file or directory:
>> '/home/bruce/poky/build/tmp/sysroots-components/x86_64/python3-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>> ->
>> '/home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/recipe-sysroot-
>> native/usr/lib/python3.5/sqlite3/__pycache__/dump.cpython-35.opt-
>> 1.pyc'
>>
>> ERROR: gnu-efi-3.0.6-r0 do_package: Function failed:
>> extend_recipe_sysroot
>> ERROR: Logfile of failure stored in:
>> /home/bruce/poky/build/tmp/work/core2-64-poky-linux/gnu-efi/3.0.6-
>> r0/temp/log.do_package.10495
> 
> You've deleted pyc files from python itself which is bad. I'd
> cleansstate python-native and rebuilt it to fix that. Ross meant onto
> to run it on the poky meta directory...
> 

Is this what I should expect for the error ?

It looked like qemu timed out, but that is probably just the log
in issue, correct ?

A manual log into the same image worked!

Bruce

-------------------
32105  3290 python3 /home/bruce/poky/scripts/oe-selftest -r 
runqemu.RunqemuTests.test_boot_deploy
32520 32105 /bin/sh -c runqemu nographic 
/home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 
bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" 
qemuparams="-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105"
32521 32520 python3 /home/bruce/poky/scripts/runqemu nographic 
/home/bruce/poky/build/tmp/deploy/images/qemux86-64 tcpserial=51081 
bootparams=console=tty1 console=ttyS0,115200n8 printk.time=1 
qemuparams=-serial tcp:127.0.0.1:44545 -pidfile pidfile_32105
32522 32105 python3 /home/bruce/poky/scripts/oe-selftest -r 
runqemu.RunqemuTests.test_boot_deploy
32527 32521 /bin/sh -c sudo /home/bruce/poky/scripts/runqemu-ifup 1000 
1000 
/home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32528 32527 sudo /home/bruce/poky/scripts/runqemu-ifup 1000 1000 
/home/bruce/poky/build/tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin
32529     2 [kworker/u8:1]
32530 32105 ps axww -o pid,ppid,command

Qemu ended unexpectedly, dump data from host is in 
/tmp/oe-saved-tests/201712131950_qemu
Sending SIGTERM to runqemu
Sending SIGKILL to runqemu
Output from runqemu:
runqemu - INFO - Continuing with the following parameters:

runqemu - INFO - Setting up tap interface under sudo

----------------------------------------------------------------------
2017-12-13 19:52:57,550 - oe-selftest - INFO - Ran 1 test in 257.941s
2017-12-13 19:52:57,550 - oe-selftest - INFO - FAILED
2017-12-13 19:52:57,550 - oe-selftest - INFO -  (errors=1)
2017-12-13 19:52:57,550 - oe-selftest - INFO - RESULTS:
2017-12-13 19:52:57,551 - oe-selftest - INFO - RESULTS - 
runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: ERROR
2017-12-13 19:52:57,551 - oe-selftest - INFO - SUMMARY:
2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest () - Ran 1 
test in 257.946s
2017-12-13 19:52:57,551 - oe-selftest - INFO - oe-selftest - FAIL - 
Required tests failed

> Cheers,
> 
> Richard
> 
> 