[bitbake-devel] [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 5/12/19 4:28 PM, Richard Purdie wrote:
> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>> e.g.:
>>
>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>> $ bitbake -p
>> [snip]
>> During handling of the above exception, another exception occurred:
>> [snip]
>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>>    hello
>>
>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>> exceptions.
>>
>> [YOCTO #13267]
>>
>> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
>> ---
>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py
>> index f8ae410..585edc5 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>               if multiconfig:
>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>> ), self.data)
>>   
>> -        except (SyntaxError, bb.BBHandledException):
>> +        except bb.BBHandledException:
>> +            raise
>> +        except SyntaxError:
>>               raise bb.BBHandledException
>>           except bb.data_smart.ExpansionError as e:
>>               logger.error(str(e))
> 
> Hi Robert,
> 
> This doesn't sound right, where is this exception being printed a
> second time? The point of "BBHandledException" is to say "don't print
> any further traces for this exception". If this fixes the bug, it means
> something somewhere is printing a trace for a second time when it
> should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes
    one of result_queue or parser_quit have a deadlock, then
    causes the shutdown():process.join() hanged.

2) The sub processes, parser_quit and result_queue are running dependently,
    we can't make sure that cooker.py:Parser():realrun() can get data from
    parser_quit, so there is a race issue, the Parser's subprocess maybe already
    exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator()
to fix the deadlock problem, and we may need change the working way of
parser_quit. But I'm not sure, I need your suggestions before work on it.

// Robert

> 
> Cheers,
> 
> Richard
> 
>