[oe] [meta-oe][PATCH] protobuf-c: fix race condition
akuster808
akuster808 at gmail.com
Wed Oct 23 15:50:10 UTC 2019
On 10/22/19 2:37 PM, Martin Jansa wrote:
> On Fri, Oct 18, 2019 at 09:06:54PM +0200, Martin Jansa wrote:
>> One more data point is that it might be fixed completely with 1.3.2 version
>> used in zeus/master and failing less often then without this patch in 1.3.1
>> used in thud/warrior.
>>
>> I was rebuilding it 300 times in thud/warrior/zeus and got 5-6 failures in
>> thud/warrior but none in zeus.
>>
>> I've started another for loop to rebuilt it 1000 time with zeus and if it
>> works then I'll retest warrior and thud with 1.3.2 upgrade backported to
>> them - if it works then we can either backport whole upgrade or check the
>> 1.3.1 1.3.2 diff to find out what else we need to backport from there.
> 1000 times in zeus = 0 failures
> 100 times in thud with 1.3.2 backported ~ 10 failures
> 1000 times in thud with 1.3.2 as well as protobuf-native 3.9.2 = 0 failures
>
> The protobuf-c upgrade isn't need to be backported to thud, there aren't
> many changes:
> https://github.com/protobuf-c/protobuf-c/compare/269771b4b45d3aba04e59569f53600003db8d9ff...1390409f4ee4e26d0635310995b516eb702c3f9e
>
> protobuf on the other hand was upgraded from 3.6.1 to 3.9.2 in zeus and
> the diff is big (1028 commits):
> https://github.com/protocolbuffers/protobuf/compare/48cb18e5c419ddd23d9badcfe4e9df7bde1979b2...52b2447247f535663ac1c292e088b4b27d2910ef
>
> quick grep in git log didn't reveal what might be causing the race
> condition, upgrading whole protobuf in warrior and thud isn't an option
> as well.
>
> Anyone interested in digging a bit deeper?
>
> I don't want to block this change, because it seems to completely fix it
> for zeus/master and partially improve it also for warrior and thud.
>
> We don't use protobuf much, so for our builds I think I'll just backport
> whole protobuf upgrade to thud once protobuf-c fix is in thud.
So is https://patchwork.openembedded.org/patch/165905/ still a go for
warrior?
- armin
> Regards,
>
>> On Fri, Oct 18, 2019 at 11:07 AM Khem Raj <raj.khem at gmail.com> wrote:
>>
>>> Yes I have seen same build failures on master even after this fix on
>>> machine with more cpus so the fix while seems to address the problem is not
>>> fixing is completely when I get a chance I will look into the logic myself
>>> too but I think we have to delve into it a bit More
>>>
>>> On Fri, Oct 18, 2019 at 3:41 AM Martin Jansa <martin.jansa at gmail.com>
>>> wrote:
>>>
>>>> Hmm I spoke too soon, at least rebuilding protobuf-c-native with this
>>>> change backported to thud still failed once with:
>>>>
>>>> ../git/t/generated-code2/cxx-generate-packed-data.cc:28:17: error: ‘foo’
>>>> is
>>>> not a namespace-name
>>>> using namespace foo;
>>>> ^~~
>>>> ../git/t/generated-code2/cxx-generate-packed-data.cc:28:20: error:
>>>> expected
>>>> namespace-name before ‘;’ token
>>>> using namespace foo;
>>>> ^
>>>> ../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
>>>> ‘TestEnumSmall’ does not name a type
>>>> #define TEST_ENUM_SMALL_TYPE_NAME TestEnumSmall
>>>> ^
>>>> ../git/t/generated-code2/common-test-arrays.h:44:1: note: in expansion of
>>>> macro ‘TEST_ENUM_SMALL_TYPE_NAME’
>>>> TEST_ENUM_SMALL_TYPE_NAME enum_small_0[] = { TEST_ENUM_SMALL(VALUE) };
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>> ../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
>>>> ‘TestEnumSmall’ does not name a type
>>>> #define TEST_ENUM_SMALL_TYPE_NAME TestEnumSmall
>>>> ^
>>>> ../git/t/generated-code2/common-test-arrays.h:45:1: note: in expansion of
>>>> macro ‘TEST_ENUM_SMALL_TYPE_NAME’
>>>> TEST_ENUM_SMALL_TYPE_NAME enum_small_1[] = { TEST_ENUM_SMALL(OTHER_VALUE)
>>>> };
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>> ../git/t/generated-code2/cxx-generate-packed-data.cc:31:36: error:
>>>> ‘TestEnumSmall’ does not name a type
>>>> #define TEST_ENUM_SMALL_TYPE_NAME TestEnumSmall
>>>> ^
>>>> ../git/t/generated-code2/common-test-arrays.h:47:1: note: in expansion of
>>>> macro ‘TEST_ENUM_SMALL_TYPE_NAME’
>>>> TEST_ENUM_SMALL_TYPE_NAME enum_small_random[] =
>>>>
>>>> {T(0),T(1),T(1),T(0),T(0),T(1),T(1),T(1),T(0),T(0),T(0),T(0),T(0),T(1),T(1),T(1),T(1),T(1),T(1),T(0),T(1),T(1),T(0),T(1),T(1),T(0)
>>>> };
>>>> ^~~~~~~~~~~~~~~~~~~~~~~~~
>>>> ...
>>>> so it probably doesn't fix the issue completely.
>>>>
>>>>
>>>> On Thu, Oct 17, 2019 at 10:44 PM Martin Jansa <martin.jansa at gmail.com>
>>>> wrote:
>>>>
>>>>> On Sun, Sep 29, 2019 at 05:29:56PM +0800, Chen Qi wrote:
>>>>>> Signed-off-by: Chen Qi <Qi.Chen at windriver.com>
>>>>>> ---
>>>>>> .../0001-avoid-race-condition.patch | 36
>>>> +++++++++++++++++++
>>>>>> .../protobuf/protobuf-c_1.3.2.bb | 1 +
>>>>>> 2 files changed, 37 insertions(+)
>>>>>> create mode 100644
>>>> meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
>>>>>> diff --git
>>>> a/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
>>>> b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
>>>>>> new file mode 100644
>>>>>> index 000000000..4fc7703d8
>>>>>> --- /dev/null
>>>>>> +++
>>>> b/meta-oe/recipes-devtools/protobuf/protobuf-c/0001-avoid-race-condition.patch
>>>>>> @@ -0,0 +1,36 @@
>>>>>> +From 216e31260b618ec73862f9f5336597f391444dac Mon Sep 17 00:00:00
>>>> 2001
>>>>>> +From: Chen Qi <Qi.Chen at windriver.com>
>>>>>> +Date: Sun, 29 Sep 2019 17:20:42 +0800
>>>>>> +Subject: [PATCH] avoid race condition
>>>>>> +
>>>>>> +It's possible that the cxx-generate-packed-data.cc is compiled
>>>>>> +while the t/test-full.pb.h is being generated. This will result
>>>>>> +the following error.
>>>>>> +
>>>>>> + DEBUG: ./t/test-full.pb.h:4:0: error: unterminated #ifndef
>>>>>> + ./t/test-full.pb.h:4:0: error: unterminated #ifndef
>>>>>> +
>>>>>> +Add a dependency to avoid such problem.
>>>>>> +
>>>>>> +Upstream-Status: Pending
>>>>> Thanks for this patch, it seems to indeed fix this issue, I've reported
>>>>> a while ago:
>>>>>
>>>>>
>>>> https://www.mail-archive.com/openembedded-devel@lists.openembedded.org/msg64363.html
>>>>> I think the upstream would be interested in this fix as well, see:
>>>>>
>>>>>
>>>> https://github.com/protobuf-c/protobuf-c/commit/b0bb56303366e2c072ee38eb5f1f11251b07239c
>>>>> Cheers,
>>>>>
>>>>>> +
>>>>>> +Signed-off-by: Chen Qi <Qi.Chen at windriver.com>
>>>>>> +---
>>>>>> + Makefile.am | 1 +
>>>>>> + 1 file changed, 1 insertion(+)
>>>>>> +
>>>>>> +diff --git a/Makefile.am b/Makefile.am
>>>>>> +index b0cb065..1608ae0 100644
>>>>>> +--- a/Makefile.am
>>>>>> ++++ b/Makefile.am
>>>>>> +@@ -156,6 +156,7 @@ noinst_PROGRAMS += \
>>>>>> + t_generated_code2_cxx_generate_packed_data_SOURCES = \
>>>>>> + t/generated-code2/cxx-generate-packed-data.cc \
>>>>>> + t/test-full.pb.cc
>>>>>> ++t/generated-code2/cxx-generate-packed-data.cc: t/test-full.pb.h
>>>>>> + $(t_generated_code2_cxx_generate_packed_data_OBJECTS):
>>>> t/test-full.pb.h
>>>>>> + t_generated_code2_cxx_generate_packed_data_CXXFLAGS = \
>>>>>> + $(AM_CXXFLAGS) \
>>>>>> +--
>>>>>> +2.17.1
>>>>>> +
>>>>>> diff --git a/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb
>>>>> b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb
>>>>>> index 6d1ffc3f4..b92f82dec 100644
>>>>>> --- a/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb
>>>>>> +++ b/meta-oe/recipes-devtools/protobuf/protobuf-c_1.3.2.bb
>>>>>> @@ -15,6 +15,7 @@ DEPENDS = "protobuf-native protobuf"
>>>>>> SRCREV = "1390409f4ee4e26d0635310995b516eb702c3f9e"
>>>>>>
>>>>>> SRC_URI = "git://github.com/protobuf-c/protobuf-c.git \
>>>>>> + file://0001-avoid-race-condition.patch \
>>>>>> "
>>>>>>
>>>>>> S = "${WORKDIR}/git"
>>>>>> --
>>>>>> 2.17.1
>>>>>>
>>>>>> --
>>>>>> _______________________________________________
>>>>>> Openembedded-devel mailing list
>>>>>> Openembedded-devel at lists.openembedded.org
>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>>>> --
>>>>> Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
>>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-devel mailing list
>>>> Openembedded-devel at lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>>>
>
More information about the Openembedded-devel
mailing list