<div dir="ltr"><div dir="ltr"><div dir="ltr">My team employs a Jenkins Pipeline that performs 3 concurrent bitbake builds, one for each of our "arches", in parallel steps/stages. Previously, we cloned poky separately for each of these builds and each was performed in separate, but identical, working trees. Our project structure is designed to keep all of the arches' recipes maintained in separate, distinct folders.<div><br></div><div>We recently "improved" this process to perform a single clone of the repo and to perform the parallel builds on the same local working tree, and we started seeing intermittent bitbake errors that appear to be caused by races between these concurrent processes in producing, and importing/consuming, pyshtables.py. These manifest as simple SyntaxErrors raised at import time, caused by pyshtables.py being instantaneously malformed. Presumably, one process is writing pyshtables.py while another is importing it. Since this file appears to be written to and read from a shared location - bitbake/lib/bb/pysh/pyshtables.py - we have a race condition.
</div><div><br></div><div>We understand that this generated file is a fundamental requirement for any bitbake process to proceed. I would like to get feedback from the developers on the assumptions we have made, and the quality of the workaround we arrived at through trial and error.</div><div><br></div><div>What we have empirically determined (since all googling of this file produces only issues related to .gitignoring it and permissions errors while writing it) is that bitbake unconditionally generates this file, but that it will "fall back" to writing it in the local build directory if it fails to write it in
bitbake/lib/bb/pysh, e.g. if it does not have write permissions in that dir.</div><div><br></div><div>So, our workaround is to simply remove write permissions on that folder:</div><div><br></div><div> chmod a-w ${pokyPath}/bitbake/lib/bb/pysh</div><div><br></div><div>..and let each of our concurrent bitbake processes fallback to writing this (and others?) file into their distinct build subfolders. This appears to work, but we are looking for confirmation that:</div><div><br></div><div>1. these assumptions are correct</div><div>2. when bitbake falls back to generating files in the local build directory, these files are actually used from this location (through python path adjustment?)</div><div>3. Are there any other such generated files or similar concerns related to concurrent bitbake builds in the same poky working tree that we should be worried about here?</div><div><br></div><div>Thanks for your consideration and expertise,</div><div>Tim Black</div></div></div></div>