Syntax and Operators

- Bitbake-Style Python Functions Versus Python Functions + BitBake-Style Python Functions Versus Python Functions Following are some important differences between @@ -1895,7 +1895,7 @@ Sometimes, it is useful to be able to obtain information from the original execution environment. - Bitbake saves a copy of the original environment into + BitBake saves a copy of the original environment into a special variable named BB_ORIGENV. @@ -2616,7 +2616,7 @@ It is often necessary to access variables in the BitBake datastore using Python functions. - The Bitbake datastore has an API that allows you this + The BitBake datastore has an API that allows you this access. Here is a list of available operations: diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml index aca6741c..bae01d90 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml @@ -1780,7 +1780,7 @@ - Bitbake normally issues a warning when building two + BitBake normally issues a warning when building two different recipes where each provides the same output. This scenario is usually something the user does not want. -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== From rpjday at crashcourse.ca Fri Feb 7 09:52:07 2020 From: rpjday at crashcourse.ca (Robert P. J. Day) Date: Fri, 7 Feb 2020 04:52:07 -0500 (EST) Subject: [bitbake-devel] [PATCH] doc: Explain dynamic calculation of BB_NUMBER_THREADS Message-ID: While it's useful for the reader to know about BB_NUMBER_THREADS, make it clear that that value will be calculated automatically by default. Signed-off-by: Robert P. J. Day --- diff --git a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml index 46dafeee..f22ab36e 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml @@ -53,6 +53,15 @@ A possibly simpler solution is that some Linux distributions (e.g. Debian and Ubuntu) provide the ncpus command. + + + If you don't explicitly set this variable, OpenEmbedded + will do it for you via the default bitbake.conf + file by running an OE-supplied utility routine: + + BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" + + @@ -569,7 +578,8 @@ It is worth noting that you can greatly speed up the build time by properly setting - the BB_NUMBER_THREADS variable. + the BB_NUMBER_THREADS variable although, if you don't, + its value will be determined dynamically by an OpenEmbedded utility routine. -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== From rpjday at crashcourse.ca Fri Feb 7 10:51:11 2020 From: rpjday at crashcourse.ca (Robert P. J. Day) Date: Fri, 7 Feb 2020 05:51:11 -0500 (EST) Subject: [bitbake-devel] [PATCH] doc: minor tweaks to ch 1 of BB user manual Message-ID: Tweaks include: - hyphenation - rewording for brevity or clarification - adding markup where appropriate Signed-off-by: Robert P. J. Day --- diff --git a/doc/bitbake-user-manual/bitbake-user-manual-intro.xml b/doc/bitbake-user-manual/bitbake-user-manual-intro.xml index 8f2a960c..3d11b60b 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-intro.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-intro.xml @@ -127,7 +127,7 @@ (e.g. Cygwin, the BSDs, and so forth). - Be self contained, rather than tightly + Be self-contained, rather than tightly integrated into the build machine's root filesystem. @@ -221,6 +221,8 @@ them How to configure and compile the source code + How to assemble the generated artifacts into + one or more installable packages Where on the target machine to install the package or packages created @@ -229,7 +231,7 @@ Within the context of BitBake, or any project utilizing BitBake as its build system, files with the .bb - extension are referred to as recipes. + extension are referred to as recipes. The term "package" is also commonly used to describe recipes. However, since the same word is used to describe packaged @@ -252,9 +254,9 @@ various configuration variables that govern the project's build process. These files fall into several areas that define - machine configuration options, distribution configuration - options, compiler tuning options, general common - configuration options, and user configuration options. + machine configuration, distribution configuration, + possible compiler tuning, general common + configuration, and user configuration. The main configuration file is the sample bitbake.conf file, which is located within the BitBake source tree @@ -292,7 +294,7 @@ Layers allow you to isolate different types of customizations from each other. While you might find it tempting to keep everything in one layer - when working on a single project, the more modular you organize + when working on a single project, the more modular your metadata, the easier it is to cope with future changes. @@ -300,8 +302,8 @@ To illustrate how you can use layers to keep things modular, consider customizations you might make to support a specific target machine. These types of customizations typically reside in a special layer, - rather than a general layer, called a Board Support Package (BSP) - Layer. + rather than a general layer, called a Board Support Package (BSP) + layer. Furthermore, the machine customizations should be isolated from recipes and metadata that support a new GUI environment, for example. -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== From peter.kjellerstedt at axis.com Fri Feb 7 16:02:07 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Fri, 7 Feb 2020 16:02:07 +0000 Subject: [bitbake-devel] [PATCH] doc: Explain dynamic calculation of BB_NUMBER_THREADS In-Reply-To: References: Message-ID: > -----Original Message----- > From: bitbake-devel-bounces at lists.openembedded.org bounces at lists.openembedded.org> On Behalf Of Robert P. J. Day > Sent: den 7 februari 2020 10:52 > To: BitBake developer list > Subject: [bitbake-devel] [PATCH] doc: Explain dynamic calculation of > BB_NUMBER_THREADS > > > While it's useful for the reader to know about BB_NUMBER_THREADS, make > it clear that that value will be calculated automatically by default. > > Signed-off-by: Robert P. J. Day > > --- > > diff --git a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > index 46dafeee..f22ab36e 100644 > --- a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > +++ b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > @@ -53,6 +53,15 @@ > A possibly simpler solution is that some Linux distributions > (e.g. Debian and Ubuntu) provide the ncpus command. > > + > + > + If you don't explicitly set this variable, OpenEmbedded > + will do it for you via the default bitbake.conf > + file by running an OE-supplied utility routine: > + > + BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" > + > + > > > > @@ -569,7 +578,8 @@ > > > It is worth noting that you can greatly speed up the build time by properly setting > - the BB_NUMBER_THREADS variable. > + the BB_NUMBER_THREADS variable although, if you don't, > + its value will be determined dynamically by an OpenEmbedded utility routine. > > > > > -- Since this is the BitBake manual, I do not think it is appropriate to refer to OE specific behavior here. //Peter From richard.purdie at linuxfoundation.org Sat Feb 8 16:01:09 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sat, 8 Feb 2020 16:01:09 +0000 Subject: [bitbake-devel] [PATCH 1/3] siggen: Optimise get_unihash disk based cache handling Message-ID: <20200208160111.13305-1-richard.purdie@linuxfoundation.org> Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 50 +++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index ded1da020f..209a342883 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -44,6 +44,7 @@ class SignatureGenerator(object): self.file_checksum_values = {} self.taints = {} self.unitaskhashes = {} + self.tidtopn = {} self.setscenetasks = set() def finalise(self, fn, d, varient): @@ -79,19 +80,19 @@ class SignatureGenerator(object): return def get_taskdata(self): - return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks) + return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) def set_taskdata(self, data): - self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks = data + self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data def reset(self, data): self.__init__(data) def get_taskhashes(self): - return self.taskhash, self.unitaskhashes + return self.taskhash, self.unitaskhashes, self.tidtopn def set_taskhashes(self, hashes): - self.taskhash, self.unitaskhashes = hashes + self.taskhash, self.unitaskhashes, self.tidtopn = hashes def save_unitaskhashes(self): return @@ -124,9 +125,10 @@ class SignatureGeneratorBasic(SignatureGenerator): else: self.checksum_cache = None - self.unihash_cache = bb.cache.SimpleCache("1") + self.unihash_cache = bb.cache.SimpleCache("3") self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {}) self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split() + self.tidtopn = {} def init_rundepcheck(self, data): self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None @@ -210,6 +212,9 @@ class SignatureGeneratorBasic(SignatureGenerator): self.runtaskdeps[tid] = [] self.file_checksum_values[tid] = [] recipename = dataCache.pkg_fn[fn] + + self.tidtopn[tid] = recipename + for dep in sorted(deps, key=clean_basepath): (depmc, _, deptaskname, depfn) = bb.runqueue.split_tid_mcfn(dep) if mc != depmc: @@ -407,24 +412,35 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server) return self._client - def __get_task_unihash_key(self, tid): - # TODO: The key only *needs* to be the taskhash, the tid is just - # convenient - return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid]) - def get_stampfile_hash(self, tid): if tid in self.taskhash: # If a unique hash is reported, use it as the stampfile hash. This # ensures that if a task won't be re-run if the taskhash changes, # but it would result in the same output hash - unihash = self.unitaskhashes.get(self.__get_task_unihash_key(tid), None) + unihash = self._get_unihash(tid) if unihash is not None: return unihash return super().get_stampfile_hash(tid) def set_unihash(self, tid, unihash): - self.unitaskhashes[self.__get_task_unihash_key(tid)] = unihash + (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) + key = mc + ":" + self.tidtopn[tid] + ":" + taskname + self.unitaskhashes[key] = (self.taskhash[tid], unihash) + + def _get_unihash(self, tid, checkkey=None): + if tid not in self.tidtopn: + return None + (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) + key = mc + ":" + self.tidtopn[tid] + ":" + taskname + if key not in self.unitaskhashes: + return None + if not checkkey: + checkkey = self.taskhash[tid] + (key, unihash) = self.unitaskhashes[key] + if key != checkkey: + return None + return unihash def get_unihash(self, tid): taskhash = self.taskhash[tid] @@ -433,11 +449,9 @@ class SignatureGeneratorUniHashMixIn(object): if self.setscenetasks and tid not in self.setscenetasks: return taskhash - key = self.__get_task_unihash_key(tid) - # TODO: This cache can grow unbounded. It probably only needs to keep # for each task - unihash = self.unitaskhashes.get(key, None) + unihash = self._get_unihash(tid) if unihash is not None: return unihash @@ -472,7 +486,7 @@ class SignatureGeneratorUniHashMixIn(object): except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) - self.unitaskhashes[key] = unihash + self.set_unihash(tid, unihash) return unihash def report_unihash(self, path, task, d): @@ -484,13 +498,13 @@ class SignatureGeneratorUniHashMixIn(object): tempdir = d.getVar('T') fn = d.getVar('BB_FILENAME') tid = fn + ':do_' + task - key = tid.rsplit("/", 1)[1] + ':' + taskhash + key = tid + ':' + taskhash if self.setscenetasks and tid not in self.setscenetasks: return # Sanity checks - cache_unihash = self.unitaskhashes.get(key, None) + cache_unihash = self._get_unihash(tid, checkkey=taskhash) if cache_unihash is None: bb.fatal('%s not in unihash cache. Please report this error' % key) -- 2.20.1 From richard.purdie at linuxfoundation.org Sat Feb 8 16:01:10 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sat, 8 Feb 2020 16:01:10 +0000 Subject: [bitbake-devel] [PATCH 2/3] siggen: Cache unihash values to avoid cache lookup In-Reply-To: <20200208160111.13305-1-richard.purdie@linuxfoundation.org> References: <20200208160111.13305-1-richard.purdie@linuxfoundation.org> Message-ID: <20200208160111.13305-2-richard.purdie@linuxfoundation.org> Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 209a342883..96807c46cd 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -40,6 +40,7 @@ class SignatureGenerator(object): def __init__(self, data): self.basehash = {} self.taskhash = {} + self.unihash = {} self.runtaskdeps = {} self.file_checksum_values = {} self.taints = {} @@ -80,19 +81,19 @@ class SignatureGenerator(object): return def get_taskdata(self): - return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) + return (self.runtaskdeps, self.taskhash, self.unihash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) def set_taskdata(self, data): - self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data + self.runtaskdeps, self.taskhash, self.unihash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data def reset(self, data): self.__init__(data) def get_taskhashes(self): - return self.taskhash, self.unitaskhashes, self.tidtopn + return self.taskhash, self.unihash, self.unitaskhashes, self.tidtopn def set_taskhashes(self, hashes): - self.taskhash, self.unitaskhashes, self.tidtopn = hashes + self.taskhash, self.unihash, self.unitaskhashes, self.tidtopn = hashes def save_unitaskhashes(self): return @@ -108,6 +109,7 @@ class SignatureGeneratorBasic(SignatureGenerator): def __init__(self, data): self.basehash = {} self.taskhash = {} + self.unihash = {} self.taskdeps = {} self.runtaskdeps = {} self.file_checksum_values = {} @@ -256,7 +258,13 @@ class SignatureGeneratorBasic(SignatureGenerator): data = self.basehash[tid] for dep in self.runtaskdeps[tid]: - data = data + self.get_unihash(dep) + if dep in self.unihash: + if self.unihash[dep] is None: + data = data + self.taskhash[dep] + else: + data = data + self.unihash[dep] + else: + data = data + self.get_unihash(dep) for (f, cs) in self.file_checksum_values[tid]: if cs: @@ -427,6 +435,7 @@ class SignatureGeneratorUniHashMixIn(object): (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) key = mc + ":" + self.tidtopn[tid] + ":" + taskname self.unitaskhashes[key] = (self.taskhash[tid], unihash) + self.unihash[tid] = unihash def _get_unihash(self, tid, checkkey=None): if tid not in self.tidtopn: @@ -447,12 +456,14 @@ class SignatureGeneratorUniHashMixIn(object): # If its not a setscene task we can return if self.setscenetasks and tid not in self.setscenetasks: + self.unihash[tid] = None return taskhash # TODO: This cache can grow unbounded. It probably only needs to keep # for each task unihash = self._get_unihash(tid) if unihash is not None: + self.unihash[tid] = unihash return unihash # In the absence of being able to discover a unique hash from the @@ -487,6 +498,7 @@ class SignatureGeneratorUniHashMixIn(object): bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) self.set_unihash(tid, unihash) + self.unihash[tid] = unihash return unihash def report_unihash(self, path, task, d): -- 2.20.1 From richard.purdie at linuxfoundation.org Sat Feb 8 16:01:11 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sat, 8 Feb 2020 16:01:11 +0000 Subject: [bitbake-devel] [PATCH 3/3] siggen: Avoid cache mismatch issues with locked sigs In-Reply-To: <20200208160111.13305-1-richard.purdie@linuxfoundation.org> References: <20200208160111.13305-1-richard.purdie@linuxfoundation.org> Message-ID: <20200208160111.13305-3-richard.purdie@linuxfoundation.org> If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie --- lib/bb/siggen.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 96807c46cd..ffd8fcaf36 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -515,6 +515,10 @@ class SignatureGeneratorUniHashMixIn(object): if self.setscenetasks and tid not in self.setscenetasks: return + # This can happen if locked sigs are in action. Detect and just abort + if taskhash != self.taskhash[tid]: + return + # Sanity checks cache_unihash = self._get_unihash(tid, checkkey=taskhash) if cache_unihash is None: -- 2.20.1 From rpjday at crashcourse.ca Sat Feb 8 19:43:38 2020 From: rpjday at crashcourse.ca (Robert P. J. Day) Date: Sat, 8 Feb 2020 14:43:38 -0500 (EST) Subject: [bitbake-devel] [PATCH] doc: correct typo of 'BitBack' Message-ID: Signed-off-by: Robert P. J. Day --- diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml b/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml index 421364c2..455f5bf3 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@ -1864,7 +1864,7 @@ accessing your $HOME/.ccache directory. The following command "whitelists" the environment variable - CCACHE_DIR causing BitBack to allow that + CCACHE_DIR causing BitBake to allow that variable into the datastore: export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR" -- ======================================================================== Robert P. J. Day Ottawa, Ontario, CANADA http://crashcourse.ca Twitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday ======================================================================== From pkj at axis.com Sun Feb 9 08:50:34 2020 From: pkj at axis.com (Peter Kjellerstedt) Date: Sun, 9 Feb 2020 09:50:34 +0100 Subject: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene tasks again Message-ID: <20200209085034.16349-1-pkj@axis.com> Track both scene tasks and run tasks to provide a complete progress information for all tasks. This avoids the progress bar from stating "Currently XX running tasks (0 of 0)" while progressing through thousands of setscene tasks, obviously confusing the user. Signed-off-by: Peter Kjellerstedt --- bitbake/lib/bb/ui/uihelper.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/bitbake/lib/bb/ui/uihelper.py b/bitbake/lib/bb/ui/uihelper.py index 48d808ae28..ac4ac5a882 100644 --- a/bitbake/lib/bb/ui/uihelper.py +++ b/bitbake/lib/bb/ui/uihelper.py @@ -16,6 +16,10 @@ class BBUIHelper: self.running_pids = [] self.failed_tasks = [] self.pidmap = {} + self.scenetasks_current = 0 + self.scenetasks_total = 0 + self.runtasks_current = 0 + self.runtasks_total = 0 self.tasknumber_current = 0 self.tasknumber_total = 0 @@ -49,9 +53,17 @@ class BBUIHelper: tid = event._fn + ":" + event._task removetid(event.pid, tid) self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)}) + elif isinstance(event, bb.runqueue.sceneQueueTaskStarted): + self.scenetasks_current = event.stats.completed + event.stats.active + event.stats.failed + 1 + self.scenetasks_total = event.stats.total + self.tasknumber_current = self.scenetasks_current + self.runtasks_current + self.tasknumber_total = self.scenetasks_total + self.runtasks_total + self.needUpdate = True elif isinstance(event, bb.runqueue.runQueueTaskStarted): - self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1 - self.tasknumber_total = event.stats.total + self.runtasks_current = event.stats.completed + event.stats.active + event.stats.failed + 1 + self.runtasks_total = event.stats.total + self.tasknumber_current = self.scenetasks_current + self.runtasks_current + self.tasknumber_total = self.scenetasks_total + self.runtasks_total self.needUpdate = True elif isinstance(event, bb.build.TaskProgress): if event.pid > 0 and event.pid in self.pidmap: -- 2.21.1 From akuster808 at gmail.com Sun Feb 9 16:03:08 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:08 -0800 Subject: [bitbake-devel] [1.44 0/7] patch review Message-ID: Here are a few more changes targeted for the zeus 3.0.2 release. Please have comments back by Tuesday. The following changes since commit 1a026d4aca47ed1b0b74a8a818635e7520e8f9c8: runqueue: Use a set for the setscene tasks list (2020-01-18 12:20:38 -0800) are available in the Git repository at: git://git.openembedded.org/bitbake-contrib stable/1.44-nut http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.44-nut Chris Laplante (3): fetch2: do not suffix srcrev cache key with PN fetch2/git: _revision_key: collapse adjacent slashes persist_data.py: Immediately get exclusive lock in __setitem__ Frazer Clews (1): cooker/toaster: replaced deprecated method warn() with warning() Richard Purdie (3): siggen: Optimise get_unihash disk based cache handling siggen: Cache unihash values to avoid cache lookup siggen: Avoid cache mismatch issues with locked sigs lib/bb/cooker.py | 2 +- lib/bb/fetch2/__init__.py | 3 +- lib/bb/fetch2/git.py | 4 +- lib/bb/persist_data.py | 3 + lib/bb/siggen.py | 68 +++++++++++++------ .../management/commands/runbuilds.py | 6 +- 6 files changed, 60 insertions(+), 26 deletions(-) -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:09 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:09 -0800 Subject: [bitbake-devel] [1.44 1/7] fetch2: do not suffix srcrev cache key with PN In-Reply-To: References: Message-ID: <63556259f2e34003ad0618a8fe19e1cfe8aef5c9.1581264053.git.akuster808@gmail.com> From: Chris Laplante Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit 6c938e6fd29beebe09b32be839dae008fe6491d2) Signed-off-by: Armin Kuster --- lib/bb/fetch2/__init__.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index 731c1608..4a781bde 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -1567,8 +1567,7 @@ class FetchMethod(object): return True, str(latest_rev) def generate_revision_key(self, ud, d, name): - key = self._revision_key(ud, d, name) - return "%s-%s" % (key, d.getVar("PN") or "") + return self._revision_key(ud, d, name) def latest_versionstring(self, ud, d): """ -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:10 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:10 -0800 Subject: [bitbake-devel] [1.44 2/7] fetch2/git: _revision_key: collapse adjacent slashes In-Reply-To: References: Message-ID: <716cdf737bc536f84ed1254d464c9f286e0d5a9a.1581264053.git.akuster808@gmail.com> From: Chris Laplante >From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit 425e21c14955dd38868c6e97637df3bbe0f89fac) Signed-off-by: Armin Kuster --- lib/bb/fetch2/git.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py index fa41b078..9ba9cdcc 100644 --- a/lib/bb/fetch2/git.py +++ b/lib/bb/fetch2/git.py @@ -594,7 +594,9 @@ class Git(FetchMethod): """ Return a unique key for the url """ - return "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] + # Collapse adjacent slashes + slash_re = re.compile(r"/+") + return "git:" + ud.host + slash_re.sub(".", ud.path) + ud.unresolvedrev[name] def _lsremote(self, ud, d, search): """ -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:11 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:11 -0800 Subject: [bitbake-devel] [1.44 3/7] persist_data.py: Immediately get exclusive lock in __setitem__ In-Reply-To: References: Message-ID: <2ba2f224b50956313f5c2ba01942143b4d6098f2.1581264053.git.akuster808@gmail.com> From: Chris Laplante To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. Signed-off-by: Chris Laplante Signed-off-by: Richard Purdie (cherry picked from commit feb43e7c30f5bfab75d718896c45df621810d06f) Signed-off-by: Armin Kuster --- lib/bb/persist_data.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py index de8f87a8..7357ab2d 100644 --- a/lib/bb/persist_data.py +++ b/lib/bb/persist_data.py @@ -179,6 +179,9 @@ class SQLTable(collections.MutableMapping): elif not isinstance(value, str): raise TypeError('Only string values are supported') + # Ensure the entire transaction (including SELECT) executes under write lock + cursor.execute("BEGIN EXCLUSIVE") + cursor.execute("SELECT * from %s where key=?;" % self.table, [key]) row = cursor.fetchone() if row is not None: -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:12 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:12 -0800 Subject: [bitbake-devel] [1.44 4/7] cooker/toaster: replaced deprecated method warn() with warning() In-Reply-To: References: Message-ID: From: Frazer Clews Removed the deprecated methods as it will only cause problems later on, and since warn() just calls warning(), it shouldn't change anything Signed-off-by: Frazer Clews Signed-off-by: Richard Purdie (cherry picked from commit a194f275235f22411cb2368f06a44f61ceb6a0f3) Signed-off-by: Armin Kuster --- lib/bb/cooker.py | 2 +- lib/toaster/bldcontrol/management/commands/runbuilds.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index b74affa7..911805a6 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1788,7 +1788,7 @@ class CookerCollectFiles(object): # When constructing an older style single regex, it's possible for BBMASK # to end up beginning with '|', which matches and masks _everything_. if mask.startswith("|"): - collectlog.warn("BBMASK contains regular expression beginning with '|', fixing: %s" % mask) + collectlog.warning("BBMASK contains regular expression beginning with '|', fixing: %s" % mask) mask = mask[1:] try: re.compile(mask) diff --git a/lib/toaster/bldcontrol/management/commands/runbuilds.py b/lib/toaster/bldcontrol/management/commands/runbuilds.py index 50ec4098..19f659ec 100644 --- a/lib/toaster/bldcontrol/management/commands/runbuilds.py +++ b/lib/toaster/bldcontrol/management/commands/runbuilds.py @@ -168,17 +168,17 @@ class Command(BaseCommand): try: self.cleanup() except Exception as e: - logger.warn("runbuilds: cleanup exception %s" % str(e)) + logger.warning("runbuilds: cleanup exception %s" % str(e)) try: self.archive() except Exception as e: - logger.warn("runbuilds: archive exception %s" % str(e)) + logger.warning("runbuilds: archive exception %s" % str(e)) try: self.schedule() except Exception as e: - logger.warn("runbuilds: schedule exception %s" % str(e)) + logger.warning("runbuilds: schedule exception %s" % str(e)) def handle(self, **options): pidfile_path = os.path.join(os.environ.get("BUILDDIR", "."), -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:13 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:13 -0800 Subject: [bitbake-devel] [1.44 5/7] siggen: Optimise get_unihash disk based cache handling In-Reply-To: References: Message-ID: <8f4f6c2f9acab23bc795ffe389c4cd74711d10ff.1581264053.git.akuster808@gmail.com> From: Richard Purdie Currently the cache can grow huge since any previously used hash is retained in the cache. This change moves to use one hash per task which improves the speed of the functions considerably. Currently performance is an issue, as are very large cache files and cache load time. By moving to a single hash per task, the shorted filename as a key is no longer usable as the same recipe has multiple variants for the same filename so this has to change. Signed-off-by: Richard Purdie (cherry picked from commit ed764e7fcf04b6d0ba6b4cac7415b1ee8f492865) Signed-off-by: Armin Kuster --- lib/bb/siggen.py | 50 +++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index ded1da02..209a3428 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -44,6 +44,7 @@ class SignatureGenerator(object): self.file_checksum_values = {} self.taints = {} self.unitaskhashes = {} + self.tidtopn = {} self.setscenetasks = set() def finalise(self, fn, d, varient): @@ -79,19 +80,19 @@ class SignatureGenerator(object): return def get_taskdata(self): - return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks) + return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) def set_taskdata(self, data): - self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.setscenetasks = data + self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data def reset(self, data): self.__init__(data) def get_taskhashes(self): - return self.taskhash, self.unitaskhashes + return self.taskhash, self.unitaskhashes, self.tidtopn def set_taskhashes(self, hashes): - self.taskhash, self.unitaskhashes = hashes + self.taskhash, self.unitaskhashes, self.tidtopn = hashes def save_unitaskhashes(self): return @@ -124,9 +125,10 @@ class SignatureGeneratorBasic(SignatureGenerator): else: self.checksum_cache = None - self.unihash_cache = bb.cache.SimpleCache("1") + self.unihash_cache = bb.cache.SimpleCache("3") self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {}) self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split() + self.tidtopn = {} def init_rundepcheck(self, data): self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None @@ -210,6 +212,9 @@ class SignatureGeneratorBasic(SignatureGenerator): self.runtaskdeps[tid] = [] self.file_checksum_values[tid] = [] recipename = dataCache.pkg_fn[fn] + + self.tidtopn[tid] = recipename + for dep in sorted(deps, key=clean_basepath): (depmc, _, deptaskname, depfn) = bb.runqueue.split_tid_mcfn(dep) if mc != depmc: @@ -407,24 +412,35 @@ class SignatureGeneratorUniHashMixIn(object): self._client = hashserv.create_client(self.server) return self._client - def __get_task_unihash_key(self, tid): - # TODO: The key only *needs* to be the taskhash, the tid is just - # convenient - return '%s:%s' % (tid.rsplit("/", 1)[1], self.taskhash[tid]) - def get_stampfile_hash(self, tid): if tid in self.taskhash: # If a unique hash is reported, use it as the stampfile hash. This # ensures that if a task won't be re-run if the taskhash changes, # but it would result in the same output hash - unihash = self.unitaskhashes.get(self.__get_task_unihash_key(tid), None) + unihash = self._get_unihash(tid) if unihash is not None: return unihash return super().get_stampfile_hash(tid) def set_unihash(self, tid, unihash): - self.unitaskhashes[self.__get_task_unihash_key(tid)] = unihash + (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) + key = mc + ":" + self.tidtopn[tid] + ":" + taskname + self.unitaskhashes[key] = (self.taskhash[tid], unihash) + + def _get_unihash(self, tid, checkkey=None): + if tid not in self.tidtopn: + return None + (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) + key = mc + ":" + self.tidtopn[tid] + ":" + taskname + if key not in self.unitaskhashes: + return None + if not checkkey: + checkkey = self.taskhash[tid] + (key, unihash) = self.unitaskhashes[key] + if key != checkkey: + return None + return unihash def get_unihash(self, tid): taskhash = self.taskhash[tid] @@ -433,11 +449,9 @@ class SignatureGeneratorUniHashMixIn(object): if self.setscenetasks and tid not in self.setscenetasks: return taskhash - key = self.__get_task_unihash_key(tid) - # TODO: This cache can grow unbounded. It probably only needs to keep # for each task - unihash = self.unitaskhashes.get(key, None) + unihash = self._get_unihash(tid) if unihash is not None: return unihash @@ -472,7 +486,7 @@ class SignatureGeneratorUniHashMixIn(object): except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) - self.unitaskhashes[key] = unihash + self.set_unihash(tid, unihash) return unihash def report_unihash(self, path, task, d): @@ -484,13 +498,13 @@ class SignatureGeneratorUniHashMixIn(object): tempdir = d.getVar('T') fn = d.getVar('BB_FILENAME') tid = fn + ':do_' + task - key = tid.rsplit("/", 1)[1] + ':' + taskhash + key = tid + ':' + taskhash if self.setscenetasks and tid not in self.setscenetasks: return # Sanity checks - cache_unihash = self.unitaskhashes.get(key, None) + cache_unihash = self._get_unihash(tid, checkkey=taskhash) if cache_unihash is None: bb.fatal('%s not in unihash cache. Please report this error' % key) -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:14 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:14 -0800 Subject: [bitbake-devel] [1.44 6/7] siggen: Cache unihash values to avoid cache lookup In-Reply-To: References: Message-ID: <1f5a662176fd174c4793e3e28d1b4e95616b1715.1581264053.git.akuster808@gmail.com> From: Richard Purdie Add unihash cache of values to speed up cache lookup. This avoids the overhead of the disk based check functions. Signed-off-by: Richard Purdie (cherry picked from commit 5c9cc45b60904a1c355db9bf9c4495f1b25aca37) Signed-off-by: Armin Kuster --- lib/bb/siggen.py | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 209a3428..96807c46 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -40,6 +40,7 @@ class SignatureGenerator(object): def __init__(self, data): self.basehash = {} self.taskhash = {} + self.unihash = {} self.runtaskdeps = {} self.file_checksum_values = {} self.taints = {} @@ -80,19 +81,19 @@ class SignatureGenerator(object): return def get_taskdata(self): - return (self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) + return (self.runtaskdeps, self.taskhash, self.unihash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks) def set_taskdata(self, data): - self.runtaskdeps, self.taskhash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data + self.runtaskdeps, self.taskhash, self.unihash, self.file_checksum_values, self.taints, self.basehash, self.unitaskhashes, self.tidtopn, self.setscenetasks = data def reset(self, data): self.__init__(data) def get_taskhashes(self): - return self.taskhash, self.unitaskhashes, self.tidtopn + return self.taskhash, self.unihash, self.unitaskhashes, self.tidtopn def set_taskhashes(self, hashes): - self.taskhash, self.unitaskhashes, self.tidtopn = hashes + self.taskhash, self.unihash, self.unitaskhashes, self.tidtopn = hashes def save_unitaskhashes(self): return @@ -108,6 +109,7 @@ class SignatureGeneratorBasic(SignatureGenerator): def __init__(self, data): self.basehash = {} self.taskhash = {} + self.unihash = {} self.taskdeps = {} self.runtaskdeps = {} self.file_checksum_values = {} @@ -256,7 +258,13 @@ class SignatureGeneratorBasic(SignatureGenerator): data = self.basehash[tid] for dep in self.runtaskdeps[tid]: - data = data + self.get_unihash(dep) + if dep in self.unihash: + if self.unihash[dep] is None: + data = data + self.taskhash[dep] + else: + data = data + self.unihash[dep] + else: + data = data + self.get_unihash(dep) for (f, cs) in self.file_checksum_values[tid]: if cs: @@ -427,6 +435,7 @@ class SignatureGeneratorUniHashMixIn(object): (mc, fn, taskname, taskfn) = bb.runqueue.split_tid_mcfn(tid) key = mc + ":" + self.tidtopn[tid] + ":" + taskname self.unitaskhashes[key] = (self.taskhash[tid], unihash) + self.unihash[tid] = unihash def _get_unihash(self, tid, checkkey=None): if tid not in self.tidtopn: @@ -447,12 +456,14 @@ class SignatureGeneratorUniHashMixIn(object): # If its not a setscene task we can return if self.setscenetasks and tid not in self.setscenetasks: + self.unihash[tid] = None return taskhash # TODO: This cache can grow unbounded. It probably only needs to keep # for each task unihash = self._get_unihash(tid) if unihash is not None: + self.unihash[tid] = unihash return unihash # In the absence of being able to discover a unique hash from the @@ -487,6 +498,7 @@ class SignatureGeneratorUniHashMixIn(object): bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) self.set_unihash(tid, unihash) + self.unihash[tid] = unihash return unihash def report_unihash(self, path, task, d): -- 2.17.1 From akuster808 at gmail.com Sun Feb 9 16:03:15 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Sun, 9 Feb 2020 08:03:15 -0800 Subject: [bitbake-devel] [1.44 7/7] siggen: Avoid cache mismatch issues with locked sigs In-Reply-To: References: Message-ID: <95687be83e716220eb3893b67428f97fd59fc2c5.1581264053.git.akuster808@gmail.com> From: Richard Purdie If locked sigs are in use this function makes little sense, need to avoid generating mismatch warnings. Signed-off-by: Richard Purdie (cherry picked from commit 27ad9c1d468fba858a4adeb56b605227b415ae0f) Signed-off-by: Armin Kuster --- lib/bb/siggen.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index 96807c46..ffd8fcaf 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -515,6 +515,10 @@ class SignatureGeneratorUniHashMixIn(object): if self.setscenetasks and tid not in self.setscenetasks: return + # This can happen if locked sigs are in action. Detect and just abort + if taskhash != self.taskhash[tid]: + return + # Sanity checks cache_unihash = self._get_unihash(tid, checkkey=taskhash) if cache_unihash is None: -- 2.17.1 From Joao.CCabral at parceiros.nos.pt Sun Feb 9 23:26:55 2020 From: Joao.CCabral at parceiros.nos.pt (Joao Carlos Cabral (P)) Date: Sun, 9 Feb 2020 23:26:55 +0000 Subject: [bitbake-devel] [HELP] Create a new image recipe based on a (changed) existing one In-Reply-To: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> References: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> Message-ID: <1581290815745.54479@parceiros.nos.pt> Hi, I'm trying to create a image recipe based on a existing one. I've searched online, and I could find such an example. My problem is that the image I what to serve as a base is changed in another layer, and I want to include that changes. Ex: (tree) Prj_root -> layer1 -> recipes-core -> images -> my-image.bb [This is the base recipe] -> layer2 -> recipes-core -> images -> my-image.bbappend [This is the changed one] -> layer3 -> recipes-core -> images -> my-image-dev.bb [this is the new image, based on my-image] my-image.bbappend is something like this: --------------------- my-image.bbappend --------- PROVIDES = "product" IMAGE_INSTALL += " \ \ " IMAGE_INSTALL_remove += " \ " (+ other functions related with signing the image) _________________________________________________ The final image is created with 'bitbake product'. And, I want to create a dev image that must be based on the current "product" image (my-image.bb + my-image.bbappend) with some dev packages. So, I need to include the changes that my-image.bbapend does to this dev image I'm trying to create. Tried simple create a new image that should be a copy of the working one, and then add the needed packages for development. So I did the following... ---------------- my-image-dev.bbappend ---------- SUMMARY = "dev Image" require layer1/recipes-core/images/my-image.bb ------------------------------------------------ I can run 'bitbake my-image-dev' and it runs, but doesn't produce the same output as 'bitbake product'. Also, If I run 'bitbake my-image-dev' 1st, it finishes fine. Running 'bitbake product' after makes a bunch of other tasks and finishes fine. Running again 'bitbake my-image-dev' finishes with an error on a specific task. ( It shows that they are not building the same thing) How can I do what I need? NOTE: I can't change anything in the layer1 and layer2. PS. Also tried to require the bbappend file, but bitbake fails saying that it's not a bitbake file. Thanks in advanced. Jo?o Cabral From Joao.CCabral at parceiros.nos.pt Sun Feb 9 23:17:54 2020 From: Joao.CCabral at parceiros.nos.pt (Joao Carlos Cabral (P)) Date: Sun, 9 Feb 2020 23:17:54 +0000 Subject: [bitbake-devel] [HELP] Create a new image recipe based on a (changed) existing one Message-ID: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> Hi, I'm trying to create a image recipe based on a existing one. I've searched online, and I could find such an example. My problem is that the image I what to serve as a base is changed in another layer, and I want to include that changes. Ex: (tree) Prj_root -> layer1 -> recipes-core -> images -> my-image.bb [This is the base recipe] -> layer2 -> recipes-core -> images -> my-image.bbappend [This is the changed one] -> layer3 -> recipes-core -> images -> my-image-dev.bb [this is the new image, based on my- image] my-image.bbappend is something like this: --------------------- my-image.bbappend --------- PROVIDES = "product" IMAGE_INSTALL += " \ \ " IMAGE_INSTALL_remove += " \ " (+ other functions related with signing the image) _________________________________________________ The final image is created with 'bitbake product'. And, I want to create a dev image that must be based on the current "product" image (my-image.bb + my-image.bbappend) with some dev packages. So, I need to include the changes that my-image.bbapend does to this dev image I'm trying to create. Tried simple create a new image that should be a copy of the working one, and then add the needed packages for development. So I did the following... ---------------- my-image-dev.bbappend ---------- SUMMARY = "dev Image" require layer1/recipes-core/images/my-image.bb ------------------------------------------------ I can run 'bitbake my-image-dev' and it runs, but doesn't produce the same output as 'bitbake product'. Also, If I run 'bitbake my-image-dev' 1st, it finishes fine. Running 'bitbake product' after makes a bunch of other tasks and finishes fine. Running again 'bitbake my-image-dev' finishes with an error on a specific task. ( It shows that they are not building the same thing) How can I do what I need? NOTE: I can't change anything in the layer1 and layer2. PS. Also tried to require the bbappend file, but bitbake fails saying that it's not a bitbake file. Thanks in advanced. Jo?o Cabral From richard.purdie at linuxfoundation.org Mon Feb 10 14:17:35 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 10 Feb 2020 14:17:35 +0000 Subject: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene tasks again In-Reply-To: <20200209085034.16349-1-pkj@axis.com> References: <20200209085034.16349-1-pkj@axis.com> Message-ID: <29677f1ba34d00b61ad1375b38f56b20c3e98d50.camel@linuxfoundation.org> On Sun, 2020-02-09 at 09:50 +0100, Peter Kjellerstedt wrote: > Track both scene tasks and run tasks to provide a complete progress > information for all tasks. This avoids the progress bar from stating > "Currently XX running tasks (0 of 0)" while progressing through > thousands of setscene tasks, obviously confusing the user. There was a reason this was removed, this can't work unfortunately. The code was changed such that setscene tasks can potentially be rerun so we have no idea what "complete" looks like or how many will actually run. I'm not happy about that but there wasn't an alternative. Cheers, Richard From peter.kjellerstedt at axis.com Mon Feb 10 15:01:24 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Mon, 10 Feb 2020 15:01:24 +0000 Subject: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene tasks again In-Reply-To: <29677f1ba34d00b61ad1375b38f56b20c3e98d50.camel@linuxfoundation.org> References: <20200209085034.16349-1-pkj@axis.com> <29677f1ba34d00b61ad1375b38f56b20c3e98d50.camel@linuxfoundation.org> Message-ID: <68698044e91d4c4dbaa60d63260ac7f7@XBOX03.axis.com> > -----Original Message----- > From: Richard Purdie > Sent: den 10 februari 2020 15:18 > To: Peter Kjellerstedt ; bitbake- > devel at lists.openembedded.org > Subject: Re: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene > tasks again > > On Sun, 2020-02-09 at 09:50 +0100, Peter Kjellerstedt wrote: > > Track both scene tasks and run tasks to provide a complete progress > > information for all tasks. This avoids the progress bar from stating > > "Currently XX running tasks (0 of 0)" while progressing through > > thousands of setscene tasks, obviously confusing the user. > > There was a reason this was removed, this can't work unfortunately. > > The code was changed such that setscene tasks can potentially be rerun > so we have no idea what "complete" looks like or how many will actually > run. I'm not happy about that but there wasn't an alternative. > > Cheers, > > Richard If a setscene task is rerun, won't that just mean that the counters don't change due to this? So in this case there won't be any progress to the total progress bar even if a task is running? That should still be better than the current situation when they are never counted. //Peter From richard.purdie at linuxfoundation.org Mon Feb 10 15:17:48 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 10 Feb 2020 15:17:48 +0000 Subject: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene tasks again In-Reply-To: <68698044e91d4c4dbaa60d63260ac7f7@XBOX03.axis.com> References: <20200209085034.16349-1-pkj@axis.com> <29677f1ba34d00b61ad1375b38f56b20c3e98d50.camel@linuxfoundation.org> <68698044e91d4c4dbaa60d63260ac7f7@XBOX03.axis.com> Message-ID: On Mon, 2020-02-10 at 15:01 +0000, Peter Kjellerstedt wrote: > If a setscene task is rerun, won't that just mean that the counters > don't change due to this? So in this case there won't be any > progress to the total progress bar even if a task is running? That > should still be better than the current situation when they are > never counted. No, that isn't how the code is written. You'd see "task 1000 of 500". We got some pretty horrible numbers when testing this originally, I got lots of complaints and its why it was removed. The code has changed a bit since and we probably don't rerun as much but we get complains when we print any invalid info... Cheers, Richard From peter.kjellerstedt at axis.com Mon Feb 10 15:36:12 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Mon, 10 Feb 2020 15:36:12 +0000 Subject: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene tasks again In-Reply-To: References: <20200209085034.16349-1-pkj@axis.com> <29677f1ba34d00b61ad1375b38f56b20c3e98d50.camel@linuxfoundation.org> <68698044e91d4c4dbaa60d63260ac7f7@XBOX03.axis.com> Message-ID: <63038209783f4414ab6f3ed7b08ff1d8@XBOX03.axis.com> > -----Original Message----- > From: Richard Purdie > Sent: den 10 februari 2020 16:18 > To: Peter Kjellerstedt ; bitbake- > devel at lists.openembedded.org > Subject: Re: [bitbake-devel] [master][zeus][PATCH] uihelper: Track scene > tasks again > > On Mon, 2020-02-10 at 15:01 +0000, Peter Kjellerstedt wrote: > > If a setscene task is rerun, won't that just mean that the counters > > don't change due to this? So in this case there won't be any > > progress to the total progress bar even if a task is running? That > > should still be better than the current situation when they are > > never counted. > > No, that isn't how the code is written. You'd see "task 1000 of 500". > We got some pretty horrible numbers when testing this originally, I got > lots of complaints and its why it was removed. > > The code has changed a bit since and we probably don't rerun as much > but we get complains when we print any invalid info... > > Cheers, > > Richard Hmm, ok. I get loud complaints that we are printing "(0 of 0)" while thousands of setscene tasks are being processed, so that is unfortunately not much better... Would it be possible to change the accounting so that we do not end up with "task 1000 of 500"? For the record, we are not using the hash equivalence server, so I guess we should not have any setscene tasks that are being rerun, but I have a feeling you do not want to treat the two cases differently. //Peter From ross at burtonini.com Mon Feb 10 15:57:04 2020 From: ross at burtonini.com (Ross Burton) Date: Mon, 10 Feb 2020 15:57:04 +0000 Subject: [bitbake-devel] git-lfs validation In-Reply-To: References: Message-ID: Please do send a patch, or ideally two separate patches. Ross On Mon, 10 Feb 2020 at 14:14, Mauro Queir?s wrote: > > Hi, > > > > I noticed two things related to git-lfs on poky/bitbake/lib/bb/fetch2/git.py, which I think may me improved: > > > > First, git.py seems to always check the master branch when looking for lfs objects (_contains_lfs function), even when the branch is specified in SRC_URI; > > > > Second, the warning message "Repository %s has LFS content but it is not being fetched" is printed even when git-lfs is available and "lfs=1" is set. > > > > I could also provide a patch if you think these two issues make sense. > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel at lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel From armccurdy at gmail.com Mon Feb 10 21:12:34 2020 From: armccurdy at gmail.com (Andre McCurdy) Date: Mon, 10 Feb 2020 13:12:34 -0800 Subject: [bitbake-devel] [HELP] Create a new image recipe based on a (changed) existing one In-Reply-To: <1581290815745.54479@parceiros.nos.pt> References: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> <1581290815745.54479@parceiros.nos.pt> Message-ID: On Mon, Feb 10, 2020 at 6:15 AM Joao Carlos Cabral (P) wrote: > > Hi, > > I'm trying to create a image recipe based on a existing one. > I've searched online, and I could find such an example. > > My problem is that the image I what to serve as a base is changed in > another layer, and I want to include that changes. > > Ex: > > (tree) > > Prj_root > -> layer1 > -> recipes-core > -> images > -> my-image.bb [This is the base recipe] > -> layer2 > -> recipes-core > -> images > -> my-image.bbappend [This is the changed one] > -> layer3 > -> recipes-core > -> images > -> my-image-dev.bb [this is the new image, based on my-image] > > my-image.bbappend is something like this: > > --------------------- my-image.bbappend --------- > PROVIDES = "product" > > IMAGE_INSTALL += " \ > \ > " > IMAGE_INSTALL_remove += " \ > > " > > (+ other functions related with signing the image) > _________________________________________________ > > The final image is created with 'bitbake product'. > And, I want to create a dev image that must be based on the current > "product" image (my-image.bb + my-image.bbappend) with some dev > packages. > So, I need to include the changes that my-image.bbapend does to this > dev image I'm trying to create. > > > Tried simple create a new image that should be a copy of the working > one, and then add the needed packages for development. > So I did the following... > > ---------------- my-image-dev.bbappend ---------- > > SUMMARY = "dev Image" > > require layer1/recipes-core/images/my-image.bb > ------------------------------------------------ > > I can run 'bitbake my-image-dev' and it runs, but doesn't produce the > same output as 'bitbake product'. > > Also, If I run 'bitbake my-image-dev' 1st, it finishes fine. Running > 'bitbake product' after makes a bunch of other tasks and finishes fine. > Running again 'bitbake my-image-dev' finishes with an error on a > specific task. ( It shows that they are not building the same thing) > > How can I do what I need? > NOTE: I can't change anything in the layer1 and layer2. For the future, the clean way to do this would be to for the base image to include a package group for dev tools. The dev tools package group would be defined by a recipe in layer 1 and empty by default. When you want to add dev tools you can create a .bbappend (in layer 3) for the dev tools recipe. If you can't modify layers 1 or 2 you can archieve something similar with a .bbappend in layer 3 which adds the dev tools to one of the package groups which _is_ being included in the image. For example, if you look in the manifest for the image you should see a number of package groups named packagegroup-base-ipv6, packagegroup-base-nfs, etc which are being included in the image. You can add packages to these package groups by using a .bbappend for the " packagegroup-base.bb" recipe. From akuster808 at gmail.com Tue Feb 11 01:47:43 2020 From: akuster808 at gmail.com (Armin Kuster) Date: Mon, 10 Feb 2020 17:47:43 -0800 Subject: [bitbake-devel] [1.44 0/7] Pull request Message-ID: Consolidate pull request, Cover letter only. The following changes since commit 1a026d4aca47ed1b0b74a8a818635e7520e8f9c8: runqueue: Use a set for the setscene tasks list (2020-01-18 12:20:38 -0800) are available in the Git repository at: git://git.openembedded.org/bitbake-contrib stable/1.44-next http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.44-next Chris Laplante (3): fetch2: do not suffix srcrev cache key with PN fetch2/git: _revision_key: collapse adjacent slashes persist_data.py: Immediately get exclusive lock in __setitem__ Frazer Clews (1): cooker/toaster: replaced deprecated method warn() with warning() Richard Purdie (3): siggen: Optimise get_unihash disk based cache handling siggen: Cache unihash values to avoid cache lookup siggen: Avoid cache mismatch issues with locked sigs lib/bb/cooker.py | 2 +- lib/bb/fetch2/__init__.py | 3 +- lib/bb/fetch2/git.py | 4 +- lib/bb/persist_data.py | 3 + lib/bb/siggen.py | 68 +++++++++++++------ .../management/commands/runbuilds.py | 6 +- 6 files changed, 60 insertions(+), 26 deletions(-) -- 2.17.1 From dl9pf at gmx.de Fri Feb 14 14:22:00 2020 From: dl9pf at gmx.de (dl9pf at gmx.de) Date: Fri, 14 Feb 2020 15:22:00 +0100 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch Message-ID: <20200214142200.28757-1-dl9pf@gmx.de> From: Jan-Simon Moeller Bitbake-layerindex would not respect the branch given with -b . This fixes the clone command. Signed-off-by: Jan-Simon Moeller --- lib/bblayers/layerindex.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py index 57cd9027..093434ac 100644 --- a/lib/bblayers/layerindex.py +++ b/lib/bblayers/layerindex.py @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): This class inherits ActionPlugin to get do_add_layer. """ - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer): + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): layername = self.get_layer_name(url) if os.path.splitext(layername)[1] == '.git': layername = os.path.splitext(layername)[0] @@ -32,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin): layerdir = os.path.join(repodir, subdir) if not os.path.exists(repodir): if fetch_layer: - result = subprocess.call(['git', 'clone', url, repodir]) + result = subprocess.call(['git', 'clone', '-b', branch, url, repodir]) if result: logger.error("Failed to download %s" % url) return None, None, None @@ -70,9 +70,9 @@ class LayerIndexPlugin(ActionPlugin): # Set the default... if args.branch: - branches = [args.branch] + branch = [args.branch] else: - branches = (self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master').split() + branch = (self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master').split() logger.debug(1, 'Trying branches: %s' % branches) ignore_layers = [] @@ -171,7 +171,7 @@ class LayerIndexPlugin(ActionPlugin): subdir, name, layerdir = self.get_fetch_layer(fetchdir, layerBranch.layer.vcs_url, layerBranch.vcs_subdir, - not args.show_only) + not args.show_only, branch) if not name: # Error already shown return 1 -- 2.11.0 From dl9pf at gmx.de Fri Feb 14 14:47:42 2020 From: dl9pf at gmx.de (dl9pf at gmx.de) Date: Fri, 14 Feb 2020 15:47:42 +0100 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch Message-ID: <20200214144742.28943-1-dl9pf@gmx.de> From: Jan-Simon Moeller Bitbake-layerindex would not respect the branch given with -b . This fixes the clone command. V2: Corrected 'branch' back to 'branches' and added a FIXME on multiple LAYERSERIES_CORENAMES. Signed-off-by: Jan-Simon Moeller --- lib/bblayers/layerindex.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py index 57cd9027..32454c8a 100644 --- a/lib/bblayers/layerindex.py +++ b/lib/bblayers/layerindex.py @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): This class inherits ActionPlugin to get do_add_layer. """ - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer): + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): layername = self.get_layer_name(url) if os.path.splitext(layername)[1] == '.git': layername = os.path.splitext(layername)[0] @@ -32,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin): layerdir = os.path.join(repodir, subdir) if not os.path.exists(repodir): if fetch_layer: - result = subprocess.call(['git', 'clone', url, repodir]) + result = subprocess.call(['git', 'clone', '-b', branch, url, repodir]) if result: logger.error("Failed to download %s" % url) return None, None, None @@ -71,8 +71,12 @@ class LayerIndexPlugin(ActionPlugin): # Set the default... if args.branch: branches = [args.branch] + checkoutbranch = args.branch else: branches = (self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master').split() + # FIXME: multple branches is not deterministic when cloning the repository + # for now pull master ... + checkoutbranch = 'master' logger.debug(1, 'Trying branches: %s' % branches) ignore_layers = [] @@ -171,7 +175,7 @@ class LayerIndexPlugin(ActionPlugin): subdir, name, layerdir = self.get_fetch_layer(fetchdir, layerBranch.layer.vcs_url, layerBranch.vcs_subdir, - not args.show_only) + not args.show_only, checkoutbranch) if not name: # Error already shown return 1 -- 2.11.0 From dl9pf at gmx.de Fri Feb 14 15:01:09 2020 From: dl9pf at gmx.de (dl9pf at gmx.de) Date: Fri, 14 Feb 2020 16:01:09 +0100 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch Message-ID: <20200214150109.29296-1-dl9pf@gmx.de> From: Jan-Simon Moeller Bitbake-layerindex would not respect the branch given with -b . This fixes the clone command. V2: Corrected 'branch' back to 'branches' and added a FIXME on multiple LAYERSERIES_CORENAMES. V3: After discussion on IRC use branches[-1] as well educated guess for now but leave the FIXME in place. Signed-off-by: Jan-Simon Moeller --- lib/bblayers/layerindex.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py index 57cd9027..096ea4f8 100644 --- a/lib/bblayers/layerindex.py +++ b/lib/bblayers/layerindex.py @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): This class inherits ActionPlugin to get do_add_layer. """ - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer): + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): layername = self.get_layer_name(url) if os.path.splitext(layername)[1] == '.git': layername = os.path.splitext(layername)[0] @@ -32,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin): layerdir = os.path.join(repodir, subdir) if not os.path.exists(repodir): if fetch_layer: - result = subprocess.call(['git', 'clone', url, repodir]) + result = subprocess.call(['git', 'clone', '-b', branch, url, repodir]) if result: logger.error("Failed to download %s" % url) return None, None, None @@ -73,6 +73,10 @@ class LayerIndexPlugin(ActionPlugin): branches = [args.branch] else: branches = (self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master').split() + + # FIXME: multple branches is not deterministic when cloning the repository + # for now we pull the last entry + checkoutbranch = branches[-1] logger.debug(1, 'Trying branches: %s' % branches) ignore_layers = [] @@ -171,7 +175,7 @@ class LayerIndexPlugin(ActionPlugin): subdir, name, layerdir = self.get_fetch_layer(fetchdir, layerBranch.layer.vcs_url, layerBranch.vcs_subdir, - not args.show_only) + not args.show_only, checkoutbranch) if not name: # Error already shown return 1 -- 2.11.0 From peter.kjellerstedt at axis.com Fri Feb 14 15:32:07 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Fri, 14 Feb 2020 15:32:07 +0000 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch In-Reply-To: <20200214150109.29296-1-dl9pf@gmx.de> References: <20200214150109.29296-1-dl9pf@gmx.de> Message-ID: <8650a200f4274ea79ba23e1e6604004c@XBOX03.axis.com> > -----Original Message----- > From: bitbake-devel-bounces at lists.openembedded.org bounces at lists.openembedded.org> On Behalf Of dl9pf at gmx.de > Sent: den 14 februari 2020 16:01 > To: bitbake-devel at lists.openembedded.org > Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the > requested branch > > From: Jan-Simon Moeller > > Bitbake-layerindex would not respect the branch given with -b . > This fixes the clone command. > > V2: Corrected 'branch' back to 'branches' and added > a FIXME on multiple LAYERSERIES_CORENAMES. > V3: After discussion on IRC use branches[-1] as well > educated guess for now but leave the FIXME in place. Information about the various patch revisions should go below the "---" below as they relate to the patch process and not the commit that eventually ends up in the Git repository. > Signed-off-by: Jan-Simon Moeller > --- > lib/bblayers/layerindex.py | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) //Peter From mark.hatle at kernel.crashing.org Fri Feb 14 18:08:27 2020 From: mark.hatle at kernel.crashing.org (Mark Hatle) Date: Fri, 14 Feb 2020 12:08:27 -0600 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch In-Reply-To: <20200214150109.29296-1-dl9pf@gmx.de> References: <20200214150109.29296-1-dl9pf@gmx.de> Message-ID: <80610678-1f40-8b81-f852-4d48a950ef73@kernel.crashing.org> This issue was already found (and I thought fixed?) Maybe the submission hadn't been accepted. See: http://lists.openembedded.org/pipermail/bitbake-devel/2019-December/020645.html Also comments below inline: On 2/14/20 9:01 AM, dl9pf at gmx.de wrote: > From: Jan-Simon Moeller > > Bitbake-layerindex would not respect the branch given with -b . > This fixes the clone command. > > V2: Corrected 'branch' back to 'branches' and added > a FIXME on multiple LAYERSERIES_CORENAMES. > V3: After discussion on IRC use branches[-1] as well > educated guess for now but leave the FIXME in place. > > Signed-off-by: Jan-Simon Moeller > --- > lib/bblayers/layerindex.py | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) > > diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py > index 57cd9027..096ea4f8 100644 > --- a/lib/bblayers/layerindex.py > +++ b/lib/bblayers/layerindex.py > @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): > This class inherits ActionPlugin to get do_add_layer. > """ > > - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer): > + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): > layername = self.get_layer_name(url) > if os.path.splitext(layername)[1] == '.git': > layername = os.path.splitext(layername)[0] > @@ -32,7 +32,7 @@ class LayerIndexPlugin(ActionPlugin): > layerdir = os.path.join(repodir, subdir) > if not os.path.exists(repodir): > if fetch_layer: > - result = subprocess.call(['git', 'clone', url, repodir]) > + result = subprocess.call(['git', 'clone', '-b', branch, url, repodir]) > if result: > logger.error("Failed to download %s" % url) There are cases where it might be called if the branch is blank. So the repo URL should be called instead. Thus -b should not be forced to always be there. (odd use-case, but it does exist.) > return None, None, None > @@ -73,6 +73,10 @@ class LayerIndexPlugin(ActionPlugin): > branches = [args.branch] > else: > branches = (self.tinfoil.config_data.getVar('LAYERSERIES_CORENAMES') or 'master').split() > + > + # FIXME: multple branches is not deterministic when cloning the repository > + # for now we pull the last entry > + checkoutbranch = branches[-1] The branch here is the name of the branch(es) to query the layerindex for. They are in order of priority, so the first found would be the one that is to be used. > logger.debug(1, 'Trying branches: %s' % branches) > > ignore_layers = [] > @@ -171,7 +175,7 @@ class LayerIndexPlugin(ActionPlugin): > subdir, name, layerdir = self.get_fetch_layer(fetchdir, > layerBranch.layer.vcs_url, > layerBranch.vcs_subdir, > - not args.show_only) > + not args.show_only, checkoutbranch) This is incorrect, it must use the branch value from the layerBranch.actual_branch. Since the branch in the repository and the branch (corename) being questioned can and are often different. --Mark > if not name: > # Error already shown > return 1 > -- > 2.11.0 > From armccurdy at gmail.com Fri Feb 14 23:33:04 2020 From: armccurdy at gmail.com (Andre McCurdy) Date: Fri, 14 Feb 2020 15:33:04 -0800 Subject: [bitbake-devel] [HELP] Create a new image recipe based on a (changed) existing one In-Reply-To: <3cb24a9342e09cea61c0bde03c26d165588faf96.camel@parceiros.nos.pt> References: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> <1581290815745.54479@parceiros.nos.pt> <3cb24a9342e09cea61c0bde03c26d165588faf96.camel@parceiros.nos.pt> Message-ID: On Thu, Feb 13, 2020 at 9:34 AM Joao Carlos Cabral (P) wrote: > > Thanks for your answer. > > Using your method, how could I create the normal image and the dev one? > > * bitbake product - would produce the normal one? > * bitbake my-image - would produce the dev one? > > I still need to create the "normal" when ending a develop phase. Assuming that you can't change layers 1 or 2 then the ONLY thing you can build which will include both the my-image recipe in layer 1 and its .bbappend in layer 2 is the my-image recipe itself. If the development image is created by adding to the contents of my-image by using a .bbappend in layer 3 then to switch back to the normal image you would need to disable that .bbappend. e.g. If you have a .bbappend which adds to a package group under the control of a variable then you could switch between production and development by setting that variable from a global config file (e.g. local.conf) and rebuilding my-image. A different approach which _would_ allow you to switch between production and development by building a different image would be to duplicate the .bbappend from layer 2 in your layer 3. In layer 3 you could then have a new image recipe which directly includes the layer 1 recipe (ie what you tried originally) plus a new .bbappend which makes the same changes to your new image recipe as the .bbappend in layer 2 makes to the layer 1 image recipe. If the contents of the .bbappend don't need to be changed then you could perhaps make the layer 3 .bbappend a symlink pointing back to the layer 2 .bbappend, rather than just copying and renaming the whole file. From pkj at axis.com Sat Feb 15 03:29:53 2020 From: pkj at axis.com (Peter Kjellerstedt) Date: Sat, 15 Feb 2020 04:29:53 +0100 Subject: [bitbake-devel] [master][zeus][PATCH 1/3] knotty: Make the bb.command.CommandExit event terminate bitbake Message-ID: <20200215032955.3958-1-pkj@axis.com> This matches the other bb.command.Command* events and without it, running `bitbake --revisions-changed` will hang indefinitely if there are changed revisions. Signed-off-by: Peter Kjellerstedt --- bitbake/lib/bb/ui/knotty.py | 1 + 1 file changed, 1 insertion(+) diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index a0340dfc20..cbb289b05f 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -590,6 +590,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, bb.command.CommandExit): if not return_value: return_value = event.exitcode + main.shutdown = 2 continue if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): main.shutdown = 2 -- 2.21.1 From pkj at axis.com Sat Feb 15 03:29:54 2020 From: pkj at axis.com (Peter Kjellerstedt) Date: Sat, 15 Feb 2020 04:29:54 +0100 Subject: [bitbake-devel] [master][zeus][PATCH 2/3] fetch2: Make fetcher_compare_revisions() work In-Reply-To: <20200215032955.3958-1-pkj@axis.com> References: <20200215032955.3958-1-pkj@axis.com> Message-ID: <20200215032955.3958-2-pkj@axis.com> This seems to have been broken for a very long time. Now it also works regardless of BB_SRCREV_POLICY. Signed-off-by: Peter Kjellerstedt --- bitbake/lib/bb/fetch2/__init__.py | 33 +++++++++++++------------------ 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 1b7f573642..56462767e5 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -485,17 +485,22 @@ def fetcher_init(d): Called to initialize the fetchers once the configuration data is known. Calls before this must not hit the cache. """ + + revs = bb.persist_data.persist('BB_URI_HEADREVS', d) + try: + # fetcher_init is called multiple times, so make sure we only save the + # revs the first time it is called. + if not bb.fetch2.saved_headrevs: + bb.fetch2.saved_headrevs = dict(revs) + except: + pass + # When to drop SCM head revisions controlled by user policy srcrev_policy = d.getVar('BB_SRCREV_POLICY') or "clear" if srcrev_policy == "cache": logger.debug(1, "Keeping SRCREV cache due to cache policy of: %s", srcrev_policy) elif srcrev_policy == "clear": logger.debug(1, "Clearing SRCREV cache due to cache policy of: %s", srcrev_policy) - revs = bb.persist_data.persist('BB_URI_HEADREVS', d) - try: - bb.fetch2.saved_headrevs = revs.items() - except: - pass revs.clear() else: raise FetchError("Invalid SRCREV cache policy of: %s" % srcrev_policy) @@ -514,22 +519,12 @@ def fetcher_parse_done(): def fetcher_compare_revisions(d): """ - Compare the revisions in the persistant cache with current values and - return true/false on whether they've changed. + Compare the revisions in the persistent cache with the saved values from + when bitbake was started and return true if they have changed. """ - data = bb.persist_data.persist('BB_URI_HEADREVS', d).items() - data2 = bb.fetch2.saved_headrevs - - changed = False - for key in data: - if key not in data2 or data2[key] != data[key]: - logger.debug(1, "%s changed", key) - changed = True - return True - else: - logger.debug(2, "%s did not change", key) - return False + headrevs = dict(bb.persist_data.persist('BB_URI_HEADREVS', d)) + return headrevs != bb.fetch2.saved_headrevs def mirror_from_string(data): mirrors = (data or "").replace('\\n',' ').split() -- 2.21.1 From pkj at axis.com Sat Feb 15 03:29:55 2020 From: pkj at axis.com (Peter Kjellerstedt) Date: Sat, 15 Feb 2020 04:29:55 +0100 Subject: [bitbake-devel] [master][zeus][PATCH 3/3] fetch2: Allow ${AUTOREV} to be used when BB_SRCREV_POLICY is "cache" In-Reply-To: <20200215032955.3958-1-pkj@axis.com> References: <20200215032955.3958-1-pkj@axis.com> Message-ID: <20200215032955.3958-3-pkj@axis.com> Mark any keys used to cache the srcrevs for a recipe as "dontcache" if BB_DONT_CACHE is set for the recipe. Remove any such keys upon the next bitbake run even if BB_SRCREV_POLICY is set to "cache". This will make sure the srcrev is updated as expected if ${AUTOREV} is used. Signed-off-by: Peter Kjellerstedt --- bitbake/lib/bb/fetch2/__init__.py | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py index 56462767e5..b83347ad90 100644 --- a/bitbake/lib/bb/fetch2/__init__.py +++ b/bitbake/lib/bb/fetch2/__init__.py @@ -499,6 +499,11 @@ def fetcher_init(d): srcrev_policy = d.getVar('BB_SRCREV_POLICY') or "clear" if srcrev_policy == "cache": logger.debug(1, "Keeping SRCREV cache due to cache policy of: %s", srcrev_policy) + # We still need to remove keys that are marked with "dontcache". + for key in list(revs.keys()): + if key.startswith("dontcache-"): + logger.debug(1, "Removing SRCREV key: %s" % key) + revs.pop(key) elif srcrev_policy == "clear": logger.debug(1, "Clearing SRCREV cache due to cache policy of: %s", srcrev_policy) revs.clear() @@ -729,9 +734,8 @@ def subprocess_setup(): signal.signal(signal.SIGPIPE, signal.SIG_DFL) def get_autorev(d): - # only not cache src rev in autorev case - if d.getVar('BB_SRCREV_POLICY') != "cache": - d.setVar('BB_DONT_CACHE', '1') + # Do not cache the srcrev in the autorev case + d.setVar('BB_DONT_CACHE', '1') return "AUTOINC" def get_srcrev(d, method_name='sortable_revision'): @@ -1594,7 +1598,13 @@ class FetchMethod(object): return True, str(latest_rev) def generate_revision_key(self, ud, d, name): - return self._revision_key(ud, d, name) + key = self._revision_key(ud, d, name) + if d.getVar('BB_DONT_CACHE'): + # Mark the key so it can be removed on the next bitbake run even if + # BB_SRCREV_POLICY is "cache". + return "dontcache-%s" % key + else: + return key def latest_versionstring(self, ud, d): """ -- 2.21.1 From Joao.CCabral at parceiros.nos.pt Thu Feb 13 17:34:05 2020 From: Joao.CCabral at parceiros.nos.pt (Joao Carlos Cabral (P)) Date: Thu, 13 Feb 2020 17:34:05 +0000 Subject: [bitbake-devel] [HELP] Create a new image recipe based on a (changed) existing one In-Reply-To: References: <61f1517446c596d50dfa4aa434e1b3934ab5696c.camel@parceiros.nos.pt> <1581290815745.54479@parceiros.nos.pt> Message-ID: <3cb24a9342e09cea61c0bde03c26d165588faf96.camel@parceiros.nos.pt> Thanks for your answer. Using your method, how could I create the normal image and the dev one? * bitbake product - would produce the normal one? * bitbake my-image - would produce the dev one? I still need to create the "normal" when ending a develop phase. Thanks Jo?o Cabral On Mon, 2020-02-10 at 13:12 -0800, Andre McCurdy wrote: > On Mon, Feb 10, 2020 at 6:15 AM Joao Carlos Cabral (P) > wrote: > > > > Hi, > > > > I'm trying to create a image recipe based on a existing one. > > I've searched online, and I could find such an example. > > > > My problem is that the image I what to serve as a base is changed > > in > > another layer, and I want to include that changes. > > > > Ex: > > > > (tree) > > > > Prj_root > > -> layer1 > > -> recipes-core > > -> images > > -> my-image.bb [This is the base recipe] > > -> layer2 > > -> recipes-core > > -> images > > -> my-image.bbappend [This is the changed one] > > -> layer3 > > -> recipes-core > > -> images > > -> my-image-dev.bb [this is the new image, based on > > my-image] > > > > my-image.bbappend is something like this: > > > > --------------------- my-image.bbappend --------- > > PROVIDES = "product" > > > > IMAGE_INSTALL += " \ > > \ > > " > > IMAGE_INSTALL_remove += " \ > > > > " > > > > (+ other functions related with signing the image) > > _________________________________________________ > > > > The final image is created with 'bitbake product'. > > And, I want to create a dev image that must be based on the current > > "product" image (my-image.bb + my-image.bbappend) with some dev > > packages. > > So, I need to include the changes that my-image.bbapend does to > > this > > dev image I'm trying to create. > > > > > > Tried simple create a new image that should be a copy of the > > working > > one, and then add the needed packages for development. > > So I did the following... > > > > ---------------- my-image-dev.bbappend ---------- > > > > SUMMARY = "dev Image" > > > > require layer1/recipes-core/images/my-image.bb > > ------------------------------------------------ > > > > I can run 'bitbake my-image-dev' and it runs, but doesn't produce > > the > > same output as 'bitbake product'. > > > > Also, If I run 'bitbake my-image-dev' 1st, it finishes fine. > > Running > > 'bitbake product' after makes a bunch of other tasks and finishes > > fine. > > Running again 'bitbake my-image-dev' finishes with an error on a > > specific task. ( It shows that they are not building the same > > thing) > > > > How can I do what I need? > > NOTE: I can't change anything in the layer1 and layer2. > > For the future, the clean way to do this would be to for the base > image to include a package group for dev tools. The dev tools package > group would be defined by a recipe in layer 1 and empty by default. > When you want to add dev tools you can create a .bbappend (in layer > 3) > for the dev tools recipe. > > If you can't modify layers 1 or 2 you can archieve something similar > with a .bbappend in layer 3 which adds the dev tools to one of the > package groups which _is_ being included in the image. For example, > if > you look in the manifest for the image you should see a number of > package groups named packagegroup-base-ipv6, packagegroup-base-nfs, > etc which are being included in the image. You can add packages to > these package groups by using a .bbappend for the " > packagegroup-base.bb" recipe. From JanSimon.Moeller at gmx.de Fri Feb 14 21:07:52 2020 From: JanSimon.Moeller at gmx.de (=?UTF-8?Q?=22Jan-Simon_M=C3=B6ller=22?=) Date: Fri, 14 Feb 2020 22:07:52 +0100 Subject: [bitbake-devel] [PATCH] Fix bitbake-layerindex to checkout the requested branch In-Reply-To: <80610678-1f40-8b81-f852-4d48a950ef73@kernel.crashing.org> References: <20200214150109.29296-1-dl9pf@gmx.de> <80610678-1f40-8b81-f852-4d48a950ef73@kernel.crashing.org> Message-ID: Hi Mark! Thanks for reviewing. I had hoped to find the right variables. A pity it did not land earlier. Can we backport this to zeus, please ? ? ? > This issue was already found (and I thought fixed?) Maybe the submission hadn't > been accepted. See: > > http://lists.openembedded.org/pipermail/bitbake-devel/2019-December/020645.html Please ignore my patchset then. Best, JS From bkylerussell at gmail.com Mon Feb 17 15:25:52 2020 From: bkylerussell at gmail.com (Kyle Russell) Date: Mon, 17 Feb 2020 10:25:52 -0500 Subject: [bitbake-devel] [][PATCH] bitbake: don't allow renameVar calls with equivalent keys Message-ID: <20200217152552.31991-1-bkylerussell@gmail.com> While usually a programming error, the behavior can cause a Parser instance to eventually gobble up a significant amount of memory, greatly affecting system performance. Try to avoid getting into that situation and alert the user about what they attempted to do. Signed-off-by: Kyle Russell --- lib/bb/data_smart.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py index dd5c6185..63a32bc6 100644 --- a/lib/bb/data_smart.py +++ b/lib/bb/data_smart.py @@ -607,6 +607,10 @@ class DataSmart(MutableMapping): """ Rename the variable key to newkey """ + if key == newkey: + bb.warn("Calling renameVar with equivalent keys (%s) is invalid" % key) + return + if '_remote_data' in self.dict: connector = self.dict["_remote_data"]["_content"] res = connector.renameVar(key, newkey) -- 2.17.1 From richard.purdie at linuxfoundation.org Mon Feb 17 16:23:07 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 17 Feb 2020 16:23:07 +0000 Subject: [bitbake-devel] [PATCH] cooker/siggen: Empty siggen cache during parsing Message-ID: <20200217162307.36082-1-richard.purdie@linuxfoundation.org> When parsing recipes its apparent the memory usage of bitbake rises linearly with number of recipes parsed. It shouldn't. Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the dependency information left behind in siggen was the culprit. Add a new method to allow us to drop this information. We don't need it after the recipe has been parsed and hashes calculated (at runtime its different but only the currently executing task would be in memory). This should give signficant memory usage improvements for bitbake and that in turn should help speed on more constrained systems, as well as when used in multiconfig environments. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 3 ++- lib/bb/siggen.py | 11 +++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 1124332186..a05630d52d 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1940,7 +1940,8 @@ class Parser(multiprocessing.Process): except IndexError: break result = self.parse(*job) - + # Clear the siggen cache after parsing to control memory usage, its huge + bb.parse.siggen.postparsing_clean_cache() try: self.results.put(result, timeout=0.25) except queue.Full: diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py index ffd8fcaf36..c2d0c736cf 100644 --- a/lib/bb/siggen.py +++ b/lib/bb/siggen.py @@ -51,6 +51,9 @@ class SignatureGenerator(object): def finalise(self, fn, d, varient): return + def postparsing_clean_cache(self): + return + def get_unihash(self, tid): return self.taskhash[tid] @@ -188,6 +191,14 @@ class SignatureGeneratorBasic(SignatureGenerator): for task in taskdeps: d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task]) + def postparsing_clean_cache(self): + # + # After parsing we can remove some things from memory to reduce our memory footprint + # + self.gendeps = {} + self.lookupcache = {} + self.taskdeps = {} + def rundep_check(self, fn, recipename, task, dep, depname, dataCache): # Return True if we should keep the dependency, False to drop it # We only manipulate the dependencies for packages not in the whitelist -- 2.25.0 From sno at netbsd.org Tue Feb 18 05:40:53 2020 From: sno at netbsd.org (Jens Rehsack) Date: Tue, 18 Feb 2020 06:40:53 +0100 Subject: [bitbake-devel] [OE-core][PATCH REBASED] bitbake: fetch2: svn: care for path_spec Message-ID: <20200218054053.2891-1-sno@netbsd.org> From: Jens Rehsack Documentation says: "path_spec": A specific directory in which to checkout the specified svn module. but existing svn fetcher uses "module" always as path of checked out svn-module, regardless whether path_spec was given or not. Signed-off-by: Jens Rehsack --- bitbake/lib/bb/fetch2/svn.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py index 6c8caf5fb9..385a6b2cf1 100644 --- a/bitbake/lib/bb/fetch2/svn.py +++ b/bitbake/lib/bb/fetch2/svn.py @@ -47,7 +47,7 @@ class Svn(FetchMethod): svndir = d.getVar("SVNDIR") or (d.getVar("DL_DIR") + "/svn") relpath = self._strip_leading_slashes(ud.path) ud.pkgdir = os.path.join(svndir, ud.host, relpath) - ud.moddir = os.path.join(ud.pkgdir, ud.module) + ud.moddir = os.path.join(ud.pkgdir, ud.path_spec) # Protects the repository from concurrent updates, e.g. from two # recipes fetching different revisions at the same time ud.svnlock = os.path.join(ud.pkgdir, "svn.lock") -- 2.17.1 From liezhi.yang at windriver.com Tue Feb 18 10:48:01 2020 From: liezhi.yang at windriver.com (Robert Yang) Date: Tue, 18 Feb 2020 18:48:01 +0800 Subject: [bitbake-devel] [PATCH 0/1] bitbake/lib/bb/event.py: Remove duplicated ones from close matches Message-ID: The following changes since commit 925800570f9093cc8de90091008759998e5cd4f2: bitbake: cooker/siggen: Empty siggen cache during parsing (2020-02-17 23:11:38 +0000) are available in the git repository at: git://git.yoctoproject.org/poky-contrib rbt/close http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rbt/close Robert Yang (1): bitbake/lib/bb/event.py: Remove duplicated ones from close matches bitbake/lib/bb/event.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.7.4 From liezhi.yang at windriver.com Tue Feb 18 10:48:02 2020 From: liezhi.yang at windriver.com (Robert Yang) Date: Tue, 18 Feb 2020 18:48:02 +0800 Subject: [bitbake-devel] [PATCH 1/1] bitbake/lib/bb/event.py: Remove duplicated ones from close matches In-Reply-To: References: Message-ID: It printed duplicated ones when there are multiple similar recipes in differrent layers, for example, if python-lockfile in different layers, and there is no python3-lockfile: $ bitbake python3-lockfile ERROR: Nothing PROVIDES 'python3-lockfile'. Close matches: python-lockfile python-lockfile python3-aiofiles Remove the duplicated ones to fix the problem. Signed-off-by: Robert Yang --- bitbake/lib/bb/event.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py index 5cfbf36..d1359f0 100644 --- a/bitbake/lib/bb/event.py +++ b/bitbake/lib/bb/event.py @@ -508,7 +508,7 @@ class NoProvider(Event): extra = '' if not self._reasons: if self._close_matches: - extra = ". Close matches:\n %s" % '\n '.join(self._close_matches) + extra = ". Close matches:\n %s" % '\n '.join(sorted(set(self._close_matches))) if self._dependees: msg = "Nothing %sPROVIDES '%s' (but %s %sDEPENDS on or otherwise requires it)%s" % (r, self._item, ", ".join(self._dependees), r, extra) -- 2.7.4 From dl9pf at gmx.de Tue Feb 18 13:55:54 2020 From: dl9pf at gmx.de (Jan-Simon Moeller) Date: Tue, 18 Feb 2020 14:55:54 +0100 Subject: [bitbake-devel] [PATCH] bitbake: layerindex: allow clones to be shallow Message-ID: <20200218135554.19362-1-dl9pf@gmx.de> When bitbake-layers fetch-layerindex clones the repositories, these are full clones. Allow the user to specify '-s' and do shallow clones instead for faster downloads. Signed-off-by: Jan-Simon Moeller --- lib/bblayers/layerindex.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py index aa3b682d..95b67a66 100644 --- a/lib/bblayers/layerindex.py +++ b/lib/bblayers/layerindex.py @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): This class inherits ActionPlugin to get do_add_layer. """ - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch, shallow=False): layername = self.get_layer_name(url) if os.path.splitext(layername)[1] == '.git': layername = os.path.splitext(layername)[0] @@ -32,10 +32,12 @@ class LayerIndexPlugin(ActionPlugin): layerdir = os.path.join(repodir, subdir) if not os.path.exists(repodir): if fetch_layer: - cmd = ['git', 'clone', '-b' , branch, url, repodir] - if not branch: - # Branch really shouldn't be empty, but use the repo default if it is - cmd = ['git', 'clone', url, repodir] + cmd = ['git', 'clone'] + if shallow: + cmd.extend(['--depth', '1']) + if branch: + cmd.extend(['-b' , branch]) + cmd.extend([url, repodir]) result = subprocess.call(cmd) if result: logger.error("Failed to download %s (%s)" % (url, branch)) @@ -176,7 +178,8 @@ class LayerIndexPlugin(ActionPlugin): layerBranch.layer.vcs_url, layerBranch.vcs_subdir, not args.show_only, - layerBranch.actual_branch) + layerBranch.actual_branch, + args.shallow) if not name: # Error already shown return 1 @@ -209,6 +212,7 @@ class LayerIndexPlugin(ActionPlugin): parser_layerindex_fetch = self.add_command(sp, 'layerindex-fetch', self.do_layerindex_fetch, parserecipes=False) parser_layerindex_fetch.add_argument('-n', '--show-only', help='show dependencies and do nothing else', action='store_true') parser_layerindex_fetch.add_argument('-b', '--branch', help='branch name to fetch') + parser_layerindex_fetch.add_argument('-s', '--shallow', help='do only shallow clones (--depth=1)', action='store_true') parser_layerindex_fetch.add_argument('-i', '--ignore', help='assume the specified layers do not need to be fetched/added (separate multiple layers with commas, no spaces)', metavar='LAYER') parser_layerindex_fetch.add_argument('layername', nargs='+', help='layer to fetch') -- 2.11.0 From akuster808 at gmail.com Tue Feb 18 15:44:47 2020 From: akuster808 at gmail.com (akuster808) Date: Tue, 18 Feb 2020 07:44:47 -0800 Subject: [bitbake-devel] [PATCH] cooker/siggen: Empty siggen cache during parsing In-Reply-To: <20200217162307.36082-1-richard.purdie@linuxfoundation.org> References: <20200217162307.36082-1-richard.purdie@linuxfoundation.org> Message-ID: <781da9cf-0229-1d3f-ab03-e064cd27265a@gmail.com> On 2/17/20 8:23 AM, Richard Purdie wrote: > When parsing recipes its apparent the memory usage of bitbake rises linearly > with number of recipes parsed. It shouldn't. This seems back port worthy to 1.44, 1.42 - armin > > Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the > dependency information left behind in siggen was the culprit. Add a new > method to allow us to drop this information. We don't need it after the recipe > has been parsed and hashes calculated (at runtime its different but only the > currently executing task would be in memory). > > This should give signficant memory usage improvements for bitbake and that > in turn should help speed on more constrained systems, as well as when used in > multiconfig environments. > > Signed-off-by: Richard Purdie > --- > lib/bb/cooker.py | 3 ++- > lib/bb/siggen.py | 11 +++++++++++ > 2 files changed, 13 insertions(+), 1 deletion(-) > > diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py > index 1124332186..a05630d52d 100644 > --- a/lib/bb/cooker.py > +++ b/lib/bb/cooker.py > @@ -1940,7 +1940,8 @@ class Parser(multiprocessing.Process): > except IndexError: > break > result = self.parse(*job) > - > + # Clear the siggen cache after parsing to control memory usage, its huge > + bb.parse.siggen.postparsing_clean_cache() > try: > self.results.put(result, timeout=0.25) > except queue.Full: > diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py > index ffd8fcaf36..c2d0c736cf 100644 > --- a/lib/bb/siggen.py > +++ b/lib/bb/siggen.py > @@ -51,6 +51,9 @@ class SignatureGenerator(object): > def finalise(self, fn, d, varient): > return > > + def postparsing_clean_cache(self): > + return > + > def get_unihash(self, tid): > return self.taskhash[tid] > > @@ -188,6 +191,14 @@ class SignatureGeneratorBasic(SignatureGenerator): > for task in taskdeps: > d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task]) > > + def postparsing_clean_cache(self): > + # > + # After parsing we can remove some things from memory to reduce our memory footprint > + # > + self.gendeps = {} > + self.lookupcache = {} > + self.taskdeps = {} > + > def rundep_check(self, fn, recipename, task, dep, depname, dataCache): > # Return True if we should keep the dependency, False to drop it > # We only manipulate the dependencies for packages not in the whitelist From richard.purdie at linuxfoundation.org Tue Feb 18 15:45:32 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Tue, 18 Feb 2020 15:45:32 +0000 Subject: [bitbake-devel] bitbake memory usage debugging using tracemalloc Message-ID: <64d9a5e38cb0159c1cce6f75b66f46b0d532c7fc.camel@linuxfoundation.org> I wanted to quickly share an example of how we can use tracemalloc to debug memory usage in bitbake. What I'd noticed was that for each loop through the parsing code in the parsing thread, the RSS increased by a lot more than I'd expect. In other words it appeared to be leaking. I've given a code example below, you can see the RSS increase using psutil or resource which basically show the same thing. guppy3 is able to show there are a much larger number of sets and strings at the end compared to the start but not *where* they're hiding. Using tracemalloc, you can snapshot at the start, then at the finish and then get a difference. You run bitbake as "PYTHONTRACEMALLOC=1 bitbake -p". Some example output is: WARNING: /media/build1/poky/bitbake/lib/bb/data.py:364: size=45.0 MiB (+45.0 MiB), count=136542 (+136542), average=346 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:388: size=10.1 MiB (+10.1 MiB), count=320 (+320), average=32.3 KiB WARNING: /media/build1/poky/bitbake/lib/bb/data.py:419: size=3868 KiB (+3868 KiB), count=9634 (+9634), average=411 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:93: size=3643 KiB (+3639 KiB), count=61407 (+61340), average=61 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:357: size=2828 KiB (+2828 KiB), count=37541 (+37541), average=77 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py:163: size=936 KiB (-933 KiB), count=11639 (-736), average=82 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py:83: size=0 B (-858 KiB), count=0 (-9278) WARNING: /media/build1/poky/bitbake/lib/bb/data.py:276: size=786 KiB (+786 KiB), count=3725 (+3725), average=216 B WARNING: /media/build1/poky/bitbake/lib/bb/siggen.py:150: size=682 KiB (+682 KiB), count=4998 (+4998), average=140 B WARNING: /usr/lib/python3.8/copy.py:76: size=730 KiB (+646 KiB), count=1962 (+1761), average=381 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:89: size=633 KiB (+612 KiB), count=3001 (+2901), average=216 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:88: size=633 KiB (+612 KiB), count=3001 (+2901), average=216 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:426: size=552 KiB (+552 KiB), count=4998 (+4998), average=113 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:84: size=422 KiB (+407 KiB), count=6002 (+5793), average=72 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:158: size=793 KiB (+372 KiB), count=956 (+680), average=850 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:296: size=334 KiB (+334 KiB), count=963 (+963), average=355 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:193: size=724 KiB (+309 KiB), count=554 (+158), average=1338 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:798: size=296 KiB (+296 KiB), count=435 (+435), average=697 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:145: size=322 KiB (-256 KiB), count=1964 (-1558), average=168 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:353: size=252 KiB (+252 KiB), count=3428 (+3428), average=75 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:23: size=1600 B (-242 KiB), count=25 (-4654), average=64 B WARNING: :580: size=8878 KiB (+222 KiB), count=107185 (+2382), average=85 B WARNING: /media/build1/poky/bitbake/lib/bb/utils.py:305: size=653 KiB (+185 KiB), count=7046 (+2206), average=95 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:90: size=188 KiB (+181 KiB), count=3002 (+2891), average=64 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:427: size=178 KiB (+178 KiB), count=164 (+164), average=1112 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/__init__.py:163: size=185 KiB (+156 KiB), count=12 (+10), average=15.4 KiB WARNING: /media/build1/poky/bitbake/lib/bb/data.py:331: size=149 KiB (+149 KiB), count=2235 (+2235), average=68 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:379: size=145 KiB (+145 KiB), count=9 (+9), average=16.1 KiB WARNING: /media/build1/poky/bitbake/lib/bb/siggen.py:159: size=144 KiB (+144 KiB), count=1 (+1), average=144 KiB WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:409: size=137 KiB (+127 KiB), count=1445 (+1358), average=97 B (I'd changed the mutiple set() generations in data.py into one single one at the end of the function just to confirm it was that which was persisting in this chart, and to check I was understanding the trace). The remaining thing was to realise those sets allocated in that data.py function were being stored in the siggen code, which is apparent when you trace the function call paths or know the code. Compared to after my recent patch: WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py:163: size=499 KiB (-1369 KiB), count=6297 (-6078), average=81 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py:83: size=0 B (-858 KiB), count=0 (-9278) WARNING: /media/build1/poky/bitbake/lib/bb/siggen.py:150: size=682 KiB (+682 KiB), count=4998 (+4998), average=140 B WARNING: /media/build1/poky/bitbake/lib/bb/data.py:426: size=552 KiB (+552 KiB), count=4998 (+4998), average=113 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:89: size=381 KiB (+360 KiB), count=1807 (+1707), average=216 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:88: size=381 KiB (+360 KiB), count=1807 (+1707), average=216 B WARNING: /usr/lib/python3.8/copy.py:76: size=433 KiB (+348 KiB), count=978 (+777), average=453 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:145: size=315 KiB (-263 KiB), count=1918 (-1602), average=168 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:23: size=1792 B (-242 KiB), count=28 (-4651), average=64 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:84: size=254 KiB (+239 KiB), count=3614 (+3405), average=72 B WARNING: :580: size=8723 KiB (+222 KiB), count=105611 (+2379), average=85 B WARNING: /media/build1/poky/bitbake/lib/bb/utils.py:305: size=653 KiB (+184 KiB), count=7042 (+2203), average=95 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:465: size=333 KiB (-168 KiB), count=3791 (-1747), average=90 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:193: size=252 KiB (-163 KiB), count=237 (-159), average=1089 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:158: size=274 KiB (-147 KiB), count=212 (-64), average=1324 B WARNING: /media/build1/poky/bitbake/lib/bb/siggen.py:159: size=144 KiB (+144 KiB), count=1 (+1), average=144 KiB WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:90: size=113 KiB (+106 KiB), count=1808 (+1697), average=64 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/ast.py:287: size=456 B (-106 KiB), count=1 (-1947), average=456 B WARNING: /media/build1/poky/bitbake/lib/bb/utils.py:395: size=203 KiB (+102 KiB), count=395 (+172), average=525 B WARNING: /usr/lib/python3.8/tracemalloc.py:532: size=92.5 KiB (+92.4 KiB), count=1631 (+1630), average=58 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/__init__.py:163: size=106 KiB (+77.7 KiB), count=7 (+5), average=15.2 KiB WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:750: size=85.0 KiB (+76.8 KiB), count=518 (+468), average=168 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:409: size=72.2 KiB (+62.4 KiB), count=729 (+642), average=101 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:1064: size=656 B (-61.7 KiB), count=1 (-1129), average=656 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:587: size=153 KiB (-61.2 KiB), count=2508 (-1010), average=62 B WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:839: size=59.1 KiB (+54.7 KiB), count=9 (+5), average=6720 B WARNING: /media/build1/poky/bitbake/lib/bb/codeparser.py:38: size=57.0 KiB (+54.3 KiB), count=517 (+493), average=113 B WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py:165: size=0 B (-48.7 KiB), count=0 (-137) WARNING: /media/build1/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py:171: size=0 B (-45.7 KiB), count=0 (-186) WARNING: /media/build1/poky/bitbake/lib/bb/data_smart.py:489: size=45.5 KiB (+45.2 KiB), count=7 (+6), average=6650 B so 45MB of data in just one object is removed, the overall gains were higher. I'm quite impressed with this module and I think we could use it to find other issues in bitbake. Cheers, Richard diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index 2d937a413cb..f4b25c336ed 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -1890,6 +1890,11 @@ class ParsingFailure(Exception): self.recipe = recipe Exception.__init__(self, realexception, recipe) +import psutil +import resource +import guppy +import tracemalloc + class Parser(multiprocessing.Process): def __init__(self, jobs, results, quit, init, profile): self.jobs = jobs @@ -1922,6 +1927,11 @@ class Parser(multiprocessing.Process): if self.init: self.init() + psprocess = psutil.Process(os.getpid()) + bb.warn("Init %d %s %s" % (os.getpid(), str(psprocess.memory_info().rss), str(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss))) + + snapshot1 = tracemalloc.take_snapshot() + pending = [] while True: try: @@ -1942,11 +1952,26 @@ class Parser(multiprocessing.Process): result = self.parse(*job) # Clear the siggen cache after parsing to control memory usage, its huge bb.parse.siggen.postparsing_clean_cache() + + psprocess = psutil.Process(os.getpid()) + bb.warn("Mid %d %s %s" % (os.getpid(), str(psprocess.memory_info().rss), str(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss))) + try: self.results.put(result, timeout=0.25) except queue.Full: pending.append(result) + snapshot2 = tracemalloc.take_snapshot() + top_stats = snapshot2.compare_to(snapshot1, 'lineno') + for stat in top_stats[:30]: + bb.warn(str(stat)) + + psprocess = psutil.Process(os.getpid()) + bb.warn("Exit %d %s %s" % (os.getpid(), str(psprocess.memory_info().rss), str(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss))) + + bb.warn(str(guppy.hpy().heap())) + + def parse(self, filename, appends): try: origfilter = bb.event.LogHandler.filter From richard.purdie at linuxfoundation.org Tue Feb 18 15:48:01 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Tue, 18 Feb 2020 15:48:01 +0000 Subject: [bitbake-devel] [PATCH] cooker/siggen: Empty siggen cache during parsing In-Reply-To: <781da9cf-0229-1d3f-ab03-e064cd27265a@gmail.com> References: <20200217162307.36082-1-richard.purdie@linuxfoundation.org> <781da9cf-0229-1d3f-ab03-e064cd27265a@gmail.com> Message-ID: On Tue, 2020-02-18 at 07:44 -0800, akuster808 wrote: > > On 2/17/20 8:23 AM, Richard Purdie wrote: > > When parsing recipes its apparent the memory usage of bitbake rises > > linearly > > with number of recipes parsed. It shouldn't. > This seems back port worthy to 1.44, 1.42 Yes, I did it already! :) (With my bitbake maintainer hat on) Cheers, Richard From richard.purdie at linuxfoundation.org Tue Feb 18 22:00:54 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Tue, 18 Feb 2020 22:00:54 +0000 Subject: [bitbake-devel] [PATCH] cooker: Reset loghandler Message-ID: <20200218220054.74923-1-richard.purdie@linuxfoundation.org> When parsing, reset the loghandler when finished, else the messages can be misleading. Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index a05630d52d..2d937a413c 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1949,6 +1949,7 @@ class Parser(multiprocessing.Process): def parse(self, filename, appends): try: + origfilter = bb.event.LogHandler.filter # Record the filename we're parsing into any events generated def parse_filter(self, record): record.taskpid = bb.event.worker_pid @@ -1971,6 +1972,8 @@ class Parser(multiprocessing.Process): # a SystemExit event for example. except BaseException as exc: return True, ParsingFailure(exc, filename) + finally: + bb.event.LogHandler.filter = origfilter class CookerParser(object): def __init__(self, cooker, filelist, masked): -- 2.25.0 From mark.hatle at kernel.crashing.org Tue Feb 18 22:06:07 2020 From: mark.hatle at kernel.crashing.org (Mark Hatle) Date: Tue, 18 Feb 2020 16:06:07 -0600 Subject: [bitbake-devel] [PATCH] bitbake: layerindex: allow clones to be shallow In-Reply-To: <20200218135554.19362-1-dl9pf@gmx.de> References: <20200218135554.19362-1-dl9pf@gmx.de> Message-ID: <503a475e-e96d-2bd7-c132-a0df0b886be6@kernel.crashing.org> Ack, this looks fine. It would be nice (in the future) to have a way to specify an optional depth, that could be passed through the system. But as implemented it's fine. --Mark On 2/18/20 7:55 AM, Jan-Simon Moeller wrote: > When bitbake-layers fetch-layerindex clones the repositories, these are > full clones. Allow the user to specify '-s' and do shallow clones > instead for faster downloads. > > Signed-off-by: Jan-Simon Moeller > --- > lib/bblayers/layerindex.py | 16 ++++++++++------ > 1 file changed, 10 insertions(+), 6 deletions(-) > > diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py > index aa3b682d..95b67a66 100644 > --- a/lib/bblayers/layerindex.py > +++ b/lib/bblayers/layerindex.py > @@ -24,7 +24,7 @@ class LayerIndexPlugin(ActionPlugin): > This class inherits ActionPlugin to get do_add_layer. > """ > > - def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch): > + def get_fetch_layer(self, fetchdir, url, subdir, fetch_layer, branch, shallow=False): > layername = self.get_layer_name(url) > if os.path.splitext(layername)[1] == '.git': > layername = os.path.splitext(layername)[0] > @@ -32,10 +32,12 @@ class LayerIndexPlugin(ActionPlugin): > layerdir = os.path.join(repodir, subdir) > if not os.path.exists(repodir): > if fetch_layer: > - cmd = ['git', 'clone', '-b' , branch, url, repodir] > - if not branch: > - # Branch really shouldn't be empty, but use the repo default if it is > - cmd = ['git', 'clone', url, repodir] > + cmd = ['git', 'clone'] > + if shallow: > + cmd.extend(['--depth', '1']) > + if branch: > + cmd.extend(['-b' , branch]) > + cmd.extend([url, repodir]) > result = subprocess.call(cmd) > if result: > logger.error("Failed to download %s (%s)" % (url, branch)) > @@ -176,7 +178,8 @@ class LayerIndexPlugin(ActionPlugin): > layerBranch.layer.vcs_url, > layerBranch.vcs_subdir, > not args.show_only, > - layerBranch.actual_branch) > + layerBranch.actual_branch, > + args.shallow) > if not name: > # Error already shown > return 1 > @@ -209,6 +212,7 @@ class LayerIndexPlugin(ActionPlugin): > parser_layerindex_fetch = self.add_command(sp, 'layerindex-fetch', self.do_layerindex_fetch, parserecipes=False) > parser_layerindex_fetch.add_argument('-n', '--show-only', help='show dependencies and do nothing else', action='store_true') > parser_layerindex_fetch.add_argument('-b', '--branch', help='branch name to fetch') > + parser_layerindex_fetch.add_argument('-s', '--shallow', help='do only shallow clones (--depth=1)', action='store_true') > parser_layerindex_fetch.add_argument('-i', '--ignore', help='assume the specified layers do not need to be fetched/added (separate multiple layers with commas, no spaces)', metavar='LAYER') > parser_layerindex_fetch.add_argument('layername', nargs='+', help='layer to fetch') > > -- > 2.11.0 > From sno at netbsd.org Wed Feb 19 06:27:20 2020 From: sno at netbsd.org (Jens Rehsack) Date: Wed, 19 Feb 2020 07:27:20 +0100 Subject: [bitbake-devel] [PATCH REBASED] bitbake: fetch2: svn: care for path_spec Message-ID: <20200219062720.4509-1-sno@netbsd.org> From: Jens Rehsack Documentation says: "path_spec": A specific directory in which to checkout the specified svn module. but existing svn fetcher uses "module" always as path of checked out svn-module, regardless whether path_spec was given or not. Signed-off-by: Jens Rehsack --- bitbake/lib/bb/fetch2/svn.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py index 6c8caf5fb9..385a6b2cf1 100644 --- a/bitbake/lib/bb/fetch2/svn.py +++ b/bitbake/lib/bb/fetch2/svn.py @@ -47,7 +47,7 @@ class Svn(FetchMethod): svndir = d.getVar("SVNDIR") or (d.getVar("DL_DIR") + "/svn") relpath = self._strip_leading_slashes(ud.path) ud.pkgdir = os.path.join(svndir, ud.host, relpath) - ud.moddir = os.path.join(ud.pkgdir, ud.module) + ud.moddir = os.path.join(ud.pkgdir, ud.path_spec) # Protects the repository from concurrent updates, e.g. from two # recipes fetching different revisions at the same time ud.svnlock = os.path.join(ud.pkgdir, "svn.lock") -- 2.17.1 From richard.purdie at linuxfoundation.org Wed Feb 19 18:44:37 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Wed, 19 Feb 2020 18:44:37 +0000 Subject: [bitbake-devel] [PATCH] fetch2/svn: Avoid UnboundLocalError exception Message-ID: <20200219184437.131258-1-richard.purdie@linuxfoundation.org> The update codepath would trigger: Exception: UnboundLocalError: local variable 'svnfetchcmd' referenced before assignment Fix this so the code functions as intended in both fetch and update cases. [YOCTO #13798] Signed-off-by: Richard Purdie --- lib/bb/fetch2/svn.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/bb/fetch2/svn.py b/lib/bb/fetch2/svn.py index 385a6b2cf1..971a5add4a 100644 --- a/lib/bb/fetch2/svn.py +++ b/lib/bb/fetch2/svn.py @@ -122,30 +122,30 @@ class Svn(FetchMethod): try: if os.access(os.path.join(ud.moddir, '.svn'), os.R_OK): - svnupdatecmd = self._buildsvncommand(ud, d, "update") + svncmd = self._buildsvncommand(ud, d, "update") logger.info("Update " + ud.url) # We need to attempt to run svn upgrade first in case its an older working format try: runfetchcmd(ud.basecmd + " upgrade", d, workdir=ud.moddir) except FetchError: pass - logger.debug(1, "Running %s", svnupdatecmd) - bb.fetch2.check_network_access(d, svnupdatecmd, ud.url) - runfetchcmd(svnupdatecmd, d, workdir=ud.moddir) + logger.debug(1, "Running %s", svncmd) + bb.fetch2.check_network_access(d, svncmd, ud.url) + runfetchcmd(svncmd, d, workdir=ud.moddir) else: - svnfetchcmd = self._buildsvncommand(ud, d, "fetch") + svncmd = self._buildsvncommand(ud, d, "fetch") logger.info("Fetch " + ud.url) # check out sources there bb.utils.mkdirhier(ud.pkgdir) - logger.debug(1, "Running %s", svnfetchcmd) - bb.fetch2.check_network_access(d, svnfetchcmd, ud.url) - runfetchcmd(svnfetchcmd, d, workdir=ud.pkgdir) + logger.debug(1, "Running %s", svncmd) + bb.fetch2.check_network_access(d, svncmd, ud.url) + runfetchcmd(svncmd, d, workdir=ud.pkgdir) if not ("externals" in ud.parm and ud.parm["externals"] == "nowarn"): # Warn the user if this had externals (won't catch them all) output = runfetchcmd("svn propget svn:externals || true", d, workdir=ud.moddir) if output: - if "--ignore-externals" in svnfetchcmd.split(): + if "--ignore-externals" in svncmd.split(): bb.warn("%s contains svn:externals." % ud.url) bb.warn("These should be added to the recipe SRC_URI as necessary.") bb.warn("svn fetch has ignored externals:\n%s" % output) -- 2.25.0 From inigovicentewalino at gmail.com Fri Feb 21 11:23:55 2020 From: inigovicentewalino at gmail.com (=?UTF-8?Q?I=C3=B1igo_Vicente_Wali=C3=B1o?=) Date: Fri, 21 Feb 2020 12:23:55 +0100 Subject: [bitbake-devel] bitbake error Message-ID: Hi, I'm trying to build yocto project ZEUS for imx8m mini. I have this configuration EXTRA_IMAGE_FEATURES= ? ssh-server-openssh tools-sdk tools-debug dbg-pkgs dev-pkgs doc-pkgs package-management tools-profile? in my build/conf/local.con file. But it reports error when executing *bitbake core-image-base*. Host environment: Ubuntu 18.04.3 LTS Can any one explain how to resolve this? The error log is below. Thanks, best regards I?igo. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- bitbake core-image-base Loading cache: 100% |######################################################################| Time: 0:00:02 Loaded 3447 entries from dependency cache. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "1.44.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "aarch64-fslc-linux" MACHINE = "imx8mmevk" DISTRO = "fslc-xwayland" DISTRO_VERSION = "3.0" TUNE_FEATURES = "aarch64 cortexa53 crc crypto" TARGET_FPU = "" meta meta-poky = "HEAD:c204cc47d0a319d0435c4a9ba06e241c2e75ff1e" meta-oe meta-multimedia meta-python = "HEAD:bb65c27a772723dfe2c15b5e1b27bcc1a1ed884c" meta-freescale = "HEAD:3a3b13bef12c3a46da976fbf3b666310f8b694a7" meta-freescale-3rdparty = "HEAD:aea3771baa77e74762358ceb673d407e36637e5f" meta-freescale-distro = "HEAD:ca27d12e4964d1336e662bcc60184bbff526c857" Initialising tasks: 100% |#################################################################| Time: 0:00:06 Sstate summary: Wanted 1034 Found 0 Missed 1034 Current 764 (0% match, 42% complete) NOTE: Executing Tasks NOTE: Setscene tasks completed ERROR: perf-1.0-r9 do_compile: oe_runmake failed ERROR: perf-1.0-r9 do_compile: Execution of '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/temp/run.do_compile.13076' failed with exit code 1: make: Entering directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' BUILD: Doing 'make -j1' parallel build Warning: Kernel ABI header at 'tools/include/uapi/drm/drm.h' differs from latest version at 'include/uapi/drm/drm.h' Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h' Warning: Kernel ABI header at 'tools/arch/x86/include/asm/disabled-features.h' differs from latest version at 'arch/x86/include/asm/disabled-features.h' Warning: Kernel ABI header at 'tools/arch/x86/include/asm/required-features.h' differs from latest version at 'arch/x86/include/asm/required-features.h' Warning: Kernel ABI header at 'tools/arch/x86/include/asm/cpufeatures.h' differs from latest version at 'arch/x86/include/asm/cpufeatures.h' Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm/include/uapi/asm/perf_regs.h' Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm64/include/uapi/asm/perf_regs.h' Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/powerpc/include/uapi/asm/perf_regs.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/x86/include/uapi/asm/perf_regs.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm_perf.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/svm.h' differs from latest version at 'arch/x86/include/uapi/asm/svm.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/unistd.h' differs from latest version at 'arch/x86/include/uapi/asm/unistd.h' Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/vmx.h' differs from latest version at 'arch/x86/include/uapi/asm/vmx.h' Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/kvm.h' differs from latest version at 'arch/powerpc/include/uapi/asm/kvm.h' Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm.h' Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm_perf.h' Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/sie.h' differs from latest version at 'arch/s390/include/uapi/asm/sie.h' Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm/include/uapi/asm/kvm.h' Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm64/include/uapi/asm/kvm.h' Warning: Kernel ABI header at 'tools/arch/x86/lib/memcpy_64.S' differs from latest version at 'arch/x86/lib/memcpy_64.S' Warning: Kernel ABI header at 'tools/arch/x86/lib/memset_64.S' differs from latest version at 'arch/x86/lib/memset_64.S' Auto-detecting system features: ... dwarf: [ on ] ... dwarf_getlocations: [ on ] ... glibc: [ on ] ... gtk2: [ OFF ] ... libaudit: [ OFF ] ... libbfd: [ on ] ... libelf: [ on ] ... libnuma: [ OFF ] ... numa_num_possible_cpus: [ OFF ] ... libperl: [ OFF ] ... libpython: [ on ] ... libslang: [ on ] ... libcrypto: [ on ] ... libunwind: [ on ] ... libdw-dwarf-unwind: [ on ] ... zlib: [ on ] ... lzma: [ on ] ... get_cpuid: [ OFF ] ... bpf: [ on ] Makefile.config:391: DWARF support is off, BPF prologue is disabled Makefile.config:588: Missing perl devel files. Disabling perl scripting support, please install perl-ExtUtils-Embed/libperl-dev Makefile.config:626: Python 3 is not yet supported; please set Makefile.config:627: PYTHON and/or PYTHON_CONFIG appropriately. Makefile.config:628: If you also have Python 2 installed, then Makefile.config:629: try something like: Makefile.config:630: Makefile.config:631: make PYTHON=python2 Makefile.config:632: Makefile.config:633: Otherwise, disable Python support entirely: Makefile.config:634: Makefile.config:635: make NO_LIBPYTHON=1 Makefile.config:636: Makefile.config:637: *** . Stop. Makefile.perf:203: recipe for target 'sub-make' failed make[1]: *** [sub-make] Error 2 Makefile:69: recipe for target 'all' failed make: *** [all] Error 2 make: Leaving directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' WARNING: exit code 1 from a shell command. ERROR: Logfile of failure stored in: /home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/temp/log.do_compile.13076 Log data follows: | DEBUG: Executing shell function do_compile | NOTE: make -j 1 -C /home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf O=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0 CROSS_COMPILE=aarch64-fslc-linux- ARCH=arm64 CC=aarch64-fslc-linux-gcc -mcpu=cortex-a53+crc+crypto -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9=/usr/src/debug/perf/1.0-r9 -fdebug-prefix-map=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/recipe-sysroot= -fdebug-prefix-map=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/recipe-sysroot-native= --sysroot=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/recipe-sysroot AR=aarch64-fslc-linux-gcc-ar LD=aarch64-fslc-linux-ld --sysroot=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/recipe-sysroot EXTRA_CFLAGS=-ldw EXTRA_LDFLAGS= perfexecdir=/usr/libexec NO_GTK2=1 NO_LIBAUDIT=1 NO_DWARF=1 NO_JVMTI=1 NO_LIBNUMA=1 NO_SDT=1 TMPDIR=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0 LIBUNWIND_DIR=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/recipe-sysroot/usr DESTDIR=/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/image prefix=/usr bindir=/usr/bin sharedir=/usr/share sysconfdir=/etc perfexecdir=/usr/libexec/perf-core ETC_PERFCONFIG=../etc sharedir=share mandir=share/man infodir=share/info PYTHON=python3 PYTHON_CONFIG=python3-config all | make: Entering directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' | BUILD: Doing 'make -j1' parallel build | Warning: Kernel ABI header at 'tools/include/uapi/drm/drm.h' differs from latest version at 'include/uapi/drm/drm.h' | Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/disabled-features.h' differs from latest version at 'arch/x86/include/asm/disabled-features.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/required-features.h' differs from latest version at 'arch/x86/include/asm/required-features.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/cpufeatures.h' differs from latest version at 'arch/x86/include/asm/cpufeatures.h' | Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm64/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/powerpc/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/x86/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm_perf.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/svm.h' differs from latest version at 'arch/x86/include/uapi/asm/svm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/unistd.h' differs from latest version at 'arch/x86/include/uapi/asm/unistd.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/vmx.h' differs from latest version at 'arch/x86/include/uapi/asm/vmx.h' | Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/kvm.h' differs from latest version at 'arch/powerpc/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm_perf.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/sie.h' differs from latest version at 'arch/s390/include/uapi/asm/sie.h' | Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm64/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/lib/memcpy_64.S' differs from latest version at 'arch/x86/lib/memcpy_64.S' | Warning: Kernel ABI header at 'tools/arch/x86/lib/memset_64.S' differs from latest version at 'arch/x86/lib/memset_64.S' | | Auto-detecting system features: | ... dwarf: [ on ] | ... dwarf_getlocations: [ on ] | ... glibc: [ on ] | ... gtk2: [ OFF ] | ... libaudit: [ OFF ] | ... libbfd: [ on ] | ... libelf: [ on ] | ... libnuma: [ OFF ] | ... numa_num_possible_cpus: [ OFF ] | ... libperl: [ OFF ] | ... libpython: [ on ] | ... libslang: [ on ] | ... libcrypto: [ on ] | ... libunwind: [ on ] | ... libdw-dwarf-unwind: [ on ] | ... zlib: [ on ] | ... lzma: [ on ] | ... get_cpuid: [ OFF ] | ... bpf: [ on ] | | Makefile.config:391: DWARF support is off, BPF prologue is disabled | Makefile.config:588: Missing perl devel files. Disabling perl scripting support, please install perl-ExtUtils-Embed/libperl-dev | Makefile.config:626: Python 3 is not yet supported; please set | Makefile.config:627: PYTHON and/or PYTHON_CONFIG appropriately. | Makefile.config:628: If you also have Python 2 installed, then | Makefile.config:629: try something like: | Makefile.config:630: | Makefile.config:631: make PYTHON=python2 | Makefile.config:632: | Makefile.config:633: Otherwise, disable Python support entirely: | Makefile.config:634: | Makefile.config:635: make NO_LIBPYTHON=1 | Makefile.config:636: | Makefile.config:637: *** . Stop. | Makefile.perf:203: recipe for target 'sub-make' failed | make[1]: *** [sub-make] Error 2 | ERROR: oe_runmake failed | Makefile:69: recipe for target 'all' failed | make: *** [all] Error 2 | make: Leaving directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' | WARNING: exit code 1 from a shell command. | ERROR: Execution of '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/temp/run.do_compile.13076' failed with exit code 1: | make: Entering directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' | BUILD: Doing 'make -j1' parallel build | Warning: Kernel ABI header at 'tools/include/uapi/drm/drm.h' differs from latest version at 'include/uapi/drm/drm.h' | Warning: Kernel ABI header at 'tools/include/uapi/linux/kvm.h' differs from latest version at 'include/uapi/linux/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/disabled-features.h' differs from latest version at 'arch/x86/include/asm/disabled-features.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/required-features.h' differs from latest version at 'arch/x86/include/asm/required-features.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/asm/cpufeatures.h' differs from latest version at 'arch/x86/include/asm/cpufeatures.h' | Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/arm64/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/powerpc/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/perf_regs.h' differs from latest version at 'arch/x86/include/uapi/asm/perf_regs.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/x86/include/uapi/asm/kvm_perf.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/svm.h' differs from latest version at 'arch/x86/include/uapi/asm/svm.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/unistd.h' differs from latest version at 'arch/x86/include/uapi/asm/unistd.h' | Warning: Kernel ABI header at 'tools/arch/x86/include/uapi/asm/vmx.h' differs from latest version at 'arch/x86/include/uapi/asm/vmx.h' | Warning: Kernel ABI header at 'tools/arch/powerpc/include/uapi/asm/kvm.h' differs from latest version at 'arch/powerpc/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/kvm_perf.h' differs from latest version at 'arch/s390/include/uapi/asm/kvm_perf.h' | Warning: Kernel ABI header at 'tools/arch/s390/include/uapi/asm/sie.h' differs from latest version at 'arch/s390/include/uapi/asm/sie.h' | Warning: Kernel ABI header at 'tools/arch/arm/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/arm64/include/uapi/asm/kvm.h' differs from latest version at 'arch/arm64/include/uapi/asm/kvm.h' | Warning: Kernel ABI header at 'tools/arch/x86/lib/memcpy_64.S' differs from latest version at 'arch/x86/lib/memcpy_64.S' | Warning: Kernel ABI header at 'tools/arch/x86/lib/memset_64.S' differs from latest version at 'arch/x86/lib/memset_64.S' | | Auto-detecting system features: | ... dwarf: [ on ] | ... dwarf_getlocations: [ on ] | ... glibc: [ on ] | ... gtk2: [ OFF ] | ... libaudit: [ OFF ] | ... libbfd: [ on ] | ... libelf: [ on ] | ... libnuma: [ OFF ] | ... numa_num_possible_cpus: [ OFF ] | ... libperl: [ OFF ] | ... libpython: [ on ] | ... libslang: [ on ] | ... libcrypto: [ on ] | ... libunwind: [ on ] | ... libdw-dwarf-unwind: [ on ] | ... zlib: [ on ] | ... lzma: [ on ] | ... get_cpuid: [ OFF ] | ... bpf: [ on ] | | Makefile.config:391: DWARF support is off, BPF prologue is disabled | Makefile.config:588: Missing perl devel files. Disabling perl scripting support, please install perl-ExtUtils-Embed/libperl-dev | Makefile.config:626: Python 3 is not yet supported; please set | Makefile.config:627: PYTHON and/or PYTHON_CONFIG appropriately. | Makefile.config:628: If you also have Python 2 installed, then | Makefile.config:629: try something like: | Makefile.config:630: | Makefile.config:631: make PYTHON=python2 | Makefile.config:632: | Makefile.config:633: Otherwise, disable Python support entirely: | Makefile.config:634: | Makefile.config:635: make NO_LIBPYTHON=1 | Makefile.config:636: | Makefile.config:637: *** . Stop. | Makefile.perf:203: recipe for target 'sub-make' failed | make[1]: *** [sub-make] Error 2 | Makefile:69: recipe for target 'all' failed | make: *** [all] Error 2 | make: Leaving directory '/home/iv/imx/fsl-community-bsp/build/tmp/work/imx8mmevk-fslc-linux/perf/1.0-r9/perf-1.0/tools/perf' | WARNING: exit code 1 from a shell command. | ERROR: Task (/home/iv/imx/fsl-community-bsp/sources/poky/meta/recipes-kernel/perf/perf.bb:do_compile) failed with exit code '1' NOTE: Tasks Summary: Attempted 3274 tasks of which 3271 didn't need to be rerun and 1 failed. NOTE: Writing buildhistory NOTE: Writing buildhistory took: 2 seconds Summary: 1 task failed: /home/iv/imx/fsl-community-bsp/sources/poky/meta/recipes-kernel/perf/perf.bb:do_compile Summary: There were 2 ERROR messages shown, returning a non-zero exit code. From raj.khem at gmail.com Fri Feb 21 19:53:48 2020 From: raj.khem at gmail.com (Khem Raj) Date: Fri, 21 Feb 2020 11:53:48 -0800 Subject: [bitbake-devel] bitbake error In-Reply-To: References: Message-ID: On Fri, Feb 21, 2020 at 3:54 AM I?igo Vicente Wali?o wrote: > > Hi, > > I'm trying to build yocto project ZEUS for imx8m mini. > > I have this configuration EXTRA_IMAGE_FEATURES= ? ssh-server-openssh tools-sdk tools-debug dbg-pkgs dev-pkgs doc-pkgs package-management tools-profile? in my build/conf/local.con file. > > But it reports error when executing bitbake core-image-base. > Host environment: Ubuntu 18.04.3 LTS > > Can any one explain how to resolve this? > perf uses kernel sources to build and its perhaps related to BSP kernel you are using to build the image that it might need certain tweaks to compile perf, it seems it asking for using python2 so perhaps you can inherit pythonnative in the perf recipe and that might help > The error log is below. > > > Thanks, best regards > > I?igo. > > > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel at lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel From bunk at stusta.de Fri Feb 21 20:52:13 2020 From: bunk at stusta.de (Adrian Bunk) Date: Fri, 21 Feb 2020 22:52:13 +0200 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console Message-ID: <20200221205213.16130-1-bunk@stusta.de> Signed-off-by: Adrian Bunk --- bitbake/lib/bb/runqueue.py | 8 ++++---- bitbake/lib/bb/siggen.py | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 71108eeed7..f6bd8db8ce 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -2054,7 +2054,7 @@ class RunQueueExecute: self.update_holdofftasks() if not self.sq_live and not self.sqdone and not self.sq_deferred and not self.updated_taskhash_queue and not self.holdoff_tasks: - logger.info("Setscene tasks completed") + logger.debug(1, "Setscene tasks completed") err = self.summarise_scenequeue_errors() if err: @@ -2259,7 +2259,7 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - logger.info("Task %s unihash changed to %s" % (tid, unihash)) + logger.debug(1, "Task %s unihash changed to %s" % (tid, unihash)) self.rqdata.runtaskentries[tid].unihash = unihash bb.parse.siggen.set_unihash(tid, unihash) toprocess.add(tid) @@ -2304,7 +2304,7 @@ class RunQueueExecute: elif tid in self.scenequeue_covered or tid in self.sq_live: # Already ran this setscene task or it running. Report the new taskhash bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) - logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + logger.debug(1, "Already covered setscene for %s so ignoring rehash (remap)" % (tid)) remapped = True if not remapped: @@ -2405,7 +2405,7 @@ class RunQueueExecute: for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: - logger.info("Setscene task %s became valid" % tid) + logger.debug(1, "Setscene task %s became valid" % tid) if harddepfail: self.sq_task_failoutright(tid) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index c2d0c736cf..d5c5c5c0af 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -606,7 +606,7 @@ class SignatureGeneratorUniHashMixIn(object): method = method + self.extramethod[tid] data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) - bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + bb.debug(1, 'Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None: bb.warn("Server unable to handle unihash report") @@ -615,9 +615,9 @@ class SignatureGeneratorUniHashMixIn(object): finalunihash = data['unihash'] if finalunihash == current_unihash: - bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + bb.debug(1, 'Task %s unihash %s unchanged by server' % (tid, finalunihash)) elif finalunihash == wanted_unihash: - bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + bb.debug(1, 'Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) self.set_unihash(tid, finalunihash) return True else: -- 2.17.1 From daisuke.yamane at cybertrust.co.jp Sat Feb 22 06:17:14 2020 From: daisuke.yamane at cybertrust.co.jp (Daisuke Yamane) Date: Sat, 22 Feb 2020 06:17:14 +0000 Subject: [bitbake-devel] [PATCH] lib/bb/utils.py: Preserve ownership of symlink Message-ID: <20200222061714.15922-1-daisuke.yamane@cybertrust.co.jp> Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. Signed-off-by: Daisuke Yamane --- bitbake/lib/bb/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index 47805d02cf..5f5767c1da 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -864,7 +864,7 @@ def copyfile(src, dest, newmtime = None, sstat = None): if destexists and not stat.S_ISDIR(dstat[stat.ST_MODE]): os.unlink(dest) os.symlink(target, dest) - #os.lchown(dest,sstat[stat.ST_UID],sstat[stat.ST_GID]) + os.lchown(dest,sstat[stat.ST_UID],sstat[stat.ST_GID]) return os.lstat(dest) except Exception as e: logger.warning("copyfile: failed to create symlink %s to %s (%s)" % (dest, target, e)) -- 2.17.1 From schnitzeltony at gmail.com Sat Feb 22 11:57:47 2020 From: schnitzeltony at gmail.com (=?UTF-8?Q?Andreas_M=C3=BCller?=) Date: Sat, 22 Feb 2020 12:57:47 +0100 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <20200221205213.16130-1-bunk@stusta.de> References: <20200221205213.16130-1-bunk@stusta.de> Message-ID: On Fri, Feb 21, 2020 at 10:00 PM Adrian Bunk wrote: > > Signed-off-by: Adrian Bunk > --- > bitbake/lib/bb/runqueue.py | 8 ++++---- > bitbake/lib/bb/siggen.py | 6 +++--- > 2 files changed, 7 insertions(+), 7 deletions(-) > > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py > index 71108eeed7..f6bd8db8ce 100644 > --- a/bitbake/lib/bb/runqueue.py > +++ b/bitbake/lib/bb/runqueue.py > @@ -2054,7 +2054,7 @@ class RunQueueExecute: > self.update_holdofftasks() > > if not self.sq_live and not self.sqdone and not self.sq_deferred and not self.updated_taskhash_queue and not self.holdoff_tasks: > - logger.info("Setscene tasks completed") > + logger.debug(1, "Setscene tasks completed") ^ Could you please leave this one - I consider it helpful > > err = self.summarise_scenequeue_errors() > if err: > @@ -2259,7 +2259,7 @@ class RunQueueExecute: > self.updated_taskhash_queue.remove((tid, unihash)) > > if unihash != self.rqdata.runtaskentries[tid].unihash: > - logger.info("Task %s unihash changed to %s" % (tid, unihash)) > + logger.debug(1, "Task %s unihash changed to %s" % (tid, unihash)) > self.rqdata.runtaskentries[tid].unihash = unihash > bb.parse.siggen.set_unihash(tid, unihash) > toprocess.add(tid) > @@ -2304,7 +2304,7 @@ class RunQueueExecute: > elif tid in self.scenequeue_covered or tid in self.sq_live: > # Already ran this setscene task or it running. Report the new taskhash > bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) > - logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) > + logger.debug(1, "Already covered setscene for %s so ignoring rehash (remap)" % (tid)) > remapped = True > > if not remapped: > @@ -2405,7 +2405,7 @@ class RunQueueExecute: > > for (tid, harddepfail, origvalid) in update_tasks: > if tid in self.sqdata.valid and not origvalid: > - logger.info("Setscene task %s became valid" % tid) > + logger.debug(1, "Setscene task %s became valid" % tid) > if harddepfail: > self.sq_task_failoutright(tid) > > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py > index c2d0c736cf..d5c5c5c0af 100644 > --- a/bitbake/lib/bb/siggen.py > +++ b/bitbake/lib/bb/siggen.py > @@ -606,7 +606,7 @@ class SignatureGeneratorUniHashMixIn(object): > method = method + self.extramethod[tid] > > data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) > - bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) > + bb.debug(1, 'Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) > > if data is None: > bb.warn("Server unable to handle unihash report") > @@ -615,9 +615,9 @@ class SignatureGeneratorUniHashMixIn(object): > finalunihash = data['unihash'] > > if finalunihash == current_unihash: > - bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash)) > + bb.debug(1, 'Task %s unihash %s unchanged by server' % (tid, finalunihash)) > elif finalunihash == wanted_unihash: > - bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) > + bb.debug(1, 'Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) > self.set_unihash(tid, finalunihash) > return True > else: > -- > 2.17.1 > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel at lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel From richard.purdie at linuxfoundation.org Sat Feb 22 15:54:54 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sat, 22 Feb 2020 15:54:54 +0000 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <20200221205213.16130-1-bunk@stusta.de> References: <20200221205213.16130-1-bunk@stusta.de> Message-ID: On Fri, 2020-02-21 at 22:52 +0200, Adrian Bunk wrote: > Signed-off-by: Adrian Bunk > --- > bitbake/lib/bb/runqueue.py | 8 ++++---- > bitbake/lib/bb/siggen.py | 6 +++--- > 2 files changed, 7 insertions(+), 7 deletions(-) The trouble here is that if we remove these messages (and I can understand why people want to), if there is a problem with hashequiv, I can simply look at the result and say "yes, it broke" with zero ability to understand why as we can't tell what happened in the build. The correct fix is probably to change the console log messages to log differently to the on disk log file. Even then, it means we can't debug autobuilder failures unless we save those logs. So in summary, there is a choice: a) We can compel me to merge patches like this. I will then just have to close all hashequiv bug reports as WORKSFORME. b) We actually fix/improve the way logging works. I'd prefer b). Cheers, Richard From bunk at stusta.de Sat Feb 22 22:05:18 2020 From: bunk at stusta.de (Adrian Bunk) Date: Sun, 23 Feb 2020 00:05:18 +0200 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: References: <20200221205213.16130-1-bunk@stusta.de> Message-ID: <20200222220518.GD9628@localhost> On Sat, Feb 22, 2020 at 12:57:47PM +0100, Andreas M?ller wrote: > On Fri, Feb 21, 2020 at 10:00 PM Adrian Bunk wrote: >... > > --- a/bitbake/lib/bb/runqueue.py > > +++ b/bitbake/lib/bb/runqueue.py > > @@ -2054,7 +2054,7 @@ class RunQueueExecute: > > self.update_holdofftasks() > > > > if not self.sq_live and not self.sqdone and not self.sq_deferred and not self.updated_taskhash_queue and not self.holdoff_tasks: > > - logger.info("Setscene tasks completed") > > + logger.debug(1, "Setscene tasks completed") > ^ Could you please leave this one - I consider it helpful >... It would be helpful if it would be shown only once. I've seen it 10 times during a build where "Setscene task .. became valid" and "Already covered setscene for ..." also happened a few times. cu Adrian From bunk at stusta.de Sun Feb 23 19:51:48 2020 From: bunk at stusta.de (Adrian Bunk) Date: Sun, 23 Feb 2020 21:51:48 +0200 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: References: <20200221205213.16130-1-bunk@stusta.de> Message-ID: <20200223195148.GB3444@localhost> On Sat, Feb 22, 2020 at 03:54:54PM +0000, Richard Purdie wrote: >... > b) We actually fix/improve the way logging works. > > I'd prefer b). Makes sense, I've submitted this as bug #13813. > Cheers, > > Richard cu Adrian From richard.purdie at linuxfoundation.org Sun Feb 23 23:53:57 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sun, 23 Feb 2020 23:53:57 +0000 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <20200223195148.GB3444@localhost> References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> Message-ID: <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> On Sun, 2020-02-23 at 21:51 +0200, Adrian Bunk wrote: > On Sat, Feb 22, 2020 at 03:54:54PM +0000, Richard Purdie wrote: > > ... > > b) We actually fix/improve the way logging works. > > > > I'd prefer b). > > Makes sense, I've submitted this as bug #13813. I note you believe this has to be fixed before we reach M3. I guess I'd agree, I just don't know who is going to do it. Bitbake bugs get auto assigned to me. I physically cannot actually keep up with the incoming bugs despite working more than I should. I was actually trying to do something about my bug count as its seriously depressing me. For each one I try and fix I seem to find three more intermittent issues on the autobuilder :(. I know you didn't mean it this way and we need to track the issues, I just don't know how we're going to handle it. Cheers, Richard From bunk at stusta.de Mon Feb 24 05:49:45 2020 From: bunk at stusta.de (Adrian Bunk) Date: Mon, 24 Feb 2020 07:49:45 +0200 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> Message-ID: <20200224054945.GB6683@localhost> On Sun, Feb 23, 2020 at 11:53:57PM +0000, Richard Purdie wrote: > On Sun, 2020-02-23 at 21:51 +0200, Adrian Bunk wrote: > > On Sat, Feb 22, 2020 at 03:54:54PM +0000, Richard Purdie wrote: > > > ... > > > b) We actually fix/improve the way logging works. > > > > > > I'd prefer b). > > > > Makes sense, I've submitted this as bug #13813. > > I note you believe this has to be fixed before we reach M3. I guess I'd > agree, I just don't know who is going to do it. I would have said M4, but that doesn't really change anything. > Bitbake bugs get auto assigned to me. I physically cannot actually keep > up with the incoming bugs despite working more than I should. > > I was actually trying to do something about my bug count as its > seriously depressing me. For each one I try and fix I seem to find > three more intermittent issues on the autobuilder :(. I know this might sound brutal, but in hindsight the hash equivalence work was not a good idea with the current resourcing. I fully understand why it is attractive to properly fix a long-standing problem, but the hash equivalence work and followup fixes might have been the difference between an OKish workload and working more than you should have done. > I know you didn't mean it this way and we need to track the issues, I > just don't know how we're going to handle it. >From my side the problem is that my initial patch is what I can do with reasonable effort. > Cheers, > > Richard cu Adrian From richard.purdie at linuxfoundation.org Mon Feb 24 09:12:06 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 24 Feb 2020 09:12:06 +0000 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <20200224054945.GB6683@localhost> References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> <20200224054945.GB6683@localhost> Message-ID: On Mon, 2020-02-24 at 07:49 +0200, Adrian Bunk wrote: > On Sun, Feb 23, 2020 at 11:53:57PM +0000, Richard Purdie wrote: > > On Sun, 2020-02-23 at 21:51 +0200, Adrian Bunk wrote: > > > On Sat, Feb 22, 2020 at 03:54:54PM +0000, Richard Purdie wrote: > > > > ... > > > > b) We actually fix/improve the way logging works. > > > > > > > > I'd prefer b). > > > > > > Makes sense, I've submitted this as bug #13813. > > > > I note you believe this has to be fixed before we reach M3. I guess > > I'd > > agree, I just don't know who is going to do it. > > I would have said M4, but that doesn't really change anything. Technically its feature work although I guess it can be argued both ways as its also a usability/bug fix. > > Bitbake bugs get auto assigned to me. I physically cannot actually > > keep > > up with the incoming bugs despite working more than I should. > > > > I was actually trying to do something about my bug count as its > > seriously depressing me. For each one I try and fix I seem to find > > three more intermittent issues on the autobuilder :(. > > I know this might sound brutal, but in hindsight the hash > equivalence work was not a good idea with the current resourcing. Not brutal, in hindsight it hasn't been such a good idea. > I fully understand why it is attractive to properly fix a long- > standing > problem, but the hash equivalence work and followup fixes might have > been the difference between an OKish workload and working more than > you should have done. Agreed, and now why I kind of feel guilty if I don't resolve some of the issues. We could just turn it off again I guess, I keep wondering about that. > > I know you didn't mean it this way and we need to track the issues, > > I > > just don't know how we're going to handle it. > > From my side the problem is that my initial patch is what I can do > with reasonable effort. Part of the reason I'm highlighting this is that I have a lot of feedback that says people don't believe there is any problem with our resourcing. Unless I can show where there are issues, its not going to change :/. Cheers, Richard From chris.laplante at agilent.com Mon Feb 24 15:44:01 2020 From: chris.laplante at agilent.com (Chris Laplante) Date: Mon, 24 Feb 2020 10:44:01 -0500 Subject: [bitbake-devel] [PATCH] runqueue: teach runonly/runall to accept "do_task" as well as "task" Message-ID: <1582559041-78575-1-git-send-email-chris.laplante@agilent.com> Previously --runonly=do_task would give a misleading error like: ERROR: Could not find any tasks with the tasknames ['do_task'] to run within the recipes of the taskgraphs of the targets... The problem is that BitBake tried to find "do_do_task". So teach it to only add the do_ prefix if it's not already there. Signed-off-by: Chris Laplante --- lib/bb/runqueue.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 71108ee..32966b4 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -920,9 +920,11 @@ class RunQueueData: runq_build = {} for task in self.cooker.configuration.runall: + if not task.startswith("do_"): + task = "do_{0}".format(task) runall_tids = set() for tid in list(self.runtaskentries): - wanttid = fn_from_tid(tid) + ":do_%s" % task + wanttid = "{0}:{1}".format(fn_from_tid(tid), task) if wanttid in delcount: self.runtaskentries[wanttid] = delcount[wanttid] if wanttid in self.runtaskentries: @@ -949,7 +951,9 @@ class RunQueueData: runq_build = {} for task in self.cooker.configuration.runonly: - runonly_tids = { k: v for k, v in self.runtaskentries.items() if taskname_from_tid(k) == "do_%s" % task } + if not task.startswith("do_"): + task = "do_{0}".format(task) + runonly_tids = { k: v for k, v in self.runtaskentries.items() if taskname_from_tid(k) == task } for tid in list(runonly_tids): mark_active(tid,1) -- 2.7.4 From pbarker at konsulko.com Mon Feb 24 20:16:29 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 24 Feb 2020 20:16:29 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs Message-ID: <20200224201629.13774-1-pbarker@konsulko.com> When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked so that the .gitmodules file can be examined. Signed-off-by: Paul Barker --- lib/bb/fetch2/gitsm.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index aa121cbe..98f5dae5 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -169,6 +169,12 @@ class GitSM(Git): raise Git.download(self, ud, d) + + # If we're using a shallow mirror tarball it needs to be unpacked so + # that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): + bb.utils.mkdirhier(ud.clonedir) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) self.process_submodules(ud, ud.clonedir, download_submodule, d) def unpack(self, ud, destdir, d): -- 2.20.1 From richard.purdie at linuxfoundation.org Mon Feb 24 21:32:33 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 24 Feb 2020 21:32:33 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: <20200224201629.13774-1-pbarker@konsulko.com> References: <20200224201629.13774-1-pbarker@konsulko.com> Message-ID: On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote: > When a shallow mirror tarball is used to satisfy a gitsm URI it needs to > be unpacked so that the .gitmodules file can be examined. > > Signed-off-by: Paul Barker > --- > lib/bb/fetch2/gitsm.py | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > index aa121cbe..98f5dae5 100644 > --- a/lib/bb/fetch2/gitsm.py > +++ b/lib/bb/fetch2/gitsm.py > @@ -169,6 +169,12 @@ class GitSM(Git): > raise > > Git.download(self, ud, d) > + > + # If we're using a shallow mirror tarball it needs to be unpacked so > + # that we can examine the .gitmodules file > + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): > + bb.utils.mkdirhier(ud.clonedir) > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) > self.process_submodules(ud, ud.clonedir, download_submodule, d) > > def unpack(self, ud, destdir, d): Does this mean we're missing a test? Cheers, Richard From pbarker at konsulko.com Mon Feb 24 21:46:31 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 24 Feb 2020 21:46:31 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com> Message-ID: On Mon, 24 Feb 2020 at 21:32, Richard Purdie wrote: > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote: > > When a shallow mirror tarball is used to satisfy a gitsm URI it needs to > > be unpacked so that the .gitmodules file can be examined. > > > > Signed-off-by: Paul Barker > > --- > > lib/bb/fetch2/gitsm.py | 6 ++++++ > > 1 file changed, 6 insertions(+) > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > index aa121cbe..98f5dae5 100644 > > --- a/lib/bb/fetch2/gitsm.py > > +++ b/lib/bb/fetch2/gitsm.py > > @@ -169,6 +169,12 @@ class GitSM(Git): > > raise > > > > Git.download(self, ud, d) > > + > > + # If we're using a shallow mirror tarball it needs to be unpacked so > > + # that we can examine the .gitmodules file > > + if ud.shallow and os.path.exists(ud.fullshallow) and Git.need_update(self, ud, d): > > + bb.utils.mkdirhier(ud.clonedir) > > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=ud.clonedir) > > self.process_submodules(ud, ud.clonedir, download_submodule, d) > > > > def unpack(self, ud, destdir, d): > > Does this mean we're missing a test? Possibly. Maybe a two step test with a recipe using a `gitsm://` URL: 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror tarballs out of the downloads directory to a local mirror directory. Throw away DL_DIR, SSTATE_DIR and TMPDIR. 2) Using the mirror generated in step 1 try to fetch and unpack the recipe. Does that sound like a sensible test? I think it would have to go in oeqa rather than the bitbake tests if we do it that way. From richard.purdie at linuxfoundation.org Mon Feb 24 22:09:57 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Mon, 24 Feb 2020 22:09:57 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com> Message-ID: On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote: > On Mon, 24 Feb 2020 at 21:32, Richard Purdie > wrote: > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote: > > > When a shallow mirror tarball is used to satisfy a gitsm URI it > > > needs to > > > be unpacked so that the .gitmodules file can be examined. > > > > > > Signed-off-by: Paul Barker > > > --- > > > lib/bb/fetch2/gitsm.py | 6 ++++++ > > > 1 file changed, 6 insertions(+) > > > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > > index aa121cbe..98f5dae5 100644 > > > --- a/lib/bb/fetch2/gitsm.py > > > +++ b/lib/bb/fetch2/gitsm.py > > > @@ -169,6 +169,12 @@ class GitSM(Git): > > > raise > > > > > > Git.download(self, ud, d) > > > + > > > + # If we're using a shallow mirror tarball it needs to be > > > unpacked so > > > + # that we can examine the .gitmodules file > > > + if ud.shallow and os.path.exists(ud.fullshallow) and > > > Git.need_update(self, ud, d): > > > + bb.utils.mkdirhier(ud.clonedir) > > > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, > > > workdir=ud.clonedir) > > > self.process_submodules(ud, ud.clonedir, > > > download_submodule, d) > > > > > > def unpack(self, ud, destdir, d): > > > > Does this mean we're missing a test? > > Possibly. Maybe a two step test with a recipe using a `gitsm://` URL: > 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, > BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror > tarballs out of the downloads directory to a local mirror directory. > Throw away DL_DIR, SSTATE_DIR and TMPDIR. > 2) Using the mirror generated in step 1 try to fetch and unpack the > recipe. > > Does that sound like a sensible test? I think it would have to go in > oeqa rather than the bitbake tests if we do it that way. It sounds like it might be reasonable, I'm not sure why we can't do that in lib/bb/tests/fetch.py though? There are other tests which do similar sequences of commands. You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a datastore and files in between but I can't see why it would need to be in oeqa? Sorry to push testing btw but the git fetcher is one of our better tested pieces of code so I do want to try and maintain good test coverage! Cheers, Richard From pbarker at konsulko.com Mon Feb 24 22:29:34 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 24 Feb 2020 22:29:34 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com> Message-ID: On Mon, 24 Feb 2020 at 22:10, Richard Purdie wrote: > > On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote: > > On Mon, 24 Feb 2020 at 21:32, Richard Purdie > > wrote: > > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote: > > > > When a shallow mirror tarball is used to satisfy a gitsm URI it > > > > needs to > > > > be unpacked so that the .gitmodules file can be examined. > > > > > > > > Signed-off-by: Paul Barker > > > > --- > > > > lib/bb/fetch2/gitsm.py | 6 ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > > > index aa121cbe..98f5dae5 100644 > > > > --- a/lib/bb/fetch2/gitsm.py > > > > +++ b/lib/bb/fetch2/gitsm.py > > > > @@ -169,6 +169,12 @@ class GitSM(Git): > > > > raise > > > > > > > > Git.download(self, ud, d) > > > > + > > > > + # If we're using a shallow mirror tarball it needs to be > > > > unpacked so > > > > + # that we can examine the .gitmodules file > > > > + if ud.shallow and os.path.exists(ud.fullshallow) and > > > > Git.need_update(self, ud, d): > > > > + bb.utils.mkdirhier(ud.clonedir) > > > > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, > > > > workdir=ud.clonedir) > > > > self.process_submodules(ud, ud.clonedir, > > > > download_submodule, d) > > > > > > > > def unpack(self, ud, destdir, d): > > > > > > Does this mean we're missing a test? > > > > Possibly. Maybe a two step test with a recipe using a `gitsm://` URL: > > 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, > > BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror > > tarballs out of the downloads directory to a local mirror directory. > > Throw away DL_DIR, SSTATE_DIR and TMPDIR. > > 2) Using the mirror generated in step 1 try to fetch and unpack the > > recipe. > > > > Does that sound like a sensible test? I think it would have to go in > > oeqa rather than the bitbake tests if we do it that way. > > It sounds like it might be reasonable, I'm not sure why we can't do > that in lib/bb/tests/fetch.py though? There are other tests which do > similar sequences of commands. > > You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a > datastore and files in between but I can't see why it would need to be > in oeqa? Ok, I've looked more closely at GitMakeShallowTest in lib/bb/tests/fetch.py. We could write a new test_shallow_submodule_mirrors function based on test_shallow_submodules and test_shallow_mirrors. The case I would want to catch is do_unpack failing. I guess we could check that fetcher.unpack() works as a proxy for that. From kergoth at gmail.com Mon Feb 24 23:06:00 2020 From: kergoth at gmail.com (Christopher Larson) Date: Mon, 24 Feb 2020 16:06:00 -0700 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com>

Message-ID: I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary. On Mon, Feb 24, 2020 at 3:29 PM Paul Barker wrote: > On Mon, 24 Feb 2020 at 22:10, Richard Purdie > wrote: > > > > On Mon, 2020-02-24 at 21:46 +0000, Paul Barker wrote: > > > On Mon, 24 Feb 2020 at 21:32, Richard Purdie > > > wrote: > > > > On Mon, 2020-02-24 at 20:16 +0000, Paul Barker wrote: > > > > > When a shallow mirror tarball is used to satisfy a gitsm URI it > > > > > needs to > > > > > be unpacked so that the .gitmodules file can be examined. > > > > > > > > > > Signed-off-by: Paul Barker > > > > > --- > > > > > lib/bb/fetch2/gitsm.py | 6 ++++++ > > > > > 1 file changed, 6 insertions(+) > > > > > > > > > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > > > > > index aa121cbe..98f5dae5 100644 > > > > > --- a/lib/bb/fetch2/gitsm.py > > > > > +++ b/lib/bb/fetch2/gitsm.py > > > > > @@ -169,6 +169,12 @@ class GitSM(Git): > > > > > raise > > > > > > > > > > Git.download(self, ud, d) > > > > > + > > > > > + # If we're using a shallow mirror tarball it needs to be > > > > > unpacked so > > > > > + # that we can examine the .gitmodules file > > > > > + if ud.shallow and os.path.exists(ud.fullshallow) and > > > > > Git.need_update(self, ud, d): > > > > > + bb.utils.mkdirhier(ud.clonedir) > > > > > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, > > > > > workdir=ud.clonedir) > > > > > self.process_submodules(ud, ud.clonedir, > > > > > download_submodule, d) > > > > > > > > > > def unpack(self, ud, destdir, d): > > > > > > > > Does this mean we're missing a test? > > > > > > Possibly. Maybe a two step test with a recipe using a `gitsm://` URL: > > > 1) Fetch the recipe with BB_GENERATE_MIRROR_TARBALLS, > > > BB_GENERATE_SHALLOW_TARBALLS and BB_GIT_SHALLOW set. Copy the mirror > > > tarballs out of the downloads directory to a local mirror directory. > > > Throw away DL_DIR, SSTATE_DIR and TMPDIR. > > > 2) Using the mirror generated in step 1 try to fetch and unpack the > > > recipe. > > > > > > Does that sound like a sensible test? I think it would have to go in > > > oeqa rather than the bitbake tests if we do it that way. > > > > It sounds like it might be reasonable, I'm not sure why we can't do > > that in lib/bb/tests/fetch.py though? There are other tests which do > > similar sequences of commands. > > > > You'd do it with calls to the fetch and unpack Fetcerh APIs and alter a > > datastore and files in between but I can't see why it would need to be > > in oeqa? > > Ok, I've looked more closely at GitMakeShallowTest in > lib/bb/tests/fetch.py. We could write a new > test_shallow_submodule_mirrors function based on > test_shallow_submodules and test_shallow_mirrors. > > The case I would want to catch is do_unpack failing. I guess we could > check that fetcher.unpack() works as a proxy for that. > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel at lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > -- Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Senior Software Engineer, Mentor Graphics -------------- next part -------------- An HTML attachment was scrubbed... URL: From pbarker at konsulko.com Mon Feb 24 23:14:55 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 24 Feb 2020 23:14:55 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com>

Message-ID: On Mon, 24 Feb 2020 at 23:06, Christopher Larson wrote: > > I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary. Any suggestions on where we can unpack the shallow tarball then? We need it to be unpacked somewhere so that we can examine the .gitmodules file. From daisuke.yamane at cybertrust.co.jp Tue Feb 25 09:23:56 2020 From: daisuke.yamane at cybertrust.co.jp (Daisuke Yamane) Date: Tue, 25 Feb 2020 09:23:56 +0000 Subject: [bitbake-devel] [1.44, 1.42, 1.40] lib/bb/utils.py: Preserve ownership of symlink Message-ID: <20200225092356.898-1-daisuke.yamane@cybertrust.co.jp> Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. Signed-off-by: Daisuke Yamane Signed-off-by: Richard Purdie --- lib/bb/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bb/utils.py b/lib/bb/utils.py index 47805d02..5f5767c1 100644 --- a/lib/bb/utils.py +++ b/lib/bb/utils.py @@ -864,7 +864,7 @@ def copyfile(src, dest, newmtime = None, sstat = None): if destexists and not stat.S_ISDIR(dstat[stat.ST_MODE]): os.unlink(dest) os.symlink(target, dest) - #os.lchown(dest,sstat[stat.ST_UID],sstat[stat.ST_GID]) + os.lchown(dest,sstat[stat.ST_UID],sstat[stat.ST_GID]) return os.lstat(dest) except Exception as e: logger.warning("copyfile: failed to create symlink %s to %s (%s)" % (dest, target, e)) -- 2.17.1 From pbarker at konsulko.com Tue Feb 25 10:33:49 2020 From: pbarker at konsulko.com (Paul Barker) Date: Tue, 25 Feb 2020 10:33:49 +0000 Subject: [bitbake-devel] [PATCH] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: References: <20200224201629.13774-1-pbarker@konsulko.com>

Message-ID: On Mon, 24 Feb 2020 at 23:14, Paul Barker wrote: > > On Mon, 24 Feb 2020 at 23:06, Christopher Larson wrote: > > > > I don't think this is right. This unpacks the shallow tarball into ud.clonedir. The fetcher is specifically done the way it is to avoid ud.clonedir ever being shallow so as to avoid having to maintain that state and convert it when necessary. > > Any suggestions on where we can unpack the shallow tarball then? We > need it to be unpacked somewhere so that we can examine the > .gitmodules file. So my downloads directory blew up and I saw lots of fetch errors for git SRC_URIs on builds overnight after testing this patch. We could extract to a temporary directory as needed when downloading a gitsm shallow mirror tarball and then throw this away once we've checked .gitmodules. That would mean repeating the tarball extraction if the download method is executed again but that should be fairly rare. How does that sound? The alternative would be to create a new 'gitsmshallow' directory under DL_DIR for unpacking gitsm shallow mirror tarballs only. The names of directories under 'gitsmshallow' would need to include the commit hash given in the shallow tarball filename so that we don't have clashes when DL_DIR is shared between builds which use different SRCREVs of the same gitsm SRC_URI. I also have a test case ready to go which I'll include in v2 of this patch once we've decided where to extract the tarballs. Thanks, Paul From rpjday at crashcourse.ca Tue Feb 25 21:22:45 2020 From: rpjday at crashcourse.ca (rpjday at crashcourse.ca) Date: Tue, 25 Feb 2020 16:22:45 -0500 Subject: [bitbake-devel] is there a bitbake equivalent to "touch"ing a file for gnu make? Message-ID: <20200225162245.Horde.i6DmVIhAUjVbUGUQKtyMoua@crashcourse.ca> colleague emails me the following puzzler ? current project based on wind river linux 9 (so effectively morty 2.2), so it has bitbake 1.32. apparently, for a specific recipe, there are intermittent build errors which she suspects are caused by a race condition in the build of that recipe. *but* that build error only occurs when something about the underlying layers have changed; as in, someone checked *something* in somewhere forcing at least something to be rebuilt, and that error does not even occur on *all* rebuilds, just some of them. so the question is, which underlying recipe change and rebuild could be triggering this? she *suspects* it might be openssl, as their project is making regular changes to openssl, and that recipe seems to be an ultimate dependency for quite a number of recipes, so she's just guessing that might be the cause. in order to test this, she is currently running through a loop wherein, in order to kick off at least some recipe rebuilds, she makes some innocuous change to openssl, causing it to be rebuilt, which then rebuilds all recipes that depend on it and so on. yuck. on the other hand, i couldn't immediately suggest anything cleaner. my thought was to (somehow) fake a rebuild of openssl (perhaps by invalidating some timestamp), but that's not going to work as once it's rebuilt, it's just going to look like it did before, so nothing else will be rebuilt. is there an easier way to do this? i don't have a system with that old a YP version at hand, so i can't test this, and nothing leaps to mind. is there a "make touch" equivalent that would drive a full rebuild of all recipes that depend on a given recipe, without having to actually mess with it? i figured, "ah, this can't be that hard," but i haven't come up with anything. rday p.s. simply cleaning the offending recipe and rebuilding it won't trigger the build error (at least that's what I've been told) ? one must change the underlying recipes at least a little in order to *force* a rebuild allegedly based on dependencies. thoughts? From armccurdy at gmail.com Wed Feb 26 00:31:12 2020 From: armccurdy at gmail.com (Andre McCurdy) Date: Tue, 25 Feb 2020 16:31:12 -0800 Subject: [bitbake-devel] is there a bitbake equivalent to "touch"ing a file for gnu make? In-Reply-To: <20200225162245.Horde.i6DmVIhAUjVbUGUQKtyMoua@crashcourse.ca> References: <20200225162245.Horde.i6DmVIhAUjVbUGUQKtyMoua@crashcourse.ca> Message-ID: On Tue, Feb 25, 2020 at 1:22 PM wrote: > > colleague emails me the following puzzler ? current project based on > wind river linux 9 (so effectively morty 2.2), so it has bitbake 1.32. > apparently, for a specific recipe, there are intermittent build errors > which she suspects are caused by a race condition in the build of that > recipe. > > *but* that build error only occurs when something about the underlying > layers have changed; as in, someone checked *something* in somewhere > forcing at least something to be rebuilt, and that error does not > even occur on *all* rebuilds, just some of them. so the question is, > which underlying recipe change and rebuild could be triggering this? > > she *suspects* it might be openssl, as their project is making > regular changes to openssl, and that recipe seems to be an ultimate > dependency for quite a number of recipes, so she's just guessing > that might be the cause. > > in order to test this, she is currently running through a loop > wherein, in order to kick off at least some recipe rebuilds, she > makes some innocuous change to openssl, causing it to be rebuilt, > which then rebuilds all recipes that depend on it and so on. > > yuck. > > on the other hand, i couldn't immediately suggest anything > cleaner. my thought was to (somehow) fake a rebuild of openssl > (perhaps by invalidating some timestamp), but that's not going > to work as once it's rebuilt, it's just going to look like it > did before, so nothing else will be rebuilt. Not sure what you mean by "fake" a rebuild. You can force a rebuild by e.g. manually changing something in the recipe such as PR, running "bitbake -c cleansstate openssl", etc, but then it's a real a rebuild. It doesn't matter if the output of the rebuild is identical to a previous build, anything which depends on openssl will get rebuilt too (at least for OE 2.2). > is there an easier way to do this? i don't have a system > with that old a YP version at hand, so i can't test this, and > nothing leaps to mind. is there a "make touch" equivalent > that would drive a full rebuild of all recipes that depend > on a given recipe, without having to actually mess with it? > > i figured, "ah, this can't be that hard," but i haven't > come up with anything. > > rday > > p.s. simply cleaning the offending recipe and rebuilding it > won't trigger the build error (at least that's what I've been > told) ? one must change the underlying recipes at least a > little in order to *force* a rebuild allegedly based on > dependencies. Right, clean doesn't remove sstate, so just running clean won't force something to be rebuilt. You need to use cleansstate instead. Anyway, it sounds like you just have a missing build dependency somewhere. Making a change to openssl causes a lot of stuff to get rebuilt (but perhaps in a different order than a full rebuild from scratch) which is highlighting the missing dependency. Best way to debug is to find out which recipe fails intermittently and then try building just that recipe after running cleansstate for that recipe and removing the entire tmp directory. It should then fail consistently. From rpjday at crashcourse.ca Wed Feb 26 01:07:27 2020 From: rpjday at crashcourse.ca (Robert P. J. Day) Date: Tue, 25 Feb 2020 20:07:27 -0500 (EST) Subject: [bitbake-devel] is there a bitbake equivalent to "touch"ing a file for gnu make? In-Reply-To: References: <20200225162245.Horde.i6DmVIhAUjVbUGUQKtyMoua@crashcourse.ca> Message-ID: On Tue, 25 Feb 2020, Andre McCurdy wrote: > On Tue, Feb 25, 2020 at 1:22 PM wrote: > > > > colleague emails me the following puzzler ? current project based on > > wind river linux 9 (so effectively morty 2.2), so it has bitbake 1.32. > > apparently, for a specific recipe, there are intermittent build errors > > which she suspects are caused by a race condition in the build of that > > recipe. > > > > *but* that build error only occurs when something about the underlying > > layers have changed; as in, someone checked *something* in somewhere > > forcing at least something to be rebuilt, and that error does not > > even occur on *all* rebuilds, just some of them. so the question is, > > which underlying recipe change and rebuild could be triggering this? > > > > she *suspects* it might be openssl, as their project is making > > regular changes to openssl, and that recipe seems to be an ultimate > > dependency for quite a number of recipes, so she's just guessing > > that might be the cause. > > > > in order to test this, she is currently running through a loop > > wherein, in order to kick off at least some recipe rebuilds, she > > makes some innocuous change to openssl, causing it to be rebuilt, > > which then rebuilds all recipes that depend on it and so on. > > > > yuck. > > > > on the other hand, i couldn't immediately suggest anything > > cleaner. my thought was to (somehow) fake a rebuild of openssl > > (perhaps by invalidating some timestamp), but that's not going > > to work as once it's rebuilt, it's just going to look like it > > did before, so nothing else will be rebuilt. > > Not sure what you mean by "fake" a rebuild. You can force a rebuild > by e.g. manually changing something in the recipe such as PR, > running "bitbake -c cleansstate openssl", ... are you sure about that? i can test this when i get home, but i thought just cleaning the sstate would not force rebuilds of the rest of those recipes, just that one. rday From armccurdy at gmail.com Wed Feb 26 01:31:52 2020 From: armccurdy at gmail.com (Andre McCurdy) Date: Tue, 25 Feb 2020 17:31:52 -0800 Subject: [bitbake-devel] is there a bitbake equivalent to "touch"ing a file for gnu make? In-Reply-To: References: <20200225162245.Horde.i6DmVIhAUjVbUGUQKtyMoua@crashcourse.ca> Message-ID: On Tue, Feb 25, 2020 at 5:07 PM Robert P. J. Day wrote: > > On Tue, 25 Feb 2020, Andre McCurdy wrote: > > > On Tue, Feb 25, 2020 at 1:22 PM wrote: > > > > > > colleague emails me the following puzzler ? current project based on > > > wind river linux 9 (so effectively morty 2.2), so it has bitbake 1.32. > > > apparently, for a specific recipe, there are intermittent build errors > > > which she suspects are caused by a race condition in the build of that > > > recipe. > > > > > > *but* that build error only occurs when something about the underlying > > > layers have changed; as in, someone checked *something* in somewhere > > > forcing at least something to be rebuilt, and that error does not > > > even occur on *all* rebuilds, just some of them. so the question is, > > > which underlying recipe change and rebuild could be triggering this? > > > > > > she *suspects* it might be openssl, as their project is making > > > regular changes to openssl, and that recipe seems to be an ultimate > > > dependency for quite a number of recipes, so she's just guessing > > > that might be the cause. > > > > > > in order to test this, she is currently running through a loop > > > wherein, in order to kick off at least some recipe rebuilds, she > > > makes some innocuous change to openssl, causing it to be rebuilt, > > > which then rebuilds all recipes that depend on it and so on. > > > > > > yuck. > > > > > > on the other hand, i couldn't immediately suggest anything > > > cleaner. my thought was to (somehow) fake a rebuild of openssl > > > (perhaps by invalidating some timestamp), but that's not going > > > to work as once it's rebuilt, it's just going to look like it > > > did before, so nothing else will be rebuilt. > > > > Not sure what you mean by "fake" a rebuild. You can force a rebuild > > by e.g. manually changing something in the recipe such as PR, > > running "bitbake -c cleansstate openssl", ... > > are you sure about that? i can test this when i get home, but i > thought just cleaning the sstate would not force rebuilds of the rest > of those recipes, just that one. Not 100% sure. Changing PR in the recipe should work though. But anyway you probably want to focus on the specific recipe which fails intermittently. Forcing it to be rebuilt after removing tmp should at least tell you if it's a missing build dependency. If it fails then it's a missing dependency and it should be obvious from the build error (missing header, library, etc) what the missing dependency is. If it passes then it suggests that recipe has an unconstrained configure option which is behaving differently based on whether it autodetects something in the sysroot (which is shared in OE 2.2). In that case it should fail consistently if you run the full build (so that sysroot in tmp is fully populated) and then cleansstate and rebuild the failing recipe. From rpjday at crashcourse.ca Fri Feb 28 13:33:36 2020 From: rpjday at crashcourse.ca (rpjday at crashcourse.ca) Date: Fri, 28 Feb 2020 08:33:36 -0500 Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? Message-ID: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> looked around and i don't see an immediately obvious way to do this ... is there a way to generate a list of *only* those recipes that were used in the current build that had associated .bbappend files? "bitbake-layers show-appends" will show all recipes with .bbappend files, including those that are not included and processed for the current build/image. here's the rationale, in case i'm missing something. been handed an existing vendor layer, with *tons* of bbappend'ed recipes, some basic, some from meta-openembedded and many from the vendor's proprietary layer. my go-to strategy is to always try to build a core-image-minimal to establish a working baseline and, when that works, it would be nice to know which of the recipes that contributed to that core-image-minimal build were modified in any way via a .bbappend file, then work my way out. does that make sense? is there a way to do this, other than wander down to tmp/work/... and try to figure out which recipes were modified from the original? am i overlooking something simple? rday From Mikko.Rapeli at bmw.de Fri Feb 28 14:09:19 2020 From: Mikko.Rapeli at bmw.de (Mikko.Rapeli at bmw.de) Date: Fri, 28 Feb 2020 14:09:19 +0000 Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? In-Reply-To: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> References: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> Message-ID: <20200228140919.GO104502@korppu> Hi, On Fri, Feb 28, 2020 at 08:33:36AM -0500, rpjday at crashcourse.ca wrote: > looked around and i don't see an immediately obvious way to do this ... > is there a way to generate a list of *only* those recipes that were > used in the current build that had associated .bbappend files? > > "bitbake-layers show-appends" will show all recipes with .bbappend > files, including those that are not included and processed for the > current build/image. here's the rationale, in case i'm missing > something. I'm doing "find meta* -name \*recipe\*bb\*" and reviewing every one of them. More visibility to this including parsing order would be nice to store in e.g. buildhistory. > been handed an existing vendor layer, with *tons* of bbappend'ed > recipes, some basic, some from meta-openembedded and many from > the vendor's proprietary layer. my go-to strategy is to always try > to build a core-image-minimal to establish a working baseline and, > when that works, it would be nice to know which of the recipes > that contributed to that core-image-minimal build were modified > in any way via a .bbappend file, then work my way out. > > does that make sense? is there a way to do this, other than wander > down to tmp/work/... and try to figure out which recipes were > modified from the original? am i overlooking something simple? I start off by reviewing every single file in vendor and BSP SW layers before I add them to builds. Sadly this does not catch bad things which get integrated later in updates. Resulting distro config in my projects is full of BBMASK paths to the various layers which are intrusive with their bbappends. Maybe recording all bbappends and their relative paths in the project and parsing order to buildhistory would help. I would also like to limit layers from adding bbappends to certain recipes but I don't see ways for this currently without external scripts. Maybe this helps, -Mikko From rpjday at crashcourse.ca Fri Feb 28 19:38:09 2020 From: rpjday at crashcourse.ca (Robert P. J. Day) Date: Fri, 28 Feb 2020 14:38:09 -0500 (EST) Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? In-Reply-To: <20200228140919.GO104502@korppu> References: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> <20200228140919.GO104502@korppu> Message-ID: On Fri, 28 Feb 2020, Mikko.Rapeli at bmw.de wrote: > Hi, > > On Fri, Feb 28, 2020 at 08:33:36AM -0500, rpjday at crashcourse.ca wrote: > > looked around and i don't see an immediately obvious way to do > > this ... is there a way to generate a list of *only* those recipes > > that were used in the current build that had associated .bbappend > > files? > > > > "bitbake-layers show-appends" will show all recipes with .bbappend > > files, including those that are not included and processed for the > > current build/image. here's the rationale, in case i'm missing > > something. > > I'm doing "find meta* -name \*recipe\*bb\*" and reviewing every one > of them. More visibility to this including parsing order would be > nice to store in e.g. buildhistory. thanks for clarifying that there wasn't a massively trivial way to do this that i just didn't see. > > been handed an existing vendor layer, with *tons* of bbappend'ed > > recipes, some basic, some from meta-openembedded and many from the > > vendor's proprietary layer. my go-to strategy is to always try to > > build a core-image-minimal to establish a working baseline and, > > when that works, it would be nice to know which of the recipes > > that contributed to that core-image-minimal build were modified in > > any way via a .bbappend file, then work my way out. > > > > does that make sense? is there a way to do this, other than wander > > down to tmp/work/... and try to figure out which recipes were > > modified from the original? am i overlooking something simple? > > I start off by reviewing every single file in vendor and BSP SW > layers before I add them to builds. Sadly this does not catch bad > things which get integrated later in updates. > > Resulting distro config in my projects is full of BBMASK paths to > the various layers which are intrusive with their bbappends. yup, i'd already thought of that ... a methodical way to slowly add recipes/layers to the build. > Maybe recording all bbappends and their relative paths in the > project and parsing order to buildhistory would help. > > I would also like to limit layers from adding bbappends to certain > recipes but I don't see ways for this currently without external > scripts. yeah, there's lots of cool things i can think of that would assist in a migration from an older version of OE/YP to a more recent one. rday From peter.kjellerstedt at axis.com Fri Feb 28 20:31:58 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Fri, 28 Feb 2020 20:31:58 +0000 Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? In-Reply-To: References: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> <20200228140919.GO104502@korppu> Message-ID: <52d5e8b6d3c64fe0a74d098e79651f2b@XBOX03.axis.com> > -----Original Message----- > From: bitbake-devel-bounces at lists.openembedded.org bounces at lists.openembedded.org> On Behalf Of Robert P. J. Day > Sent: den 28 februari 2020 20:38 > To: Mikko.Rapeli at bmw.de > Cc: bitbake-devel at lists.openembedded.org > Subject: Re: [bitbake-devel] can i get a list of only bbappend'ed recipes > that were *used* in a build? > > On Fri, 28 Feb 2020, Mikko.Rapeli at bmw.de wrote: > > > Hi, > > > > On Fri, Feb 28, 2020 at 08:33:36AM -0500, rpjday at crashcourse.ca wrote: > > > looked around and i don't see an immediately obvious way to do > > > this ... is there a way to generate a list of *only* those recipes > > > that were used in the current build that had associated .bbappend > > > files? > > > > > > "bitbake-layers show-appends" will show all recipes with .bbappend > > > files, including those that are not included and processed for the > > > current build/image. here's the rationale, in case i'm missing > > > something. > > > > I'm doing "find meta* -name \*recipe\*bb\*" and reviewing every one > > of them. More visibility to this including parsing order would be > > nice to store in e.g. buildhistory. > > thanks for clarifying that there wasn't a massively trivial way to > do this that i just didn't see. For each image you build, you should find a manifest in "tmp/deploy/image//-.manifest". It contains the names, architectures and versions of all packages that were installed in the image. To convert the package names from the manifest to recipes, you can use oe-pkgdata-util: oe-pkgdata-util -p tmp/pkgdata/ package-info -f (the -p tmp/pkgdata/ option isn't necessary, but you avoid starting tinfoil if you use it). This will give you one line per package that also includes the recipe name. Pass the output to `cut -f3 -d' ' | sort -u' and you should have the list of recipes that were used to build the packages in your image. Then all you have to do is take the intersection of recipes from that list and from `bitbake-layers show-appends` and you should have the list of recipes that have bbappends applied and are used to build your image. Not trivial, but it should be possible to write a relatively simple script to do the above. > > > been handed an existing vendor layer, with *tons* of bbappend'ed > > > recipes, some basic, some from meta-openembedded and many from the > > > vendor's proprietary layer. my go-to strategy is to always try to > > > build a core-image-minimal to establish a working baseline and, > > > when that works, it would be nice to know which of the recipes > > > that contributed to that core-image-minimal build were modified in > > > any way via a .bbappend file, then work my way out. > > > > > > does that make sense? is there a way to do this, other than wander > > > down to tmp/work/... and try to figure out which recipes were > > > modified from the original? am i overlooking something simple? > > > > I start off by reviewing every single file in vendor and BSP SW > > layers before I add them to builds. Sadly this does not catch bad > > things which get integrated later in updates. > > > > Resulting distro config in my projects is full of BBMASK paths to > > the various layers which are intrusive with their bbappends. > > yup, i'd already thought of that ... a methodical way to slowly add > recipes/layers to the build. > > > Maybe recording all bbappends and their relative paths in the > > project and parsing order to buildhistory would help. > > > > I would also like to limit layers from adding bbappends to certain > > recipes but I don't see ways for this currently without external > > scripts. > > yeah, there's lots of cool things i can think of that would assist > in a migration from an older version of OE/YP to a more recent one. > > rday //Peter From aaron at assonance.org Fri Feb 28 20:40:01 2020 From: aaron at assonance.org (Aaron Cohen) Date: Fri, 28 Feb 2020 15:40:01 -0500 Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? In-Reply-To: <52d5e8b6d3c64fe0a74d098e79651f2b@XBOX03.axis.com> References: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> <20200228140919.GO104502@korppu> <52d5e8b6d3c64fe0a74d098e79651f2b@XBOX03.axis.com> Message-ID: On Fri, Feb 28, 2020 at 3:32 PM Peter Kjellerstedt < peter.kjellerstedt at axis.com> wrote: > > -----Original Message----- > > From: bitbake-devel-bounces at lists.openembedded.org > bounces at lists.openembedded.org> On Behalf Of Robert P. J. Day > > Sent: den 28 februari 2020 20:38 > > To: Mikko.Rapeli at bmw.de > > Cc: bitbake-devel at lists.openembedded.org > > Subject: Re: [bitbake-devel] can i get a list of only bbappend'ed recipes > > that were *used* in a build? > > > > On Fri, 28 Feb 2020, Mikko.Rapeli at bmw.de wrote: > > > > > Hi, > > > > > > On Fri, Feb 28, 2020 at 08:33:36AM -0500, rpjday at crashcourse.ca wrote: > > > > looked around and i don't see an immediately obvious way to do > > > > this ... is there a way to generate a list of *only* those recipes > > > > that were used in the current build that had associated .bbappend > > > > files? > > > > > > > > "bitbake-layers show-appends" will show all recipes with .bbappend > > > > files, including those that are not included and processed for the > > > > current build/image. here's the rationale, in case i'm missing > > > > something. > > > > > > I'm doing "find meta* -name \*recipe\*bb\*" and reviewing every one > > > of them. More visibility to this including parsing order would be > > > nice to store in e.g. buildhistory. > > > > thanks for clarifying that there wasn't a massively trivial way to > > do this that i just didn't see. > > For each image you build, you should find a manifest in > "tmp/deploy/image//-.manifest". It contains the > names, architectures and versions of all packages that were installed in > the image. > > To convert the package names from the manifest to recipes, you can use > oe-pkgdata-util: > > oe-pkgdata-util -p tmp/pkgdata/ package-info -f manifest file> > > (the -p tmp/pkgdata/ option isn't necessary, but you avoid > starting tinfoil if you use it). This will give you one line per package > that also includes the recipe name. Pass the output to `cut -f3 -d' ' | > sort -u' and you should have the list of recipes that were used to build > the packages in your image. > > Then all you have to do is take the intersection of recipes from that > list and from `bitbake-layers show-appends` and you should have the > list of recipes that have bbappends applied and are used to build your > image. > > pipe the package list through 'xargs bitbake-layers show-appends' and it seems exactly what's asked for. oe-pkgdata-util -p tmp/pkgdata/ package-info -f tmp/deploy/images//.manifest | cut -f3 -d' ' | sort -u | xargs bitbake-layers show-appends -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter.kjellerstedt at axis.com Fri Feb 28 20:46:29 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Fri, 28 Feb 2020 20:46:29 +0000 Subject: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? In-Reply-To: References: <20200228083336.Horde.2i5ag6ZVWX3JF5j5y3bF1q2@crashcourse.ca> <20200228140919.GO104502@korppu> <52d5e8b6d3c64fe0a74d098e79651f2b@XBOX03.axis.com> Message-ID: <3e40c1258e9a4593945215f9cd41593d@XBOX03.axis.com> That was even simpler than I had thought. Nice! :) //Peter From: Aaron Cohen Sent: den 28 februari 2020 21:40 To: Peter Kjellerstedt Cc: Robert P. J. Day ; Mikko.Rapeli at bmw.de; bitbake-devel at lists.openembedded.org Subject: Re: [bitbake-devel] can i get a list of only bbappend'ed recipes that were *used* in a build? On Fri, Feb 28, 2020 at 3:32 PM Peter Kjellerstedt > wrote: > -----Original Message----- > From: bitbake-devel-bounces at lists.openembedded.org

bounces at lists.openembedded.org> On Behalf Of Robert P. J. Day > Sent: den 28 februari 2020 20:38 > To: Mikko.Rapeli at bmw.de > Cc: bitbake-devel at lists.openembedded.org > Subject: Re: [bitbake-devel] can i get a list of only bbappend'ed recipes > that were *used* in a build? > > On Fri, 28 Feb 2020, Mikko.Rapeli at bmw.de wrote: > > > Hi, > > > > On Fri, Feb 28, 2020 at 08:33:36AM -0500, rpjday at crashcourse.ca wrote: > > > looked around and i don't see an immediately obvious way to do > > > this ... is there a way to generate a list of *only* those recipes > > > that were used in the current build that had associated .bbappend > > > files? > > > > > > "bitbake-layers show-appends" will show all recipes with .bbappend > > > files, including those that are not included and processed for the > > > current build/image. here's the rationale, in case i'm missing > > > something. > > > > I'm doing "find meta* -name \*recipe\*bb\*" and reviewing every one > > of them. More visibility to this including parsing order would be > > nice to store in e.g. buildhistory. > > thanks for clarifying that there wasn't a massively trivial way to > do this that i just didn't see. For each image you build, you should find a manifest in "tmp/deploy/image//-.manifest". It contains the names, architectures and versions of all packages that were installed in the image. To convert the package names from the manifest to recipes, you can use oe-pkgdata-util: oe-pkgdata-util -p tmp/pkgdata/ package-info -f (the -p tmp/pkgdata/ option isn't necessary, but you avoid starting tinfoil if you use it). This will give you one line per package that also includes the recipe name. Pass the output to `cut -f3 -d' ' | sort -u' and you should have the list of recipes that were used to build the packages in your image. Then all you have to do is take the intersection of recipes from that list and from `bitbake-layers show-appends` and you should have the list of recipes that have bbappends applied and are used to build your image. pipe the package list through 'xargs bitbake-layers show-appends' and it seems exactly what's asked for. oe-pkgdata-util -p tmp/pkgdata/ package-info -f tmp/deploy/images//.manifest | cut -f3 -d' ' | sort -u | xargs bitbake-layers show-appends -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard.purdie at linuxfoundation.org Sat Feb 29 16:16:33 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sat, 29 Feb 2020 16:16:33 +0000 Subject: [bitbake-devel] [PATCH] fetch2: Limit shown checksums to sha256 Message-ID: <20200229161633.496077-1-richard.purdie@linuxfoundation.org> Currently bitbake will list many checksums for a recipe when none are present, encouraging users to add them all to a recipe. We don't need/want them all. We used to show md5 and sha256 but given the concerns about md5, switch to showing just sha256 going forward which seems like the sensible one to standardise upon. There will be no change to existing recipe functionality. Signed-off-by: Richard Purdie --- lib/bb/fetch2/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py index b83347ad90..9734e21126 100644 --- a/lib/bb/fetch2/__init__.py +++ b/lib/bb/fetch2/__init__.py @@ -34,6 +34,7 @@ _checksum_cache = bb.checksum.FileChecksumCache() logger = logging.getLogger("BitBake.Fetcher") CHECKSUM_LIST = [ "md5", "sha256", "sha1", "sha384", "sha512" ] +SHOWN_CHECKSUM_LIST = ["sha256"] class BBFetchException(Exception): """Class all fetch exceptions inherit from""" @@ -580,7 +581,9 @@ def verify_checksum(ud, d, precomputed={}): checksum_dict = {ci["id"] : ci["data"] for ci in checksum_infos} checksum_event = {"%ssum" % ci["id"] : ci["data"] for ci in checksum_infos} - checksum_lines = ["SRC_URI[%s] = \"%s\"" % (ci["name"], ci["data"]) for ci in checksum_infos] + for ci in checksum_infos: + if ci["id"] in SHOWN_CHECKSUM_LIST: + checksum_lines = ["SRC_URI[%s] = \"%s\"" % (ci["name"], ci["data"])] # If no checksum has been provided if ud.method.recommends_checksum(ud) and all(ci["expected"] is None for ci in checksum_infos): -- 2.25.0