From richard.purdie at linuxfoundation.org Sun Mar 1 10:58:46 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Sun, 1 Mar 2020 10:58:46 +0000 Subject: [bitbake-devel] [PATCH] cooker: Reset unpon clientComplete Message-ID: <20200301105846.525957-1-richard.purdie@linuxfoundation.org> If for example a tinfoil connection edits the datastore, a subsequent connection can be "corrupted" by those changes. By calling the cooker reset function at exit, the behaviour becomes the same as a newly setup server. This avoids problems in tests when BB_SERVER_TIMEOUT is set as the server is properly reset between connections. [YOCTO #13812] Signed-off-by: Richard Purdie --- lib/bb/cooker.py | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index 2d937a413c..2a17c67373 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -1665,6 +1665,7 @@ class BBCooker: self.command.reset() self.databuilder.reset() self.data = self.databuilder.data + self.reset() class CookerExit(bb.event.Event): -- 2.25.0 From tanuk at iki.fi Tue Mar 3 07:56:29 2020 From: tanuk at iki.fi (Tanu Kaskinen) Date: Tue, 03 Mar 2020 09:56:29 +0200 Subject: [bitbake-devel] [PATCH] fetch2: Limit shown checksums to sha256 In-Reply-To: <20200229161633.496077-1-richard.purdie@linuxfoundation.org> References: <20200229161633.496077-1-richard.purdie@linuxfoundation.org> Message-ID: <736799570e22f93fdea25f2fb0ecb129dacd5760.camel@iki.fi> On Sat, 2020-02-29 at 16:16 +0000, Richard Purdie wrote: > Currently bitbake will list many checksums for a recipe when none are > present, encouraging users to add them all to a recipe. We don't > need/want them all. > > We used to show md5 and sha256 but given the concerns about md5, > switch to showing just sha256 going forward which seems like the > sensible one to standardise upon. Should this be interpreted so that going forward, it's recommended to drop the md5 checksums from recipes when doing version upgrades? -- Tanu https://www.patreon.com/tanuk https://liberapay.com/tanuk From richard.purdie at linuxfoundation.org Tue Mar 3 08:23:59 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Tue, 03 Mar 2020 08:23:59 +0000 Subject: [bitbake-devel] [PATCH] fetch2: Limit shown checksums to sha256 In-Reply-To: <736799570e22f93fdea25f2fb0ecb129dacd5760.camel@iki.fi> References: <20200229161633.496077-1-richard.purdie@linuxfoundation.org> <736799570e22f93fdea25f2fb0ecb129dacd5760.camel@iki.fi> Message-ID: <777e3d343a1a29738dec59a3984ae63d6f7b296b.camel@linuxfoundation.org> On Tue, 2020-03-03 at 09:56 +0200, Tanu Kaskinen wrote: > On Sat, 2020-02-29 at 16:16 +0000, Richard Purdie wrote: > > Currently bitbake will list many checksums for a recipe when none > > are > > present, encouraging users to add them all to a recipe. We don't > > need/want them all. > > > > We used to show md5 and sha256 but given the concerns about md5, > > switch to showing just sha256 going forward which seems like the > > sensible one to standardise upon. > > Should this be interpreted so that going forward, it's recommended to > drop the md5 checksums from recipes when doing version upgrades? Yes, that is the intent/plan. We don't really need both and this usability fix is a good opportunity to rationalise what we're doing. Cheers, Richard From jpewhacker at gmail.com Wed Mar 4 20:02:22 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 4 Mar 2020 14:02:22 -0600 Subject: [bitbake-devel] [PATCH 0/4] bitbake: Add multiconfig to 'bitbake-layers show-recipes' Message-ID: <20200304200226.23771-1-JPEWhacker@gmail.com> Implements support for specifying a specific multiconfig to use when running 'bitbake-layers show-recipes'. This required fixing up some of the multiconfig support in tinfoil and the cooker. Joshua Watt (4): bitbake: command: Add mc parameter to findProviders command bitbake: cooker: Respect multiconfig parameter bitbake: tinfoil: Add multiconfig support bitbake: bblayers: query: Add multiconfig option bitbake/lib/bb/command.py | 6 +++++- bitbake/lib/bb/cooker.py | 6 +++--- bitbake/lib/bb/tinfoil.py | 38 ++++++++++++++++++----------------- bitbake/lib/bblayers/query.py | 20 +++++++++++------- 4 files changed, 41 insertions(+), 29 deletions(-) -- 2.17.1 From jpewhacker at gmail.com Wed Mar 4 20:02:23 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 4 Mar 2020 14:02:23 -0600 Subject: [bitbake-devel] [PATCH 1/4] bitbake: command: Add mc parameter to findProviders command In-Reply-To: <20200304200226.23771-1-JPEWhacker@gmail.com> References: <20200304200226.23771-1-JPEWhacker@gmail.com> Message-ID: <20200304200226.23771-2-JPEWhacker@gmail.com> Adds a multiconfig selection parameter to the findProviders command. This allows a client to find the providers for a specific multiconfig instead of the base configuration. Signed-off-by: Joshua Watt --- bitbake/lib/bb/command.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py index c8e1352865..b38c151b3d 100644 --- a/bitbake/lib/bb/command.py +++ b/bitbake/lib/bb/command.py @@ -414,7 +414,11 @@ class CommandsSync: getAllAppends.readonly = True def findProviders(self, command, params): - return command.cooker.findProviders() + try: + mc = params[0] + except IndexError: + mc = '' + return command.cooker.findProviders(mc) findProviders.readonly = True def findBestProvider(self, command, params): -- 2.17.1 From jpewhacker at gmail.com Wed Mar 4 20:02:24 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 4 Mar 2020 14:02:24 -0600 Subject: [bitbake-devel] [PATCH 2/4] bitbake: cooker: Respect multiconfig parameter In-Reply-To: <20200304200226.23771-1-JPEWhacker@gmail.com> References: <20200304200226.23771-1-JPEWhacker@gmail.com> Message-ID: <20200304200226.23771-3-JPEWhacker@gmail.com> The cooker had a multiconfig parameter for the findProviders() and findBestProviders() API, but it was being ignored. Signed-off-by: Joshua Watt --- bitbake/lib/bb/cooker.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index bda6d1b5c4..e527e23114 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -1023,16 +1023,16 @@ class BBCooker: bb.event.fire(bb.event.FilesMatchingFound(filepattern, matches), self.data) def findProviders(self, mc=''): - return bb.providers.findProviders(self.data, self.recipecaches[mc], self.recipecaches[mc].pkg_pn) + return bb.providers.findProviders(self.databuilder.mcdata[mc], self.recipecaches[mc], self.recipecaches[mc].pkg_pn) def findBestProvider(self, pn, mc=''): if pn in self.recipecaches[mc].providers: filenames = self.recipecaches[mc].providers[pn] - eligible, foundUnique = bb.providers.filterProviders(filenames, pn, self.data, self.recipecaches[mc]) + eligible, foundUnique = bb.providers.filterProviders(filenames, pn, self.databuilder.mcdata[mc], self.recipecaches[mc]) filename = eligible[0] return None, None, None, filename elif pn in self.recipecaches[mc].pkg_pn: - return bb.providers.findBestProvider(pn, self.data, self.recipecaches[mc], self.recipecaches[mc].pkg_pn) + return bb.providers.findBestProvider(pn, self.databuilder.mcdata[mc], self.recipecaches[mc], self.recipecaches[mc].pkg_pn) else: return None, None, None, None -- 2.17.1 From jpewhacker at gmail.com Wed Mar 4 20:02:25 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 4 Mar 2020 14:02:25 -0600 Subject: [bitbake-devel] [PATCH 3/4] bitbake: tinfoil: Add multiconfig support In-Reply-To: <20200304200226.23771-1-JPEWhacker@gmail.com> References: <20200304200226.23771-1-JPEWhacker@gmail.com> Message-ID: <20200304200226.23771-4-JPEWhacker@gmail.com> Adds support for the Tinfoil cache adaptor to be bound to a specific multiconfig and invoke the appropriate commands for that multiconfig instead of the default. The cooker adapter now creates a cache adapter for each multiconfig specified in BBMULTICONFIG so that each multiconfig is present. Signed-off-by: Joshua Watt --- bitbake/lib/bb/tinfoil.py | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py index 645f49639d..5c5be456e2 100644 --- a/bitbake/lib/bb/tinfoil.py +++ b/bitbake/lib/bb/tinfoil.py @@ -127,12 +127,13 @@ class TinfoilCookerAdapter: class TinfoilRecipeCacheAdapter: """ cooker.recipecache adapter """ - def __init__(self, tinfoil): + def __init__(self, tinfoil, mc=''): self.tinfoil = tinfoil + self.mc = mc self._cache = {} def get_pkg_pn_fn(self): - pkg_pn = defaultdict(list, self.tinfoil.run_command('getRecipes') or []) + pkg_pn = defaultdict(list, self.tinfoil.run_command('getRecipes', self.mc) or []) pkg_fn = {} for pn, fnlist in pkg_pn.items(): for fn in fnlist: @@ -151,27 +152,27 @@ class TinfoilCookerAdapter: self.get_pkg_pn_fn() return self._cache[name] elif name == 'deps': - attrvalue = defaultdict(list, self.tinfoil.run_command('getRecipeDepends') or []) + attrvalue = defaultdict(list, self.tinfoil.run_command('getRecipeDepends', self.mc) or []) elif name == 'rundeps': - attrvalue = defaultdict(lambda: defaultdict(list), self.tinfoil.run_command('getRuntimeDepends') or []) + attrvalue = defaultdict(lambda: defaultdict(list), self.tinfoil.run_command('getRuntimeDepends', self.mc) or []) elif name == 'runrecs': - attrvalue = defaultdict(lambda: defaultdict(list), self.tinfoil.run_command('getRuntimeRecommends') or []) + attrvalue = defaultdict(lambda: defaultdict(list), self.tinfoil.run_command('getRuntimeRecommends', self.mc) or []) elif name == 'pkg_pepvpr': - attrvalue = self.tinfoil.run_command('getRecipeVersions') or {} + attrvalue = self.tinfoil.run_command('getRecipeVersions', self.mc) or {} elif name == 'inherits': - attrvalue = self.tinfoil.run_command('getRecipeInherits') or {} + attrvalue = self.tinfoil.run_command('getRecipeInherits', self.mc) or {} elif name == 'bbfile_priority': - attrvalue = self.tinfoil.run_command('getBbFilePriority') or {} + attrvalue = self.tinfoil.run_command('getBbFilePriority', self.mc) or {} elif name == 'pkg_dp': - attrvalue = self.tinfoil.run_command('getDefaultPreference') or {} + attrvalue = self.tinfoil.run_command('getDefaultPreference', self.mc) or {} elif name == 'fn_provides': - attrvalue = self.tinfoil.run_command('getRecipeProvides') or {} + attrvalue = self.tinfoil.run_command('getRecipeProvides', self.mc) or {} elif name == 'packages': - attrvalue = self.tinfoil.run_command('getRecipePackages') or {} + attrvalue = self.tinfoil.run_command('getRecipePackages', self.mc) or {} elif name == 'packages_dynamic': - attrvalue = self.tinfoil.run_command('getRecipePackagesDynamic') or {} + attrvalue = self.tinfoil.run_command('getRecipePackagesDynamic', self.mc) or {} elif name == 'rproviders': - attrvalue = self.tinfoil.run_command('getRProviders') or {} + attrvalue = self.tinfoil.run_command('getRProviders', self.mc) or {} else: raise AttributeError("%s instance has no attribute '%s'" % (self.__class__.__name__, name)) @@ -182,8 +183,9 @@ class TinfoilCookerAdapter: self.tinfoil = tinfoil self.collection = self.TinfoilCookerCollectionAdapter(tinfoil) self.recipecaches = {} - # FIXME all machines self.recipecaches[''] = self.TinfoilRecipeCacheAdapter(tinfoil) + for mc in (tinfoil.config_data.getVar('BBMULTICONFIG') or '').split(): + self.recipecaches[mc] = self.TinfoilRecipeCacheAdapter(tinfoil, mc) self._cache = {} def __getattr__(self, name): # Grab these only when they are requested since they aren't always used @@ -501,11 +503,11 @@ class Tinfoil: """ return OrderedDict(self.run_command('getSkippedRecipes')) - def get_all_providers(self): - return defaultdict(list, self.run_command('allProviders')) + def get_all_providers(self, mc=''): + return defaultdict(list, self.run_command('allProviders', mc)) - def find_providers(self): - return self.run_command('findProviders') + def find_providers(self, mc=''): + return self.run_command('findProviders', mc) def find_best_provider(self, pn): return self.run_command('findBestProvider', pn) -- 2.17.1 From jpewhacker at gmail.com Wed Mar 4 20:02:26 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 4 Mar 2020 14:02:26 -0600 Subject: [bitbake-devel] [PATCH 4/4] bitbake: bblayers: query: Add multiconfig option In-Reply-To: <20200304200226.23771-1-JPEWhacker@gmail.com> References: <20200304200226.23771-1-JPEWhacker@gmail.com> Message-ID: <20200304200226.23771-5-JPEWhacker@gmail.com> Adds an option to the show-recipes subcommand that allows the user to specify which multiconfig should be shown. Signed-off-by: Joshua Watt --- bitbake/lib/bblayers/query.py | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py index 7db49c8e2a..e2cc310532 100644 --- a/bitbake/lib/bblayers/query.py +++ b/bitbake/lib/bblayers/query.py @@ -46,7 +46,7 @@ layer, with the preferred version first. Note that skipped recipes that are overlayed will also be listed, with a " (skipped)" suffix. """ - items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None) + items_listed = self.list_recipes('Overlayed recipes', None, True, args.same_version, args.filenames, False, True, None, False, None, args.mc) # Check for overlayed .bbclass files classes = collections.defaultdict(list) @@ -112,9 +112,9 @@ skipped recipes will also be listed, with a " (skipped)" suffix. title = 'Matching recipes:' else: title = 'Available recipes:' - self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist) + self.list_recipes(title, args.pnspec, False, False, args.filenames, args.recipes_only, args.multiple, args.layer, args.bare, inheritlist, args.mc) - def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits): + def list_recipes(self, title, pnspec, show_overlayed_only, show_same_ver_only, show_filenames, show_recipes_only, show_multi_provider_only, selected_layer, bare, inherits, mc): if inherits: bbpath = str(self.tinfoil.config_data.getVar('BBPATH')) for classname in inherits: @@ -123,14 +123,18 @@ skipped recipes will also be listed, with a " (skipped)" suffix. logger.error('No class named %s found in BBPATH', classfile) sys.exit(1) - pkg_pn = self.tinfoil.cooker.recipecaches[''].pkg_pn - (latest_versions, preferred_versions) = self.tinfoil.find_providers() - allproviders = self.tinfoil.get_all_providers() + pkg_pn = self.tinfoil.cooker.recipecaches[mc].pkg_pn + (latest_versions, preferred_versions) = self.tinfoil.find_providers(mc) + allproviders = self.tinfoil.get_all_providers(mc) # Ensure we list skipped recipes # We are largely guessing about PN, PV and the preferred version here, # but we have no choice since skipped recipes are not fully parsed skiplist = list(self.tinfoil.cooker.skiplist.keys()) + mcspec = 'mc:%s:' % mc + if mc: + skiplist = [s[len(mcspec):] for s in skiplist if s.startswith(mcspec)] + for fn in skiplist: recipe_parts = os.path.splitext(os.path.basename(fn))[0].split('_') p = recipe_parts[0] @@ -187,7 +191,7 @@ skipped recipes will also be listed, with a " (skipped)" suffix. # We only display once per recipe, we should prefer non extended versions of the # recipe if present (so e.g. in OpenEmbedded, openssl rather than nativesdk-openssl # which would otherwise sort first). - if realfn[1] and realfn[0] in self.tinfoil.cooker.recipecaches[''].pkg_fn: + if realfn[1] and realfn[0] in self.tinfoil.cooker.recipecaches[mc].pkg_fn: continue if inherits: @@ -496,6 +500,7 @@ NOTE: .bbappend files can impact the dependencies. parser_show_overlayed = self.add_command(sp, 'show-overlayed', self.do_show_overlayed) parser_show_overlayed.add_argument('-f', '--filenames', help='instead of the default formatting, list filenames of higher priority recipes with the ones they overlay indented underneath', action='store_true') parser_show_overlayed.add_argument('-s', '--same-version', help='only list overlayed recipes where the version is the same', action='store_true') + parser_show_overlayed.add_argument('--mc', help='use specified multiconfig', default='') parser_show_recipes = self.add_command(sp, 'show-recipes', self.do_show_recipes) parser_show_recipes.add_argument('-f', '--filenames', help='instead of the default formatting, list filenames of higher priority recipes with the ones they overlay indented underneath', action='store_true') @@ -504,6 +509,7 @@ NOTE: .bbappend files can impact the dependencies. parser_show_recipes.add_argument('-i', '--inherits', help='only list recipes that inherit the named class(es) - separate multiple classes using , (without spaces)', metavar='CLASS', default='') parser_show_recipes.add_argument('-l', '--layer', help='only list recipes from the selected layer', default='') parser_show_recipes.add_argument('-b', '--bare', help='output just names without the "(skipped)" marker', action='store_true') + parser_show_recipes.add_argument('--mc', help='use specified multiconfig', default='') parser_show_recipes.add_argument('pnspec', nargs='*', help='optional recipe name specification (wildcards allowed, enclose in quotes to avoid shell expansion)') parser_show_appends = self.add_command(sp, 'show-appends', self.do_show_appends) -- 2.17.1 From bunk at stusta.de Fri Mar 6 10:24:20 2020 From: bunk at stusta.de (Adrian Bunk) Date: Fri, 6 Mar 2020 12:24:20 +0200 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> <20200224054945.GB6683@localhost> Message-ID: <20200306102420.GB17785@localhost> On Mon, Feb 24, 2020 at 09:12:06AM +0000, Richard Purdie wrote: > On Mon, 2020-02-24 at 07:49 +0200, Adrian Bunk wrote: > > On Sun, Feb 23, 2020 at 11:53:57PM +0000, Richard Purdie wrote: >... > > > I know you didn't mean it this way and we need to track the issues, > > > I > > > just don't know how we're going to handle it. > > > > From my side the problem is that my initial patch is what I can do > > with reasonable effort. > > Part of the reason I'm highlighting this is that I have a lot of > feedback that says people don't believe there is any problem with our > resourcing. Unless I can show where there are issues, its not going to > change :/. Why does YP give users false/misleading information regarding security support of stable and LTS releases despite lack of resources to provide security support? There is no reason to believe claims that there would be any problem with resourcing when the public announcements from YP give the opposite impression. > Cheers, > > Richard cu Adrian From richard.purdie at linuxfoundation.org Fri Mar 6 10:47:49 2020 From: richard.purdie at linuxfoundation.org (Richard Purdie) Date: Fri, 06 Mar 2020 10:47:49 +0000 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: <20200306102420.GB17785@localhost> References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> <20200224054945.GB6683@localhost> <20200306102420.GB17785@localhost> Message-ID: On Fri, 2020-03-06 at 12:24 +0200, Adrian Bunk wrote: > On Mon, Feb 24, 2020 at 09:12:06AM +0000, Richard Purdie wrote: > > On Mon, 2020-02-24 at 07:49 +0200, Adrian Bunk wrote: > > > On Sun, Feb 23, 2020 at 11:53:57PM +0000, Richard Purdie wrote: > > ... > > > > I know you didn't mean it this way and we need to track the > > > > issues, > > > > I > > > > just don't know how we're going to handle it. > > > > > > From my side the problem is that my initial patch is what I can > > > do > > > with reasonable effort. > > > > Part of the reason I'm highlighting this is that I have a lot of > > feedback that says people don't believe there is any problem with > > our > > resourcing. Unless I can show where there are issues, its not going > > to > > change :/. > > Why does YP give users false/misleading information regarding > security support of stable and LTS releases despite lack of resources > to provide security support? See my other longer reply to you on oe-arch. I think the project is trying to improve the security support we can offer on stable releases as they become an LTS. No its not perfect and we're not offering guarantees but is is an improvement. The project got to pick where it allocated funding and it chose to improve LTS and security over for example helping me with day to day patch review/testing and build failure work. I don't mind, we have to start somewhere to improve things and LTS/security is important. The project does already fund some "day to day" running by funding me to work on bitbake/OE-Core. Yes, it would be helpful if it wasn't just me, maybe in the future if we can show success of the LTS we may attract more members and I can get some help and stop burning out, time will tell. > There is no reason to believe claims that there would be any problem > with resourcing when the public announcements from YP give the > opposite impression. As I've said in a few places now, there will be some comments from the project about this. I have already drafted something about resourcing but there were concerns it would cause panic and I've agreed to reword it to try not to do that. There isn't reason to panic, we do need more help and we will communicate that when its ready. You'd be amazed at the work in writing public statements and your reaction to the LTS announcement will sadly be held as an example of why I shouldn't talk about resourcing. Cheers, Richard From alex.kanavin at gmail.com Fri Mar 6 11:18:16 2020 From: alex.kanavin at gmail.com (Alexander Kanavin) Date: Fri, 6 Mar 2020 12:18:16 +0100 Subject: [bitbake-devel] [PATCH] Lower priority of debug messages that should not be shown on the console In-Reply-To: References: <20200221205213.16130-1-bunk@stusta.de> <20200223195148.GB3444@localhost> <49d09c50265086a49798d59c8569323c37bba430.camel@linuxfoundation.org> <20200224054945.GB6683@localhost> <20200306102420.GB17785@localhost> Message-ID: On Fri, 6 Mar 2020 at 11:48, Richard Purdie < richard.purdie at linuxfoundation.org> wrote: > You'd be amazed at > the work in writing public statements and your reaction to the LTS > announcement will sadly be held as an example of why I shouldn't talk > about resourcing. > I am fairly sure that example is an outlier, and not a common data point. Both in substance (claiming that it's easy to misrepresent the LTS announcement - not true), and in tone (various toxic accusations). Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From akuster808 at gmail.com Sun Mar 8 18:13:13 2020 From: akuster808 at gmail.com (akuster808) Date: Sun, 8 Mar 2020 11:13:13 -0700 Subject: [bitbake-devel] Stable life cycle status Message-ID: Hello, Thud has moved to "Community supported" today. The older releases have moved to EOL as no one has stepped up to take on the maintenance of those older branches. regards, Armin From pbarker at konsulko.com Mon Mar 9 13:42:46 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 9 Mar 2020 13:42:46 +0000 Subject: [bitbake-devel] [PATCH v2] fetch2/gitsm: Unpack shallow mirror tarballs Message-ID: <20200309134246.14067-1-pbarker@konsulko.com> When a shallow mirror tarball is used to satisfy a gitsm URI it needs to be unpacked temporarily so that the .gitmodules file can be examined. Signed-off-by: Paul Barker --- lib/bb/fetch2/gitsm.py | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py index aa121cbe..e7083001 100644 --- a/lib/bb/fetch2/gitsm.py +++ b/lib/bb/fetch2/gitsm.py @@ -20,6 +20,8 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" requires a clean of your r import os import bb import copy +import shutil +import tempfile from bb.fetch2.git import Git from bb.fetch2 import runfetchcmd from bb.fetch2 import logger @@ -130,7 +132,7 @@ class GitSM(Git): ld.setVar('SRCPV', d.getVar('SRCPV')) ld.setVar('SRCREV_FORMAT', module) - function(ud, url, module, paths[module], ld) + function(ud, url, module, paths[module], workdir, ld) return submodules != [] @@ -152,7 +154,7 @@ class GitSM(Git): return False def download(self, ud, d): - def download_submodule(ud, url, module, modpath, d): + def download_submodule(ud, url, module, modpath, workdir, d): url += ";bareclone=1;nobranch=1" # Is the following still needed? @@ -163,16 +165,25 @@ class GitSM(Git): newfetch.download() # Drop a nugget to add each of the srcrevs we've fetched (used by need_update) runfetchcmd("%s config --add bitbake.srcrev %s" % \ - (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=ud.clonedir) + (ud.basecmd, ud.revisions[ud.names[0]]), d, workdir=workdir) except Exception as e: logger.error('gitsm: submodule download failed: %s %s' % (type(e).__name__, str(e))) raise Git.download(self, ud, d) - self.process_submodules(ud, ud.clonedir, download_submodule, d) + + # If we're using a shallow mirror tarball it needs to be unpacked + # temporarily so that we can examine the .gitmodules file + if ud.shallow and os.path.exists(ud.fullshallow) and self.need_update(ud, d): + tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) + self.process_submodules(ud, tmpdir, download_submodule, d) + shutil.rmtree(tmpdir) + else: + self.process_submodules(ud, ud.clonedir, download_submodule, d) def unpack(self, ud, destdir, d): - def unpack_submodules(ud, url, module, modpath, d): + def unpack_submodules(ud, url, module, modpath, workdir, d): url += ";bareclone=1;nobranch=1" # Figure out where we clone over the bare submodules... -- 2.20.1 From pbarker at konsulko.com Mon Mar 9 13:45:13 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 9 Mar 2020 13:45:13 +0000 Subject: [bitbake-devel] [PATCH v2] fetch2/gitsm: Unpack shallow mirror tarballs In-Reply-To: <20200309134246.14067-1-pbarker@konsulko.com> References: <20200309134246.14067-1-pbarker@konsulko.com> Message-ID: Sorry, I missed the new test when copying this from a poky checkout to a bitbake checkout to generate the patch. I'll send the test as a follow-up. On Mon, 9 Mar 2020 at 13:43, Paul Barker wrote: > When a shallow mirror tarball is used to satisfy a gitsm URI it needs to > be unpacked temporarily so that the .gitmodules file can be examined. > > Signed-off-by: Paul Barker > --- > lib/bb/fetch2/gitsm.py | 21 ++++++++++++++++----- > 1 file changed, 16 insertions(+), 5 deletions(-) > > diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py > index aa121cbe..e7083001 100644 > --- a/lib/bb/fetch2/gitsm.py > +++ b/lib/bb/fetch2/gitsm.py > @@ -20,6 +20,8 @@ NOTE: Switching a SRC_URI from "git://" to "gitsm://" > requires a clean of your r > import os > import bb > import copy > +import shutil > +import tempfile > from bb.fetch2.git import Git > from bb.fetch2 import runfetchcmd > from bb.fetch2 import logger > @@ -130,7 +132,7 @@ class GitSM(Git): > ld.setVar('SRCPV', d.getVar('SRCPV')) > ld.setVar('SRCREV_FORMAT', module) > > - function(ud, url, module, paths[module], ld) > + function(ud, url, module, paths[module], workdir, ld) > > return submodules != [] > > @@ -152,7 +154,7 @@ class GitSM(Git): > return False > > def download(self, ud, d): > - def download_submodule(ud, url, module, modpath, d): > + def download_submodule(ud, url, module, modpath, workdir, d): > url += ";bareclone=1;nobranch=1" > > # Is the following still needed? > @@ -163,16 +165,25 @@ class GitSM(Git): > newfetch.download() > # Drop a nugget to add each of the srcrevs we've fetched > (used by need_update) > runfetchcmd("%s config --add bitbake.srcrev %s" % \ > - (ud.basecmd, ud.revisions[ud.names[0]]), d, > workdir=ud.clonedir) > + (ud.basecmd, ud.revisions[ud.names[0]]), d, > workdir=workdir) > except Exception as e: > logger.error('gitsm: submodule download failed: %s %s' % > (type(e).__name__, str(e))) > raise > > Git.download(self, ud, d) > - self.process_submodules(ud, ud.clonedir, download_submodule, d) > + > + # If we're using a shallow mirror tarball it needs to be unpacked > + # temporarily so that we can examine the .gitmodules file > + if ud.shallow and os.path.exists(ud.fullshallow) and > self.need_update(ud, d): > + tmpdir = tempfile.mkdtemp(dir=d.getVar("DL_DIR")) > + runfetchcmd("tar -xzf %s" % ud.fullshallow, d, workdir=tmpdir) > + self.process_submodules(ud, tmpdir, download_submodule, d) > + shutil.rmtree(tmpdir) > + else: > + self.process_submodules(ud, ud.clonedir, download_submodule, > d) > > def unpack(self, ud, destdir, d): > - def unpack_submodules(ud, url, module, modpath, d): > + def unpack_submodules(ud, url, module, modpath, workdir, d): > url += ";bareclone=1;nobranch=1" > > # Figure out where we clone over the bare submodules... > -- > 2.20.1 > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pbarker at konsulko.com Mon Mar 9 13:52:13 2020 From: pbarker at konsulko.com (Paul Barker) Date: Mon, 9 Mar 2020 13:52:13 +0000 Subject: [bitbake-devel] [PATCH] tests: Add test for gitsm fetcher with shallow mirror tarballs Message-ID: <20200309135213.14279-1-pbarker@konsulko.com> Signed-off-by: Paul Barker --- lib/bb/tests/fetch.py | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py index 45dc9e5d..d0c161a7 100644 --- a/lib/bb/tests/fetch.py +++ b/lib/bb/tests/fetch.py @@ -1691,6 +1691,47 @@ class GitShallowTest(FetcherTest): # Verify the submodule is also shallow self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule')) + def test_shallow_submodule_mirrors(self): + self.add_empty_file('a') + self.add_empty_file('b') + + smdir = os.path.join(self.tempdir, 'gitsubmodule') + bb.utils.mkdirhier(smdir) + self.git('init', cwd=smdir) + # Make this look like it was cloned from a remote... + self.git('config --add remote.origin.url "%s"' % smdir, cwd=smdir) + self.git('config --add remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"', cwd=smdir) + self.add_empty_file('asub', cwd=smdir) + self.add_empty_file('bsub', cwd=smdir) + + self.git('submodule init', cwd=self.srcdir) + self.git('submodule add file://%s' % smdir, cwd=self.srcdir) + self.git('submodule update', cwd=self.srcdir) + self.git('commit -m submodule -a', cwd=self.srcdir) + + uri = 'gitsm://%s;protocol=file;subdir=${S}' % self.srcdir + + # Fetch once to generate the shallow tarball + fetcher, ud = self.fetch(uri) + + # Set up the mirror + mirrordir = os.path.join(self.tempdir, 'mirror') + os.rename(self.dldir, mirrordir) + self.d.setVar('PREMIRRORS', 'gitsm://.*/.* file://%s/\n' % mirrordir) + + # Fetch from the mirror + bb.utils.remove(self.dldir, recurse=True) + bb.utils.remove(self.gitdir, recurse=True) + self.fetch_and_unpack(uri) + + # Verify the main repository is shallow + self.assertRevCount(1) + + # Verify the gitsubmodule directory is present + assert os.listdir(os.path.join(self.gitdir, 'gitsubmodule')) + + # Verify the submodule is also shallow + self.assertRevCount(1, cwd=os.path.join(self.gitdir, 'gitsubmodule')) if any(os.path.exists(os.path.join(p, 'git-annex')) for p in os.environ.get('PATH').split(':')): def test_shallow_annex(self): -- 2.20.1 From jpewhacker at gmail.com Mon Mar 9 16:33:38 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:38 -0500 Subject: [bitbake-devel] [PATCH 00/15] Add support for python structured logging Message-ID: <20200309163353.15362-1-JPEWhacker@gmail.com> Adds support for users to provide configuration for Python structured logging in bitbake. This gives a high degree of flexibility to the way that logs are capture and displayed, so users can generate very targeted logging if desired. The initial use case for this is to maintain the high level of hash equivalence logging on the Yocto autobuilders, while reducing the noise for most other users. I've added documentation updates for the new BB_LOGCONFIG variable, but it probably needs some grooming. Joshua Watt (15): bitbake: lib/bb/msg.py: Convert default domains to a dictionary bitbake: knotty: Handle logging messages with specific logger bitbake: lib/bb/msg.py: Use log level instead of debug count bitbake: lib/bb/msg.py: Add repr for BBLogFormatter bitbake: knotty: Add commented logging_tree code bitbake: lib/bb/msg.py: Add filter utilities bitbake: lib/bb/msg.py: Remove unused filters bitbake: lib/bb/msg.py: Add helper to set logging config bitbake: knotty: Remove dependency on format variable bitbake: knotty: Setup logs with config helper bitbake: worker: Remove unnecessary addDefaultLogFilter bitbake: Log hash equivalence with a different logger bitbake: Add autobuilder logging configuration bitbake: Lower hash equivalence logging bitbake: doc: Add documentation for BB_LOGCONFIG bitbake/bin/bitbake-worker | 3 +- bitbake/contrib/autobuilderlog.json | 27 +++ .../bitbake-user-manual-execution.xml | 97 ++++++++++ .../bitbake-user-manual-ref-variables.xml | 11 ++ bitbake/lib/bb/__init__.py | 7 +- bitbake/lib/bb/msg.py | 172 +++++++++++++---- bitbake/lib/bb/runqueue.py | 13 +- bitbake/lib/bb/siggen.py | 17 +- bitbake/lib/bb/tinfoil.py | 4 +- bitbake/lib/bb/ui/knotty.py | 173 ++++++++++++------ 10 files changed, 419 insertions(+), 105 deletions(-) create mode 100644 bitbake/contrib/autobuilderlog.json -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:39 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:39 -0500 Subject: [bitbake-devel] [PATCH 01/15] bitbake: lib/bb/msg.py: Convert default domains to a dictionary In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-2-JPEWhacker@gmail.com> Converts the default domain variable to a dictionary where the keys are the logging domains and the values are the logging level (instead of the debug count). This makes it easier to deal with the logging domains and the awkward conversion from a list to a dictionary only needs to be done once when logging is initialized. Finally, other code has been written that already assumes this variable is a dictionary, see: f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker") Signed-off-by: Joshua Watt --- bitbake/lib/bb/__init__.py | 5 +++-- bitbake/lib/bb/msg.py | 17 +++++++---------- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index 88641e280b..acd4af13a8 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -43,12 +43,13 @@ class BBLogger(Logger): Logger.__init__(self, name) def bbdebug(self, level, msg, *args, **kwargs): + loglevel = logging.DEBUG - level + 1 if not bb.event.worker_pid: - if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]): + if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]): return if level > (bb.msg.loggerDefaultDebugLevel): return - return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs) + return self.log(loglevel, msg, *args, **kwargs) def plain(self, msg, *args, **kwargs): return self.log(logging.INFO + 1, msg, *args, **kwargs) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index 33c0e2fa19..d1b0e929d4 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -138,7 +138,7 @@ class BBLogFilterStdOut(BBLogFilter): loggerDefaultDebugLevel = 0 loggerDefaultVerbose = False loggerVerboseLogs = False -loggerDefaultDomains = [] +loggerDefaultDomains = {} def init_msgconfig(verbose, debug, debug_domains=None): """ @@ -148,15 +148,16 @@ def init_msgconfig(verbose, debug, debug_domains=None): bb.msg.loggerDefaultVerbose = verbose if verbose: bb.msg.loggerVerboseLogs = True + + bb.msg.loggerDefaultDomains = {} if debug_domains: - bb.msg.loggerDefaultDomains = debug_domains - else: - bb.msg.loggerDefaultDomains = [] + for (domainarg, iterator) in groupby(debug_domains): + dlevel = len(tuple(iterator)) + bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 def constructLogOptions(): debug = loggerDefaultDebugLevel verbose = loggerDefaultVerbose - domains = loggerDefaultDomains if debug: level = BBLogFormatter.DEBUG - debug + 1 @@ -165,11 +166,7 @@ def constructLogOptions(): else: level = BBLogFormatter.NOTE - debug_domains = {} - for (domainarg, iterator) in groupby(domains): - dlevel = len(tuple(iterator)) - debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 - return level, debug_domains + return level, loggerDefaultDomains def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): level, debug_domains = constructLogOptions() -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:40 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:40 -0500 Subject: [bitbake-devel] [PATCH 02/15] bitbake: knotty: Handle logging messages with specific logger In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-3-JPEWhacker@gmail.com> Handles the log messages from the bitbake server with the specific logger that the event originated from. This allows hierarchical logging configurations to work as expected. Signed-off-by: Joshua Watt --- bitbake/lib/bb/ui/knotty.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index cbb289b05f..a6a92b920f 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -522,7 +522,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn'): event.msg = event.fn + ': ' + event.msg - logger.handle(event) + logging.getLogger(event.name).handle(event) continue if isinstance(event, bb.build.TaskFailedSilent): -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:41 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:41 -0500 Subject: [bitbake-devel] [PATCH 03/15] bitbake: lib/bb/msg.py: Use log level instead of debug count In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-4-JPEWhacker@gmail.com> Passes around the actual logging level as the default log level variable instead of the debug count. This makes it easier to deal with logging levels since the conversion from debug count and verbose flag only has to occur once when logging is initialized and after that actual log levels can be used Signed-off-by: Joshua Watt --- bitbake/bin/bitbake-worker | 2 +- bitbake/lib/bb/__init__.py | 2 +- bitbake/lib/bb/msg.py | 22 +++++++++------------- bitbake/lib/bb/runqueue.py | 2 +- 4 files changed, 12 insertions(+), 16 deletions(-) diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index 1e641e81c2..fc28a34d8a 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -414,7 +414,7 @@ class BitbakeWorker(object): def handle_workerdata(self, data): self.workerdata = pickle.loads(data) - bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"] + bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"] bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"] bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"] bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"] diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py index acd4af13a8..dc5e91e29a 100644 --- a/bitbake/lib/bb/__init__.py +++ b/bitbake/lib/bb/__init__.py @@ -47,7 +47,7 @@ class BBLogger(Logger): if not bb.event.worker_pid: if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]): return - if level > (bb.msg.loggerDefaultDebugLevel): + if loglevel > bb.msg.loggerDefaultLogLevel: return return self.log(loglevel, msg, *args, **kwargs) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index d1b0e929d4..ea6a9543f7 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -135,7 +135,7 @@ class BBLogFilterStdOut(BBLogFilter): # Message control functions # -loggerDefaultDebugLevel = 0 +loggerDefaultLogLevel = BBLogFormatter.NOTE loggerDefaultVerbose = False loggerVerboseLogs = False loggerDefaultDomains = {} @@ -144,11 +144,17 @@ def init_msgconfig(verbose, debug, debug_domains=None): """ Set default verbosity and debug levels config the logger """ - bb.msg.loggerDefaultDebugLevel = debug bb.msg.loggerDefaultVerbose = verbose if verbose: bb.msg.loggerVerboseLogs = True + if debug: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.DEBUG - debug + 1 + elif verbose: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.VERBOSE + else: + bb.msg.loggerDefaultLogLevel = BBLogFormatter.NOTE + bb.msg.loggerDefaultDomains = {} if debug_domains: for (domainarg, iterator) in groupby(debug_domains): @@ -156,17 +162,7 @@ def init_msgconfig(verbose, debug, debug_domains=None): bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1 def constructLogOptions(): - debug = loggerDefaultDebugLevel - verbose = loggerDefaultVerbose - - if debug: - level = BBLogFormatter.DEBUG - debug + 1 - elif verbose: - level = BBLogFormatter.VERBOSE - else: - level = BBLogFormatter.NOTE - - return level, loggerDefaultDomains + return loggerDefaultLogLevel, loggerDefaultDomains def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): level, debug_domains = constructLogOptions() diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 32966b4f7c..4106fa4bc4 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1254,7 +1254,7 @@ class RunQueue: "fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs, "fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv, "sigdata" : bb.parse.siggen.get_taskdata(), - "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel, + "logdefaultlevel" : bb.msg.loggerDefaultLogLevel, "logdefaultverbose" : bb.msg.loggerDefaultVerbose, "logdefaultverboselogs" : bb.msg.loggerVerboseLogs, "logdefaultdomain" : bb.msg.loggerDefaultDomains, -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:42 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:42 -0500 Subject: [bitbake-devel] [PATCH 04/15] bitbake: lib/bb/msg.py: Add repr for BBLogFormatter In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-5-JPEWhacker@gmail.com> Adds a __repr__ function for BBLogFormatter. This allows it to get a human readable string when printed using the logging_tree module Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index ea6a9543f7..c70fd80806 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -98,6 +98,9 @@ class BBLogFormatter(logging.Formatter): def enable_color(self): self.color_enabled = True + def __repr__(self): + return "%s fmt='%s' color=%s" % (self.__class__.__name__, self._fmt, "True" if self.color_enabled else "False") + class BBLogFilter(object): def __init__(self, handler, level, debug_domains): self.stdlevel = level -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:43 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:43 -0500 Subject: [bitbake-devel] [PATCH 05/15] bitbake: knotty: Add commented logging_tree code In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-6-JPEWhacker@gmail.com> Adds a comment section that can be easily uncommented to enable dumping the logging tree. This module is extremely useful for debugging issued with logging configuration Signed-off-by: Joshua Watt --- bitbake/lib/bb/ui/knotty.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index a6a92b920f..aac12cd479 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -430,6 +430,11 @@ def main(server, eventHandler, params, tf = TerminalFilter): llevel, debug_domains = bb.msg.constructLogOptions() server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list]) + # The logging_tree module is *extremely* helpful in debugging logging + # domains. Uncomment here to dump the logging tree when bitbake starts + #import logging_tree + #logging_tree.printout() + universe = False if not params.observe_only: params.updateFromServer(server) -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:44 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:44 -0500 Subject: [bitbake-devel] [PATCH 06/15] bitbake: lib/bb/msg.py: Add filter utilities In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-7-JPEWhacker@gmail.com> Adds generic filter utilities that can be used to filter when the log level is greater than or equal to a defined value, or below a defined value, as well as a generic function to translate a string to a logging level (or bitbake logging level) Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index c70fd80806..8561826a6e 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -135,6 +135,28 @@ class BBLogFilterStdOut(BBLogFilter): return True return False +class LogFilterGEQLevel(logging.Filter): + def __init__(self, level): + self.strlevel = str(level) + self.level = stringToLevel(level) + + def __repr__(self): + return "%s level >= %s (%d)" % (self.__class__.__name__, self.strlevel, self.level) + + def filter(self, record): + return (record.levelno >= self.level) + +class LogFilterLTLevel(logging.Filter): + def __init__(self, level): + self.strlevel = str(level) + self.level = stringToLevel(level) + + def __repr__(self): + return "%s level < %s (%d)" % (self.__class__.__name__, self.strlevel, self.level) + + def filter(self, record): + return (record.levelno < self.level) + # Message control functions # @@ -175,6 +197,19 @@ def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None): cls(handler, level, debug_domains) +def stringToLevel(level): + try: + return int(level) + except ValueError: + pass + + try: + return getattr(logging, level) + except AttributeError: + pass + + return getattr(BBLogFormatter, level) + # # Message handling functions # -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:45 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:45 -0500 Subject: [bitbake-devel] [PATCH 07/15] bitbake: lib/bb/msg.py: Remove unused filters In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-8-JPEWhacker@gmail.com> Now that the filter are described using the python logging structure, these classes are no longer needed. Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index 8561826a6e..cab079e333 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -119,22 +119,6 @@ class BBLogFilter(object): return True return False -class BBLogFilterStdErr(BBLogFilter): - def filter(self, record): - if not BBLogFilter.filter(self, record): - return False - if record.levelno >= logging.ERROR: - return True - return False - -class BBLogFilterStdOut(BBLogFilter): - def filter(self, record): - if not BBLogFilter.filter(self, record): - return False - if record.levelno < logging.ERROR: - return True - return False - class LogFilterGEQLevel(logging.Filter): def __init__(self, level): self.strlevel = str(level) -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:46 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:46 -0500 Subject: [bitbake-devel] [PATCH 08/15] bitbake: lib/bb/msg.py: Add helper to set logging config In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-9-JPEWhacker@gmail.com> Adds a helper function to setup the structure logging information in bitbake. The helper function takes a default configuration and an optional path to a user config file that can be merged into the default. Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 93 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index cab079e333..6259af037f 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -13,6 +13,7 @@ Message handling infrastructure for bitbake import sys import copy import logging +import logging.config from itertools import groupby import bb import bb.event @@ -227,3 +228,95 @@ def has_console_handler(logger): if handler.stream in [sys.stderr, sys.stdout]: return True return False + +def setLoggingConfig(defaultconfig, userconfigfile=None): + logconfig = copy.deepcopy(defaultconfig) + + if userconfigfile: + with open(userconfigfile, 'r') as f: + if userconfigfile.endswith('.yml') or userconfigfile.endswith('.yaml'): + import yaml + userconfig = yaml.load(f) + elif userconfigfile.endswith('.json') or userconfigfile.endswith('.cfg'): + import json + userconfig = json.load(f) + else: + raise BaseException("Unrecognized file format: %s" % userconfigfile) + + if userconfig.get('bitbake_merge', True): + # Merge config with the default config + if userconfig.get('version') != logconfig['version']: + raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version'))) + + # Set some defaults to make merging easier + userconfig.setdefault("loggers", {}) + + # If a handler, formatter, or filter is defined in the user + # config, it will replace an existing one in the default config + for k in ("handlers", "formatters", "filters"): + logconfig.setdefault(k, {}).update(userconfig.get(k, {})) + + seen_loggers = set() + for name, l in logconfig["loggers"].items(): + # If the merge option is set, merge the handlers and + # filters. Otherwise, if it is False, this logger won't get + # add to the set of seen loggers and will replace the + # existing one + if l.get('bitbake_merge', True): + ulogger = userconfig["loggers"].setdefault(name, {}) + ulogger.setdefault("handlers", []) + ulogger.setdefault("filters", []) + + # Merge lists + l.setdefault("handlers", []).extend(ulogger["handlers"]) + l.setdefault("filters", []).extend(ulogger["filters"]) + + # Replace other properties if present + if "level" in ulogger: + l["level"] = ulogger["level"] + + if "propagate" in ulogger: + l["propagate"] = ulogger["propagate"] + + seen_loggers.add(name) + + # Add all loggers present in the user config, but not any that + # have already been processed + for name in set(userconfig["loggers"].keys()) - seen_loggers: + logconfig["loggers"][name] = userconfig["loggers"][name] + + else: + # Replace the entire default config + logconfig = userconfig + + # Convert all level parameters to integers in case users want to use the + # bitbake defined level names + for h in logconfig["handlers"].values(): + if "level" in h: + h["level"] = bb.msg.stringToLevel(h["level"]) + + for l in logconfig["loggers"].values(): + if "level" in l: + l["level"] = bb.msg.stringToLevel(l["level"]) + + logging.config.dictConfig(logconfig) + + # The user may have specified logging domains they want at a higher debug + # level than the standard. + for name, l in logconfig["loggers"].items(): + if not name.startswith("BitBake."): + continue + + if not "level" in l: + continue + + curlevel = bb.msg.loggerDefaultDomains.get(name) + # Note: level parameter should already be a int because of conversion + # above + newlevel = int(l["level"]) + if curlevel is None or newlevel < curlevel: + bb.msg.loggerDefaultDomains[name] = newlevel + + # TODO: I don't think that setting the global log level should be necessary + #if newlevel < bb.msg.loggerDefaultLogLevel: + # bb.msg.loggerDefaultLogLevel = newlevel -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:47 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:47 -0500 Subject: [bitbake-devel] [PATCH 09/15] bitbake: knotty: Remove dependency on format variable In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-10-JPEWhacker@gmail.com> Passing around the log formatter variable was unnecessary since the log levels of interest can be accesses as class members of bb.msg.BBLogFormatter. Switching to do this will make using the structured python logging much easier, since it can be difficult to extract out the formatter for a specific handler. Signed-off-by: Joshua Watt --- bitbake/lib/bb/tinfoil.py | 4 +--- bitbake/lib/bb/ui/knotty.py | 32 ++++++++++++++++++++------------ 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py index 9560eb5b49..5c5be456e2 100644 --- a/bitbake/lib/bb/tinfoil.py +++ b/bitbake/lib/bb/tinfoil.py @@ -735,11 +735,9 @@ class Tinfoil: console = handler elif handler.stream == sys.stderr: errconsole = handler - format_str = "%(levelname)s: %(message)s" - format = bb.msg.BBLogFormatter(format_str) helper.shutdown = 0 parseprogress = None - termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet) + termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, quiet=self.quiet) try: while True: try: diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index aac12cd479..d5dce7172a 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -109,12 +109,11 @@ def pluralise(singular, plural, qty): class InteractConsoleLogFilter(logging.Filter): - def __init__(self, tf, format): + def __init__(self, tf): self.tf = tf - self.format = format def filter(self, record): - if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): + if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")): return False self.tf.clearFooter() return True @@ -150,7 +149,7 @@ class TerminalFilter(object): cr = (25, 80) return cr - def __init__(self, main, helper, console, errconsole, format, quiet): + def __init__(self, main, helper, console, errconsole, quiet): self.main = main self.helper = helper self.cuu = None @@ -180,7 +179,16 @@ class TerminalFilter(object): termios.tcsetattr(fd, termios.TCSADRAIN, new) curses.setupterm() if curses.tigetnum("colors") > 2: - format.enable_color() + if console: + try: + console.formatter.enable_color() + except AttributeError: + pass + if errconsole: + try: + errconsole.formatter.enable_color() + except AttributeError: + pass self.ed = curses.tigetstr("ed") if self.ed: self.cuu = curses.tigetstr("cuu") @@ -197,9 +205,9 @@ class TerminalFilter(object): bb.note("Unable to use interactive mode for this terminal, using fallback") return if console: - console.addFilter(InteractConsoleLogFilter(self, format)) + console.addFilter(InteractConsoleLogFilter(self)) if errconsole: - errconsole.addFilter(InteractConsoleLogFilter(self, format)) + errconsole.addFilter(InteractConsoleLogFilter(self)) self.main_progress = None @@ -469,7 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): printinterval = 5000 lastprint = time.time() - termfilter = tf(main, helper, console, errconsole, format, params.options.quiet) + termfilter = tf(main, helper, console, errconsole, params.options.quiet) atexit.register(termfilter.finish) while True: @@ -508,21 +516,21 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, logging.LogRecord): lastprint = time.time() printinterval = 5000 - if event.levelno >= format.ERROR: + if event.levelno >= bb.msg.BBLogFormatter.ERROR: errors = errors + 1 return_value = 1 - elif event.levelno == format.WARNING: + elif event.levelno == bb.msg.BBLogFormatter.WARNING: warnings = warnings + 1 if event.taskpid != 0: # For "normal" logging conditions, don't show note logs from tasks # but do show them if the user has changed the default log level to # include verbose/debug messages - if event.levelno <= format.NOTE and (event.levelno < llevel or (event.levelno == format.NOTE and llevel != format.VERBOSE)): + if event.levelno <= bb.msg.BBLogFormatter.NOTE and (event.levelno < llevel or (event.levelno == bb.msg.BBLogFormatter.NOTE and llevel != bb.msg.BBLogFormatter.VERBOSE)): continue # Prefix task messages with recipe/task - if event.taskpid in helper.pidmap and event.levelno != format.PLAIN: + if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN: taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]] event.msg = taskinfo['title'] + ': ' + event.msg if hasattr(event, 'fn'): -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:49 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:49 -0500 Subject: [bitbake-devel] [PATCH 11/15] bitbake: worker: Remove unnecessary addDefaultLogFilter In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-12-JPEWhacker@gmail.com> Adding the default log filter here is unnecessary because there are no defined logging domains when it is called, which means it does no actual filtering. Signed-off-by: Joshua Watt --- bitbake/bin/bitbake-worker | 1 - 1 file changed, 1 deletion(-) diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker index fc28a34d8a..97cc0fd60f 100755 --- a/bitbake/bin/bitbake-worker +++ b/bitbake/bin/bitbake-worker @@ -65,7 +65,6 @@ if 0: format_str = "%(levelname)s: %(message)s" conlogformat = bb.msg.BBLogFormatter(format_str) consolelog = logging.FileHandler(logfilename) - bb.msg.addDefaultlogFilter(consolelog) consolelog.setFormatter(conlogformat) logger.addHandler(consolelog) -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:48 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:48 -0500 Subject: [bitbake-devel] [PATCH 10/15] bitbake: knotty: Setup logs with config helper In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-11-JPEWhacker@gmail.com> Sets up logging in knotty to use python's structured logging config and the bb.msg.setLoggingConfig() helper to setup logging. This allows the user to specify additional logging mechanism in a config file with BB_LOGCONFIG Signed-off-by: Joshua Watt --- bitbake/lib/bb/ui/knotty.py | 134 ++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 38 deletions(-) diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index d5dce7172a..386f27876d 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -370,7 +370,11 @@ def _log_settings_from_server(server, observe_only): if error: logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error) raise BaseException(error) - return includelogs, loglines, consolelogfile + logconfigfile, error = server.runCommand([cmd, "BB_LOGCONFIG"]) + if error: + logger.error("Unable to get the value of BB_LOGCONFIG variable: %s" % error) + raise BaseException(error) + return includelogs, loglines, consolelogfile, logconfigfile _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.LogRecord", "bb.build.TaskFailed", "bb.build.TaskBase", "bb.event.ParseStarted", @@ -387,7 +391,87 @@ def main(server, eventHandler, params, tf = TerminalFilter): if not params.observe_only: params.updateToServer(server, os.environ.copy()) - includelogs, loglines, consolelogfile = _log_settings_from_server(server, params.observe_only) + includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only) + + loglevel, _ = bb.msg.constructLogOptions() + + if params.options.quiet == 0: + console_loglevel = loglevel + elif params.options.quiet > 2: + console_loglevel = bb.msg.BBLogFormatter.ERROR + else: + console_loglevel = bb.msg.BBLogFormatter.WARNING + + logconfig = { + "version": 1, + "handlers": { + "BitBake.console": { + "class": "logging.StreamHandler", + "formatter": "BitBake.consoleFormatter", + "level": console_loglevel, + "stream": "ext://sys.stdout", + "filters": ["BitBake.stdoutFilter"], + }, + "BitBake.errconsole": { + "class": "logging.StreamHandler", + "formatter": "BitBake.consoleFormatter", + "level": loglevel, + "stream": "ext://sys.stderr", + "filters": ["BitBake.stderrFilter"], + }, + }, + "formatters": { + # This format instance will get color output enabled by the + # terminal + "BitBake.consoleFormatter" : { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + }, + # The file log requires a separate instance so that it doesn't get + # color enabled + "BitBake.logfileFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + "filters": { + "BitBake.stdoutFilter": { + "()": "bb.msg.LogFilterLTLevel", + "level": "ERROR" + }, + "BitBake.stderrFilter": { + "()": "bb.msg.LogFilterGEQLevel", + "level": "ERROR" + } + }, + "loggers": { + "BitBake": { + "level": loglevel, + "handlers": ["BitBake.console", "BitBake.errconsole"], + } + }, + "disable_existing_loggers": False + } + + # Enable the console log file if enabled + if consolelogfile and not params.options.show_environment and not params.options.show_versions: + logconfig["handlers"]["BitBake.consolelog"] ={ + "class": "logging.FileHandler", + "formatter": "BitBake.logfileFormatter", + "level": "INFO", + "filename": consolelogfile, + } + logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog") + + bb.utils.mkdirhier(os.path.dirname(consolelogfile)) + loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') + bb.utils.remove(loglink) + try: + os.symlink(os.path.basename(consolelogfile), loglink) + except OSError: + pass + + bb.msg.setLoggingConfig(logconfig, logconfigfile) if sys.stdin.isatty() and sys.stdout.isatty(): log_exec_tty = True @@ -396,23 +480,16 @@ def main(server, eventHandler, params, tf = TerminalFilter): helper = uihelper.BBUIHelper() - console = logging.StreamHandler(sys.stdout) - errconsole = logging.StreamHandler(sys.stderr) - format_str = "%(levelname)s: %(message)s" - format = bb.msg.BBLogFormatter(format_str) - if params.options.quiet == 0: - forcelevel = None - elif params.options.quiet > 2: - forcelevel = bb.msg.BBLogFormatter.ERROR - else: - forcelevel = bb.msg.BBLogFormatter.WARNING - bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel) - bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr) - console.setFormatter(format) - errconsole.setFormatter(format) - if not bb.msg.has_console_handler(logger): - logger.addHandler(console) - logger.addHandler(errconsole) + # Look for the specially designated handlers which need to be passed to the + # terminal handler + console = None + errconsole = None + for h in logger.handlers: + name = getattr(h, '_name', None) + if name == 'BitBake.console': + console = h + elif name == 'BitBake.errconsole': + errconsole = h bb.utils.set_process_name("KnottyUI") @@ -420,21 +497,6 @@ def main(server, eventHandler, params, tf = TerminalFilter): server.terminateServer() return - consolelog = None - if consolelogfile and not params.options.show_environment and not params.options.show_versions: - bb.utils.mkdirhier(os.path.dirname(consolelogfile)) - conlogformat = bb.msg.BBLogFormatter(format_str) - consolelog = logging.FileHandler(consolelogfile) - bb.msg.addDefaultlogFilter(consolelog) - consolelog.setFormatter(conlogformat) - logger.addHandler(consolelog) - loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') - bb.utils.remove(loglink) - try: - os.symlink(os.path.basename(consolelogfile), loglink) - except OSError: - pass - llevel, debug_domains = bb.msg.constructLogOptions() server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list]) @@ -761,8 +823,4 @@ def main(server, eventHandler, params, tf = TerminalFilter): if e.errno == errno.EPIPE: pass - if consolelog: - logger.removeHandler(consolelog) - consolelog.close() - return return_value -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:50 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:50 -0500 Subject: [bitbake-devel] [PATCH 12/15] bitbake: Log hash equivalence with a different logger In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-13-JPEWhacker@gmail.com> Switches the hash equivalence logging to use a different logger so that it can be easily filtered out with python's structured logging. Signed-off-by: Joshua Watt --- bitbake/lib/bb/runqueue.py | 11 ++++++----- bitbake/lib/bb/siggen.py | 17 +++++++++-------- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 4106fa4bc4..e84890b8b7 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -27,6 +27,7 @@ import pprint bblogger = logging.getLogger("BitBake") logger = logging.getLogger("BitBake.RunQueue") +hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv") __find_sha256__ = re.compile( r'(?i)(?" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"") - logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed))) + hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed))) for tid in changed: if tid not in self.rqdata.runq_setscene_tids: @@ -2346,7 +2347,7 @@ class RunQueueExecute: # Check no tasks this covers are running for dep in self.sqdata.sq_covered_tasks[tid]: if dep in self.runq_running and dep not in self.runq_complete: - logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid)) + hashequiv_logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid)) valid = False break if not valid: @@ -2409,7 +2410,7 @@ class RunQueueExecute: for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: - logger.info("Setscene task %s became valid" % tid) + hashequiv_logger.info("Setscene task %s became valid" % tid) if harddepfail: self.sq_task_failoutright(tid) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index c2d0c736cf..0357b544b5 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -16,6 +16,7 @@ from bb import runqueue import hashserv logger = logging.getLogger('BitBake.SigGen') +hashequiv_logger = logging.getLogger('BitBake.SigGen.HashEquiv') def init(d): siggens = [obj for obj in globals().values() @@ -502,9 +503,9 @@ class SignatureGeneratorUniHashMixIn(object): # A unique hash equal to the taskhash is not very interesting, # so it is reported it at debug level 2. If they differ, that # is much more interesting, so it is reported at debug level 1 - bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server)) + hashequiv_logger.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server)) else: - bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server)) + hashequiv_logger.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) @@ -578,12 +579,12 @@ class SignatureGeneratorUniHashMixIn(object): new_unihash = data['unihash'] if new_unihash != unihash: - bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) + hashequiv_logger.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server)) bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d) self.set_unihash(tid, new_unihash) d.setVar('BB_UNIHASH', new_unihash) else: - bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) + hashequiv_logger.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) finally: @@ -606,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object): method = method + self.extramethod[tid] data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) - bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None: bb.warn("Server unable to handle unihash report") @@ -615,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object): finalunihash = data['unihash'] if finalunihash == current_unihash: - bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash)) elif finalunihash == wanted_unihash: - bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) self.set_unihash(tid, finalunihash) return True else: # TODO: What to do here? - bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) + hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:51 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:51 -0500 Subject: [bitbake-devel] [PATCH 13/15] bitbake: Add autobuilder logging configuration In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-14-JPEWhacker@gmail.com> Adds a configuration file that the autobuilder can use to capture interesting logging domains above the ones that show up for normal users on stdout/stderr. Signed-off-by: Joshua Watt --- bitbake/contrib/autobuilderlog.json | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 bitbake/contrib/autobuilderlog.json diff --git a/bitbake/contrib/autobuilderlog.json b/bitbake/contrib/autobuilderlog.json new file mode 100644 index 0000000000..103a1141f6 --- /dev/null +++ b/bitbake/contrib/autobuilderlog.json @@ -0,0 +1,27 @@ +{ + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "autobuilder.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s" + } + }, + "loggers": { + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + } + } +} -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:52 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:52 -0500 Subject: [bitbake-devel] [PATCH 14/15] bitbake: Lower hash equivalence logging In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-15-JPEWhacker@gmail.com> Lowers the level at which hash equivalence messages are logged so as to not annoy the majority of users. The autobuilder can use a custom logging configuration to log these to a file for debugging (see contrib/autobuilderlog.json) [YOCTO #13813] Signed-off-by: Joshua Watt --- bitbake/lib/bb/runqueue.py | 6 +++--- bitbake/lib/bb/siggen.py | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index e84890b8b7..cef9b0fbbf 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -2264,7 +2264,7 @@ class RunQueueExecute: self.updated_taskhash_queue.remove((tid, unihash)) if unihash != self.rqdata.runtaskentries[tid].unihash: - hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash)) + hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash)) self.rqdata.runtaskentries[tid].unihash = unihash bb.parse.siggen.set_unihash(tid, unihash) toprocess.add(tid) @@ -2309,7 +2309,7 @@ class RunQueueExecute: elif tid in self.scenequeue_covered or tid in self.sq_live: # Already ran this setscene task or it running. Report the new taskhash bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches) - hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) + hashequiv_logger.verbose("Already covered setscene for %s so ignoring rehash (remap)" % (tid)) remapped = True if not remapped: @@ -2410,7 +2410,7 @@ class RunQueueExecute: for (tid, harddepfail, origvalid) in update_tasks: if tid in self.sqdata.valid and not origvalid: - hashequiv_logger.info("Setscene task %s became valid" % tid) + hashequiv_logger.verbose("Setscene task %s became valid" % tid) if harddepfail: self.sq_task_failoutright(tid) diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 0357b544b5..8bfc45235d 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -607,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object): method = method + self.extramethod[tid] data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data) - hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) + hashequiv_logger.verbose('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data))) if data is None: bb.warn("Server unable to handle unihash report") @@ -616,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object): finalunihash = data['unihash'] if finalunihash == current_unihash: - hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash)) + hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash)) elif finalunihash == wanted_unihash: - hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) + hashequiv_logger.verbose('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash)) self.set_unihash(tid, finalunihash) return True else: # TODO: What to do here? - hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) + hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash)) except hashserv.client.HashConnectionError as e: bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e))) -- 2.17.1 From jpewhacker at gmail.com Mon Mar 9 16:33:53 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Mon, 9 Mar 2020 11:33:53 -0500 Subject: [bitbake-devel] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG In-Reply-To: <20200309163353.15362-1-JPEWhacker@gmail.com> References: <20200309163353.15362-1-JPEWhacker@gmail.com> Message-ID: <20200309163353.15362-16-JPEWhacker@gmail.com> Adds documentation describing how to use the BB_LOGCONFIG variable to enable custom logging. Signed-off-by: Joshua Watt --- .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++ .../bitbake-user-manual-ref-variables.xml | 11 +++ 2 files changed, 108 insertions(+) diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml index 8f606676b4..3b31f748cc 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml @@ -929,4 +929,101 @@ section. + +
+ Logging + + In addition to the standard command line option to control how + verbose builds are when execute, bitbake also supports user defined + configuration of the + Python logging + facilities through the + BB_LOGCONFIG + variable. This variable defines a json or yaml + logging configuration + that will be intelligently merged into the default configuration. + The logging configuration is merged using the following rules: + + + The user defined configuration will completely replace the default + configuration if top level key + bitbake_merge is set to the value + False. In this case, all other rules + are ignored. + + + The user configuration must have a top level + version which must match the value of + the default configuration. + + + Any keys defined in the handlers, + formatters, or filters, + will be merged into the same section in the default + configuration, with the user specified keys taking + replacing a default one if there is a conflict. In + practice, this means that if both the default configuration + and user configuration specify a handler named + myhandler, the user defined one will + replace the default. To prevent the user from inadvertently + replacing a default handler, formatter, or filter, all of + the default ones are named with a prefix of + "BitBake." + + + If a logger is defined by the user with the key + bitbake_merge set to + False, that logger will be completely + replaced by user configuration. In this case, no other + rules will apply to that logger. + + + All user defined filter and + handlers properties for a given logger + will be merged with corresponding properties from the + default logger. For example, if the user configuration adds + a filter called myFilter to the + BitBake.SigGen, and the default + configuration adds a filter called + BitBake.defaultFilter, both filters + will be applied to the logger + + + + + + As an example, consider the following user logging configuration + file which logs all Hash Equivalence related messages of VERBOSE or + higher to a file called hashequiv.log + + { + "version": 1, + "handlers": { + "autobuilderlog": { + "class": "logging.FileHandler", + "formatter": "logfileFormatter", + "level": "DEBUG", + "filename": "hashequiv.log", + "mode": "w" + } + }, + "formatters": { + "logfileFormatter": { + "format": "%(name)s: %(levelname)s: %(message)s" + } + }, + "loggers": { + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["autobuilderlog"] + } + } + } + + +
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml index bae01d90c0..c4bd1f2584 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml @@ -539,6 +539,17 @@ + BB_LOGCONFIG + + + Specifies the name of a config file that contains the user + logging configuration. See + Logging for additional + information + + + + BB_LOGFMT -- 2.17.1 From jpewhacker at gmail.com Wed Mar 11 23:28:43 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 11 Mar 2020 18:28:43 -0500 Subject: [bitbake-devel] [PATCH 0/4] More Bitbake logging improvements Message-ID: <20200311232847.2874-1-JPEWhacker@gmail.com> Supplemental patches to improve logging in bitbake by using python structured logging. Fixes up some ResourceWarnings and improves the flexibility of the console and consolelog. Joshua Watt (4): bitbake: knotty: Add logging cleanup bitbake: msg: Add helper to merge logging configs bitbake: knotty: Update hash equivalence logging bitbake: runqueue: Lower setscene complete logging level bitbake/contrib/autobuilderlog.json | 18 +---- bitbake/lib/bb/msg.py | 100 ++++++++++++++++------------ bitbake/lib/bb/runqueue.py | 2 +- bitbake/lib/bb/ui/knotty.py | 70 ++++++++++++++++--- 4 files changed, 124 insertions(+), 66 deletions(-) -- 2.17.1 From jpewhacker at gmail.com Wed Mar 11 23:28:44 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 11 Mar 2020 18:28:44 -0500 Subject: [bitbake-devel] [PATCH 1/4] bitbake: knotty: Add logging cleanup In-Reply-To: <20200311232847.2874-1-JPEWhacker@gmail.com> References: <20200311232847.2874-1-JPEWhacker@gmail.com> Message-ID: <20200311232847.2874-2-JPEWhacker@gmail.com> Adds code to close all loggers when bitbake exits. This prevents unclosed file ResourceWarnings. A form of this closing existed previously, but was removed in the new logging code. Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 11 +++++++++++ bitbake/lib/bb/ui/knotty.py | 2 ++ 2 files changed, 13 insertions(+) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index ced605d96d..ee3c38b5a4 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -284,3 +284,14 @@ def setLoggingConfig(defaultconfig, userconfigfile=None): # TODO: I don't think that setting the global log level should be necessary #if newlevel < bb.msg.loggerDefaultLogLevel: # bb.msg.loggerDefaultLogLevel = newlevel + +def cleanupLogging(): + # Iterate through all the handlers and close them if possible. Fixes + # 'Unclosed resource' warnings when bitbake exits, see + # https://bugs.python.org/issue23010 + handlers = set() + for logger_iter in logging.Logger.manager.loggerDict.keys(): + handlers.update(logging.getLogger(logger_iter).handlers) + + for h in handlers: + h.close() diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index b229687ef8..2dfce90a4f 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -823,4 +823,6 @@ def main(server, eventHandler, params, tf = TerminalFilter): if e.errno == errno.EPIPE: pass + bb.msg.cleanupLogging() + return return_value -- 2.17.1 From jpewhacker at gmail.com Wed Mar 11 23:28:45 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 11 Mar 2020 18:28:45 -0500 Subject: [bitbake-devel] [PATCH 2/4] bitbake: msg: Add helper to merge logging configs In-Reply-To: <20200311232847.2874-1-JPEWhacker@gmail.com> References: <20200311232847.2874-1-JPEWhacker@gmail.com> Message-ID: <20200311232847.2874-3-JPEWhacker@gmail.com> Adds a function that can be used by UI front ends to merge logging configuration fragments. Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 89 +++++++++++++++++++++++-------------------- 1 file changed, 48 insertions(+), 41 deletions(-) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index ee3c38b5a4..c22d0d3e15 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -193,6 +193,53 @@ def has_console_handler(logger): return True return False +def mergeLoggingConfig(logconfig, userconfig): + logconfig = copy.deepcopy(logconfig) + userconfig = copy.deepcopy(userconfig) + + # Merge config with the default config + if userconfig.get('version') != logconfig['version']: + raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version'))) + + # Set some defaults to make merging easier + userconfig.setdefault("loggers", {}) + + # If a handler, formatter, or filter is defined in the user + # config, it will replace an existing one in the default config + for k in ("handlers", "formatters", "filters"): + logconfig.setdefault(k, {}).update(userconfig.get(k, {})) + + seen_loggers = set() + for name, l in logconfig["loggers"].items(): + # If the merge option is set, merge the handlers and + # filters. Otherwise, if it is False, this logger won't get + # add to the set of seen loggers and will replace the + # existing one + if l.get('bitbake_merge', True): + ulogger = userconfig["loggers"].setdefault(name, {}) + ulogger.setdefault("handlers", []) + ulogger.setdefault("filters", []) + + # Merge lists + l.setdefault("handlers", []).extend(ulogger["handlers"]) + l.setdefault("filters", []).extend(ulogger["filters"]) + + # Replace other properties if present + if "level" in ulogger: + l["level"] = ulogger["level"] + + if "propagate" in ulogger: + l["propagate"] = ulogger["propagate"] + + seen_loggers.add(name) + + # Add all loggers present in the user config, but not any that + # have already been processed + for name in set(userconfig["loggers"].keys()) - seen_loggers: + logconfig["loggers"][name] = userconfig["loggers"][name] + + return logconfig + def setLoggingConfig(defaultconfig, userconfigfile=None): logconfig = copy.deepcopy(defaultconfig) @@ -208,47 +255,7 @@ def setLoggingConfig(defaultconfig, userconfigfile=None): raise BaseException("Unrecognized file format: %s" % userconfigfile) if userconfig.get('bitbake_merge', True): - # Merge config with the default config - if userconfig.get('version') != logconfig['version']: - raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version'))) - - # Set some defaults to make merging easier - userconfig.setdefault("loggers", {}) - - # If a handler, formatter, or filter is defined in the user - # config, it will replace an existing one in the default config - for k in ("handlers", "formatters", "filters"): - logconfig.setdefault(k, {}).update(userconfig.get(k, {})) - - seen_loggers = set() - for name, l in logconfig["loggers"].items(): - # If the merge option is set, merge the handlers and - # filters. Otherwise, if it is False, this logger won't get - # add to the set of seen loggers and will replace the - # existing one - if l.get('bitbake_merge', True): - ulogger = userconfig["loggers"].setdefault(name, {}) - ulogger.setdefault("handlers", []) - ulogger.setdefault("filters", []) - - # Merge lists - l.setdefault("handlers", []).extend(ulogger["handlers"]) - l.setdefault("filters", []).extend(ulogger["filters"]) - - # Replace other properties if present - if "level" in ulogger: - l["level"] = ulogger["level"] - - if "propagate" in ulogger: - l["propagate"] = ulogger["propagate"] - - seen_loggers.add(name) - - # Add all loggers present in the user config, but not any that - # have already been processed - for name in set(userconfig["loggers"].keys()) - seen_loggers: - logconfig["loggers"][name] = userconfig["loggers"][name] - + logconfig = mergeLoggingConfig(logconfig, userconfig) else: # Replace the entire default config logconfig = userconfig -- 2.17.1 From jpewhacker at gmail.com Wed Mar 11 23:28:46 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 11 Mar 2020 18:28:46 -0500 Subject: [bitbake-devel] [PATCH 3/4] bitbake: knotty: Update hash equivalence logging In-Reply-To: <20200311232847.2874-1-JPEWhacker@gmail.com> References: <20200311232847.2874-1-JPEWhacker@gmail.com> Message-ID: <20200311232847.2874-4-JPEWhacker@gmail.com> Updates hash equivalence logging so that the interesting VERBOSE messages are always logged to the consolelog file so that issues in individual user builds can be diagnosed. The autobuilder logging config then updates this so that they also are shown on stdout, since the consolelog file is not capture there. In order to facilitate this, 2 new logging handlers were added, "BitBake.verbconsole" and "BitBake.verbconsolelog". Neither of these handlers are attached to anything by default, but they will log any messages that wouldn't otherwise be logged by the normal console or consolelog handlers. Users can attach whatever loggers the desire to this handler to get them to appear on the console or in the consolelog, as demonstrated by the autobuilderlog.json file. Signed-off-by: Joshua Watt --- bitbake/contrib/autobuilderlog.json | 18 +------- bitbake/lib/bb/ui/knotty.py | 68 +++++++++++++++++++++++++---- 2 files changed, 62 insertions(+), 24 deletions(-) diff --git a/bitbake/contrib/autobuilderlog.json b/bitbake/contrib/autobuilderlog.json index 103a1141f6..193a675a1f 100644 --- a/bitbake/contrib/autobuilderlog.json +++ b/bitbake/contrib/autobuilderlog.json @@ -1,27 +1,13 @@ { "version": 1, - "handlers": { - "autobuilderlog": { - "class": "logging.FileHandler", - "formatter": "logfileFormatter", - "level": "DEBUG", - "filename": "autobuilder.log", - "mode": "w" - } - }, - "formatters": { - "logfileFormatter": { - "format": "%(name)s: %(levelname)s: %(message)s" - } - }, "loggers": { "BitBake.SigGen.HashEquiv": { "level": "VERBOSE", - "handlers": ["autobuilderlog"] + "handlers": ["BitBake.verbconsole"] }, "BitBake.RunQueue.HashEquiv": { "level": "VERBOSE", - "handlers": ["autobuilderlog"] + "handlers": ["BitBake.verbconsole"] } } } diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index 2dfce90a4f..b4df5f6e88 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -419,6 +419,18 @@ def main(server, eventHandler, params, tf = TerminalFilter): "stream": "ext://sys.stderr", "filters": ["BitBake.stderrFilter"], }, + # This handler can be used if specific loggers should print on + # the console at a lower severity than the default. It will + # display any messages sent to it that are lower than then + # BitBake.console logging level (so as to prevent duplication of + # messages). Nothing is attached to this handler by default + "BitBake.verbconsole": { + "class": "logging.StreamHandler", + "formatter": "BitBake.consoleFormatter", + "level": 1, + "stream": "ext://sys.stdout", + "filters": ["BitBake.verbconsoleFilter"], + }, }, "formatters": { # This format instance will get color output enabled by the @@ -442,7 +454,11 @@ def main(server, eventHandler, params, tf = TerminalFilter): "BitBake.stderrFilter": { "()": "bb.msg.LogFilterGEQLevel", "level": "ERROR" - } + }, + "BitBake.verbconsoleFilter": { + "()": "bb.msg.LogFilterLTLevel", + "level": console_loglevel + }, }, "loggers": { "BitBake": { @@ -455,13 +471,49 @@ def main(server, eventHandler, params, tf = TerminalFilter): # Enable the console log file if enabled if consolelogfile and not params.options.show_environment and not params.options.show_versions: - logconfig["handlers"]["BitBake.consolelog"] ={ - "class": "logging.FileHandler", - "formatter": "BitBake.logfileFormatter", - "level": "INFO", - "filename": consolelogfile, - } - logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog") + logconfig = bb.msg.mergeLoggingConfig(logconfig, { + "version": 1, + "handlers" : { + "BitBake.consolelog": { + "class": "logging.FileHandler", + "formatter": "BitBake.logfileFormatter", + "level": loglevel, + "filename": consolelogfile, + }, + # Just like verbconsole, anything sent here will go to the + # log file, unless it would go to BitBake.consolelog + "BitBake.verbconsolelog" : { + "class": "logging.FileHandler", + "formatter": "BitBake.logfileFormatter", + "level": 1, + "filename": consolelogfile, + "filters": ["BitBake.verbconsolelogFilter"], + }, + }, + "filters": { + "BitBake.verbconsolelogFilter": { + "()": "bb.msg.LogFilterLTLevel", + "level": loglevel, + }, + }, + "loggers": { + "BitBake": { + "handlers": ["BitBake.consolelog"], + }, + + # Other interesting things that we want to keep an eye on + # in the log files in case someone has an issue, but not + # necessarily show to the user on the console + "BitBake.SigGen.HashEquiv": { + "level": "VERBOSE", + "handlers": ["BitBake.verbconsolelog"], + }, + "BitBake.RunQueue.HashEquiv": { + "level": "VERBOSE", + "handlers": ["BitBake.verbconsolelog"], + } + } + }) bb.utils.mkdirhier(os.path.dirname(consolelogfile)) loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log') -- 2.17.1 From jpewhacker at gmail.com Wed Mar 11 23:28:47 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Wed, 11 Mar 2020 18:28:47 -0500 Subject: [bitbake-devel] [PATCH 4/4] bitbake: runqueue: Lower setscene complete logging level In-Reply-To: <20200311232847.2874-1-JPEWhacker@gmail.com> References: <20200311232847.2874-1-JPEWhacker@gmail.com> Message-ID: <20200311232847.2874-5-JPEWhacker@gmail.com> Lowers the level of the log message when setscene tasks have completed. This message can occur multiple times when hash equivalence is enabled, since the runqueue switches between executing setscene tasks and normal tasks. Since this is primarily of use when debugging hash equivalence, use the hash equivalence logger at VERBOSE level. [YOCTO #13813] Signed-off-by: Joshua Watt --- bitbake/lib/bb/runqueue.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index cef9b0fbbf..16f076f3b1 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -2059,7 +2059,7 @@ class RunQueueExecute: self.update_holdofftasks() if not self.sq_live and not self.sqdone and not self.sq_deferred and not self.updated_taskhash_queue and not self.holdoff_tasks: - logger.info("Setscene tasks completed") + hashequiv_logger.verbose("Setscene tasks completed") err = self.summarise_scenequeue_errors() if err: -- 2.17.1 From jpewhacker at gmail.com Thu Mar 12 18:30:01 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Thu, 12 Mar 2020 13:30:01 -0500 Subject: [bitbake-devel] [PATCH 0/3] More logging improvements Message-ID: <20200312183004.8339-1-JPEWhacker@gmail.com> A few more logging improvements for bitbake. Joshua Watt (3): bitbake: msg: Return config object bitbake: knotty: Treat verbconsole as a console output bitbake: Use logging.shutdown() instead of bb.msg.cleanupLogging() bitbake/lib/bb/msg.py | 14 +++---------- bitbake/lib/bb/ui/knotty.py | 42 +++++++++++++++++-------------------- 2 files changed, 22 insertions(+), 34 deletions(-) -- 2.17.1 From jpewhacker at gmail.com Thu Mar 12 18:30:02 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Thu, 12 Mar 2020 13:30:02 -0500 Subject: [bitbake-devel] [PATCH 1/3] bitbake: msg: Return config object In-Reply-To: <20200312183004.8339-1-JPEWhacker@gmail.com> References: <20200312183004.8339-1-JPEWhacker@gmail.com> Message-ID: <20200312183004.8339-2-JPEWhacker@gmail.com> Returns the configuration object from setLoggingConfig(). This object has a config dictionary that contains all of the created handlers, filters and loggers, which makes it much easier to pull out items with specific names. Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index c22d0d3e15..3f95e21cc1 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -270,7 +270,8 @@ def setLoggingConfig(defaultconfig, userconfigfile=None): if "level" in l: l["level"] = bb.msg.stringToLevel(l["level"]) - logging.config.dictConfig(logconfig) + conf = logging.config.dictConfigClass(logconfig) + conf.configure() # The user may have specified logging domains they want at a higher debug # level than the standard. @@ -292,6 +293,8 @@ def setLoggingConfig(defaultconfig, userconfigfile=None): #if newlevel < bb.msg.loggerDefaultLogLevel: # bb.msg.loggerDefaultLogLevel = newlevel + return conf + def cleanupLogging(): # Iterate through all the handlers and close them if possible. Fixes # 'Unclosed resource' warnings when bitbake exits, see -- 2.17.1 From jpewhacker at gmail.com Thu Mar 12 18:30:03 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Thu, 12 Mar 2020 13:30:03 -0500 Subject: [bitbake-devel] [PATCH 2/3] bitbake: knotty: Treat verbconsole as a console output In-Reply-To: <20200312183004.8339-1-JPEWhacker@gmail.com> References: <20200312183004.8339-1-JPEWhacker@gmail.com> Message-ID: <20200312183004.8339-3-JPEWhacker@gmail.com> The BitBake.verbconsole needs to be treated like a console output logger (meaning that the TerminalFilter attaches an InteractConsoleLogFilter to it), even if it's not directly attached to the root 'BitBake' logger. First, assign a special "is_console" property to the relevant handlers, then look for the property in the handlers from the configuration object return by bb.msg.setLoggingConfig(). Finally, pass the list of all handlers to the TerminalFilter object; it doesn't care about the difference between console and errconsole, so pass all the relevant handlers as a list. This fixes cases where the console output was corrupted when messages were sent to the 'BitBake.verbconsole' handler. Signed-off-by: Joshua Watt --- bitbake/lib/bb/ui/knotty.py | 40 +++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 22 deletions(-) diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index b4df5f6e88..db4511844a 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -149,7 +149,7 @@ class TerminalFilter(object): cr = (25, 80) return cr - def __init__(self, main, helper, console, errconsole, quiet): + def __init__(self, main, helper, handlers, quiet): self.main = main self.helper = helper self.cuu = None @@ -179,14 +179,9 @@ class TerminalFilter(object): termios.tcsetattr(fd, termios.TCSADRAIN, new) curses.setupterm() if curses.tigetnum("colors") > 2: - if console: + for h in handlers: try: - console.formatter.enable_color() - except AttributeError: - pass - if errconsole: - try: - errconsole.formatter.enable_color() + h.formatter.enable_color() except AttributeError: pass self.ed = curses.tigetstr("ed") @@ -204,10 +199,9 @@ class TerminalFilter(object): self.interactive = False bb.note("Unable to use interactive mode for this terminal, using fallback") return - if console: - console.addFilter(InteractConsoleLogFilter(self)) - if errconsole: - errconsole.addFilter(InteractConsoleLogFilter(self)) + + for h in handlers: + h.addFilter(InteractConsoleLogFilter(self)) self.main_progress = None @@ -411,6 +405,9 @@ def main(server, eventHandler, params, tf = TerminalFilter): "level": console_loglevel, "stream": "ext://sys.stdout", "filters": ["BitBake.stdoutFilter"], + ".": { + "is_console": True, + }, }, "BitBake.errconsole": { "class": "logging.StreamHandler", @@ -418,6 +415,9 @@ def main(server, eventHandler, params, tf = TerminalFilter): "level": loglevel, "stream": "ext://sys.stderr", "filters": ["BitBake.stderrFilter"], + ".": { + "is_console": True, + }, }, # This handler can be used if specific loggers should print on # the console at a lower severity than the default. It will @@ -430,6 +430,9 @@ def main(server, eventHandler, params, tf = TerminalFilter): "level": 1, "stream": "ext://sys.stdout", "filters": ["BitBake.verbconsoleFilter"], + ".": { + "is_console": True, + }, }, }, "formatters": { @@ -523,7 +526,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): except OSError: pass - bb.msg.setLoggingConfig(logconfig, logconfigfile) + conf = bb.msg.setLoggingConfig(logconfig, logconfigfile) if sys.stdin.isatty() and sys.stdout.isatty(): log_exec_tty = True @@ -534,14 +537,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): # Look for the specially designated handlers which need to be passed to the # terminal handler - console = None - errconsole = None - for h in logger.handlers: - name = getattr(h, '_name', None) - if name == 'BitBake.console': - console = h - elif name == 'BitBake.errconsole': - errconsole = h + console_handlers = [h for h in conf.config['handlers'].values() if getattr(h, 'is_console', False)] bb.utils.set_process_name("KnottyUI") @@ -591,7 +587,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): printinterval = 5000 lastprint = time.time() - termfilter = tf(main, helper, console, errconsole, params.options.quiet) + termfilter = tf(main, helper, console_handlers, params.options.quiet) atexit.register(termfilter.finish) while True: -- 2.17.1 From jpewhacker at gmail.com Thu Mar 12 18:30:04 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Thu, 12 Mar 2020 13:30:04 -0500 Subject: [bitbake-devel] [PATCH 3/3] bitbake: Use logging.shutdown() instead of bb.msg.cleanupLogging() In-Reply-To: <20200312183004.8339-1-JPEWhacker@gmail.com> References: <20200312183004.8339-1-JPEWhacker@gmail.com> Message-ID: <20200312183004.8339-4-JPEWhacker@gmail.com> The logging module provides a shutdown() function that does the same thing in a much better way Signed-off-by: Joshua Watt --- bitbake/lib/bb/msg.py | 11 ----------- bitbake/lib/bb/ui/knotty.py | 2 +- 2 files changed, 1 insertion(+), 12 deletions(-) diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py index 3f95e21cc1..29f0a3999e 100644 --- a/bitbake/lib/bb/msg.py +++ b/bitbake/lib/bb/msg.py @@ -294,14 +294,3 @@ def setLoggingConfig(defaultconfig, userconfigfile=None): # bb.msg.loggerDefaultLogLevel = newlevel return conf - -def cleanupLogging(): - # Iterate through all the handlers and close them if possible. Fixes - # 'Unclosed resource' warnings when bitbake exits, see - # https://bugs.python.org/issue23010 - handlers = set() - for logger_iter in logging.Logger.manager.loggerDict.keys(): - handlers.update(logging.getLogger(logger_iter).handlers) - - for h in handlers: - h.close() diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py index db4511844a..33ee891256 100644 --- a/bitbake/lib/bb/ui/knotty.py +++ b/bitbake/lib/bb/ui/knotty.py @@ -871,6 +871,6 @@ def main(server, eventHandler, params, tf = TerminalFilter): if e.errno == errno.EPIPE: pass - bb.msg.cleanupLogging() + logging.shutdown() return return_value -- 2.17.1 From jpewhacker at gmail.com Fri Mar 13 01:42:23 2020 From: jpewhacker at gmail.com (Joshua Watt) Date: Thu, 12 Mar 2020 20:42:23 -0500 Subject: [bitbake-devel] [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG In-Reply-To: <172109D8-BB79-48CB-93EB-C65C07EAE62A@stacktrust.org> References: <20200309163353.15362-16-JPEWhacker@gmail.com> <172109D8-BB79-48CB-93EB-C65C07EAE62A@stacktrust.org> Message-ID: On Thu, Mar 12, 2020 at 6:50 PM Rich Persaud wrote: > > On Mar 9, 2020, at 12:34, Joshua Watt wrote: > > > > ?Adds documentation describing how to use the BB_LOGCONFIG variable to > > enable custom logging. > > > > Signed-off-by: Joshua Watt > > --- > > .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++ > > .../bitbake-user-manual-ref-variables.xml | 11 +++ > > 2 files changed, 108 insertions(+) > > > > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > > index 8f606676b4..3b31f748cc 100644 > > --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > > +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > > @@ -929,4 +929,101 @@ > > section. > > > > > > + > > +
> > + Logging > > + > > + In addition to the standard command line option to control how > > + verbose builds are when execute, bitbake also supports user defined > > + configuration of the > > + Python logging > > + facilities through the > > + BB_LOGCONFIG > > + variable. This variable defines a json or yaml > > + logging configuration > > + that will be intelligently merged into the default configuration. > > + The logging configuration is merged using the following rules: > > + > > + > > + The user defined configuration will completely replace the default > > + configuration if top level key > > + bitbake_merge is set to the value > > + False. In this case, all other rules > > + are ignored. > > + > > + > > + The user configuration must have a top level > > + version which must match the value of > > + the default configuration. > > + > > + > > + Any keys defined in the handlers, > > + formatters, or filters, > > + will be merged into the same section in the default > > + configuration, with the user specified keys taking > > + replacing a default one if there is a conflict. In > > + practice, this means that if both the default configuration > > + and user configuration specify a handler named > > + myhandler, the user defined one will > > + replace the default. To prevent the user from inadvertently > > + replacing a default handler, formatter, or filter, all of > > + the default ones are named with a prefix of > > + "BitBake." > > + > > + > > + If a logger is defined by the user with the key > > + bitbake_merge set to > > + False, that logger will be completely > > + replaced by user configuration. In this case, no other > > + rules will apply to that logger. > > + > > + > > + All user defined filter and > > + handlers properties for a given logger > > + will be merged with corresponding properties from the > > + default logger. For example, if the user configuration adds > > + a filter called myFilter to the > > + BitBake.SigGen, and the default > > + configuration adds a filter called > > + BitBake.defaultFilter, both filters > > + will be applied to the logger > > + > > + > > + > > + > > + > > + As an example, consider the following user logging configuration > > + file which logs all Hash Equivalence related messages of VERBOSE or > > + higher to a file called hashequiv.log > > + > > + { > > + "version": 1, > > + "handlers": { > > + "autobuilderlog": { > > + "class": "logging.FileHandler", > > + "formatter": "logfileFormatter", > > + "level": "DEBUG", > > + "filename": "hashequiv.log", > > + "mode": "w" > > + } > > + }, > > + "formatters": { > > + "logfileFormatter": { > > + "format": "%(name)s: %(levelname)s: %(message)s" > > What's the syntax for including a timestamp in the log message? Or is that implicit? It's not implicit, but you can add it easily with %(asctime)s. There are also several other options: https://docs.python.org/3/library/logging.html#logrecord-attributes > > Rich From rp at stacktrust.org Thu Mar 12 23:50:49 2020 From: rp at stacktrust.org (Rich Persaud) Date: Thu, 12 Mar 2020 19:50:49 -0400 Subject: [bitbake-devel] [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG In-Reply-To: <20200309163353.15362-16-JPEWhacker@gmail.com> References: <20200309163353.15362-16-JPEWhacker@gmail.com> Message-ID: <172109D8-BB79-48CB-93EB-C65C07EAE62A@stacktrust.org> On Mar 9, 2020, at 12:34, Joshua Watt wrote: > > ?Adds documentation describing how to use the BB_LOGCONFIG variable to > enable custom logging. > > Signed-off-by: Joshua Watt > --- > .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++ > .../bitbake-user-manual-ref-variables.xml | 11 +++ > 2 files changed, 108 insertions(+) > > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > index 8f606676b4..3b31f748cc 100644 > --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml > @@ -929,4 +929,101 @@ > section. > >
> + > +
> + Logging > + > + In addition to the standard command line option to control how > + verbose builds are when execute, bitbake also supports user defined > + configuration of the > + Python logging > + facilities through the > + BB_LOGCONFIG > + variable. This variable defines a json or yaml > + logging configuration > + that will be intelligently merged into the default configuration. > + The logging configuration is merged using the following rules: > + > + > + The user defined configuration will completely replace the default > + configuration if top level key > + bitbake_merge is set to the value > + False. In this case, all other rules > + are ignored. > + > + > + The user configuration must have a top level > + version which must match the value of > + the default configuration. > + > + > + Any keys defined in the handlers, > + formatters, or filters, > + will be merged into the same section in the default > + configuration, with the user specified keys taking > + replacing a default one if there is a conflict. In > + practice, this means that if both the default configuration > + and user configuration specify a handler named > + myhandler, the user defined one will > + replace the default. To prevent the user from inadvertently > + replacing a default handler, formatter, or filter, all of > + the default ones are named with a prefix of > + "BitBake." > + > + > + If a logger is defined by the user with the key > + bitbake_merge set to > + False, that logger will be completely > + replaced by user configuration. In this case, no other > + rules will apply to that logger. > + > + > + All user defined filter and > + handlers properties for a given logger > + will be merged with corresponding properties from the > + default logger. For example, if the user configuration adds > + a filter called myFilter to the > + BitBake.SigGen, and the default > + configuration adds a filter called > + BitBake.defaultFilter, both filters > + will be applied to the logger > + > + > + > + > + > + As an example, consider the following user logging configuration > + file which logs all Hash Equivalence related messages of VERBOSE or > + higher to a file called hashequiv.log > + > + { > + "version": 1, > + "handlers": { > + "autobuilderlog": { > + "class": "logging.FileHandler", > + "formatter": "logfileFormatter", > + "level": "DEBUG", > + "filename": "hashequiv.log", > + "mode": "w" > + } > + }, > + "formatters": { > + "logfileFormatter": { > + "format": "%(name)s: %(levelname)s: %(message)s" What's the syntax for including a timestamp in the log message? Or is that implicit? Rich From akuster808 at gmail.com Fri Mar 13 18:03:29 2020 From: akuster808 at gmail.com (akuster808) Date: Fri, 13 Mar 2020 11:03:29 -0700 Subject: [bitbake-devel] Thud community support Message-ID: The Poky thud branch is now under 'Community support' and is seeking a maintainer to continue supporting the following repos for the thud release: bitbake, core, meta-yocto and yocto-docs. There will be no more point releases. see https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#Maintainer_Procedures? to get an idea on what is expected. The one thing we (YP) don't know is if there will be AutoBuilder resources available.? Please contact me or Richard regarding that statement. If no one steps up by April 24th (six weeks -ish from now), Thud will be moved to EOL status. - Armin From mhalstead at linuxfoundation.org Fri Mar 13 23:58:30 2020 From: mhalstead at linuxfoundation.org (Michael Halstead) Date: Fri, 13 Mar 2020 16:58:30 -0700 Subject: [bitbake-devel] Mailing list platform change March 20th Message-ID: We are moving our lists from Mailman to Groups.io. E-mail to lists will be delayed during the move window. We are aiming to complete the migration during business hours in the Pacific time zone. A new account will be created for you on the Groups.io platform if you don't already have one. You can read more about the change on the wiki: https://www.openembedded.org/wiki/GroupsMigration If there are serious issues we will rollback the changes. We will e-mail all lists when work is complete. -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthias.schoepfer at googlemail.com Sun Mar 15 14:22:42 2020 From: matthias.schoepfer at googlemail.com (Matthias Schoepfer) Date: Sun, 15 Mar 2020 15:22:42 +0100 Subject: [bitbake-devel] git fetcher: does not execute git fetch --tags or similar when HEAD has not changed Message-ID: Hi! As pointed by Denys Dmytriyenko, I repost here from the yocto project mailing list. We have noticed the following issue: We keep the versions of out software by means of tags on the git repositories, i.e. during the build, somethings like git describe --tags gets called. In yocto, our recipe of SomeLibrary might look then similar to this: SRC_URI = "git://our.private.gitserver.org/git/SomeLibrary.git;tags=v${PV}" When we build the image, and *then* apply a tag to HEAD, that was already built before, it seems like the tags are not fetched (my guess is git fetcher sees that origins HEAD and local HEAD have the same hash and is fine with it. I believe this is also true for SRCREV instead of tags=). The error is, that while the correct version is built, it will report itself with a wrong version. Is this a bug? Would you welcome some patch that does git fetch --tags in any case?! Regards, ? Matthias From chris.laplante at agilent.com Sun Mar 15 16:16:10 2020 From: chris.laplante at agilent.com (Chris Laplante) Date: Sun, 15 Mar 2020 12:16:10 -0400 Subject: [bitbake-devel] [PATCH] build.py: warn of deprecated use of [noexec] or [nostamp] with value other than '1' Message-ID: <1584288970-69085-1-git-send-email-chris.laplante@agilent.com> It would be nice for something like the following to work: do_task[noexec] = "${@"1" if condition else ""}" It currently does not, because "noexec" takes effect even when the flag is present but the value is "". [YOCTO #13808] Signed-off-by: Chris Laplante --- lib/bb/build.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/bb/build.py b/lib/bb/build.py index 23b6ee4..2503515 100644 --- a/lib/bb/build.py +++ b/lib/bb/build.py @@ -798,11 +798,14 @@ def add_tasks(tasklist, d): task_deps['tasks'].append(task) flags = d.getVarFlags(task) - def getTask(name): + def getTask(name, check_deprecation=False): if not name in task_deps: task_deps[name] = {} if name in flags: deptask = d.expand(flags[name]) + # See: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13808 + if check_deprecation and (deptask != "1"): + bb.warn("Deprecated usage of [{0}] flag with a value other than '1'. In the future, [{0}] will only take effect when set to '1'.".format(name)) task_deps[name][task] = deptask getTask('mcdepends') getTask('depends') @@ -811,9 +814,9 @@ def add_tasks(tasklist, d): getTask('rdeptask') getTask('recrdeptask') getTask('recideptask') - getTask('nostamp') + getTask('nostamp', check_deprecation=True) getTask('fakeroot') - getTask('noexec') + getTask('noexec', check_deprecation=True) getTask('umask') task_deps['parents'][task] = [] if 'deps' in flags: -- 2.7.4 From peter.kjellerstedt at axis.com Mon Mar 16 11:19:26 2020 From: peter.kjellerstedt at axis.com (Peter Kjellerstedt) Date: Mon, 16 Mar 2020 11:19:26 +0000 Subject: [bitbake-devel] [master][zeus][PATCH 1/3] knotty: Make the bb.command.CommandExit event terminate bitbake In-Reply-To: <20200215032955.3958-1-pkj@axis.com> References: <20200215032955.3958-1-pkj@axis.com> Message-ID: <6b3a542faa054325940f888386e2495e@XBOX03.axis.com> > -----Original Message----- > From: bitbake-devel-bounces at lists.openembedded.org bounces at lists.openembedded.org> On Behalf Of Peter Kjellerstedt > Sent: den 15 februari 2020 04:30 > To: bitbake-devel at lists.openembedded.org > Subject: [bitbake-devel] [master][zeus][PATCH 1/3] knotty: Make the > bb.command.CommandExit event terminate bitbake > > This matches the other bb.command.Command* events and without it, > running `bitbake --revisions-changed` will hang indefinitely if there > are changed revisions. > > Signed-off-by: Peter Kjellerstedt > --- > bitbake/lib/bb/ui/knotty.py | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py > index a0340dfc20..cbb289b05f 100644 > --- a/bitbake/lib/bb/ui/knotty.py > +++ b/bitbake/lib/bb/ui/knotty.py > @@ -590,6 +590,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): > if isinstance(event, bb.command.CommandExit): > if not return_value: > return_value = event.exitcode > + main.shutdown = 2 > continue > if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): > main.shutdown = 2 > -- > 2.21.1 Please backport the three patches in this set to Zeus now that they have been integrated to master. //Peter From akuster808 at gmail.com Tue Mar 17 21:12:54 2020 From: akuster808 at gmail.com (akuster808) Date: Tue, 17 Mar 2020 14:12:54 -0700 Subject: [bitbake-devel] [poky] Thud community support In-Reply-To: <15FBEF1FC40D189D.8114@lists.yoctoproject.org> References: <15FBEF1FC40D189D.8114@lists.yoctoproject.org> Message-ID: <81fb73b4-f683-5356-ff07-1ac909afb97d@gmail.com> On 3/13/20 11:03 AM, akuster via Lists.Yoctoproject.Org wrote: > The Poky thud branch is now under 'Community support' and is seeking a > maintainer to continue supporting the following repos for the thud release: > > bitbake, core, meta-yocto and yocto-docs. > > > There will be no more point releases. > > see > https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS#Maintainer_Procedures? > to get an idea on what is expected. > > The one thing we (YP) don't know is if there will be AutoBuilder > resources available.? Please contact me or Richard regarding that statement. > > If no one steps up by April 24th (six weeks -ish from now), Thud will be > moved to EOL status. It was pointed out to me that there is no Maintainer selection process or criteria defined.? This? will have to be sorted out before we move forward on this transition. - armin > - Armin > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#11988): https://lists.yoctoproject.org/g/poky/message/11988 > Mute This Topic: https://lists.yoctoproject.org/mt/71932489/3616698 > Group Owner: poky+owner at lists.yoctoproject.org > Unsubscribe: https://lists.yoctoproject.org/g/poky/unsub [akuster808 at gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- -------------- next part -------------- An HTML attachment was scrubbed... URL: From jacob.kroon at gmail.com Wed Mar 18 23:17:36 2020 From: jacob.kroon at gmail.com (Jacob Kroon) Date: Thu, 19 Mar 2020 00:17:36 +0100 Subject: [bitbake-devel] [PATCH 1/2] bitbake-user-manual: Fix order of end tags Message-ID: <20200318231737.1169597-1-jacob.kroon@gmail.com> Fixes commit e22565968828c86983162e67f52ebb106242ca76. Signed-off-by: Jacob Kroon --- doc/bitbake-user-manual/bitbake-user-manual-execution.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml index 3b31f748..6c5234ed 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-execution.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-execution.xml @@ -976,7 +976,7 @@ False, that logger will be completely replaced by user configuration. In this case, no other rules will apply to that logger. - + All user defined filter and handlers properties for a given logger @@ -987,7 +987,7 @@ configuration adds a filter called BitBake.defaultFilter, both filters will be applied to the logger - + -- 2.25.1 From jacob.kroon at gmail.com Wed Mar 18 23:17:37 2020 From: jacob.kroon at gmail.com (Jacob Kroon) Date: Thu, 19 Mar 2020 00:17:37 +0100 Subject: [bitbake-devel] [PATCH 2/2] bitbake-user-manual: immediate-variable-expansion: Correct description In-Reply-To: <20200318231737.1169597-1-jacob.kroon@gmail.com> References: <20200318231737.1169597-1-jacob.kroon@gmail.com> Message-ID: <20200318231737.1169597-2-jacob.kroon@gmail.com> References to undefined variables are preserved as is and do not expand to nothing as in GNU Make. Signed-off-by: Jacob Kroon --- .../bitbake-user-manual-metadata.xml | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml b/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml index bb5a7f86..10b58835 100644 --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml @@ -294,17 +294,20 @@ rather than when the variable is actually used: T = "123" - A := "${B} ${A} test ${T}" + A := "test ${T}" T = "456" - B = "${T} bval" + B := "${T} ${C}" C = "cval" C := "${C}append" In this example, A contains - "test 123" because ${B} and - ${A} at the time of parsing are undefined, - which leaves "test 123". - And, the variable C + "test 123", even though the final value of T + is "456". + The variable B will end up containing "456 cvalappend". + This is because references to undefined variables are preserved as is + during (immediate)expansion. This is in contrast to GNU Make, where undefined + variables expand to nothing. + The variable C contains "cvalappend" since ${C} immediately expands to "cval". -- 2.25.1 From akuster808 at gmail.com Thu Mar 19 14:24:29 2020 From: akuster808 at gmail.com (akuster808) Date: Thu, 19 Mar 2020 07:24:29 -0700 Subject: [bitbake-devel] [master][zeus][PATCH 1/3] knotty: Make the bb.command.CommandExit event terminate bitbake In-Reply-To: <6b3a542faa054325940f888386e2495e@XBOX03.axis.com> References: <20200215032955.3958-1-pkj@axis.com> <6b3a542faa054325940f888386e2495e@XBOX03.axis.com> Message-ID: <1ffcea3b-0409-9f92-2ee6-e68a20118bba@gmail.com> got them. had to cherry pick as the patches did not apply. - armin On 3/16/20 4:19 AM, Peter Kjellerstedt wrote: >> -----Original Message----- >> From: bitbake-devel-bounces at lists.openembedded.org > bounces at lists.openembedded.org> On Behalf Of Peter Kjellerstedt >> Sent: den 15 februari 2020 04:30 >> To: bitbake-devel at lists.openembedded.org >> Subject: [bitbake-devel] [master][zeus][PATCH 1/3] knotty: Make the >> bb.command.CommandExit event terminate bitbake >> >> This matches the other bb.command.Command* events and without it, >> running `bitbake --revisions-changed` will hang indefinitely if there >> are changed revisions. >> >> Signed-off-by: Peter Kjellerstedt >> --- >> bitbake/lib/bb/ui/knotty.py | 1 + >> 1 file changed, 1 insertion(+) >> >> diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py >> index a0340dfc20..cbb289b05f 100644 >> --- a/bitbake/lib/bb/ui/knotty.py >> +++ b/bitbake/lib/bb/ui/knotty.py >> @@ -590,6 +590,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): >> if isinstance(event, bb.command.CommandExit): >> if not return_value: >> return_value = event.exitcode >> + main.shutdown = 2 >> continue >> if isinstance(event, (bb.command.CommandCompleted, bb.cooker.CookerExit)): >> main.shutdown = 2 >> -- >> 2.21.1 > Please backport the three patches in this set to Zeus now that they have > been integrated to master. > > //Peter > From mhalstead at linuxfoundation.org Thu Mar 19 16:44:43 2020 From: mhalstead at linuxfoundation.org (Michael Halstead) Date: Thu, 19 Mar 2020 09:44:43 -0700 Subject: [bitbake-devel] Mailing list platform change March 20th In-Reply-To: References: Message-ID: Everything is proceeding smoothly and this work will continue as planned. The migration starts at noon PDT tomorrow. List owners please take care of all outstanding moderation in the next 24 hours to ensure a smooth transition. Thank you, -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer On Fri, Mar 13, 2020 at 4:58 PM Michael Halstead < mhalstead at linuxfoundation.org> wrote: > We are moving our lists from Mailman to Groups.io. E-mail to lists will > be delayed during the move window. We are aiming to complete the migration > during business hours in the Pacific time zone. > > A new account will be created for you on the Groups.io platform if you > don't already have one. > > You can read more about the change on the wiki: > https://www.openembedded.org/wiki/GroupsMigration > > If there are serious issues we will rollback the changes. We will e-mail > all lists when work is complete. > > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhalstead at linuxfoundation.org Fri Mar 20 18:59:42 2020 From: mhalstead at linuxfoundation.org (Michael Halstead) Date: Fri, 20 Mar 2020 11:59:42 -0700 Subject: [bitbake-devel] Mailing list platform change March 20th In-Reply-To: References: Message-ID: The migration will begin shortly. List mail will be delayed until the process is complete. E-mail sent during downtime should eventually be delivered. However waiting to send until after the switch is recommended. Thank you, -- Michael Halstead Linux Foundation / Yocto Project Systems Operations Engineer On Thu, Mar 19, 2020 at 9:44 AM Michael Halstead < mhalstead at linuxfoundation.org> wrote: > Everything is proceeding smoothly and this work will continue as planned. > The migration starts at noon PDT tomorrow. > > List owners please take care of all outstanding moderation in the next 24 > hours to ensure a smooth transition. > > Thank you, > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer > > On Fri, Mar 13, 2020 at 4:58 PM Michael Halstead < > mhalstead at linuxfoundation.org> wrote: > >> We are moving our lists from Mailman to Groups.io. E-mail to lists will >> be delayed during the move window. We are aiming to complete the migration >> during business hours in the Pacific time zone. >> >> A new account will be created for you on the Groups.io platform if you >> don't already have one. >> >> You can read more about the change on the wiki: >> https://www.openembedded.org/wiki/GroupsMigration >> >> If there are serious issues we will rollback the changes. We will e-mail >> all lists when work is complete. >> >> -- >> Michael Halstead >> Linux Foundation / Yocto Project >> Systems Operations Engineer >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: