<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 5 Nov 2019 at 22:50, Nicolas Dechesne <<a href="mailto:nicolas.dechesne@linaro.org">nicolas.dechesne@linaro.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">hi,<br>
<br>
On Tue, Oct 8, 2019 at 8:42 PM Aníbal Limón <<a href="mailto:anibal.limon@linaro.org" target="_blank">anibal.limon@linaro.org</a>> wrote:<br>
><br>
> In some cases people/organizations are using a SRC_URI with<br>
> file:///PATH_TO_DIR that contains a git repository for different<br>
> reasons, it is useful when want to do an internal build without<br>
> clone sources from outside.<br>
><br>
> This could consume a lot of CPU time because the current taskhash<br>
> generation mechanism didn't identify that the folder is a VCS<br>
> (git, svn, cvs) and makes the cksum for every file including the<br>
> .git repository in this case.<br>
><br>
> There are different ways to improve the situation,<br>
><br>
> * Add protocol=gitscm in file:// SRC_URI but the taskhash is<br>
> calculated before the fetcher identifies the protocol, will require<br>
> some changes in bitbake codebase.<br>
> * This patch: When directory is a git repository (contains .git)<br>
> use HEAD rev + git diff to calculate checksum instead of do it<br>
> in every file, that is hackish because make some assumptions about<br>
> .git directory contents.<br>
> * Variant of this patch: Make a list of VCS directories (.git, .svn,<br>
> .cvs) and take out for cksum calculations, same as before making<br>
> assumptions about the . folders content.<br>
<br>
I've discussed with Khem and Richard today (@ELCE) about this patch.<br>
We kind of agreed that the current approach is not really good since<br>
the local fetcher isn't supposed to 'deal' with scm commands. however<br>
we agreed that the last variant proposed above might be a much better<br>
approach. The idea would be to exclude VCS directories from the<br>
checksum computation. It could potentially be extended in a slightly<br>
more generic way, like using a variable to specify a list of<br>
directories to exclude, which could be unset by default, and set in oe<br>
core as<br>
BB_LOCAL_DIRS_EXCLUDE = ".git .cvs .svn"<br>
<br>
Anibal: do you think you can give it a try?<br></blockquote><div><br></div><div>It sounds good, this new variable will be only used for cksum exclude?, if yes may be to change</div><div>for more specific variable BB_LOCAL_DIRS_TASKHASH_EXCLUDE.</div><div><br></div><div>Regards,</div><div>Anibal</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
><br>
> Signed-off-by: Aníbal Limón <<a href="mailto:anibal.limon@linaro.org" target="_blank">anibal.limon@linaro.org</a>><br>
> ---<br>
> lib/bb/checksum.py | 13 +++++++++++++<br>
> 1 file changed, 13 insertions(+)<br>
><br>
> diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py<br>
> index 5bc8a8fc..ee125cb5 100644<br>
> --- a/lib/bb/checksum.py<br>
> +++ b/lib/bb/checksum.py<br>
> @@ -86,6 +86,19 @@ class FileChecksumCache(MultiProcessCache):<br>
> return checksum<br>
><br>
> def checksum_dir(pth):<br>
> + git_dir = os.path.join(pth, '.git')<br>
> + if os.path.exists(git_dir):<br>
> + import subprocess, hashlib<br>
> + m = hashlib.md5()<br>
> + head = subprocess.check_output("cd %s && git rev-parse HEAD" % pth, shell=True)<br>
> + diff = subprocess.check_output("cd %s && git diff" % pth, shell=True)<br>
> + m.update(head)<br>
> + if diff:<br>
> + m.update(diff)<br>
> +<br>
> + return [(pth, m.hexdigest())]<br>
> +<br>
> +<br>
> # Handle directories recursively<br>
> if pth == "/":<br>
> bb.fatal("Refusing to checksum /")<br>
> --<br>
> 2.23.0<br>
><br>
</blockquote></div></div>