<div><div dir="auto">I think it’s ok to have it in Glibc/localedef not all distros have this version of util Linux yet and perhaps we should start creating a buildtools tarballs which is a prerequisite for building OE and collect these dependencies in there instead</div></div><div dir="auto">Of sticking them in unusual places </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 16, 2019 at 7:59 PM Jason Wessel <<a href="mailto:jason.wessel@windriver.com">jason.wessel@windriver.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 8/16/19 4:50 PM, <a href="mailto:richard.purdie@linuxfoundation.org" target="_blank">richard.purdie@linuxfoundation.org</a> wrote:<br>
> On Fri, 2019-08-16 at 14:29 -0700, Khem Raj wrote:<br>
>> On Fri, Aug 16, 2019 at 2:25 PM Richard Purdie<br>
>> <<a href="mailto:richard.purdie@linuxfoundation.org" target="_blank">richard.purdie@linuxfoundation.org</a>> wrote:<br>
>>> This was discussed on IRC. We really really want to avoid that as a<br>
>>> dependency if we care about build performance. Its really easy to<br>
>>> just<br>
>>> add that in but the implications are significant :(<br>
>>><br>
>> A captured version of this utility would also mean that we have to<br>
>> keep this in sync for bug fixes and security fixes etc. which could<br>
>> also be considered as expense, so how much do we lose if util-linux-<br>
>> native is brought early into dependency chain?<br>
> The tool is being used for a specific purpose at build time. Its not<br>
> exposed on target and its use is extremely constrained to a specific<br>
> set of files where its behaviour should be deterministic.<br>
><br>
> As such, yes there are questions about bug fixes and about security but<br>
> I don't think its going to be a serious burden. We'd need to carry this<br>
> until such times as the host systems we support have hardlink which is<br>
> probably a few years out but will arrive.<br>
><br>
> The impact of having util-linux-native in the dependency graph at this<br>
> point is significant. I say that as someone who has done a lot of work<br>
> in trimming down the times of our bootstrap process. This is exactly<br>
> the kind of dependency I've removed and avoided.<br>
><br>
> glibc is special in that its the worst choke point on the dependency<br>
> graph. Nothing to do with target binaries happens until it is ready and<br>
> then gcc-runtime can build. Whilst the locales are split from glibc,<br>
> they are still connected in the task graph for packaging purposes so<br>
> this is important.<br>
><br>
> Its been pointed out there are other places where util-linux-native has<br>
> crept in, in particular python3-native. My preference would be to<br>
> minimise these dependencies rather than use them as a way to justify<br>
> adding it here too.<br>
<br>
<br>
So what is the consensus here? Can we require the host system to have hardlink? It is available as a stand alone binary on Ubuntu 14.04 and up and on Fedora systems, but isn't installed by default, and we'd have to check the version because the arguments sadly are slightly different. We could certainly make util-linux a dependency, but that is why this patch existed, in order to avoid the dependency.<br>
<br>
I did find a python version at <a href="https://raw.githubusercontent.com/akaihola/hardlinkpy/master/hardlink.py" rel="noreferrer" target="_blank">https://raw.githubusercontent.com/akaihola/hardlinkpy/master/hardlink.py</a>. I made a one line modification to it and it works, but it isn't nearly as efficient. The proof is below, using the same locale-tree each time. It is important to look at both the real and user time. The difference gets more significant when you run it under pseudo, where we expect it to run 100% of the time. It is roughly 10x slower to do this in python.<br>
<br>
<br>
time recipe-sysroot-native/usr/bin/cross-localedef-hardlink -c -v locale-tree<br>
Directories: 982<br>
Objects: 7441<br>
Regular files: 6441<br>
Comparisons: 3331<br>
Linked: 3109<br>
Saved: 754675712<br>
<br>
real 0m0.641s<br>
user 0m0.120s<br>
sys 0m0.518s<br>
<br>
time python3 hardlink.py -v 0 -c locale-tree<br>
<br>
<br>
Hard linking Statistics:<br>
Directories : 982<br>
Regular files : 6441<br>
Comparisons : 31884<br>
Hardlinked this run : 3109<br>
Total hardlinks : 3109<br>
Bytes saved this run : 744089843 (709.619 mebibytes)<br>
Total bytes saved : 744089843 (709.619 mebibytes)<br>
Total run time : 2.124422073364258 seconds<br>
<br>
real 0m2.180s<br>
user 0m1.220s<br>
sys 0m0.953s<br>
<br>
<br>
============Now lets run it under pseudo ================<br>
<br>
time recipe-sysroot-native/usr/bin/cross-localedef-hardlink -c -v locale-tree<br>
Directories: 982<br>
Objects: 7441<br>
Regular files: 6441<br>
Comparisons: 3331<br>
Linked: 3109<br>
Saved: 754675712<br>
<br>
real 0m2.505s<br>
user 0m0.373s<br>
sys 0m1.286s<br>
<br>
time python3 hardlink.py -v 0 -c locale-tree<br>
<br>
<br>
Hard linking Statistics:<br>
Directories : 982<br>
Regular files : 6441<br>
Comparisons : 31884<br>
Hardlinked this run : 3109<br>
Total hardlinks : 3109<br>
Bytes saved this run : 744089843 (709.619 mebibytes)<br>
Total bytes saved : 744089843 (709.619 mebibytes)<br>
Total run time : 12.971824884414673 seconds<br>
<br>
real 0m13.046s<br>
user 0m3.633s<br>
sys 0m5.192s<br>
<br>
<br>
</blockquote></div></div>