<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello Jens,</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 9/12/2018 4:53 AM, Jens Rehsack
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8ABCF3BE-EFE8-4FE6-9886-65F6AE4E4EDC@netbsd.org">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">Am 11.09.2018 um 20:56 schrieb Alejandro Enedino
Hernandez Samaniego <<a
href="mailto:alejandro.enedino.hernandez-samaniego@xilinx.com"
class="" moz-do-not-send="true">alejandro.enedino.hernandez-samaniego@xilinx.com</a>>:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hey Jens,</p>
</div>
</div>
</blockquote>
<div>Hey <span style="caret-color: rgb(34, 34, 34); color:
rgb(34, 34, 34); font-family: arial, sans-serif; font-size:
small; background-color: rgb(255, 255, 255);" class="">Alejandro,</span></div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<div class="moz-cite-prefix">On 09/10/2018 11:58 PM, Jens
Rehsack wrote:<br class="">
</div>
<blockquote type="cite"
cite="mid:0EC3E010-56C9-4EB4-90B4-FD9A68F99AAC@netbsd.org"
class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">Am 10.09.2018 um 23:33 schrieb
Alejandro Enedino Hernandez Samaniego <<a
href="mailto:alejandro.enedino.hernandez-samaniego@xilinx.com"
class="" moz-do-not-send="true">alejandro.enedino.hernandez-samaniego@xilinx.com</a>>:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type"
content="text/html; charset=windows-1252"
class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hey Jens,</p>
<p class=""><br class="">
</p>
<p class="">As I explained before, when you
create a manifest for python (target), it uses
the native build as base (it literally runs
the native python that was just built), it is
assumed its the same version as target and
contains all the modules provided by upstream,
otherwise the missing modules cannot be
checked for dependencies, and the manifest
becomes incoherent, so its not an option to
have an incomplete python native build.</p>
</div>
</div>
</blockquote>
<div class="">In that case, uuid for target never gets
deployed, but it is. And I didn't see any packaging
issues for `python3` nor for `nativesdk-python3`</div>
</div>
</blockquote>
<br class="">
<br class="">
I don't see what that has to do with anything, fixing the
native build should not affect what gets deployed on
target, thats exactly why we have a manifest, so they user
can decide what to install and what not to install.<br
class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>The manifest isn't used for python3-native.</div>
<div>You try to argue whether there is an error to be fixed -
and I don't agree.</div>
<div>Each fix requires effort - and that's why some errors will
never become fixed. When the impact is reasonable or high
enough, fixes are more likely.</div>
<div><br class="">
</div>
<div>So: before you try to force me into "that's all ugly und
must be seriously beautified just because ..." I will not do
anything.</div>
<div>When you give me a sane reason, I try to understand and
make a decision.</div>
</div>
</blockquote>
<p><br>
</p>
<p><br>
</p>
<p>Ok first of all, I am not forcing you to do anything, and I've
never said its all ugly, so please don't reply aggressively since
I am just trying to help you get your patch merged, and it needs
to work correctly for that to happen, this is a community and if
you don't want to do something just don't do it, we're all trying
to make this work in the best possible way, whether we get paid
for it or do it for fun.<br>
</p>
<p>Second of all, I know the manifest is not used for
python3-native, I never said python3-native used the manifest, but
I am trying to explain to you why the fix is necessary, and I am
even telling you what the fix is, so you don't have to waste your
time and effort on that, here it goes once again:</p>
<p><br>
</p>
<p>There are python3-native and python3 packages.</p>
<p>The python3 package uses a manifest file, to go through the
installed files by the whole python installation from upstream,
and separates it in several packages, providing more granularity
and giving users the possibility of installing a trimmed version
of python, only with what they require on their image.</p>
<p>I think we're good until here.</p>
<p>The problem here comes because we don't control upstream python
and it changes with every release, so new files show up, for
example what I mentioned before sha3.*.so.</p>
<p>Since that file is not on the manifest, during packaging, it will
end up on python3-misc, which contains basically everything that
doesn't belong anywhere else, although clearly this file in
specific should belong in python3-crypt.</p>
<p>What this means for the user, is that he/she will install
python3-crypt and expect it to work at runtime, but this wont
happen, and when sha3 gets imported (or something that depends on
it to run), it will fail, this will cause a bug on our system,
which we would need to fix (yes, fixing bugs takes effort and
time).<br>
</p>
<p>At this point an usual fix would be to just install python3-misc
as well, because it contains the required library, but
python3-misc also contains other unnecessary stuff that the user
doesn't need, but anyways an RDEPENDS to python3-misc would
somewhat solve it. The correct fix however, is to add the
sha3.*.so file to the manifest, on the python3-crypt package, that
way, it will get installed and the user would be able to use it
correctly. This would solve one problem, but its still not enough,
because, I can see (from the log), that there are some
dependencies from other packages to sha3, which means that yes, it
will work correctly if we decide to install python3-crypt, but if
we just install one of the other packages that need it, it won't,
again, it will fail to import it and its the same story, its
introducing a bug into the system.</p>
<p>OK, up to this point I've explained how this will cause bugs in
our system, now, what does this fix that I mentioned has to do
with everything?. <br>
</p>
<p>There is a special task for the python3 package (not native),
that creates a manifest for us, this task tries to import every
module from the manifest and get its dependencies, for example, if
the python3-numbers package needed the sha3 library (this is just
an example I really don't know if numbers needs this), this will
be picked up, and the python3-numbers package will RDEPEND on
python3-crypt, hence solving both the bugs I mentioned before,
even before they exist.</p>
<p>Now why is a fix to python3-native important for python3 target?</p>
<p>Because of the way the previously mentioned script works, it
calls python3-native (it has to, because it can't use python from
host for example, because its not guaranteed its the same version
of python3 target, and we need it to contain exactly the same
files) to import all modules in an automated manner and get their
dependencies, that's how we know if there are new files and/or
rdependencies between python3 packages.</p>
<p>So what will happen if the python3-native build isn't complete?,
whatever file its missing, we wont be able to catch dependencies
to it, causing them to end up in python3-misc, causing the issue
#1, and also, in any case, there wont be an RDEPENDS created to
python3-misc (which would solve issue #1), sometimes files do
belong in python3-misc, but we need to know that so we can add an
RDEPENDS to it, but again if we just add an RDEPENDS to
python3-misc we'll be causing issue/bug #2, which we won't catch,
until someone actually tries to run something on a running image
and gets an import failure on their application.</p>
<p><br>
</p>
<p>I'm not even asking you to fix the manifest, I am simply asking
you to provide a functional native package (in this case,
functional means with all modules built), so it doesn't break the
current functionality that helps us catch these bugs. I hope its
clear now that I am not asking you to just change something
because I want "beautiful" code everywhere and I never was, I am
asking you to do it because your patch will introduce bugs into
the system, and I am trying to help you fix them.</p>
<p><br>
</p>
<p>Alejandro<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:8ABCF3BE-EFE8-4FE6-9886-65F6AE4E4EDC@netbsd.org">
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> I know you
didn't see any packaging issues, but that doesn't mean
they don't exist, just from the log I showed you, I can
tell you that the python3-crypt package is not created
correctly, for example, if you do:<br class="">
<br class="">
IMAGE_INSTALL_append = " python3-crypt"<br class="">
<br class="">
Boot the image, run python3 and you try to import sha3, it
will fail, because the sha3*.so library won't be on the
filesystem.<br class="">
<br class="">
<br class="">
And thats because the sha3.*.so library was just
introduced in this upgrade, and our manifest isn't aware
it exists, hence it'll end up on python3-misc and you have
just created an unnecessary dependency from python3-crypt
to python3-misc (and worse, a dependency which were not
even aware of, until we test it manually), which beats the
whole purpose of the manifest.<br class="">
<br class="">
The do_package function is not gonna fail just because, so
you won't see errors, but the files will be packaged
incorrectly, causing runtime errors as a consequence, the
create_manifest task tries to solve these runtime issues
before they happen.<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>That's something completely different. I'm not a python guy
and I don't get from the ChangeLog what I have to change in
the manifest.</div>
<div>I can and will do such stuff when doing the perl update.
For python, I ask for review because of things I don't know.</div>
<div><br class="">
</div>
<div>You can give me a reasonable list of such changes and I
update the manifest accordingly when I rebase the patch after
Ross PGO patches are committed.</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> Cheers,<br
class="">
<br class="">
Alejandro<br class="">
<br class="">
<br class="">
<blockquote type="cite"
cite="mid:0EC3E010-56C9-4EB4-90B4-FD9A68F99AAC@netbsd.org"
class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Yes you probably need a patch to
look at the correct directories for the h
files, as well as a dependency to make the h
files available on recipe-sysroot-native.</p>
</div>
</div>
</blockquote>
Please check the submission.</div>
</blockquote>
<br class="">
I did, its not checking inside recipe-sysroot-native.<br
class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>I know. You could check the submission anyway and find the
right patch and then argue differently.</div>
</div>
<div class=""><br class="">
</div>
Cheers<br class="">
<div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">
<div class="">-- </div>
<div class="">Jens Rehsack - <a
href="mailto:rehsack@gmail.com" class=""
moz-do-not-send="true">rehsack@gmail.com</a></div>
</div>
</div>
</div>
<br class="">
</blockquote>
</body>
</html>