<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"><span>On Oct 26, 2019, at 11:45 AM, Richard Purdie &lt;richard.purdie@linuxfoundation.org&gt; wrote:</span><br><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>On Sat, 2019-10-26 at 10:34 -0400, Rich Persaud wrote:</span><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On Oct 26, 2019, at 7:11 AM, richard.purdie@linuxfoundation.org</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>wrote:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>There may be lessons in Microsoft's move from diverse hardware to</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>VM-</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>based testing:</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>I don't disagreed, real hardware does have some distinct</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>advantages. We</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>also have to keep in mind that we're testing what the Yocto Project</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>is</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>offering. At some point YP has to trust that the underlying kernels</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>we</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>ship are "ok" and that upstream has covered some of this.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Here's a quote from John Loucaides of Eclypsium, ex-Intel,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>https://twitter.com/johnloucaides/status/1187839034661339136,</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>describing his PSEC video:</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>"While hard, I don't believe supply chain security is unsolvable. I</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>believe it's a "tragedy of the commons" problem where everyone hopes</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>someone else will fix it. By working together on tools/projects, we</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>can change the incentives and create practical solutions."</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Sure.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>As a project we struggle to do what we do with the resources we have</span><br></blockquote><blockquote type="cite"><span>available today, to the point that we're burning out people, me</span><br></blockquote><blockquote type="cite"><span>included.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I have pushed and tried to have automated real hardware testing for</span><br></blockquote><blockquote type="cite"><span>over a decade. Its proving very difficult to get traction and make it</span><br></blockquote><blockquote type="cite"><span>happen.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>We could make a significant difference under virtualised testing so</span><br></blockquote><blockquote type="cite"><span>we've done that, I've spent a considerable amount of time ensuring YP</span><br></blockquote><blockquote type="cite"><span>at least has robust testing there. There is much more I'd love to do</span><br></blockquote><blockquote type="cite"><span>but I personally don't have the hours in the day and I struggle to find</span><br></blockquote><blockquote type="cite"><span>people to help.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I'm a realist. I'd like YP LTS to happen. The bigger the barriers we</span><br></blockquote><blockquote type="cite"><span>put in its way, the less likely it is. I've always aimed to build</span><br></blockquote><blockquote type="cite"><span>systems and technologies that do allow incremental</span><br></blockquote><blockquote type="cite"><span>development/extension so there is at least options for this.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Virtual testing works well for the software stack itself which a</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>large</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>part of what YP offers, particularly where we can run upstream's</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>tests</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>for components. Where its struggles is on the variations of</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>hardware</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>that are out there as the article highlights.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>As such, I think virtual testing for any YP LTS is the realistic</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>option</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>we have right now.</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>Some upstream kernel testing is also done on virtual machines, for</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>similar reasons. They may expect downstream "distros" to have more</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>device-focused hardware coverage.</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>To avoid YP users assuming that YP and/or the upstream kernel has</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>done hardware testing for YP LTS, we may want to document specific</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>testing actions that are expected from YP LTS users. &nbsp;E.g. should YP</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>downstreams work directly with CKI via the Linux Foundation, to pool</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>hardware test results? &nbsp;Should they report hw test results to other</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>YP LTS users?</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>We do document very clearly the testing that YP has done for each</span><br></blockquote><blockquote type="cite"><span>release. Users are already expected to figure out and understand what</span><br></blockquote><blockquote type="cite"><span>additional testing their end use case may need and LTS is no different.</span><br></blockquote><blockquote type="cite"><span>Their hardware for example will look very different, even between</span><br></blockquote><blockquote type="cite"><span>users.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>Pooling results works if the base configurations can be summarised and</span><br></blockquote><blockquote type="cite"><span>reproduced. I know the automated-testing people are working on ways to</span><br></blockquote><blockquote type="cite"><span>improve the ability to do that. For the YP side we do have some helpful</span><br></blockquote><blockquote type="cite"><span>technology for example the hash representations of our configurations</span><br></blockquote><blockquote type="cite"><span>which cover most of the software supply chain.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>Business requirements for "LTS releases" include security fixes</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>and</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>non-regression of production systems, i.e. the software supply</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>chain</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>security topics discussed in [1] above. &nbsp;There were CKI [3]</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>discussions about sharing test resources (human, machine) for</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>kernel</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>testing. &nbsp;Some CKI challenges and potential solutions [4] may be</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>applicable to Yocto LTS testing of packages beyond the Linux</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>kernel.</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>VM-based testing could be supplemented with pooled test results</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>from</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>distributed testing on diverse hardware, across the Yocto</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>contributor</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>community. &nbsp;Even with VM-based testing, IOMMU PCI passthrough can</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>be</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>used to give a VM direct access to a PCI device, for device</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>driver</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>testing.</span><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>We have already build a mechanism into our test results process for</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>externally contributed test results and we are happy to collect</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>that</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>data although we haven't seen much being contributed yet. I'd love</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>to</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>see more data coming in for any LTS though (or any other release</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>for</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span>that matter).</span><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>That's good news. &nbsp;Could you share pointers/docs/samples on the</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>mechanism for external test result contributions?</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I've worked on material which I gave to our tech writer but the</span><br></blockquote><blockquote type="cite"><span>resulting docs haven't been produced yet which comes down to resources</span><br></blockquote><blockquote type="cite"><span>:(.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I've also posted many times over the last year about the Yocto QA</span><br></blockquote><blockquote type="cite"><span>developments, how the new process works and how people could contribute</span><br></blockquote><blockquote type="cite"><span>test results to that.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>The QA page on the wiki:</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>https://wiki.yoctoproject.org/wiki/QA</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>has the basics, its not extensively documented but its not a complex</span><br></blockquote><blockquote type="cite"><span>process either and it is the process Intel/WindRiver use to contribute</span><br></blockquote><blockquote type="cite"><span>test results today.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>Is this effort coordinated with ATS or CKI for pooling of hw-</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>specific test results?</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>No, its hard enough to get YP testing to work without more parties</span><br></blockquote><blockquote type="cite"><span>involved. I was actually specifically told by managers to even drop the</span><br></blockquote><blockquote type="cite"><span>generic process as it was adding too much overhead and complexity. I</span><br></blockquote><blockquote type="cite"><span>refused, despite significant pressure FWIW.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><blockquote type="cite"><span>If we are able to attract contributions, could we eventually</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>associate YP LTS external test results with public BSP definitions</span><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><span>and hardware model/revision numbers?</span><br></blockquote></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>I'd love to see it happen and I do understand and like your vision but</span><br></blockquote><blockquote type="cite"><span>sadly companies don't seem to see the value of putting resources into</span><br></blockquote><blockquote type="cite"><span>things like this.</span></blockquote><br><blockquote type="cite"><span></span></blockquote><blockquote type="cite"><br></blockquote></div><div dir="ltr">Some companies have evolved their business models to monetize complex (OSS + proprietary) process, silicon, software, model, and data supply chains.</div><div dir="ltr"><br></div><div dir="ltr">kernelci.org (announced today [1]) is now part of Linux Foundation, joining LVFS (Linux firmware updates), CCC (confidential computing hardware enclave SDK) and CHIPS Alliance (OSS EDA tools for open hardware).</div><div dir="ltr"><br></div><div dir="ltr">Google helped drive upstream Linux kernel LTS, motivated by the economics of Android support. &nbsp;Microsoft Linux has a license that includes 10 years (!) of security updates. &nbsp;Both are contributing to kernelci.org, along with RedHat's CKI.</div><div dir="ltr"><br></div><div dir="ltr">PSEC video [2] for Azure Sphere, and the final slide in the linked PDF, mentions nightly automated testing, with static analysis and fuzzing. "IoT OS" is based on Yocto. &nbsp; There's also a linked talk by Ryan Fairfax with more detail on the build system.</div><div dir="ltr"><br></div><div dir="ltr">Rich</div><div dir="ltr"><br></div><div dir="ltr">[1]&nbsp;<a href="http://tradeshownews.vporoom.com/2019-10-28-Distributed-Linux-Testing-Platform-KernelCI-Secures-Funding-and-Long-Term-Sustainability-as-New-Linux-Foundation-Project?utm_source=dlvr.it&amp;utm_medium=twitter">http://tradeshownews.vporoom.com/2019-10-28-Distributed-Linux-Testing-Platform-KernelCI-Secures-Funding-and-Long-Term-Sustainability-as-New-Linux-Foundation-Project</a></div><div dir="ltr"><br></div><div dir="ltr">[2]&nbsp;<a href="https://www.platformsecuritysummit.com/2019/speaker/seay/">https://www.platformsecuritysummit.com/2019/speaker/seay/</a></div><div dir="ltr"><br></div><div dir="ltr"><br></div></body></html>