<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr">On Oct 26, 2019, at 7:11 AM, richard.purdie@linuxfoundation.org wrote:</div><div dir="ltr"><blockquote type="cite"><br></blockquote></div><blockquote type="cite"><div dir="ltr"><span>On Fri, 2019-10-25 at 16:04 -0400, Rich Persaud wrote:</span><br><blockquote type="cite"><span>There may be lessons in Microsoft's move from diverse hardware to VM-</span><br></blockquote><blockquote type="cite"><span>based testing:</span><br></blockquote><blockquote type="cite"><span>https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/</span><br></blockquote><span></span><br><span>I don't disagreed, real hardware does have some distinct advantages. We</span><br><span>also have to keep in mind that we're testing what the Yocto Project is</span><br><span>offering. At some point YP has to trust that the underlying kernels we</span><br><span>ship are "ok" and that upstream has covered some of this.</span></div></blockquote><div><br></div><div>Here's a quote from John Loucaides of Eclypsium, ex-Intel, <a href="https://twitter.com/johnloucaides/status/1187839034661339136?s=21">https://twitter.com/johnloucaides/status/1187839034661339136</a>, describing his PSEC video:</div><div><br></div><div><div>"While hard, I don't believe supply chain security is unsolvable. I believe it's a "tragedy of the commons" problem where everyone hopes someone else will fix it. By working together on tools/projects, we can change the incentives and create practical solutions."</div></div><br><blockquote type="cite"><div dir="ltr"><span></span><span>Virtual testing works well for the software stack itself which a large</span><br><span>part of what YP offers, particularly where we can run upstream's tests</span><br><span>for components. Where its struggles is on the variations of hardware</span><br><span>that are out there as the article highlights.</span><br><span></span><br><span>As such, I think virtual testing for any YP LTS is the realistic option</span><br><span>we have right now.</span><br></div></blockquote><div><br></div><div>Some upstream kernel testing is also done on virtual machines, for similar reasons. They may expect downstream "distros" to have more device-focused hardware coverage. </div><div><br></div><div>To avoid YP users assuming that YP and/or the upstream kernel has done hardware testing for YP LTS, we may want to document specific testing actions that are expected from YP LTS users. E.g. should YP downstreams work directly with CKI via the Linux Foundation, to pool hardware test results? Should they report hw test results to other YP LTS users?</div><div><br></div><br><blockquote type="cite"><div dir="ltr"><span></span><blockquote type="cite"><span>Business requirements for "LTS releases" include security fixes and</span><br></blockquote><blockquote type="cite"><span>non-regression of production systems, i.e. the software supply chain</span><br></blockquote><blockquote type="cite"><span>security topics discussed in [1] above. There were CKI [3]</span><br></blockquote><blockquote type="cite"><span>discussions about sharing test resources (human, machine) for kernel</span><br></blockquote><blockquote type="cite"><span>testing. Some CKI challenges and potential solutions [4] may be</span><br></blockquote><blockquote type="cite"><span>applicable to Yocto LTS testing of packages beyond the Linux kernel.</span><br></blockquote><blockquote type="cite"><span></span><br></blockquote><blockquote type="cite"><span>VM-based testing could be supplemented with pooled test results from</span><br></blockquote><blockquote type="cite"><span>distributed testing on diverse hardware, across the Yocto contributor</span><br></blockquote><blockquote type="cite"><span>community. Even with VM-based testing, IOMMU PCI passthrough can be</span><br></blockquote><blockquote type="cite"><span>used to give a VM direct access to a PCI device, for device driver</span><br></blockquote><blockquote type="cite"><span>testing.</span><br></blockquote><span></span><br><span>We have already build a mechanism into our test results process for</span><br><span>externally contributed test results and we are happy to collect that</span><br><span>data although we haven't seen much being contributed yet. I'd love to</span><br><span>see more data coming in for any LTS though (or any other release for</span><br><span>that matter).</span><br></div></blockquote><br><div>That's good news. Could you share pointers/docs/samples on the mechanism for external test result contributions? Is this effort coordinated with ATS or CKI for pooling of hw-specific test results? If we are able to attract contributions, could we eventually associate YP LTS external test results with public BSP definitions and hardware model/revision numbers?</div><div><br></div><div>Rich</div></body></html>