<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 22, 2019, 9:20 AM Richard Purdie <<a href="mailto:richard.purdie@linuxfoundation.org" rel="noreferrer noreferrer" target="_blank">richard.purdie@linuxfoundation.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I wanted to summarise what my local tests with the hash server<br>
concluded.<br>
<br>
a) the opendb() changes I made in:<br>
<a href="http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ca04aaf7b51e3ee2bb04da970d5f20f2c9982cb8" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=ca04aaf7b51e3ee2bb04da970d5f20f2c9982cb8</a><br>
broke things as the database is opened for each request. I have <br>
local patches to fix that but it only helps by about 10%.<br>
<br>
b) the overhead from the separate receive and handling threads is not <br>
worth the overhead:<br>
<a href="http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d40d7e43856f176c45cf515644b5f211c708e237" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d40d7e43856f176c45cf515644b5f211c708e237</a><br>
This probably halves throughput.<br>
<br>
c) I moved the database writes to their own thread with a queue but it <br>
doesn't seem to help much other than allowing other threads to <br>
handle requests in parallel.<br>
<br>
d) the ThreadedMixin regresses performance further and is the worst <br>
change I've tested.<br>
<br>
e) Using ThreadPoolExecutor along the lines of:<br>
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=10)<br>
self.executor.submit(self.process_request_thread, request, client_address)<br>
doesn't help speed. Its faster then ThreadedMixin but slower than no threads.<br>
<br>
f) The profile data I have suggests we spend a lot of time in TCP/HTTP <br>
header overhead and connection setup which confirms Joshua's <br>
thoughts.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Based on the profiling I did, the overhead of the HTTP requests is also substantial compared to the amount of actual body data sent.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
g) The most optimal setup is therefore the original server with no <br>
threading.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Ya threading is *bad*. I don't exactly know why...</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
h) The autobuilder would need to cope with 9000*40 requests in under a <br>
minute, preferably faster. The current server does not have <br>
anywhere near that speed.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I have tried a few improvements to the client and server model, and I was able to get a 5x improvement in performance. On my desktop, this went from less than 3000 requests/sec to 15000 requests/sec. (both tests were with 40 simultaneous clients). I believe this would allow my underpowered desktop to process these requests in under 30 seconds, let alone the autobuilder server.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
My conclusion based on this is that we need to rewrite the way runqueue<br>
makes the hash computations, perhaps seeding the cache in advance so we<br>
minimise the number of single calls. We can make one query for all 9000<br>
entries to the server on a single connection/request, or batch in<br>
blocks of 1000 or similar.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Would that work? Aren't the hash calculations dependent on each other? My concern here is that we might end up making much more than 9000 requests as unihashes are changed by the server.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The other option would be a custom server/protocol but I think what<br>
we're using is fine, we just need to change how.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I've recently been investigating how to optimize the protocol for the heavy request load expirenced during signature generation. There are a few ways I have thought of that this could be done.</div><div dir="auto"><br></div><div dir="auto">1) use the HTTP upgrade header to "convert" an http connection to an endpoint to an optimized byte stream based hash lookup protocol (this is effectively how websockets work). This protocol would be a simple persistent connection where the client would send a line to the server and the server would reply with a single line that was the unihash.</div><div dir="auto"><br></div><div dir="auto">2) The same hash lookup protocol as before, but over an actual websocket.</div><div dir="auto"><br></div><div dir="auto">3) a custom TCP protocol, perhaps some sort of JSON based message protocol. This protocol would also be able to enter a "stream" mode, just like the previous two for optimal transfer speed.</div><div dir="auto"><br></div><div dir="auto">The trick here of course is that if you are leaving persistent connections open, you need a way to service the streams fairly between the clients, and you can't use threads, since we've seen that they do not scale well. Using Pythons asynchronous API (async/await) can be used to solve this and run everything on a single thread.</div><div dir="auto"><br></div><div dir="auto">The three proposed solutions have their tradeoffs. 1 & 2 would require 3rd party modules, as there is no AIO http server in vanilla python. 3 can be done with vanilla python (3.5 or later), but doesn't have any of the advantages that we get from using HTTP, such as proxy transversal, HTTPS negotiation, and others. I'm not sure how important these things are anyway.</div><div dir="auto"><br></div><div dir="auto">One advantage here is that the runqueue changes are relatively simple, and isolated to the hash equivalence mix in class.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I have some profiling code for the hashserver but its doing profiling<br>
per thread so doesn't integrate well until we decide what form the<br>
codebase should have. Simpler is looking to perform better.<br>
<br>
I haven't had a chance to work on these patches, nor will I over the<br>
next few days as I'm taking a break but I think I know what we need to<br>
do now.<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Thanks a lot for looking into this, Richard. Have a good break. We can pick it back up next week and try to figure out the best path.</div><div dir="auto"><br></div><div dir="auto">Joshua Watt</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
<br>
Richard<br>
<br>
<br>
</blockquote></div></div></div>