<div dir="ltr">Is <span style="font-size:12.8px">BB_DISKMON_DIRS enabled by default?</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Quick grep shows it only in local.conf.sample*:</span></div><div><div><span style="font-size:12.8px">meta/conf/local.conf.sample:BB_DISKMON_DIRS = "\</span></div><div><span style="font-size:12.8px">meta/conf/local.conf.sample.extended:# inode is running low, it is enabled when BB_DISKMON_DIRS is set.</span></div><div><span style="font-size:12.8px">meta/conf/local.conf.sample.extended:#BB_DISKMON_DIRS = "STOPTASKS,${TMPDIR},1G,100K WARN,${SSTATE_DIR},1G,100K"</span></div><div style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">and my jenkins builds are very close to default oe-core nodistro config, so I don't think I have that enabled.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Maybe I should enable it, or maybe it should be enabled by default if we cannot fix this exception stream.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 27, 2017 at 10:08 AM, Patrick Ohly <span dir="ltr"><<a href="mailto:patrick.ohly@intel.com" target="_blank">patrick.ohly@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Thu, 2017-06-15 at 08:48 +0200, Martin Jansa wrote:<br>
> This issue exists for very long time.<br>
><br>
><br>
> I know that when the builder runs out of disk space there are multiple<br>
> things which might go wrong (I've seen bad archives on premirrors, bad<br>
> sstate archives caused by this), so this issue isn't the main problem,<br>
> but still would be nice to fail faster.<br>
><br>
><br>
> In last build which was running for some 9 hours, it was first<br>
> building for maybe 2 hours before it run out of disk space and this<br>
> morning there is 50MB log just from bitbake output stored on the<br>
> jenkins master. Repeating following message very quickly<br>
><br>
><br>
> # grep -c "Errno 28" consoleText.txt<br>
> 42986<br>
><br>
><br>
> ERROR: Running command [['world'], 'build']<br>
> Traceback (most recent call last):<br>
> File "/home/jenkins/oe/world/shr-<wbr>core/bitbake/lib/bb/event.py", line<br>
> 211, in fire(event=<bb.event.<wbr>HeartbeatEvent object at 0x7fcfed3e96a0>,<br>
> d=<bb.data_smart.DataSmart object at 0x7fd00330b198>):<br>
><br>
> > fire_class_handlers(event, d)<br>
> if worker_fire:<br>
> File "/home/jenkins/oe/world/shr-<wbr>core/bitbake/lib/bb/event.py", line<br>
> 134, in fire_class_handlers(event=<bb.<wbr>event.HeartbeatEvent object at<br>
> 0x7fcfed3e96a0>, d=<bb.data_smart.DataSmart object at<br>
> 0x7fd00330b198>):<br>
> continue<br>
> > execute_handler(name, handler, event, d)<br>
><br>
> File "/home/jenkins/oe/world/shr-<wbr>core/bitbake/lib/bb/event.py", line<br>
> 106, in execute_handler(name='<wbr>runqueue_stats', handler=<function<br>
> runqueue_stats at 0x7fd0020c6158>, event=<bb.event.HeartbeatEvent<br>
> object at 0x7fcfed3e96a0>, d=<bb.data_smart.DataSmart object at<br>
> 0x7fd00330b198>):<br>
> try:<br>
> > ret = handler(event)<br>
> except (bb.parse.SkipRecipe, bb.BBHandledException):<br>
> File<br>
> "/home/jenkins/oe/world/shr-<wbr>core/openembedded-core/meta/<wbr>classes/buildstats.bbclass", line 212, in runqueue_stats(e=<bb.event.<wbr>HeartbeatEvent object at 0x7fcfed3e96a0>):<br>
> done = isinstance(e, bb.event.BuildCompleted)<br>
> > system_stats.sample(e, force=done)<br>
> if done:<br>
> File<br>
> "/home/jenkins/oe/world/shr-<wbr>core/openembedded-core/meta/<wbr>lib/buildstats.py", line 148, in SystemStats.sample(event=<bb.<wbr>event.HeartbeatEvent object at 0x7fcfed3e96a0>, force=False):<br>
> data +<br>
> > b'\n')<br>
> self.last_proc = now<br>
> OSError: [Errno 28] No space left on device<br>
><br>
><br>
> It would be better to exit completely when something as bad as Errno<br>
> 28 happens.<br>
<br>
</div></div>Do you have BB_DISKMON_DIRS active? Probably yes.<br>
<br>
The reason why it did not trigger here might be that the build ran out<br>
of disk space so quickly that the disk monitoring had no chance to<br>
detect the problem before system stat sampling itself started failing<br>
with the error above.<br>
<br>
System stat sampling and disk monitoring are hooking into the same<br>
event, so my theory is that once the system stat sampling fails, disk<br>
monitoring code no longer runs.<br>
<br>
I'm not sure what exactly the right fix is: detect uncaught OSError like<br>
28 in the bitbake event loop and abort the build, and/or catch the error<br>
in buildstats.py and ignore it so that the normal disk monitoring can<br>
happen?<br>
<br>
I know how to do the latter, but not the former.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Best Regards, Patrick Ohly<br>
<br>
The content of this message is my personal opinion only and although<br>
I am an employee of Intel, the statements I make here in no way<br>
represent Intel's position on the issue, nor am I authorized to speak<br>
on behalf of Intel on this matter.<br>
<br>
<br>
<br>
</font></span></blockquote></div><br></div>