<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 8, 2016 at 12:14 PM, Joshua Lock <span dir="ltr"><<a href="mailto:joshua.g.lock@intel.com" target="_blank">joshua.g.lock@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">From: Joshua Lock <<a href="mailto:joshua.lock@collabora.co.uk">joshua.lock@collabora.co.uk</a>><br>
<br>
The individual partitions created by wic are sparse but without<br>
this change the assembled image is written as one (potentially<br>
very) large file.<br>
<br>
Preserve sparseness in the assembled image by passing the sparse<br>
conversion symbol.<br>
<br>
[YOCTO #9099]<br>
<br>
Signed-off-by: Joshua Lock <<a href="mailto:joshua.lock@collabora.co.uk">joshua.lock@collabora.co.uk</a>><br>
---<br>
scripts/lib/wic/utils/partitionedfs.py | 2 +-<br>
1 file changed, 1 insertion(+), 1 deletion(-)<br>
<br>
diff --git a/scripts/lib/wic/utils/partitionedfs.py b/scripts/lib/wic/utils/partitionedfs.py<br>
index 5a103bb..3e2b420 100644<br>
--- a/scripts/lib/wic/utils/partitionedfs.py<br>
+++ b/scripts/lib/wic/utils/partitionedfs.py<br>
@@ -340,7 +340,7 @@ class Image(object):<br>
source = part['source_file']<br>
if source:<br>
# install source_file contents into a partition<br>
- cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc" % \<br>
+ cmd = "dd if=%s of=%s bs=%d seek=%d count=%d conv=notrunc,sparse" % \<br>
(source, image_file, self.sector_size,<br>
part['start'], part['size'])<br>
exec_cmd(cmd)<br></blockquote><div><br></div><div><br></div><div><br></div><div>This might be dangerous. Even in recent version of DD (coreutils 8.25), it might not detect in source file mapped/unmapped blocks properly, </div><div>(just use is_nul() function to check if block contains only zeros, instead of e.g. fiemap). </div><div>Result might be so, that in output file block which must be zeros is actually become unmapped. And as consequence, if it might not be written to destination drive.</div><div><br></div><div>Let me show example based on Ostro Project sparse images:</div><div><br></div><div><div>kad@orava:/tmp> ls -ls 2222</div><div>308416 -rw-r--r-- 1 kad users 5280628736 Apr 6 00:53 2222</div></div><div><div>kad@orava:/tmp> grep /MappedBlocksCount 2222.bmap<br></div><div> <MappedBlocksCount> 77104 </MappedBlocksCount></div></div><div><br></div><div>Now, let's copy it with dd:</div><div>kad@orava:/tmp> ~/cu/coreutils-8.25/src/dd if=2222 of=4444 conv=notrunc,sparse bs=4096</div><div>1289216+0 records in</div><div>1289216+0 records out</div><div>5280628736 bytes (5.3 GB, 4.9 GiB) copied, 1.31239 s, 4.0 GB/s</div><div>kad@orava:/tmp> bmaptool create 4444 -o 4444.bmap<br></div><div><div>kad@orava:/tmp> ls -ls 4444</div></div><div>308348 -rw-r--r-- 1 kad users 5280628736 Apr 9 13:22 4444<br></div><div><div>kad@orava:/tmp> grep /MappedBlocksCount 4444.bmap</div><div> <MappedBlocksCount> 77087 </MappedBlocksCount></div><div>kad@orava:/tmp></div></div><div><br></div><div>As you can see, in result of dd copied file we have less blocks mapped than in original.</div><div><br></div><div>Diffing block ranges: (- original file, + dd copied)</div><div><div> <BlockMap></div><div>- <Range> 0-4 </Range></div><div>- <Range> 768-3321 </Range></div><div>- <Range> 4608-7161 </Range></div><div>+ <Range> 0 </Range></div><div>+ <Range> 768-770 </Range></div><div>+ <Range> 772-774 </Range></div><div>+ <Range> 776 </Range></div><div>+ <Range> 780-3321 </Range></div><div>+ <Range> 4608-4610 </Range></div><div>+ <Range> 4612-4614 </Range></div><div>+ <Range> 4616 </Range></div><div>+ <Range> 4620-7161 </Range></div><div>- <Range> 1289211-1289215 </Range><br></div><div>+ <Range> 1289211 </Range></div><div>+ <Range> 1289215 </Range></div><div> </BlockMap></div></div><div><br></div><div>As you can see, some blocks that must be cleaned (e.g. 1-3 around MBR or in the end, 1289211-1289215 where GPT bits located) are not marked as mapped,</div><div>thus in case of sparse aware writing tool used to write to destination drive it might result as unwritten zero blocks, thus some old data would left on device.</div><div><br></div><div>I would suggest not to accept this patch right now, and do copy function that would be aware about real mapped/unmapped blocks.</div><div>Something similar that I've did for Ostro Project here: <a href="https://github.com/ostroproject/meta-ostro/commit/d883169c3266c3c44f22db5f75ae94c4cf0f2924">https://github.com/ostroproject/meta-ostro/commit/d883169c3266c3c44f22db5f75ae94c4cf0f2924</a> </div><div>but adopted to be generic for use inside wic.</div><div><br></div></div><div><br></div>-- <br><div class="gmail_signature">br, Alexander Kanevskiy</div>
</div></div>