<div dir="ltr">Changes in V2:<div><br></div><div>Added a missing "<span style="font-size:14px">do_cve_check_write_manifest[</span><wbr style="font-size:14px"><span style="font-size:14px">dirs] = "${DEPLOY_DIR_IMAGE}"</span>".</div><div><br></div><div>//Ming Liu</div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-07-24 6:47 GMT+02:00 <span dir="ltr"><<a href="mailto:liu.ming50@gmail.com" target="_blank">liu.ming50@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">From: Ming Liu <<a href="mailto:peter.x.liu@external.atlascopco.com">peter.x.liu@external.<wbr>atlascopco.com</a>><br>
<br>
The current cve_check has some drawbacks:<br>
- cve_check_write_rootfs_<wbr>manifest is being added to<br>
ROOTFS_POSTPROCESS_COMMAND, but it actually has nothing to do related<br>
to generating a rootfs, but just deploying cve reports to DEPLOY_DIR,<br>
this leads do_rootfs/do_image unnecessarily rerun when the task hash<br>
changed of cve_check_write_rootfs_<wbr>manifest.<br>
- The generated cve manifest should be image specific, but it does not<br>
work in that way so far, for instance, if the users bitbake two<br>
images at the same time, both of them will have all CVE reports even<br>
though the recipes are not being depended neither at run time nor at<br>
build time by a image, for instance, if a user run:<br>
"bitbake core-image-minimal-initramfs core-image-minimal", he/she<br>
will find the cve manifest of core-image-minimal-initramfs cantians<br>
some recipe names depended by core-image-minimal but not by itself.<br>
- do_cve_check should be excluded for image recipes, it does not make<br>
sense to be run for image.<br>
- There is a code snippet checking and cleaning cve-check recipe result<br>
before re-building, introduced by following commit 85b4941c:<br>
[ cve-check: clean cve-check recipe result before re-building ]<br>
this checking is necessary which I agree with the author, but the<br>
code snippet was put into cve_check_write_rootfs_<wbr>manifest which<br>
supposes to run in ROOTFS_POSTPROCESS_COMMAND, and it's trying to<br>
locate the cve reports by PN, then how could it work since the PN is<br>
always the image recipe name?<br>
<br>
So this patch mainly aims to fix the above drawbacks, by:<br>
- Add a inter-task do_cve_check_write_manifest to image recipes,<br>
detecting do_cve_check dependencies from BB_TASKDEPDATA, this ensures<br>
all the cve-check reports in the generated manifest are really<br>
related to a specific image build.<br>
- Add do_cve_check task only for non-image recipes.<br>
- Move the "cleaning cve-check recipe result before re-building" logic<br>
to do_cve_check.<br>
- Drop the CVE_CHECK_TMP_FILE and related processes, I think it's sort<br>
of tricky to have to delete it in a bb.cooker.CookerExit handler, we<br>
can use CVE_CHECK_DB_FILE to determine if the database had been<br>
updated correctly or not.<br>
<br>
Signed-off-by: Ming Liu <<a href="mailto:peter.x.liu@external.atlascopco.com">peter.x.liu@external.<wbr>atlascopco.com</a>><br>
---<br>
meta/classes/cve-check.bbclass | 123 ++++++++++++++-------<br>
.../cve-check-tool/<a href="http://cve-check-tool_5.6.4.bb" rel="noreferrer" target="_blank">cve-check-<wbr>tool_5.6.4.bb</a> | 9 +-<br>
2 files changed, 86 insertions(+), 46 deletions(-)<br>
<br>
diff --git a/meta/classes/cve-check.<wbr>bbclass b/meta/classes/cve-check.<wbr>bbclass<br>
index 6f5b0f5..2860afe 100644<br>
--- a/meta/classes/cve-check.<wbr>bbclass<br>
+++ b/meta/classes/cve-check.<wbr>bbclass<br>
@@ -29,7 +29,6 @@ CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvd.db"<br>
<br>
CVE_CHECK_LOCAL_DIR ?= "${WORKDIR}/cve"<br>
CVE_CHECK_LOCAL_FILE ?= "${CVE_CHECK_LOCAL_DIR}/cve.<wbr>log"<br>
-CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"<br>
<br>
CVE_CHECK_DIR ??= "${DEPLOY_DIR}/cve"<br>
CVE_CHECK_MANIFEST ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_<wbr>NAME}${IMAGE_NAME_SUFFIX}.cve"<br>
@@ -46,12 +45,27 @@ CVE_CHECK_CVE_WHITELIST = "{\<br>
'CVE-2014-2524': ('6.3','5.2',), \<br>
}"<br>
<br>
+python () {<br>
+ if bb.data.inherits_class('image'<wbr>, d):<br>
+ bb.build.addtask('do_cve_<wbr>check_write_manifest', 'do_build', None, d)<br>
+ else:<br>
+ bb.build.addtask('do_cve_<wbr>check', None, 'do_unpack', d)<br>
+ bb.build.addtask('do_cve_<wbr>checkall', 'do_build', 'do_cve_check', d)<br>
+}<br>
+<br>
python do_cve_check () {<br>
"""<br>
Check recipe for patched and unpatched CVEs<br>
"""<br>
<br>
- if os.path.exists(d.getVar("CVE_<wbr>CHECK_TMP_FILE")):<br>
+ # Remove the cve report in previous build to prevent old cves kept in<br>
+ # CVE_CHECK_DIR in case there is no result for current one, which can<br>
+ # happen on version upgrading or when cve/recipe is whitelisted.<br>
+ old_cve_file = os.path.join(d.getVar("CVE_<wbr>CHECK_DIR"), d.getVar("PN"))<br>
+ if os.path.exists(old_cve_file):<br>
+ bb.utils.remove(old_cve_file)<br>
+<br>
+ if os.path.exists(d.getVar("CVE_<wbr>CHECK_DB_FILE")):<br>
patched_cves = get_patches_cves(d)<br>
patched, unpatched = check_cves(d, patched_cves)<br>
if patched or unpatched:<br>
@@ -61,53 +75,86 @@ python do_cve_check () {<br>
bb.note("Failed to update CVE database, skipping CVE check")<br>
}<br>
<br>
-addtask cve_check after do_unpack before do_build<br>
do_cve_check[depends] = "cve-check-tool-native:do_<wbr>populate_sysroot cve-check-tool-native:do_<wbr>populate_cve_db"<br>
do_cve_check[nostamp] = "1"<br>
<br>
-python cve_check_cleanup () {<br>
- """<br>
- Delete the file used to gather all the CVE information.<br>
- """<br>
-<br>
- bb.utils.remove(e.data.getVar(<wbr>"CVE_CHECK_TMP_FILE"))<br>
+# Dependency placeholder<br>
+do_cve_checkall () {<br>
+ :<br>
}<br>
<br>
-addhandler cve_check_cleanup<br>
-cve_check_cleanup[eventmask] = "bb.cooker.CookerExit"<br>
+do_cve_checkall[recrdeptask] = "do_cve_checkall do_cve_check"<br>
+do_cve_checkall[noexec] = "1"<br>
<br>
-python cve_check_write_rootfs_<wbr>manifest () {<br>
+python do_cve_check_write_manifest () {<br>
"""<br>
Create CVE manifest when building an image<br>
"""<br>
<br>
- import shutil<br>
-<br>
- if d.getVar("CVE_CHECK_COPY_<wbr>FILES") == "1":<br>
- deploy_file = os.path.join(d.getVar("CVE_<wbr>CHECK_DIR"), d.getVar("PN"))<br>
- if os.path.exists(deploy_file):<br>
- bb.utils.remove(deploy_file)<br>
-<br>
- if os.path.exists(d.getVar("CVE_<wbr>CHECK_TMP_FILE")):<br>
- bb.note("Writing rootfs CVE manifest")<br>
+ if d.getVar("CVE_CHECK_CREATE_<wbr>MANIFEST") != "1":<br>
+ return<br>
+<br>
+ from datetime import datetime<br>
+<br>
+ depends = set()<br>
+ rdepends = set()<br>
+ cve_dir = d.getVar("CVE_CHECK_DIR")<br>
+ manifest_name = d.getVar("CVE_CHECK_MANIFEST")<br>
+ taskdepdata = d.getVar("BB_TASKDEPDATA", False)<br>
+ for taskdep in taskdepdata:<br>
+ if taskdepdata[taskdep][1] == "do_cve_check_write_manifest":<br>
+ for dep in taskdepdata[taskdep][3]:<br>
+ if taskdepdata[dep][1] == "do_cve_check":<br>
+ pn = taskdepdata[dep][0]<br>
+ # Skip the recipes do not have cve report<br>
+ if not os.path.exists(os.path.join(<wbr>cve_dir, pn)):<br>
+ continue<br>
+ if pn.find('-native') != -1 or pn.find('-cross') != -1:<br>
+ depends.add(pn)<br>
+ else:<br>
+ rdepends.add(pn)<br>
+<br>
+ # Sort the results<br>
+ depends = sorted(depends)<br>
+ rdepends = sorted(rdepends)<br>
+<br>
+ bb.note("Writing CVE manifest")<br>
+ if rdepends or depends:<br>
+ utctime = datetime.utcfromtimestamp(os.<wbr>path.getmtime(d.getVar("CVE_<wbr>CHECK_DB_FILE")))<br>
+ summary = "CVE database was updated on %s UTC\n" % utctime<br>
+ summary += "Patched/Unpatched CVEs found in target recipes: %s\n" % " ".join(rdepends) if rdepends else ""<br>
+ summary += "Patched/Unpatched CVEs found in native/cross recipes: %s\n" % " ".join(depends) if depends else ""<br>
+ summary += "Check out the details below\n\n"<br>
+ with open(manifest_name, 'a') as f:<br>
+ f.write(summary)<br>
+<br>
+ # First, write CVEs impacting target system, then write CVEs impacting build system<br>
+ for pn in rdepends + depends:<br>
+ cve_file = os.path.join(cve_dir, pn)<br>
+ if os.path.exists(cve_file):<br>
+ with open(cve_file, 'r') as input:<br>
+ cve_text = input.read()<br>
+ with open(manifest_name, 'a') as output:<br>
+ output.write(cve_text)<br>
+<br>
+ if manifest_name and os.path.exists(manifest_name):<br>
deploy_dir = d.getVar("DEPLOY_DIR_IMAGE")<br>
link_name = d.getVar("IMAGE_LINK_NAME")<br>
- manifest_name = d.getVar("CVE_CHECK_MANIFEST")<br>
- cve_tmp_file = d.getVar("CVE_CHECK_TMP_FILE")<br>
-<br>
- shutil.copyfile(cve_tmp_file, manifest_name)<br>
-<br>
- if manifest_name and os.path.exists(manifest_name):<br>
- manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)<br>
- # If we already have another manifest, update symlinks<br>
- if os.path.exists(os.path.<wbr>realpath(manifest_link)):<br>
- os.remove(manifest_link)<br>
- os.symlink(os.path.basename(<wbr>manifest_name), manifest_link)<br>
- bb.plain("Image CVE report stored in: %s" % manifest_name)<br>
+ manifest_link = os.path.join(deploy_dir, "%s.cve" % link_name)<br>
+ # If we already have another manifest, update symlinks<br>
+ if os.path.exists(os.path.<wbr>realpath(manifest_link)):<br>
+ os.remove(manifest_link)<br>
+ os.symlink(os.path.basename(<wbr>manifest_name), manifest_link)<br>
+ bb.plain("Image CVE report stored in: %s" % manifest_name)<br>
+<br>
+ if d.getVar("CVE_CHECK_COPY_<wbr>FILES") != "1":<br>
+ bb.utils.remove(d.getVar('CVE_<wbr>CHECK_DIR'), True)<br>
}<br>
<br>
-ROOTFS_POSTPROCESS_COMMAND_<wbr>prepend = "${@'cve_check_write_rootfs_<wbr>manifest; ' if d.getVar('CVE_CHECK_CREATE_<wbr>MANIFEST') == '1' else ''}"<br>
-do_rootfs[recrdeptask] += "${@'do_cve_check' if d.getVar('CVE_CHECK_CREATE_<wbr>MANIFEST') == '1' else ''}"<br>
+do_cve_check_write_manifest[<wbr>vardepsexclude] = "BB_TASKDEPDATA"<br>
+do_cve_check_write_manifest[<wbr>recrdeptask] = "do_cve_checkall do_cve_check"<br>
+do_cve_check_write_manifest[<wbr>dirs] = "${DEPLOY_DIR_IMAGE}"<br>
+do_cve_check_write_manifest[<wbr>nostamp] = "1"<br>
<br>
def get_patches_cves(d):<br>
"""<br>
@@ -264,13 +311,9 @@ def cve_write_data(d, patched, unpatched, cve_data):<br>
bb.note("Writing file %s with CVE information" % cve_file)<br>
f.write(write_string)<br>
<br>
- if d.getVar("CVE_CHECK_COPY_<wbr>FILES") == "1":<br>
+ if d.getVar("CVE_CHECK_COPY_<wbr>FILES") == "1" or d.getVar("CVE_CHECK_CREATE_<wbr>MANIFEST") == "1":<br>
cve_dir = d.getVar("CVE_CHECK_DIR")<br>
bb.utils.mkdirhier(cve_dir)<br>
deploy_file = os.path.join(cve_dir, d.getVar("PN"))<br>
with open(deploy_file, "w") as f:<br>
f.write(write_string)<br>
-<br>
- if d.getVar("CVE_CHECK_CREATE_<wbr>MANIFEST") == "1":<br>
- with open(d.getVar("CVE_CHECK_TMP_<wbr>FILE"), "a") as f:<br>
- f.write("%s" % write_string)<br>
diff --git a/meta/recipes-devtools/cve-<wbr>check-tool/<a href="http://cve-check-tool_5.6.4.bb" rel="noreferrer" target="_blank">cve-check-tool_5.6.<wbr>4.bb</a> b/meta/recipes-devtools/cve-<wbr>check-tool/<a href="http://cve-check-tool_5.6.4.bb" rel="noreferrer" target="_blank">cve-check-tool_5.6.<wbr>4.bb</a><br>
index 4829b11..55fe690 100644<br>
--- a/meta/recipes-devtools/cve-<wbr>check-tool/<a href="http://cve-check-tool_5.6.4.bb" rel="noreferrer" target="_blank">cve-check-tool_5.6.<wbr>4.bb</a><br>
+++ b/meta/recipes-devtools/cve-<wbr>check-tool/<a href="http://cve-check-tool_5.6.4.bb" rel="noreferrer" target="_blank">cve-check-tool_5.6.<wbr>4.bb</a><br>
@@ -36,20 +36,17 @@ do_populate_cve_db() {<br>
<br>
# In case we don't inherit cve-check class, use default values defined in the class.<br>
cve_dir="${CVE_CHECK_DB_DIR}"<br>
- cve_file="${CVE_CHECK_TMP_<wbr>FILE}"<br>
-<br>
[ -z "${cve_dir}" ] && cve_dir="${DL_DIR}/CVE_CHECK"<br>
- [ -z "${cve_file}" ] && cve_file="${TMPDIR}/cve_check"<br>
<br>
bbdebug 2 "Updating cve-check-tool database located in $cve_dir"<br>
# --cacert works around curl-native not finding the CA bundle<br>
- if cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-<wbr>certificates.crt -d "$cve_dir" ; then<br>
- printf "CVE database was updated on %s UTC\n\n" "$(LANG=C date --utc +'%F %T')" > "$cve_file"<br>
- else<br>
+ if ! cve-check-update --cacert ${sysconfdir}/ssl/certs/ca-<wbr>certificates.crt -d "$cve_dir" ; then<br>
bbwarn "Error in executing cve-check-update"<br>
if [ "${@'1' if bb.data.inherits_class('cve-<wbr>check', d) else '0'}" -ne 0 ] ; then<br>
bbwarn "Failed to update cve-check-tool database, CVEs won't be checked"<br>
fi<br>
+ # Clean up leftovers of a unsuccessful updating<br>
+ rm -rf $cve_dir/*<br>
fi<br>
}<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
2.7.4<br>
<br>
</font></span></blockquote></div><br></div>