[OE-core] [PATCH 2/2] cve-update-db-native: Remove hash column from database.

[Burton, Ross]

I must have failed to actually post them, and this is now in next.
I'll rebase and send instead!

Ross

On Thu, 18 Jul 2019 at 14:56, Pierre Le Magourou <lemagoup at gmail.com> wrote:
>
> Hello Ross,
>
> > Can you rebase this on top of the patches I sent yesterday to change
> > the path construction to use os.path.join() please.
>
> I can't find the patches your are referring to. My patches are rebased
> on the last master, and I don't see a patch from you in master-next.
>
> Pierre
>
> Le jeu. 18 juil. 2019 à 15:10, Burton, Ross <ross.burton at intel.com> a écrit :
> >
>
> >
> > Ross
> >
> > On Thu, 18 Jul 2019 at 13:41, Pierre Le Magourou <lemagoup at gmail.com> wrote:
> > >
> > > From: Pierre Le Magourou <pierre.lemagourou at softbankrobotics.com>
> > >
> > > djb2 hash algorithm was found to do collisions, so the database was
> > > sometime missing data. Remove this hash mechanism, clear and populate
> > > elements from scratch in PRODUCTS table if the current year needs an
> > > update.
> > >
> > > Signed-off-by: Pierre Le Magourou <pierre.lemagourou at softbankrobotics.com>
> > > ---
> > >  meta/classes/cve-check.bbclass                 | 12 ++++++------
> > >  meta/recipes-core/meta/cve-update-db-native.bb | 21 +++++++--------------
> > >  2 files changed, 13 insertions(+), 20 deletions(-)
> > >
> > > diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass
> > > index 512d4c7302..c00d2910be 100644
> > > --- a/meta/classes/cve-check.bbclass
> > > +++ b/meta/classes/cve-check.bbclass
> > > @@ -26,7 +26,7 @@ CVE_PRODUCT ??= "${BPN}"
> > >  CVE_VERSION ??= "${PV}"
> > >
> > >  CVE_CHECK_DB_DIR ?= "${DL_DIR}/CVE_CHECK"
> > > -CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve.db"
> > > +CVE_CHECK_DB_FILE ?= "${CVE_CHECK_DB_DIR}/nvdcve_1.0.db"
> > >
> > >  CVE_CHECK_LOG ?= "${T}/cve.log"
> > >  CVE_CHECK_TMP_FILE ?= "${TMPDIR}/cve_check"
> > > @@ -200,11 +200,11 @@ def check_cves(d, patched_cves):
> > >              c.execute("SELECT * FROM PRODUCTS WHERE PRODUCT IS ?", (product,))
> > >
> > >          for row in c:
> > > -            cve = row[1]
> > > -            version_start = row[4]
> > > -            operator_start = row[5]
> > > -            version_end = row[6]
> > > -            operator_end = row[7]
> > > +            cve = row[0]
> > > +            version_start = row[3]
> > > +            operator_start = row[4]
> > > +            version_end = row[5]
> > > +            operator_end = row[6]
> > >
> > >              if cve in cve_whitelist:
> > >                  bb.note("%s-%s has been whitelisted for %s" % (product, pv, cve))
> > > diff --git a/meta/recipes-core/meta/cve-update-db-native.bb b/meta/recipes-core/meta/cve-update-db-native.bb
> > > index 72d1f48835..3519beae5f 100644
> > > --- a/meta/recipes-core/meta/cve-update-db-native.bb
> > > +++ b/meta/recipes-core/meta/cve-update-db-native.bb
> > > @@ -30,7 +30,7 @@ python do_populate_cve_db() {
> > >      YEAR_START = 2002
> > >
> > >      db_dir = d.getVar("DL_DIR") + '/CVE_CHECK'
> > > -    db_file = db_dir + '/nvdcve.db'
> > > +    db_file = db_dir + '/nvdcve_1.0.db'
> > >      json_tmpfile = db_dir + '/nvd.json.gz'
> > >      proxy = d.getVar("https_proxy")
> > >      cve_f = open(d.getVar("TMPDIR") + '/cve_check', 'a')
> > > @@ -65,6 +65,10 @@ python do_populate_cve_db() {
> > >          c.execute("select DATE from META where YEAR = ?", (year,))
> > >          meta = c.fetchone()
> > >          if not meta or meta[0] != last_modified:
> > > +            # Clear products table entries corresponding to current year
> > > +            cve_year = 'CVE-' + str(year) + '%'
> > > +            c.execute("delete from PRODUCTS where ID like ?", (cve_year,))
> > > +
> > >              # Update db with current year json file
> > >              req = urllib.request.Request(json_url)
> > >              if proxy:
> > > @@ -91,27 +95,16 @@ python do_populate_cve_db() {
> > >      conn.close()
> > >  }
> > >
> > > -# DJB2 hash algorithm
> > > -def hash_djb2(s):
> > > -    hash = 5381
> > > -    for x in s:
> > > -        hash = (( hash << 5) + hash) + ord(x)
> > > -
> > > -    return hash & 0xFFFFFFFF
> > > -
> > >  def initialize_db(c):
> > >      c.execute("CREATE TABLE IF NOT EXISTS META (YEAR INTEGER UNIQUE, DATE TEXT)")
> > >      c.execute("CREATE TABLE IF NOT EXISTS NVD (ID TEXT UNIQUE, SUMMARY TEXT, \
> > >          SCOREV2 TEXT, SCOREV3 TEXT, MODIFIED INTEGER, VECTOR TEXT)")
> > > -    c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (HASH INTEGER UNIQUE, ID TEXT, \
> > > +    c.execute("CREATE TABLE IF NOT EXISTS PRODUCTS (ID TEXT, \
> > >          VENDOR TEXT, PRODUCT TEXT, VERSION_START TEXT, OPERATOR_START TEXT, \
> > >          VERSION_END TEXT, OPERATOR_END TEXT)")
> > >
> > >  def insert_elt(c, db_values):
> > > -    product_str = db_values[0] + db_values[1] + db_values[2] + db_values[3]
> > > -    hashstr = hash_djb2(product_str)
> > > -    db_values.insert(0, hashstr)
> > > -    query = "insert or replace into PRODUCTS values (?, ?, ?, ?, ?, ?, ?, ?)"
> > > +    query = "insert into PRODUCTS values (?, ?, ?, ?, ?, ?, ?)"
> > >      c.execute(query, db_values)
> > >
> > >  def parse_node_and_insert(c, node, cveId):
> > > --
> > > 2.11.0
> > >
> > > --
> > > _______________________________________________
> > > Openembedded-core mailing list
> > > Openembedded-core at lists.openembedded.org
> > > http://lists.openembedded.org/mailman/listinfo/openembedded-core