[bitbake-devel] RFC - avoid "database is locked" exception in persistent_data.py

[Lu, Lianhao]

Hi guys,

We sometimes met "OperationalError: database is locked" exceptions in persist_data.py 
during the bitbake parsing the recipes. It seems this is caused by a burst of reads/writes to the 
database during the recipe parsing time. http://bugzilla.pokylinux.org/show_bug.cgi?id=1761 

It seems to me that the only way to avoid this kind of exception completely in bitbake is to
serialize the access to the database by using some kind of multiprocess lock in SQLTable._execute() 
to make sure only one process is executing the SQL statement. Though this has performance penalty,
but in bitbake case, it only increase the parsing time less than 0.3 seconds out of a total 21 seconds.

Another way is to increase the current retry count from 500 to a very large number or to infinite. But I'm
not sure whether there are other situations might raise the same exception in which case we do need to
throw it out instead of to continuously retry.

Any comment?

Best Regards,
-Lianhao Lu