[bitbake-devel] How bitbake works(Draft)
Robert Yang
liezhi.yang at windriver.com
Mon Dec 5 03:27:07 UTC 2011
Hi Richard,
Here is a doc about how bitbake works, it may help the bitbake
newbie to understand how bitbake works, I'm very glad to add it
to the public bitbake repository as documentation if it is useful.
please feel free to give you comments.
How bitbake works
Robert Yang
The document tries to explain a common bitbake build, it is based on
yocto 1.2 and the bitbake version is:
$ bitbake --version
BitBake Build Tool Core version 1.15.0, bitbake version 1.15.0
The entrance of the bitbake is the main() function in bin/bitbake, the
main() function does the following things:
1) Parse command line options
2) Initialize the configuration from the command line options
3) Get the ui and server from the configuration
4) Initialize logger
5) Initialize server
6) Register the idle functions
7) Initialize cooker
8) Parse command line action
9) Add the cooker to the server
10) Start the server and waiting for tasks to run
11) Start the ui
12) The ui runs command on the server
Then the build will start, and stop when the build is done.
1, Parse commandline options
-> options (will be used by BBConfiguration)
/
/
parser.parse_args (return two values)
\
\ -> args (will be saved to configuration.pkgs_to_build)
bitbake uses the python moudule optparse to parse the options:
options, args = parser.parse_args(sys.argv)
For example:
$bitbake gzip -cpatch
After the parse, the "options" is a class' instance, and:
options.cmd = "patch"
There are also many other members in options which would have default
values.
and:
args = ['/buildarea/lyang1/poky/bitbake/bin/bitbake', 'gzip']
The args saves the arguments which has not been matched by the
option.
2, Initialize the configuration from the commandline options
BBConfiguration
options -----------------------> configuration
configuration = BBConfiguration(options)
BBConfiguration will convert the dictionary of "options" to the
member of BBConfiguration, and add a member called pkgs_to_build.
configuration.pkgs_to_build.extend(args[1:])
Add the arguments to configuration.pkgs_to_build except args[0] (which
is the bitbake itself)
3, Get the ui and server from the configuration
-> ui_main (will be used by server.launchUI)
/
/
configuration
\
\-> server (will be used to run tasks)
ui_main = get_ui(configuration)
The get_ui is defined in bin/bitbake, it would use the knotty
(lib/bb/ui/knotty.py) by default, and use __import__ to import the
module at runtime. (The "import" can't import the module
dynamically). The ui_main is the main() function in
lib/bb/ui/knotty.py
The similar to the server, the server is lib/bb/server/process.py by
default.
4) Initialize logger
* set the log level.
---------> verbose ------->
/ \
/ \
configuration ------------> debug --------> bb.msg.init_msgconfig
\ /
\ /
--> debug_domains ------->
bb.msg.init_msgconfig(configuration.verbose, configuration.debug,
configuration.debug_domains)
* set the log handler
handler = bb.event.LogHandler()
logger.addHandler(handler)
There are a lot of log handlers, for example: FileHandler,
SocketHandler, SMTPHandler and so on, but bitbake uses it's own
handler bb.event.LogHandler to handle the messages. The Loghandler
is defined in lib/bb/event.py, it re-defines the function emit which
is used to send the message to the UI(fire(record, None)), and
redefines the function filter, which is used to filter the messages,
determine whether to emit the message(all messages would be emitted
).
5, Initialize server
server = server.BitBakeServer()
server.initServer()
The BitBakeServer() is defined in lib/bb/server/process.py, and the
server.initServer() will init the server:
--> ui_channel
/
Pipe ()
\
--> server_channel ->\ --> command_channel
\ / --> event_queue
----> def waitEvent -> ProcessServer ----> event
/ / \ ---> _idlefunctions
ProcessEventQueue --------->/ ---> keep_running
\
----> def getEvent
The Pipe () from multiprocessing returns two duplex connection
objects connected by a pipe here, The ui_channel is used for ui to
send and receive messages with the server, and the server_channel is
used for cooker to send and receive messages with the server.
The ProcessEventQueue is defined in lib/bb/server/process.py: It is
the subclass of multiprocessing.queues.Queue, the
multiprocessing.queues.Queue has not defined waitEvent and getEvent
which are needed by the UI, so define them here.
The ProcessServer is the real server that we used, it is the
subclass of Process which is from the multiprocessing:
* command_channel = server_channal
* event_queue = ProcessEventQueue
* event = EventAdapter(event_queue)
The EventAdapter is used for adapting the API of the queue, the
bb.event need call a send() method, but our actual queue only has
put(), so convert the send() to put().
* The _idlefunctions is used for saving the functions which would be
run by the server, the register_idle_function will set the value.
* keep_running = Event()
The Event is from multiprocessing, it will run the
keep_running.set(), the set() is to set the internal flag to true.
All processes waiting for it to become true are awakened. Processes
that call wait() once the flag is true will not block at all.
6, Register the idle functions
idle = server.getServerIdleCB()
getServerIdleCB --> register_idle_function --> _idlefunctions
This is used for assigning a value for _idlefunctions in the
server.
The function buildFile and buildTargets will use
server_registration_cb, these are the most important two functions
for building. And the configuration is one of the most important
data which saves the configuration data for a build.
7, Initialize cooker
cooker = bb.cooker.BBCooker(configuration, idle, initialenv)
The BBCooker class is defined in lib/bb/cooker.py, and the
initialization is:
+----------------+
|stateShutdown |
|stateStop |
|getCmdLineAction|
BBCooker -> cmds_sync ->|getVariable |
/ | \ / |setVariable |
/ | \ / |resetCooker |
server_registration_cb | command +----------------+
configuration | \
configuration.event_data| \
| -> cmds_async (see list below)
V
LoadConfiguration
/ | \
/ | \
configuration.data | configuration.cmd
(bb.data.init) | (BB_DEFAULT_TASK or build)
V
parseConfigurationFiles
|
|
V
+---------------------------------------------------+
| bb.parse.init_parser --> bb.siggen.init |
| bblayers.conf (Signature generator)|
| BBLAYERS |
| BBPATH |
| conf/bitbake.conf |
| base.bblcass |
| INHERIT |
| register BBHANDLERS |
| bb.codeparser.parser_cache_init |
| save data to configuration.data |
+---------------------------------------------------+
* loadConfigurationData
Parse the config file(.conf, base.bbclass, and some .bbclass
used by the INHERIT), the data will be saved in configuration.data
(bb.data.init()), we can think it as a dictionary.
* command
Bitbake divides the commands into two kinds: cmds_sync and
cmds_async, cmds_sync has been list in the graph, and cmds_async
list is:
buildFile, buildTargets, generateDepTreeEvent, generateDotGraph,
generateTargetsTree, findConfigFiles, findFilesMatchingInDir,
findConfigFilePath, showVersions, showEnvironmentTarget,
showEnvironment, parseFiles, reparseFiles, compareRevisions
These commands are defined in lib/bb/command.py.
* The configuration.event_data would be used by the function
fire().
8, Parse command line action
cooker.parseCommandLine()
The parseCommandLine() will save the command line action in the
dictionary cook.commandlineAction, the default action is
buildTargets:
self.commandlineAction['action'] = ["buildTargets",
self.configuration.pkgs_to_build, self.configuration.cmd]
The ui will use the server to run "getCmdLineAction", and then
run buildTargets(self.configuration.pkgs_to_build, self.configuration.cmd)
to build the target. The buildTargets is defined in BBCooker.
9, Add the cooker to the server:
server.addcooker(cooker)
server.cooker = cooker
server.server.cooker = cooker
Add the cooker to BitBakeServer and ProcessServer, the first
"sever" is the "BitBakeServer", and the second server is the
real server "ProcessServer".
10, Start the server
server.detach(cooker_logfile)
This will invoke the detach which is defined in
lib/bb/server/process.py::BitBakeServer:
def detach(self, cooker_logfile):
self.server.start()
return
The start() is defined in multiprocessing, which means start the
process’ activity.
11, Start the ui
server_connection = server.establishConnection()
The BitBakeServerConnection will be used to communicate between
the server and the ui
server.launchUI(ui_main, server_connection.connection,
server_connection.events)
+-----------------+
launchUI ----> server_main ----> |main in knotty.py|
+-----------------+
* The default ui is knotty.py
* The main function in knotty.py is running now.
12, The ui runs command on the server
* Get the command line action
* Run the command one the server
* Waiting for the result from the server and handle them.
cmdline = server.runCommand(["command"])
If we run "bitbake target", then the flow is:
+-----------------------------------------------------------+
| server.runCommand(["getCmdLineAction"]) |
| +========+ | |
| | The ui | | return "buildTargets" |
| +========+ | |
| V |
| server.runCommand(["buildTargets"]) |
+-----------------------------------------------------------+
| ^
V |
command.py::buildTargets |
| |
V | return
+------------------------------------------------+ | the
| cooker.py::buildTargets | | result
| +======+ | | |
| |cooker| V | |
| +======+ updateCache | |
| (.bb and .bbclass files are parsed) | |
| taskdata = bb.taskdata.TaskData | |
| rq = bb.runqueue.RunQueue | |
| server_registration_cb(buildTargetsIdle, rq) | |
+------------------------------------------------+ |
| |
| |
V |
+----------------------------------------------------------+
| +===========+ process.py::run |
| |The server | | |
| |is running | | |
| +===========+ V |
| process.py::main |
+----------------------------------------------------------+
| ^
| | return the result
V |
+----------------------------------------------------------+
| +========+ cooker.py::buildTargetsIdle |
| | cooker | | |
| +========+ V |
| rq.execute_runqueue |
| (the tasks are in rq, and begin to run |
+----------------------------------------------------------+
// Robert
More information about the bitbake-devel
mailing list