[bitbake-devel] How bitbake works(Draft)
Robert Yang
liezhi.yang at windriver.com
Thu Mar 8 08:20:39 UTC 2012
Hi Richard,
Here is an updated one.
Changes:
1) Remove the function's arguments as many as possible, try
to make it easy to maintain.
2) Change the name from "How bitbake works" to
"A Brief introduction to bitbake internal"
3) Try to explain rather than put the code there.
4) Some other trivial fix
Please see the document at the end of the email.
// Robert
On 12/07/2011 09:01 PM, Richard Purdie wrote:
> Hi Robert,
>
> On Mon, 2011-12-05 at 11:27 +0800, Robert Yang wrote:
>> Here is a doc about how bitbake works, it may help the bitbake
>> newbie to understand how bitbake works, I'm very glad to add it
>> to the public bitbake repository as documentation if it is useful.
>> please feel free to give you comments.
>
> I like this and I do think we need to improve the bitbake documentation.
> We do need to ensure there is some consistent style and format for the
> documents though so anything going into the repository really needs to
> be added to the existing manual.
>
> I'm also a little concerned that in its current form (listing arguments
> to functions), this documentation would be high maintenance. If we could
> separate it from the code implementation details a little bit, I think
> it would make it less likely to bitrot quite as quickly.
>
> Cheers,
>
> Richard
>
>
A Brief introduction to bitbake internal
The document tries to explain how a common bitbake build runs, it is
based on the yocto project.
The entrance of the bitbake is the main() function in bin/bitbake, the
main() function does the following things:
1, Parse command line options
* options are something like -b, -f, -c and so on.
2, Initialize the configuration
* The configuration saves the options data here, and will save more
later, the metadata(.conf, .inc, .bbclass and so on) will be saved
in configuration.data.
3, Get the ui and server from the configuration
* The ui is the user interface, you can specify which interface you
prefer.
* The server is used for running the tasks.
4, Initialize the logger
* The logger is used for logging the messages.
5, Initialize server
* Prepare the server for running
6, Register the idle functions
* Register the functions to the server, and the server will invoke
the function when it is idle.
7, Initialize cooker
* The cooker manages one bitbake build run, and it will save the
metadata to configuration.data here.
8, Parse command line action
* The actions are: fetch, unpack, patch and so on, the default is
build.
9, Add the cooker to the server
* Let the server can use the data in the cooker
10, Start the server and waiting for tasks to come to run
11, Start the ui
12, The ui submits commands to the server
* Then the build will start, and stop when it is done.
Let's see them one by one:
1, Parse command line options
* parser.parse_args
It returns two values: options and args, the options will be used
by the BBConfiguration
options, args = parser.parse_args
For example:
$bitbake gzip -cpatch
After the parse:
options.cmd = "patch"
The args saves the arguments which has not been matched by the
option:
args = ['/buildarea/lyang1/poky/bitbake/bin/bitbake', 'gzip']
2, Initialize the configuration from the commandline options
BBConfiguration
options -----------------------> configuration
configuration = BBConfiguration
BBConfiguration converts the dictionary of the "options" to the
member of BBConfiguration, and add a member called pkgs_to_build,
which will be used for saving the build pkg:
configuration.pkgs_to_build.extend
The configuration and configuration.data are the most important data
during a build, they save everything of the build.
3, Get the ui and server from the configuration
There are several UI and server in yocto, the default UI and server
are knotty(lib/bb/ui/knotty.py) and (lib/bb/server/process.py), you
can specify another ui by "bitbake -u <ui>", and "bitbake -t <server>"
for server.
The ui will be started by the server: server.launchUI.
The server will be used for running the tasks.
4, Initialize the logger
* set the log level.
bb.msg.init_msgconfig
* set the log handler
handler = bb.event.LogHandler()
logger.addHandler
There are lots of log handlers, for example: FileHandler,
SocketHandler, SMTPHandler and so on, but bitbake uses it's own
handler bb.event.LogHandler to handle the messages. This Loghandler
is defined in lib/bb/event.py, it re-defines the function emit
which is used to send the message to the UI, and redefines the
function filter which is used for determining whether to emit
the message(all messages would be emitted).
5, Initialize server
server = server.BitBakeServer()
server.initServer()
The BitBakeServer() is defined in lib/bb/server/process.py, and the
server.initServer() will init the server:
--> ui_channel
/
Pipe ()
\
--> server_channel ->\ --> command_channel
\ / --> event_queue
----> def waitEvent -> ProcessServer ----> event
/ / \ ---> _idlefunctions
ProcessEventQueue --------->/ ---> keep_running
\
----> def getEvent
The Pipe () from multiprocessing returns two duplex connection
objects connected by a pipe here, The ui_channel is used for ui to
send and receive messages with the server, and the server_channel is
used for cooker to send and receive messages with the server.
The ProcessEventQueue is defined in lib/bb/server/process.py: It is
the subclass of multiprocessing.queues.Queue, the
multiprocessing.queues.Queue has not defined waitEvent and getEvent
which are needed by the UI, so define them here.
The ProcessServer is the real server that we used, it is the
subclass of Process which is from the multiprocessing:
* command_channel = server_channal
To receive the command to run.
* event_queue = ProcessEventQueue
Supply the API waitEvent and getEvent, so that the ui can use the
same API for different servers.
* event = EventAdapter
The EventAdapter is used for adapting the API of the queue, the
bb.event need call a send() method, but our actual queue only has
put(), so convert the send() to put().
* The _idlefunctions is used for saving the functions which would be
run by the server, the register_idle_function will set the value.
* keep_running = Event()
The Event is from multiprocessing, it will run the
keep_running.set(), the set() is to set the internal flag to true.
All processes waiting for it to become true are awakened. Processes
that call wait() once the flag is true will not block at all.
6, Register the idle functions
idle = server.getServerIdleCB()
getServerIdleCB --> register_idle_function --> _idlefunctions
This is used for assigning a value for _idlefunctions in the
server.
The function buildFile and buildTargets will use
server_registration_cb, these are the most important two functions
for building.
7, Initialize cooker
cooker = bb.cooker.BBCooker
The BBCooker class is defined in lib/bb/cooker.py, and the
initialization is:
+----------------+
|stateShutdown |
|stateStop |
|getCmdLineAction|
BBCooker -> cmds_sync ->|getVariable |
/ | \ / |setVariable |
/ | \ / |resetCooker |
server_registration_cb | command +----------------+
configuration | \
configuration.event_data| \
| -> cmds_async (see list below)
V
LoadConfiguration
/ | \
/ | \
configuration.data | configuration.cmd
(bb.data.init) | (BB_DEFAULT_TASK or build)
V
parseConfigurationFiles
|
|
V
+---------------------------------------------------+
| bb.parse.init_parser --> bb.siggen.init |
| bblayers.conf (Signature generator)|
| BBLAYERS |
| BBPATH |
| conf/bitbake.conf |
| base.bblcass |
| INHERIT |
| register BBHANDLERS |
| bb.codeparser.parser_cache_init |
| save metadata to configuration.data |
+---------------------------------------------------+
* loadConfigurationData
Parse the config file(.conf, base.bbclass, and some .bbclass
used by the INHERIT), the data will be saved in configuration.data
(bb.data.init()), we can think it as a dictionary.
* command
Bitbake divides the commands into two kinds: cmds_sync and
cmds_async, cmds_sync has been list in the graph, and cmds_async
list is:
buildFile, buildTargets, generateDepTreeEvent, generateDotGraph,
generateTargetsTree, findConfigFiles, findFilesMatchingInDir,
findConfigFilePath, showVersions, showEnvironmentTarget,
showEnvironment, parseFiles, reparseFiles, compareRevisions
These commands are defined in lib/bb/command.py.
* The configuration.event_data would be used by the function
fire().
8, Parse command line action
cooker.parseCommandLine()
The parseCommandLine() will save the command line action in the
dictionary cook.commandlineAction, the default action is
buildTargets:
self.commandlineAction['action'] = ["buildTargets",
self.configuration.pkgs_to_build, self.configuration.cmd]
The ui will use the server to run "getCmdLineAction", and then
run buildTargets to build the target. The buildTargets is defined
in BBCooker.
9, Add the cooker to the server:
server.cooker = cooker
Add the cooker to BitBakeServer, so the server can use the data in
the cooker
10, Start the server and waiting for tasks to come to run
server.detach
This will invoke the function server.start() which is in
lib/bb/server/process.py::BitBakeServer:
The server.start() is defined in multiprocessing, which means start
the process’ activity.
11, Start the ui
server_connection = server.establishConnection()
The BitBakeServerConnection will be used to communicate between
the server and the ui
server.launchUI
* The main() function in knotty.py is running now.
12, The ui submits commands to the server
* Get the command line action
* Run the command one the server
* Waiting for the result from the server and handle them.
cmdline = server.runCommand
If we run "bitbake target", then the flow is:
+-----------------------------------------------------------+
| server.runCommand |
| +========+ | |
| | The ui | | return "buildTargets" |
| +========+ | |
| V |
| server.runCommand(["buildTargets"]) |
+-----------------------------------------------------------+
| ^
V |
command.py::buildTargets |
| |
V | return
+------------------------------------------------+ | the
| cooker.py::buildTargets | | result
| +======+ | | |
| |cooker| V | |
| +======+ updateCache | |
| (.bb and .bbclass files are parsed) | |
| taskdata = bb.taskdata.TaskData | |
| rq = bb.runqueue.RunQueue | |
| server_registration_cb | |
+------------------------------------------------+ |
| |
| |
V |
+----------------------------------------------------------+
| +===========+ process.py::run |
| |The server | | |
| |is running | | |
| +===========+ V |
| process.py::main |
+----------------------------------------------------------+
| ^
| | return the result
V |
+----------------------------------------------------------+
| +========+ cooker.py::buildTargetsIdle |
| | cooker | | |
| +========+ V |
| rq.execute_runqueue |
| (the tasks are in rq, and begin to run) |
+----------------------------------------------------------+
The build will stop when the tasks in the runqueue has been done.
More information about the bitbake-devel
mailing list