Eric: Undo revision 3267 by Yjytalamago (Talk)

2010-11-24T07:55:23Z

Undo revision 3267 by Yjytalamago (Talk)

← Older revision	Revision as of 07:55, 24 November 2010
(No difference)

Mparkachov: Building OE images in EC2 cloud

2010-09-13T19:30:30Z

Building OE images in EC2 cloud

← Older revision		Revision as of 19:30, 13 September 2010
Line 115:		Line 115:
	===<span style="color:red">Caution</span>===		===<span style="color:red">Caution</span>===
	Don't forget to terminate your EC2 instance after build, otherwise you'll be paying for running instance.		Don't forget to terminate your EC2 instance after build, otherwise you'll be paying for running instance.

			[[Category:User]]

Mparkachov: Building openembedded images on the Amazon EC2 infrastructure

2010-09-07T09:46:58Z

Building openembedded images on the Amazon EC2 infrastructure

New page

== Prerequisites ==

In order to run instances (virtual machines) on an Amazon cloud infrastructure one need to get corresponding account and register on the [http://aws.amazon.com AWS (Amazon Web Service)]. [http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/ Getting Started Guide] will help to get up and running, or basics could be checked on the [http://www.ubuntu.com/cloud/public/deploy Ubuntu clould guide].

The procedure uses Cloud Config utility from Ubuntu EC2 images to configure an instance and perform build. [http://ubuntu-smoser.blogspot.com/2010/03/introducing-cloud-inits-cloud-config.html Here] is an introduction of the utility with links to examples and documenation.

The build described here uses [http://www.angstrom-distribution.org/ Ångström distribution] and based on the setup-scripts described [http://www.angstrom-distribution.org/building-angstrom here].

=== Selecting AMI ===

The build was tested on 06 Septmeber 2010 with 32-bit Ubuntu 10.10 beta image, AMI ami-be6288d7. At the time of testing the 10.04 (LTS) image was not working due to the [https://bugs.launchpad.net/ubuntu/+source/gzip/+bug/524366 bug in shipped gzip]. The status of the cloud config in the older versions of the ubuntu and debian is not clear. The list of the released images for ubuntu could be found [http://uec-images.ubuntu.com/query/ here].

The procedure completely configures the image, so there is no need to use customized images or EBS-based images. The S3-based image provides more than enough "ephemerial" storage to perform the build.

=== Revisions ===

The test was performed for beagleboard machine with minimal-image. The build is based on 41569c5f4eb81baeb26f3a429602c8568d1c24fe commit, which corresponds to the [http://cgit.openembedded.org/cgit.cgi/openembedded/tag/?id=testing_2010-08-23 testing_2010-08-23 branch].

== Build procedure ==

* Sign in to the [https://console.aws.amazon.com/s3/home AWS Management Console]

* Request either spot or on demand instance.

* Select AMI ami-be6288d7

* In the User data paste following cloud configuration data

<pre>
#cloud-config
runcmd:
- dpkg-reconfigure -f noninteractive dash
- chown ubuntu /mnt
- [ su, -c, "( git clone git://gitorious.org/angstrom/angstrom-setup-scripts.git /mnt/oe )", ubuntu ]
- [ su, -c, "( cd /mnt/oe; sh oebb.sh config beagleboard )", ubuntu ]
- [ su, -c, "sed -i 's/^#PARALLEL/PARALLEL/g' /mnt/oe/build/conf/local.conf", ubuntu ]
- ln -s /mnt/oe /var/www/oe
- echo "<html><body><h2>Angstrom environment</h2><p>Browse building <a href='oe/'>directory</a></p><p>or watch <a href='oe/bitbake.log'>bitbake.log</a></p><br/><pre>---------------- local.conf ----------------" >/var/www/index.html
- cat /mnt/oe/build/conf/local.conf >>/var/www/index.html
- echo "</pre></body></html>" >>/var/www/index.html
- [ sh, -c, echo "========= Starting build =========" ]
- [ su, -c, "( cd /mnt/oe; sh oebb.sh update commit 41569c5f4eb81baeb26f3a429602c8568d1c24fe 2>&1 >/mnt/oe/update.log ; source ~/.oe/environment ; nohup bitbake minimal-image 2>&1 >/mnt/oe/bitbake.log & )", ubuntu ]

debconf_selections: |
debconf debconf/priority select critical
dash dash/sh boolean false

packages:
- sed
- wget
- cvs
- subversion
- git-core
- coreutils
- unzip
- texi2html
- texinfo
- libsdl1.2-dev
- docbook-utils
- gawk
- python-pysqlite2
- diffstat
- help2man
- make
- gcc
- build-essential
- g++
- desktop-file-utils
- chrpath
- libxml2-utils
- xmlto
- python-psyco
- apache2

</pre>

* Submit request

After around 15 minutes required to initialize, start and install software one could point the browser to primitive page on the public DNS address of the instance to browse the build directory or get log of the bitbake.

The resulting machine deploy image could be downloaded directly from the browser, so no additional software is required.

=== Performance ===

The test was run on the c1.medium instance, which is correspondingly accessible and powerful enough. Spot instance was used to further reduce price of the build. The build was using parallel make with 2 threads and parallel bitbake with 2 threads as well.

The complete cycle from starting the instance to finished build took 2 hours 20 minutes.

The m1.small type of virtual machine proved to be underpowered (the build was cancelled after 5 hours).

The procedure retrieves quite a lot of data (sources) from the internet, but at the time of testing an incoming traffic was free and outgound traffic was negligibly small. After November 1, the inbound traffic will be priced as well, which will introduce additional costs. Caching of the downloaded files on an EBS volume could be considered in the future.

=== Changing targets ===

There are 3 important parameters to the build process, which could be changed.

* Machine architecture
Following line in the user data configures machine, which is in this case is beagleboard
<pre><nowiki>
sh oebb.sh config beagleboard
</nowiki></pre>

* State of OE repository
The commit could be specified to get repeatable tests. For the trunk build <code>commit 41569c5f4eb81baeb26f3a429602c8568d1c24fe</code> should be removed.
<pre><nowiki>
sh oebb.sh update commit 41569c5f4eb81baeb26f3a429602c8568d1c24fe
</nowiki></pre>

* Target image
The target image (in this case minimal-image) could be specified on the following line.
<pre><nowiki>
nohup bitbake minimal-image
</nowiki></pre>

===<span style="color:red">Caution</span>===
Don't forget to terminate your EC2 instance after build, otherwise you'll be paying for running instance.

EC2CloudBuild - Revision history

Eric: Undo revision 3267 by Yjytalamago (Talk)

Mparkachov: Building OE images in EC2 cloud

Mparkachov: Building openembedded images on the Amazon EC2 infrastructure