Custom Ubuntu Chef AMI with Ruby Enterprise
This tutorial will guide you through setting up your own Ubuntu AMI that runs a Chef client to point at any Chef server. It assumes you’ve worked with EC2 before, but have never used Chef to manage your EC2 servers. This is an alternative approach to using knife ec2 server create, and is based on the steps at Opscode, with added explanation and alternative approaches.
We’ve found that knife’s EC2 support isn’t very reliable yet, and in any case its bootstrapping is designed to run on fresh instances, which means waiting a long time when you want to try out your Chef recipes on a fresh instance. This approach means you don’t wait so long on fresh instances, as Chef and RubyGems are already installed.
This is an example that you should analyse and change as you see fit. NB especially the step that installs Ruby Enterprise Edition, as you might not want this. Use the Opscode instructions as a fallback.
Being authentic
You should have the EC2 API tools installed and the following environment variables set up. For example, in your ~/.profile or ~/.bashrc:
export EC2_HOME=$HOME/bin/ec2-api-tools-1.3-46266
export EC2_PRIVATE_KEY=$HOME/.ec2/pk-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
export EC2_CERT=$HOME/.ec2/cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
export EC2_REGION=eu-west-1
export AMAZON_ACCESS_KEY_ID='ABCDEFGHIJKLMNOP'
export AMAZON_SECRET_ACCESS_KEY='ABCDEFGHIJKLMNOP'
When logging into your instances, you’ll need to use your Amazon key pair, which you should retrieve from your AWS Management Console (under EC2). You can pass the path to the .pem file using ssh’s -i option, or you can fix it more permanently into your ~/.ssh/config:
host *.eu-west-1.compute.amazonaws.com
User ubuntu
IdentityFile /Users/andrew/.ssh/camelpunch.pem
StrictHostKeyChecking no
Replace eu-west-1 with your region (or use *.compute.amazonaws.com if you like). Notice I’ve also turned off StrictHostKeyChecking for EC2 servers. This saves me typing ‘yes’ every time I log into a fresh EC2 machine. I’ve also set the user to ‘ubuntu’, so you don’t have to specify that when connecting.
Create the template instance
We need to create a new instance based on the latest Ubuntu EC2 AMI that we can take a snapshot of and use as the base for all of our servers managed with Chef. This is our opportunity to bake in packages that we don’t want to have to install every time a new Chef client is created. We shouldn’t go too crazy though: Chef is intended as a configuration management tool, and the more we customise our base AMI, the more we’ll potentially have to undo in scripts if we don’t want it later.
Decide which Ubuntu base AMI you want to use. Ubuntu publish different versions depending on architecture, region and root device storage. Note: in our tests, the new t1.micro instance type did not wake up properly after following these instructions.
To start a new instance, visit the previous link, copy the AMI number you wish to use and substitute it for the one below. Substitute ‘camelpunch’ with the name of your own key pair. I’m specifying -z, the availability zone, because we have reservations in specific zones that should be used to save money. You might not care which zone is used in your region, so you could leave the option out. You’ll want to use the -t option if you want something other than the m1.small instance type.
ec2run -k camelpunch -z eu-west-1b ami-16794c62
This command will return some output. The second column of the second row is the new instance ID, which begins ‘i-‘. Pass this into ec2din:
ec2din i-fb18518c
The fourth column of the second row of output should begin ‘ec2-‘. This is the public address of the new instance. If you don’t see it, your machine isn’t ready yet, so wait a while. Log in:
ssh ec2-46-51-135-170.eu-west-1.compute.amazonaws.com
The following steps are lifted directly from the above-linked Opscode instructions. This stops everything we’re about to type from being permanently stored in the AMI, and switches to the root user:
export HISTSIZE=0
sudo su -
export HISTSIZE=0
We now enable the multiverse repositories to make more Ubuntu packages available for use. Edit /etc/apt/sources.list and uncomment the appropriate lines.
Update the package cache and upgrade the installed packages, then install what we need:
apt-get update
apt-get upgrade
apt-get install -y runit build-essential # only need build-essential if you want to compile stuff on your servers
Install Ruby Enterprise Edition
Since we use Ruby Enterprise Edition for all of our development and deployment, we prefer to have only one version of Ruby installed on the box. Chef works fine with this version, but be warned: some of the Opscode cookbooks (you’ll learn about cookbooks in future tutorials) expect MRI Ruby to be installed, and some depend on installing REE in addition to MRI. If you want to use MRI or another version of Ruby for the system-wide Ruby installation, go ahead and install that instead of this step. Also, be sure to replace the package URL with the appropriate one from the REE download page. This will depend on the architecture you have chosen for your EC2 instance.
cd /tmp
wget http://rubyforge.org/frs/download.php/71100/ruby-enterprise_1.8.7-2010.02_i386_ubuntu10.04.deb
dpkg -i ruby-enterprise_1.8.7-2010.02_i386_ubuntu10.04.deb
Install Chef and make it start at boot
REE comes with RubyGems, so you don’t need to install it separately. We do need the Chef gems, though:
gem install chef --no-ri --no-rdoc
We then need the files from the standard Opscode install (check their gist for the latest client.rb):
mkdir -p /etc/chef /var/log/chef /var/chef /etc/sv/chef-client/log/main
curl https://raw.github.com/gist/319106/c1b2fecced45228f97de8a7e50559c94d4a5664b/ec2_client.rb > /etc/chef/client.rb
# You might like a different interval from the default 1800 seconds (30 mins). If so, edit /etc/sv/chef-client/run.
curl https://raw.github.com/gist/601385/545a16ab8319db592be1aa6be3d9b8ab2c9dd44d/run > /etc/sv/chef-client/run
curl https://raw.github.com/gist/601385/5ea8b75cad724e29fc0057832c399db71ff9f673/log-run > /etc/sv/chef-client/log/run
chmod +x /etc/sv/chef-client/log/run /etc/sv/chef-client/run
ln -s /etc/sv/chef-client /etc/service/chef-client
ln -s /usr/bin/sv /etc/init.d/chef-client
sleep 5
sv stop chef-client
And the same clean up operation, which gives a few harmless warnings:
rm ~root/.ssh/authorized_keys
rm ~ubuntu/.ssh/authorized_keys
rm ~root/.gemrc ~ubuntu/.gemrc
rm -rf ~/.gem /tmp/rubygems*
rm /var/cache/apt/archives/*deb
rm /etc/sv/chef-client/log/main/current /var/log/chef/client.log
Build the AMI
At this point, you can easily build an AMI from the running instance by right-clicking your running ‘template instance’ from your Amazon EC2 console and choosing “Create Image”. The Opscode instructions show you the hard way. We like easy ways.
Once this has been selected, your instance will reboot. Leave it alone until the new AMI has popped up in your EC2 console.
Using the AMI
You’re now free to use the AMI for new Chef-managed servers. But how do you do that? The clever bit about setting up your AMI to start chef-client at boot is that you can pass in instance options to configure the box to use the Chef server you are using, and also specify a run list. So, gem install chef on your own machine and use knife to get the instance data you need to pass to ec2run:
knife ec2 instance data -F json -r 'role[rails]' > roles/rails-data.json # assuming you have a role called 'rails'
When starting a server with ec2run you pass in this information with the -f option, the path to the JSON file, above. When the server starts up, Chef knows to read this information and start churning through the run list. It also registers with the Chef server specified in the JSON file so your server knows it has a new client.
Starting to write cookbooks
All this is well and good, but if you haven’t written or installed any cookbooks, you have nothing to use Chef for. The next tutorial will cover customising existing recipes and writing your own.
In the meantime, check out the Opscode help pages. Sadly, blogs about Chef are hard to come by, but Googling might help!