Jason Ardell's blog

I've been using Chef with the Opscode Platform to orchestrate provisioning infrastructure on Amazon EC2. Until recently I was flying solo and didn't need to share Opscode or Amazon accounts with anybody. But as we got closer to rolling out the new infrastructure we needed to pull in a fully-qualified sysadmin to tune and lock down the servers.

Sharing an Opscode Organization

The problem with collaborating on Chef is that if more than one person is using the same Opscode Platform account then it's possible to overwrite each others' cookbooks/roles/etc. Our solution is to view Opscode Platform accounts similarly to environments (dev, staging, etc), and never have two developers working in the same Opscode "organization". Instead, each developer should write and debug in their own development organization, then promote code to an official organization.

One challenge of this approach is if you're working on multiple projects it'd be possible to accidentally get cookbooks (e.g. apache2) and roles (staging) mixed between projects on your dev organization. It could lead to some painful debugging.

Sharing an AWS Account

Another challenge of sharing an Amazon Web Services account is that AWS doesn't allow account delegation, and requires login with a user's Amazon username and password. Needless to say the guy I work for didn't care to share his Amazon log in credentials. Enter ElasticFox, a Firefox extension that mimics the AWS Management Console, but only requires your AWS Access Key Id and Secret Access Key (which Chef already requires). ElasticFox lets you terminate servers, add keys, assign Elastic IPs, and all sorts of other things--just the ticket I needed.

Thoughts? Suggestions?

How do you collaborate on Chef development? Particularly, how do you use Opscode's organizations to sandbox your development? Do you use a separate repository for each Chef project you're working on, or do you use a single git repo across all your projects?

Lately I've been working on deploying a multi-tier architecture to Amazon EC2 using Chef. Chef helps you boot and configure infrastructure on the cloud--one line of code to install Apache, PHP, or PostgreSQL for example.

When I first started with Chef I found that there was a pretty high learning curve, and I was in need of a best-practice workflow for building recipes. Through much weeping and gnashing of teeth I've fallen into what I feel is a pretty stable workflow, which is what this post documents.

Installing Chef

I've only done this once; Opscode's quick-start is pretty good with a few caveats...

On the "Install Ruby" step

I'd recommend using RVM. Then you'll need to install a Ruby, and I highly recommend creating a gemset while you're at it:

rvm install 1.8.7-p173
rvm gemset create chef

Then switch to your newly-installed Ruby and gemset. You'll make great use of this later when we get to installing the Chef gems.

rvm use 1.8.7-p173@chef

On the "Install Chef" step

Skip the "Install Chef" step for a second; we'll get around to it after we install git. [go install git]

Instead of installing the chef gem manually we'll do it in the awesome Rails 3 way: using bundler. Clone my chef repo and set up a .rvmrc

git clone https://github.com/ardell/Chef.git
mv .rvmrc.sample .rvmrc

Now install the bundler gem and bundle install. It'll install all the required Chef gems for you (as defined in Gemfile).

gem install bundler
bundle install

I've taken the liberty of tricking out that repo with a sample .chef/knife.rb file and some other wisdom I've picked up along the way. Copy the sample .chef/knife.rb and customize it with your credentials.

mv .chef/knife.rb.sample .chef/knife.rb

After you successfully get past the "Connect to the Opscode Platform" step, abandon the rest of the getting started guide; it'll just confuse you.

Set up AWS/EC2

You'll need to create a private key on Amazon Web Services since they use keys to authenticate you. Save it in your repo at .chef/[name-of-keypair].pem, you'll use it in the next step. See this blog post for help.

Booting Servers & Building Recipes

Here's where we get to a developer's workflow. The basic process is:

1. Import, create, and edit cookbooks
2. Add and update roles
3. Create or update servers

Lather, rinse repeat. When I'm developing I'll run steps 1, 2, and 3 in order many, many times until the chef run succeeds setting up a perfect server on the initial run.

Step 1: Import, create, and edit cookbooks

Cookbooks are what Chef calls collections of recipes--for example the Apache2 cookbook contains an Apache server recipe and several recipes to install various Apache modules (e.g. mod_ssl, mod_expires). To import a cookbook from Opscode's "blessed" cookbooks, rub the following command where [cookbook-name] is the name of a cookbook such as "apache2". Once you've imported a cookbook it's yours to edit however you like.

knife cookbook site vendor [cookbook-name]

Vendoring a cookbook only imports it into your local repository--chef server does not know anything about it yet. In order for you to use any cookbook on your server you must run:

knife cookbook upload [cookbook-name]

Note that when you vendor a cookbook, knife will actually run git commands without your permission. It will create a branch, pull down the cookbook, switch to master (not your previous branch!!) then merge in changes. I hate that knife does this, and I hope it is changed, but it is what it is.

Step 2: Add and update roles

Each server you boot will be a node, and the right way to assign recipes to nodes is by using roles. In practice roles are like tags, so for a given node you'll have an environment role (e.g. "staging"), and one or more functional roles (e.g. "webserver"). When chef-client runs, it determines which recipes to run based on what roles it has.

Create roles/webserver.rb and add the recipes you want into the run_list declaration. For example for a PHP/Apache role, webserver.rb might look like this:

name "webserver"
description "Apache2/PHP web server."
run_list(
  "recipe[postgresql::client]",
  "recipe[php]",
  "recipe[php::module_pgsql]",
  "recipe[apache2]",
  "recipe[apache2::mod_php5]"
)

Note that each recipe you reference in your run list will have to be defined (using knife cookbook site vendor [cookbook-name] or created by you), then uploaded using knife cookbook upload [cookbook-name].

When you're done editing your role, and after each time you edit it, you'll need to push those changes to chef server so your nodes can fetch them when they update. Executing the following command will parse the role and add/update it on chef server.

knife role from file webserver.rb

Step 3: Creating and Updating Servers

This is the exciting part. You should now be able to boot a server on ec2 by running the following command... and it's a long command...

knife ec2 server create 'role[webserver]' --ssh-key ec2-keypair --identity-file .chef/ec2-keypair.pem --ssh-user ubuntu --groups default --image ami-88f504e1 --flavor m1.small -Z us-east-1a

Part of the power of Chef is that you don't have to spin up a new server every time you update your recipes. Make sure your changes have been uploaded using knife cookbook upload [cookbook-name] or knife role from file [role].rb, then running the following command will update all the webservers in the staging environment:

knife ssh 'role:staging AND role:webserver' 'sudo chef-client -l debug' -a ec2.public_hostname --ssh-user ubuntu

What this knife command does under the covers is:

1. ask chef server to return a list of all webservers in the staging environment
2. ssh into each of these servers
3. run sudo chef-client -l debug (log level=debug) on each one

You can use knife ssh to interrogate your servers anytime, but remember you should be using sudo chef-client along with recipes to install packages and recipes in an automated fashion.

Flags for knife ec2 server create

• --flavor m1.small is the size of the instance we want
• --image ami-480df921 says use the canonical ubuntu image
• --ssh-key ec2-keypair is the name of the keypair you created on aws
• --ssh-user ubuntu says connect via ssh with the "ubuntu" username (instead of default=root)
• --identity-file .chef/ec2-keypair.pem says use the key at this location
• -Z us-east-1a Amazon doesn't have capacity of m1.small servers at us-east-1b (which is our default data center), so we use this one instead
• -a ec2.public_hostname tells chef to connect to the server using the ec2.public_hostname key, not the fully-qualified domain name (default), which is a address local to ec2
• --groups default the user groups that the created user should belong to

Conclusion

I'm learning more about using chef properly each time I use it, but this workflow has been pretty stable for me so far. Do you have a different workflow, or other commands that you prefer?

I've been using CloudKick to monitor some servers I've booted using Chef.  And they have one UX element that is so darn easy to use and so obvious that I can't imagine designing a site without stealing it.

In the footer of every page, they have a single text box with the label "I wish this page would".  The text box is small so I don't feel like I need to write a novel; this is crucial because I don't want to write a long bug report either.  The element doesn't have a submit button either--just the enter key.  When I submit a suggestion it thanks me for my feedback and doesn't interrupt what I was doing.  Simple and unobtrusive.

I wish every site did feedback like this.  When I'm using your site I don't want to be sent off to create an account at Uservoice or Get Satisfaction, and I don't care to upvote all the problems that other people had.  But it does give me satisfaction to know that you know about my problem.

One of my favorite services right now is Twilio, the cloud telephony provider who lets you build applications on top of phone or SMS. Their API is super simple and their service just works. I love Twilio.

That said, one area where they could improve is their error handling. Currently they require you to log in to their web interface and check for errors. I'd much prefer a push system that would notify me by email anytime Twilio couldn't parse the XML I gave it (for example).

So I spent an hour building a simple PHP command-line error reporter for Twilio. It's intended to be run from a cron script, and piped to a mailer. It also has an option to delete old notifications so you won't get the same errors over and over again.

Here it is: https://github.com/ardell/twilio-notifier

The readme gives more details, installation, and usage, but the upshot is that it's 1 command to install (thanks to Pearfarm) and 1 command to run it. It is built with Alan Pinstein's awesome CLImax PHP Command-Line Framework, which made parsing arguments super simple.

Give it a try and let me know what you think!

Joshua Silver and I are leading a "Getting Started with PHP and MySQL" workshop tonight at the Georgia Tech College of Computing from 6-8pm.  If you've been looking to get into web development you should join us.  There will be free pizza!

Below are the notes/walkthrough for our talk.

------

Get started here:
http://book.cakephp.org/view/219/Blog

--- 10.1 ---
We've already got a web server running with PHP and MySQL on Rackspace, so skip this part.

--- 10.1.1 ---
We already installed Cake for you.  Just go to http://[your.ip.address] and make sure the install was successful.

--- 10.1.2 ---

Here we go.  SSH into your box:
$> ssh root@[your.ip.address]
[enter password]

Now we'll connect to MySQL through the command-line client.  This means "connect to mysql with user 'cakephpuser', prompt me for a password, and use database 'cakephpdb'".
$> mysql -u cakephpuser -p cakephpdb
[password is "foo"]

Now let's set up our "students" table:
mysql> CREATE TABLE students (
mysql>   id            INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
mysql>   first_name    VARCHAR(50),
mysql>   last_name     VARCHAR(50),
mysql>   phone_number  VARCHAR(20),
mysql>   created       DATETIME DEFAULT NULL,
mysql>   modified      DATETIME DEFAULT NULL
mysql> );

You can use the following commands to make sure everything went smoothly:
mysql> show tables;
mysql> describe students;

Let's insert a record so we'll have a piece of test data:
mysql> INSERT INTO students (first_name, last_name, phone_number, created)
mysql> VALUES ('Foo', 'Bar', '404-555-1212', NOW());

Exit out of MySQL:
mysql> exit

--- 10.1.3 ---
We already did this for you.

--- 10.1.4 ---
We already did this for you.

--- 10.1.5 ---
We already did this for you.

--- 10.1.6 ---
Let's head over to our cakephp directory:
$> cd /var/www/cakephp

Now we'll create a Model for our Student object.  Feel free to use your favorite editor (vim, etc).
$> pico app/models/student.php

And add the following code:
<code>
<?php

class Student extends AppModel {

  var $name = 'Student';

}
</code>

--- 10.1.7 ---
Create a Controller for Students.
$> pico app/controllers/students_controller.php

And add the following code:
<code>
<?php

class StudentsController extends AppController {

  var $name = 'Students';

  function index() {
    $this->set('students', $this->Student->find('all'));
  }

}
</code>

--- 10.1.9 ---
We need to make a directory for app/views/students.
$> mkdir -p app/views/students

Create a View for our StudentsController::index action.
$> pico app/views/students/index.ctp

And add the following code:
<code>
<h1>Students</h1>
 
<table>
  <tr>
    <th>Id</th>
    <th>First Name</th>
    <th>Last Name</th>
    <th>Phone Number</th>
    <th>Created</th>
  </tr>
 
  <!-- Here is where we loop through our $students array, printing out student info -->
  <?php foreach ($students as $student) { ?>
    <tr>
      <td><?php echo $student['Student']['id']; ?></td>
      <td><?php echo $student['Student']['first_name']; ?></td>
      <td><?php echo $student['Student']['last_name']; ?></td>
      <td><?php echo $student['Student']['phone_number']; ?></td>
      <td><?php echo $student['Student']['created']; ?></td>
    </tr>
  <?php } ?>
</table>
</code>

Now you should have a page at your ip address displaying the list of students.  Go here in your web browser
http://[your.ip.address]/students

--- 10.1.9 ---
Edit your StudentsController.
$> pico app/controllers/students_controller.php

Add the following code right below your index action:
<code>
  function add() {
    if (!empty($this->data)) {
      if ($this->Student->save($this->data)) {
        $this->Session->setFlash('Your student has been saved.');
        $this->redirect(array('action' => 'index'));
      }
    }
  }
</code>

--- 10.1.10 ---
Create a view for the StudentsController::add action:
$> pico app/views/students/add.ctp

And add the following code:
<code>
<h1>Add Student</h1>
 
<?php
echo $form->create('Student');
echo $form->input('first_name');
echo $form->input('last_name');
echo $form->input('phone_number');
echo $form->end('Save Student');
?>
</code>

Now try it out by heading to:
http://[your.ip.address]/students/add

--- 10.1.11 ---
We'll skip the rest of the steps from this one on in the interest of time.

Pearfarm is an open way to share PHP code, pretty much what PEAR should have been from the start.  And it's super easy to host your own packages on Pearfarm.

I just now set up Clusterer (http://github.com/ardell/clusterer/) on Pearfarm, which can now be installed like so: "pear install channel://ardell.pearfarm.org/Clusterer".

[ create an account at pearfarm.org ]

> sudo pear install pearfarm.pearfarm.org/pearfarm
> pearfarm keygen
[ copy and paste key into your profile ]

> cd /my/repo
> pearfarm init
[ edit pearfarm.spec ]
channel should be: your-user-name.pearfarm.org

> sudo pear channel-discover your-user-name.pearfarm.org
> pearfarm build
> pearfarm push


That's it!

We typically do test-driven development where I work, which is an essential part of a "red-green refactor". The idea is that you write a test that fails (red) before you write any feature code (green).

This makes sure both that your code works as specified by your test, and perhaps more importantly that your test is actually testing something non-trivial.

Recently I forgot to write a test first. (gasp) Using Git I was able to go back in time before I had written the functionality, write failing tests, then add in my feature code to make the tests pass.

$> git checkout -b topic
[ write feature code ]
$> git commit -am "Feature without tests"
$> git checkout -b tests HEAD^
[ write failing tests ]
$> git commit -am "Failing tests"
$> git checkout topic
$> git rebase tests
[ tests should pass now! ]

I don't recommend using this for your regular workflow, but in a pinch if you've forgotten to test first this will bring you back to your more agile workflow.