Amazon’s ECS (EC2 Container Service) is a great way to manage and deploy docker containers. We at Lumos Labs were drawn to it for its ease of setup and tight integration with other AWS services. Using ECS’s blue-green deployment, Application Load Balancers, and CloudWatch, we were able to quickly stand up production-level services running at scale. Unfortunately, triggering deployments consistently and reliably became an unexpected pain point.

To answer to this problem, we built and open-sourced Broadside, a command-line tool that lets developers confidently manage deployments for any kind of application.

Example Usage

Assume your application has a configured broadside.conf.rb:

Broadside.configure do |config|
  config.application = 'railsapp'
  config.default_docker_image = 'quay.io/lumoslabs/railsapp'
  config.targets = {
    production_web: {
      cluster: 'production-cluster',
      scale: 4,
      command: ['bundle', 'exec', 'unicorn', '-c', 'config/unicorn.conf.rb'],
      env_file: '.env.production',
      predeploy_commands: [
        ['bundle', 'exec', 'rake', 'db:migrate']
      ]
    },
    staging_worker: {
      cluster: 'staging-cluster',
      scale: 1,
      command: ['bundle', 'exec', 'rake', 'resque:work'],
      env_file: '.env.staging'
    }
  }
end

Running:

broadside deploy short --target staging_worker --tag feature_branch

would deploy the feature_branch tagged image from quay.io/lumoslabs/railsapp to the existing ECS service called railsapp_staging_worker at a scale of 1, importing any environment variables defined in .env.staging into the ECS task definition.

When using the following command and flags:

broadside deploy full --target production_web --tag v2.0

Broadside will first launch one-off tasks sequentially running the configured predeploy_commands, then deploy to the railsapp_production_web service at a scale of 4.

How it works

Under the hood, Broadside is simply leveraging ECS’s blue green deployment. The ECS service is updated to use a new task definition revision created with the provided tag and configurations. Deployment events from AWS are continually polled until the service reaches a stable state (i.e. the new containers are able to start and respond to health checks). In the event of a timeout or error, Broadside will simply mark the deployed revision as “inactive” and the previous revision will be deployed. This will ensure that the most recent task definition revision is always the configuration that is in service.

Other Useful Commands

Running many applications on a large cluster can complicate administration. For this reason we developed the following utility commands to manage deployed services:

• broadside targets - displays a summarized table of information for all deploy targets for the current application
• broadside ssh --target example_target - SSH onto the host machine running the docker container
• broadside bash --target example_target - brings up a bash shell inside the container directly
• broadside status --target example_target - displays the service’s instances that current containers are running on, environment variables, and much more
• broadside logtail --target example_target - tails logs on a running container for the specified service

Running one-off commands

Broadside allows you to spin up a container with a given tag, execute a command on it, and then spin it down. For example, if you wanted to run a database migration using a specific version of the application, you could run:

broadside run --command 'bundle exec rake db:migrate' --tag v1.9.8 --target example_target

Broadside will start an ECS task running bundle exec rake db:migrate that will display logs upon completion.

Bootstrapping a new service and task_definition

In our prior examples, it was assumed you had already defined a service and initial task_definition in the Amazon GUI. With the bootstrap command, you can tell Broadside to do that for you by running:

broadside bootstrap --target example_target

Simply configure your broadside.conf.rb with the additional values in service_config and task_definition_config:

Broadside.configure do |config|
  config.application = 'myapp'
  config.default_docker_image = 'quay.io/example/image_path'
  config.aws.ecs_default_cluster = 'mycluster'
  config.targets = {
    example_target: {
      scale: 2,
      command: ['java', '-cp', '*:.', 'com.example.application.myapp.StartApp'],
      env_file: '.env.production'
      service_config: {
        deployment_configuration: {
          minimum_healthy_percent: 50
        }
      },
      task_definition_config: {
        container_definitions: [
          {
            cpu: 512,
            memory: 2048
          }
        ]
      }
    }
  }
end

An initial task_definition will be created with additional configurations at the task_definition_config key, which accepts valid ECS task_definition options. Following that, a service called myapp_example_target will be created with the configuration at the :service_config key, which may contain any valid ECS service options.

Postscript

We have found Broadside to dramatically simplify a lot of the workflow involved with using ECS. If you use ECS, consider trying out Broadside and let us know what you think!