Approximately a year ago, we decided to switch away from using a third-party service for sending out emails to our customers in favor of building our own service, which we called MailFiend.

Why did we build our own service?

We had three main motivations: financial, technical, and personal.

Financially, we thought were paying too much for our third-party service. It didn’t make sense for us to pay tens of thousands of dollars every month when we knew we could create a better solution on our own.

Technically, we were being forced to replicate huge parts of our production and data warehouse databases every night to a database maintainted by our ESP (Email Service Provider), and keeping everything in sync was becoming an increasingly huge nightmare. In order to target key recipients with our mailings, it was necessary to sync the relevant data to their service as often as possible so that our emails would be sent to the correct recipients. Creating our own service to handle all the opt-outs and other email preferences would allow us to just read from our own database when necessary, sidestepping all the nasty synchronization issues. In addition, keeping this in-house allowed us access to more data to send more personalized emails, with data accurate as of the moment the email was fired.

Personally, we didn’t like their user interface. Since their tool was designed for far more use cases than we needed, there were certain abstractions that didn’t make sense to us, causing confusion when it came time for anyone to become familiar with the tool. By creating our own service, we could control the UI and tailor it for our own needs. We decided that we could build an intuitive interface ourselves, using Rails, and farm out only the actual email-send.

What did we build?

Templates

First off, we needed a way for our email team to build templates to send to our users. This required a way to edit the template without having to go into Github and changing an actual file. For this, we used ActiveAdmin, storing our template as an ActiveRecord model.

We also embedded ACE editor in order to allow the team to modify HTML more easily. This looked a little like:

function createAceEditor(parent_node) {
  var editor = $('#editor'),                         //editor is a pre element
  html_source = $('#template_html_source'),          //html_source is a textarea
  aceEditor = ace.edit(editor),
  editorSession = aceEditor.getSession(),
  HTMLMode = require('ace/mode/html').Mode;

  editor.appendTo(html_source.parent());
  editor.height(html_source.height()).width(html_source.width());

  editorSession.setMode(new HTMLMode());
  editorSession.setValue(html_source.val());
  editorSession.on('change', function() {
    html_source.val(editorSession.getValue());
  });
  editorSession.setTabSize(2);
  editorSession.setUseSoftTabs(true);

  html_source.hide();
}

Of course, it’s not enough to send the exact same email to every user; we needed to interpolate values based on the user we were sending it to. Because we frequently modify and change our templates, we wanted a way to simply specify what variables were available, and have those (and only those) variables be available to the template. We found Liquid, a templating engine developed by Shopify, that seemed to be exactly what we needed. Liquid allowed us to have the designers write something like Hello {{ first_name }}, we love you! , and have that render, while not allowing them to execute arbitrary ruby code, like Hello {{ MailFiend::Template.all.map(&:destroy) }}, we love you! .

Images

Next, we needed to be able to embed images into our emails. Whereas we used to upload these images to our third party service beforehand, we needed to deal with these ourselves now that we were attempting a homegrown solution. We decided to use Carrierwave, which allowed us to easily upload these images to Amazon S3 via ActiveAdmin and turn them into img tags by means of a Liquid filter (which looked like {{ 'image_name' | imgize: size: '100x100'}} ). We simply included Carrierwave in our Gemfile, ran rails generate uploader Image, added mount_uploader :image, ImageUploader, and it worked like a charm! (once we fiddled with some configs).

We got a model that looked like:

module MailFiend
  class Image < ActiveRecord::Base
    validates :name, :presence => true, :uniqueness => true
    validates :image, :presence => true

    mount_uploader :image, ImageUploader

    ...

To facilitate the transition from our old service to our new one, we created a simple tool that would let them change img tags into imgize filters:

More Interpolation

In the process of designing emails, we discovered that mere variable interpolation wasn’t enough to suit our needs. We were getting a lot of repeated HTML, such as in the header and footer, and wanted a way to abstract that out. To fulfill that need, we created a model called Copy that would store the lines of HTML. To get this HTML into the actual template we used Liquid again. By creating a filter called render_copy and a copy called footer, we could simply have our designers put {{'footer' | render_copy}} into a template and it would expand that into the footer for them.

Internationlization

Our last big step was to internationalize these emails. Since we were rolling out to multiple locales at once, we needed a way to have a translation for not just every template, but for every image as well, as some of them had English text in them. We used a gem called Globalize, which allowed us to have translated versions of specified models and make tabs for each locale available to the admin. By just adding the line translates :subject, :html_source, :text_source in the model, and having a migration to run Template.create_translation_table!({subject: :string, html_source: :text, text_source: :text}, those fields now supported translations. To make tabs for each locale, we did something that looked like:

= semantic_form_for [:admin_email, @email_template], child_index: 'template-translation' do |f|
  //f.globalize_inputs is a method on ActiveAdmin::FormBuilder that creates tabs for each translation
  = f.globalize_inputs :translations do |g|
    = g.inputs 'Translated Content' do
      = g.input :subject
      = g.input :text_source
      = g.input :html_source
      = g.input :locale, as: :hidden

Globalize had some problems playing with ActiveAdmin; in order to make the form work, we also had to add the line accepts_nested_attributes_for :translations; without the line, the form would get confused, and the params hash would have only empty inputs coming in. We also ran into some trouble internationalizing images. When we added globalize into our images admin, like:

= semantic_form_for [:admin_email, @email_image], child_index: 'image-translation' do |f|
  = f.inputs 'Image' do
    = f.input :templates, :as => :check_boxes, :collection => MailFiend::Template.all.sort_by{|template| template.name.downcase}

  = f.globalize_inputs :translations do |g|
    = g.inputs 'Translated Content' do
      - if g.object.persisted?
        = g.input :image, :as => :file, :hint => display_image(g.object.image.url(:thumb))
      - else
        = g.input :image, :as => :file
      = g.input :locale, as: :hidden

we were getting an undefined method attribute_will_change! error. Changing mount_uploader :image, ImageUploader to Translation.mount_uploader :image, ImageUploader and adding accepts_nested_attributes_for :translations like a Carrierwave pull request suggested changed this error to undefined method 'name' for nil:NilClass. Eventually, we discovered that this error was due to ordering: once we moved translates :image to before Translation.mount_uploader :image, ImageUploader, the problem went away.

Architecture

The architecture is a little too complicated to go into fully in this post, but some major points were:

• An embedded engine that acts like a separate app but with a few well-defined touch points into the rest of the app for convenience/simplicity
• A direct database connection to a separate reporting database for analytics-derived user facts
• A gem-encapsulated communication channel over HTTP API with our analytics-built query system on a separate server
• A Resque-scheduler powered with tools like resque_spec and timecop for thorough timing coverage for mass mailings

Conclusion

When just starting out, it makes sense to use a third party system, as building an email system on your own is a huge hassle. However, the more data you have to deal with, the more sense it makes to start building an in-house system, to avoid the costs of synchronization as well as allowing the most customization and flexibility.