Health checks with Stethoscope behind an SSL Elastic Load Balancer.

I am rather fond of the Stethoscope gem for monitoring, since it lives inside the runtime Ruby process and can directly check on Rails without being subject to it. Returning HTML or JSON as required it also makes a nice responder for health checks from Nagios and AWS Elastic Load Balancers.

Using Stethoscope is as simple as adding the gem in your Gemfile and installing an initializer, e.g.:

However, in this modern paranoid age, many apps (including practically everything I write) include config.force_ssl true in config/environments/production.rb to require SSL, HSTS and secure cookies. Using this behind an SSL-terminating load balancer (such as the AWS ELB) is a problem though, because the health check is only available to the ELB via HTTP. This results in a 301 redirect that Rails will return as a result of the SSL forcing but that the LB can’t follow.

One solution might be to configure an SSL endpoint on the application servers and use an HTTPS ping, but this introduces complexity, increases the attack surface by distributing the private key far more widely, increases latency, and increases the difference between the health check request path and the real application path. I’m filing that solution under “no thanks”.

A better way to fix this is simple but relies on knowledge of request processing. Rails is a Rack application and processes requests via a stack of Rack middleware that terminates with route processing. Rack is an elegant internal protocol for Ruby web applications and understanding it will serve you well.

If you dive into the Rails source code you’ll find that config.force_ssl true results in a piece of Rack middleware being installed by the default Rails middleware stack. This is ActionDispatch::SSL and it is inserted (c.f. rake middleware) at the top of the stack, which is the source of our problem because it captures the request and returns a redirect so Stethoscope never gets a look in.

Fortunately it is not hard to rearrange the middleware stack and now we know the problem, the solution is easy; just add this to the Stethoscope initializer:

# Place Stethoscope at top of stack.
Rails.configuration.middleware.delete Stethoscope
Rails.configuration.middleware.unshift Stethoscope

Et voila. Stethoscope is now the first piece of middleware in the Rack processing and not subject to ActionDispatch::SSL’s intervention.

There is a small downside: the health check won’t be subject to SSL forcing. However it is still available via SSL, and since it is mounted on a secret URL and only ever invoked by explicitly configured tools of your own, this shouldn’t be a problem.

Hook AWS notifications into Slack with a Lambda function

We have AWS RDS instances that send lifecycle notifications to an SNS topic. This was ending up in email, but I prefer to receive notifications in a Slack channel. Fortunately they are easy to integrate using AWS Lambda and Slack’s webhooks.

Here’s the Lambda function I’m using[1], which parses the message object (if it can) and formats the notifications before posting.

Note: the following links will require both AWS and Slack logins.
To do it yourself, create the webhook in Slack. Then publish the code above as a Lambda function in your AWS account (a basic execution role will be fine). Update line 5 with your own webhook URL. Have RDS notify a SNS topic and subscribe the Lambda function to that topic.

You can check the integration using Lambda’s standard SNS test event. For an end-to-end & event parsing test I suggest creating and deleting a RDS snapshot.

[1] derived with thanks from

Using arrays of hstore with Rails 4

UPDATE: The patch below has now been merged into edge rails and should appear in the 4.1 release as well as, hopefully, a backport into 4.0.3.

One of the attractive new things in Rails 4 is the enhanced support for PostgreSQL array and hstore column types, amongst other things. There are quite a few articles describing their use – here’s a clear one – and there are subtleties with strong parameters. However they don’t quite work together out of the box – but very nearly, and with a little work we can store arrays of hashes directly in a single PostgreSQL column.


We were already able to create array-of-hstore columns using the new syntax; for example, with the following migration:

class CreateServers < ActiveRecord::Migration
  def change
    enable_extension "hstore"
    create_table :servers do |t|
      t.string :name
      t.hstore :interfaces, array: true

So far, so good; running rake db:migrate will create the interfaces[] array-of-hstore column. But trying to use it falls flat; any kind of insert or update will produce an error from PostgreSQL, because the hstore needs to be encoded inside the array for the wire protocol.

The workaround

Applying a small patch to edge rails (see, we can now create a record with an array-of-hash values in the interfaces column:

server = Server.create(name: "server01", interfaces: [
  { "name" => "bge0", "ipv4" => "", "state" => "up" },
  { "name" => "de0", "state" => "disabled", "by" => "misha" },
  { "name" => "fe0", "state" => "up" },

Now we’re getting somewhere! And loading that record back will produce the expected array-of-hash from server.interfaces, which we can use like any other:

server.interfaces[0]["state"]             # => "up" { |i| i["name"] }   # => ["bge0", "de0", "fe0"]

Updates & change tracking

Because the hash values themselves are not Active Record model objects, changing a value requires that you notify Rails with attr_will_change; for example:

server.interfaces[0]["state"] = "down"
server.interfaces[0]["by"] = "misha"


The syntax for hstore-array querying is not obvious; we have to unroll the array and use it like a WHERE IN clause. The PostgreSQL array operators ANY and ALL are available to us, and allow us to find records by the values stored under the hash keys in array elements. For example, declaring this in your model:

class Server < ActiveRecord::Base
  scope :where_any, ->(column, key, value) { where("? = ANY (SELECT unnest(\"#{column}\") -> ?)", value, key) }
  scope :where_all, ->(column, key, value) { where("? = ALL (SELECT unnest(\"#{column}\") -> ?)", value, key) }

will allow us to find our server above with:

Server.where_any(:interfaces, :state, "up")

however, it will not match this relation:

Server.where_all(:interfaces, :state, "up")

because one of the array elements doesn’t match (in this example, the disabled interface).

Use with forms and Strong Parameters

Also new in Rails 4 are Strong Parameters. These will still be unfamiliar to many, but they allow structural validation of complex input with a fairly straightforward syntax. For example, in a controller, the following will permit form-based submission of our array-of-hash structure:

  def create
    @server =
    # ...
  def server_params
    params.require(:server).permit(:name, :interfaces => [:name, :ipv4, :state, :by])

The form itself needs to use an OpenStruct to wrap each hash, so that ActionView is able to pull fields from it; e.g.

<% @server.interfaces.each do |interface| %>
  <%= f.fields_for :interfaces,, index: "" do |interface_form| %>
    <%= render "interface_fields", f: interface_form %>
  <% end %>
<% end %>

where _interface_fields.html.erb is:

  <%= f.text_field :name %>
  <%= f.text_field :ipv4 %>
  <%= f.text_field :state %>
  <%= f.text_field :by %>

Note that if you need to dynamically add/remove array elements, this is also compatible with the technique for nested forms – however, a little extra infrastructure is required. I’ll save that for a future post.


I haven’t worked this out yet. In PostgreSQL you can normally index arrays, and index hstores – but my attempts to index an array-of-hstore column have so far produced errors. It may be necessary to define an operator class for the type. One might argue that even needing this suggests that a more conventional relational structure should be used, but it’ll remain an open performance question. When I get to the bottom of that particular rabbit hole, I’ll let you know.

author’s note: a previous version of this article used symbolic keys for the hash, but this changed to string keys, for consistency with other access methods, by request of the rails team during review of the pull request.