Health checks with Stethoscope behind an SSL Elastic Load Balancer

I am rather fond of the Stethoscope gem for monitoring, since it lives inside the runtime Ruby process and can directly check on Rails without being subject to it. Returning HTML or JSON as required it also makes a nice responder for health checks from Nagios and AWS Elastic Load Balancers.

Using Stethoscope is as simple as adding the gem in your Gemfile and installing an initializer in e.g. config/initializers/stethoscope.rb:

Stethoscope.url = "/stethoscope_myapp_ae53unit"

Stethoscope.check :database do |resp|
  resp[:last_migration] = ActiveRecord::Migrator.current_version
  resp[:pg_version] = ActiveRecord::Base.connection.select_value('SELECT version()')
end

Stethoscope.check :release do |resp|
  resp[:version] = "#{MyApp.config.version} (#{Rails.env})"
  resp[:git_revision] = `git rev-parse HEAD`.chomp
end

uname = `uname -a`.chomp
Stethoscope.check :platform do |resp|
  resp[:ruby] = RUBY_DESCRIPTION
  resp[:rails] = Rails::version
  resp[:linux] = uname
end

However, in this modern paranoid age, many apps (including practically everything I write) include config.force_ssl true in config/environments/production.rb to require SSL, HSTS and secure cookies. Using this behind an SSL-terminating load balancer (such as the AWS ELB) is a problem though, because the health check is only available to the ELB via HTTP. This results in a 301 redirect that Rails will return as a result of the SSL forcing but that the LB can’t follow.

One solution might be to configure an SSL endpoint on the application servers and use an HTTPS ping, but this introduces complexity, increases the attack surface by distributing the private key far more widely, increases latency, and increases the difference between the health check request path and the real application path. I’m filing that solution under “no thanks”.

A better way to fix this is simple but relies on knowledge of request processing. Rails is a Rack application and processes requests via a stack of Rack middleware that terminates with route processing. Rack is an elegant internal protocol for Ruby web applications and understanding it will serve you well.

If you dive into the Rails source code you’ll find that config.force_ssl true results in a piece of Rack middleware being installed by the default Rails middleware stack. This is ActionDispatch::SSL and it is inserted (c.f. rake middleware) at the top of the stack, which is the source of our problem because it captures the request and returns a redirect so Stethoscope never gets a look in.

Fortunately it is not hard to rearrange the middleware stack and now we know the problem, the solution is easy; just add this to the Stethoscope initializer:

# Place Stethoscope at top of stack.
Rails.configuration.middleware.delete Stethoscope
Rails.configuration.middleware.unshift Stethoscope

Et voila. Stethoscope is now the first piece of middleware in the Rack processing and not subject to ActionDispatch::SSL’s intervention.

There is a small downside: the health check won’t be subject to SSL forcing. However it is still available via SSL, and since it is mounted on a secret URL and only ever invoked by explicitly configured tools of your own, this shouldn’t be a problem.