Privilege escalation

Back in the dim & distant past – late 1999, although no records capture the exact date – I was asked to compromise a server and gain root access. I said yes.

This is the first and only time I have deliberately cracked a live, production server.

This was not as questionable an undertaking as it sounds. I knew of the machine in question, and I knew its operator, and he still had a working secure shell login. But sudo had not been configured, and the root password had been forgotten. Whoops. Could I help? I was willing to try.

Why not simply reboot the machine in single user mode? Because it was a busy DNS server and downtime was undesirable. And it wasn’t physically nearby.

I remember looking into current known vulnerabilities. The server was running FreeBSD 3.x-RELEASE and there weren’t many. Moreover I did not trust any “off the shelf” root kits, considering them a source of hidden malware of their own, and I did not have time to learn & practice x86 assembler to develop & discover anything myself. And remembering the brief – I didn’t want to accidentally DoS the server by fumbling a kernel vulnerability.

Fortunately we had the OS source code revision history. I went hunting for recently fixed bugs, in the hope of exploiting them. And found one.

(We didn’t have github back then. But I was subscribed to the FreeBSD commit mailing list).

The issue was in the fts(3) system library and its use by find(1). There was a daily periodic script that ran as root, and invoked find(1) to traverse the entire filesystem. The bug in fts(3) was a buffer overrun in pointer arithmetic when handling a memory reallocation caused by deep recursion, resulting in a segmentation error. Which due to a little bit of sloppiness in the periodic script, meant a core dump.

Even better, it meant a core dump to a file in the current working directory, named find.core. Et voila. There was our hook: a process memory dump, written as root, to a known filename.

The trick was to make find.core a symlink, in this case to /root/.ssh/authorized_keys. The idea being that the dump will be written through the symlink to that destination filename. And to ensure that our SSH key would appear in the resulting file, simply make a deep, deep directory tree with long directories named for that key (with appropriate newline padding), each one containing that symlink.

The long, deep directory hierarchy would trigger the bug, resulting in the core dump, which being a memory image should necessarily include the directory name – i.e., our key would be in the file.

A very small exploit script was run to create just such a structure.

We woke up the next morning, and behold – the periodic task had executed, root’s authorized_keys file contained a core dump, and amongst the garbage was our public key, for which we had the private part and which sshd(8) would honour.


Et voila, root access gained, privilege escalated, and (recalling the brief) as safely as possible, since the erroneous pointer arithmetic occurred in userland, not in kernel space.

Health checks with Stethoscope behind an SSL Elastic Load Balancer.

I am rather fond of the Stethoscope gem for monitoring, since it lives inside the runtime Ruby process and can directly check on Rails without being subject to it. Returning HTML or JSON as required it also makes a nice responder for health checks from Nagios and AWS Elastic Load Balancers.

Using Stethoscope is as simple as adding the gem in your Gemfile and installing an initializer, e.g.:

However, in this modern paranoid age, many apps (including practically everything I write) include config.force_ssl true in config/environments/production.rb to require SSL, HSTS and secure cookies. Using this behind an SSL-terminating load balancer (such as the AWS ELB) is a problem though, because the health check is only available to the ELB via HTTP. This results in a 301 redirect that Rails will return as a result of the SSL forcing but that the LB can’t follow.

One solution might be to configure an SSL endpoint on the application servers and use an HTTPS ping, but this introduces complexity, increases the attack surface by distributing the private key far more widely, increases latency, and increases the difference between the health check request path and the real application path. I’m filing that solution under “no thanks”.

A better way to fix this is simple but relies on knowledge of request processing. Rails is a Rack application and processes requests via a stack of Rack middleware that terminates with route processing. Rack is an elegant internal protocol for Ruby web applications and understanding it will serve you well.

If you dive into the Rails source code you’ll find that config.force_ssl true results in a piece of Rack middleware being installed by the default Rails middleware stack. This is ActionDispatch::SSL and it is inserted (c.f. rake middleware) at the top of the stack, which is the source of our problem because it captures the request and returns a redirect so Stethoscope never gets a look in.

Fortunately it is not hard to rearrange the middleware stack and now we know the problem, the solution is easy; just add this to the Stethoscope initializer:

# Place Stethoscope at top of stack.
Rails.configuration.middleware.delete Stethoscope
Rails.configuration.middleware.unshift Stethoscope

Et voila. Stethoscope is now the first piece of middleware in the Rack processing and not subject to ActionDispatch::SSL’s intervention.

There is a small downside: the health check won’t be subject to SSL forcing. However it is still available via SSL, and since it is mounted on a secret URL and only ever invoked by explicitly configured tools of your own, this shouldn’t be a problem.

Hook AWS notifications into Slack with a Lambda function

We have AWS RDS instances that send lifecycle notifications to an SNS topic. This was ending up in email, but I prefer to receive notifications in a Slack channel. Fortunately they are easy to integrate using AWS Lambda and Slack’s webhooks.

Here’s the Lambda function I’m using[1], which parses the message object (if it can) and formats the notifications before posting.

Note: the following links will require both AWS and Slack logins.
To do it yourself, create the webhook in Slack. Then publish the code above as a Lambda function in your AWS account (a basic execution role will be fine). Update line 5 with your own webhook URL. Have RDS notify a SNS topic and subscribe the Lambda function to that topic.

You can check the integration using Lambda’s standard SNS test event. For an end-to-end & event parsing test I suggest creating and deleting a RDS snapshot.

[1] derived with thanks from

Using arrays of hstore with Rails 4

UPDATE: The patch below has now been merged into edge rails and should appear in the 4.1 release as well as, hopefully, a backport into 4.0.3.

One of the attractive new things in Rails 4 is the enhanced support for PostgreSQL array and hstore column types, amongst other things. There are quite a few articles describing their use – here’s a clear one – and there are subtleties with strong parameters. However they don’t quite work together out of the box – but very nearly, and with a little work we can store arrays of hashes directly in a single PostgreSQL column.


We were already able to create array-of-hstore columns using the new syntax; for example, with the following migration:

class CreateServers < ActiveRecord::Migration
  def change
    enable_extension "hstore"
    create_table :servers do |t|
      t.string :name
      t.hstore :interfaces, array: true

So far, so good; running rake db:migrate will create the interfaces[] array-of-hstore column. But trying to use it falls flat; any kind of insert or update will produce an error from PostgreSQL, because the hstore needs to be encoded inside the array for the wire protocol.

The workaround

Applying a small patch to edge rails (see, we can now create a record with an array-of-hash values in the interfaces column:

server = Server.create(name: "server01", interfaces: [
  { "name" => "bge0", "ipv4" => "", "state" => "up" },
  { "name" => "de0", "state" => "disabled", "by" => "misha" },
  { "name" => "fe0", "state" => "up" },

Now we’re getting somewhere! And loading that record back will produce the expected array-of-hash from server.interfaces, which we can use like any other:

server.interfaces[0]["state"]             # => "up" { |i| i["name"] }   # => ["bge0", "de0", "fe0"]

Updates & change tracking

Because the hash values themselves are not Active Record model objects, changing a value requires that you notify Rails with attr_will_change; for example:

server.interfaces[0]["state"] = "down"
server.interfaces[0]["by"] = "misha"


The syntax for hstore-array querying is not obvious; we have to unroll the array and use it like a WHERE IN clause. The PostgreSQL array operators ANY and ALL are available to us, and allow us to find records by the values stored under the hash keys in array elements. For example, declaring this in your model:

class Server < ActiveRecord::Base
  scope :where_any, ->(column, key, value) { where("? = ANY (SELECT unnest(\"#{column}\") -> ?)", value, key) }
  scope :where_all, ->(column, key, value) { where("? = ALL (SELECT unnest(\"#{column}\") -> ?)", value, key) }

will allow us to find our server above with:

Server.where_any(:interfaces, :state, "up")

however, it will not match this relation:

Server.where_all(:interfaces, :state, "up")

because one of the array elements doesn’t match (in this example, the disabled interface).

Use with forms and Strong Parameters

Also new in Rails 4 are Strong Parameters. These will still be unfamiliar to many, but they allow structural validation of complex input with a fairly straightforward syntax. For example, in a controller, the following will permit form-based submission of our array-of-hash structure:

  def create
    @server =
    # ...
  def server_params
    params.require(:server).permit(:name, :interfaces => [:name, :ipv4, :state, :by])

The form itself needs to use an OpenStruct to wrap each hash, so that ActionView is able to pull fields from it; e.g.

<% @server.interfaces.each do |interface| %>
  <%= f.fields_for :interfaces,, index: "" do |interface_form| %>
    <%= render "interface_fields", f: interface_form %>
  <% end %>
<% end %>

where _interface_fields.html.erb is:

  <%= f.text_field :name %>
  <%= f.text_field :ipv4 %>
  <%= f.text_field :state %>
  <%= f.text_field :by %>

Note that if you need to dynamically add/remove array elements, this is also compatible with the technique for nested forms – however, a little extra infrastructure is required. I’ll save that for a future post.


I haven’t worked this out yet. In PostgreSQL you can normally index arrays, and index hstores – but my attempts to index an array-of-hstore column have so far produced errors. It may be necessary to define an operator class for the type. One might argue that even needing this suggests that a more conventional relational structure should be used, but it’ll remain an open performance question. When I get to the bottom of that particular rabbit hole, I’ll let you know.

author’s note: a previous version of this article used symbolic keys for the hash, but this changed to string keys, for consistency with other access methods, by request of the rails team during review of the pull request.