Adding filters to stock Devise controllers

Have some before_filters or after_filters you want to add to some or all of the Devise controller actions without having to subclass all the Devise controllers?

Here's a little trick I use: directly call the before_filter or after_filter class methods in the devise.rb initializer. For example, I have a prepare_for_mobile method that I want to apply as a before_filter to all of the Devise controller actions. This is what I have at the bottom of config/initializers/devise.rb:



Note that this is exactly the same as if you had added the before_filter call directly inside these controllers. That means if you wanted to be more selective, you could just as easily do something like:




IMPORTANT NOTE: This tactic only works reliably in the production environment. That is because in development mode, the Devise controller classes get reloaded on each request, but this config file that we've edited does not.

UPDATED 2/16/2012: See my follow-up post where I revisit this issue with a solution that works in both production and development environments.

Rails incorrectly parses HTTP Accept header for single media with range

Try hitting your Rails app with an HTTP header like this:

Accept: text/html;q=0.7

You will most likely get an exception that looks something like this:

Missing template home/index with {:locale=>[:en, :en],  :formats=>["text/html;q=0.7"], :handlers=>[:rhtml, :erb, :haml,  :rxml, :builder, :rjs]} in view paths  "/path/to/my_app/app/views"

This is because the ActionPack Mime::Type class is buggy in the way it parses this header when there is only one media type specified, and that media type has a range specified (the ";q=0.7"). This is perfectly valid HTML, and ActionPack actually handles the ranges correctly when there are more than one type specified.

An issue has been open for a while on this: https://github.com/rails/rails/issues/736 -- working patches have been proposed but not applied, yet the issue got closed for some reason.

I prefer not to patch my own version of Rails, so instead I am using a small monkey patch that works around this problem. I've created a file named config/initializers/rails_issue_736.rb that fixes the Mime::Type.lookup method to only use the part that comes before the semi-colon:

Encrypted Cookie Store that works with Rails 3.0

If you like the benefits using Rails's CookieStore, but want to store possibly sensitive data in the cookie, then encrypted_cookie_store is for you. It works like CookieStore, but encrypts the cookie payload so that it cannot be read by the client.

If you are using Ruby on Rails 3.0.0 or higher (but not yet Rails 3.1), then you'll need my fork of encrypted_cookie_store that fixes it for Rails 3.

To install it:

gem install scottwb-encrypted_cookie_store

Then add to your bundler Gemfile:

gem 'scottwb-encrypted_cookie_store', :require => 'encrypted_cookie_store'

Then read the rest of the installation/configuration instructions.

Background

For Rails 2.3, the folks at Phusion created encrypted_cookie_store, which you'd install as a plugin, and it worked great. I used it for a long time.

However, they never updated it for Rails 3. That's where Ben Sales came in. He forked this project and made it work for Rails 3...in the pre-release days, that is. Ben did all the work to get it packaged up as a gem and updated it to work with Rails 3 railties and initializers.

Unfortunately, sometime between Rails 3.0.0.beta3 and 3.0.0.beta4, the layout of AbstractStore and CookieStore changed quite a bit, pushing a lot of the functionality out to Rack, and breaking the encrypted_cookie_store gem.

That's where I come in. I basically did the minimal amount of work required to get it to work with Rails 3.0 (tested on 3.0.0, 3.0.7, and 3.0.8.rc4), got all the specs working again, and created a new gem called 'scottwb-encrypted_session_cookie'.

It doesn't work in Rails 3.1, but I'll probably remedy that once Rails 3.1 officially releases. I'm also happy to accept patches if anyone else onces to tackle that.

This is a nice gem. Maybe some day I'll make a push to clean it up and lobby to have it as one of the packaged options that ships with Rails...

Workaround an IE bug: Redirecting subdomains to www while preserving virtual hosts (Nginx, Rails)

I ran into an interesting redirect configuration problem that was exacerbated by Internet Explorer's handing of cookies, which led me to move all subdomain redirection logic out of my Rails app and into Nginx. In the end, this feels like a good thing anyway. If you just want the conclusion, skip to the solution section.

Background

My original setup was to have nginx have virtual hosts for www.example.com, test.example.com, and staging.example.com, where the www.example.com was the default one that would catch anything else such as foo.example.com, or just example.com. Then the Rails app at www.example.com would check the URI in a before_filter and redirect to www.example.com, using a solution similar to the one described here: http://stackoverflow.com/questions/327122/redirect-myapp-com-to-www-myapp-com-in-rails-without-using-htaccess.

Problem

The problem arose when an Internet Explorer user hit the site via http://example.com (i.e.: without the "www." prefix). Here is what happens when someone hits example.com in this setup:

1. The default virtual host for www.example.com picks up the request and hands it to the Rails app.
2. The Rails app's before_filter sees that the request.host does not start with 'www', and does a redirect.
3. Somewhere the ActionController internals set a session cookie on the domain "example.com" -- even though this before_filter redirects BEFORE any code tries to touch the session. (Perhaps this is an artifact of using CookieStore.)
4. The browser stores the session cookie for example.com, the follows the redirect to www.example.com, sending that cookie (which is basically an empty session at this point).
5. The user logs in. The Rails app sets stuff in the session and returns a response that sets the new session cookie with the new session, for www.example.com.
6. The browser now has two session cookies: One not-logged-in one for example.com, and one logged-in one for www.example.com.
7. The next page load is where the problems start...

On Safari, Chrome, and Firefox, the next page load, on www.example.com, only sends the session cookie that is for www.example.com...and all is well. However, on Internet Explorer (tested on IE8 and IE9 beta), the browser sends BOTH session cookies, which both have the same name. In my observation, Rails always choose the first one, which is always the LEAST specific one - the one for example.com. Now, even though the user just logged in, he is not logged in.

I don't know what the HTTP RFC for user agents says about this, but I am just gonna go ahead and call this in IE bug. According to an MSDN FAQ about IE's cookie-handling, IE is different than other browsers, and they're just fine with that (see Q3 on that page). Also see Q1 where they say they don't even try to support the RFC for cookies.

Similar problems have been documented on the web. I read somewhere that if your Rails app redirects before touching the session, that you won't have this problem. That's not the case for me, but I suspect that may have to do with using CookieStore. I can imagine that CookieStore touches the session by reading it when it comes time to generate the cookie, and that this gets triggered even after a redirect.

Redirect with Nginx

Since I could not get Rails to NOT set a cookie for example.com, I decided I would do this redirect in nginx. There are plenty of examples on the web about redirecting example.com to www.example.com, or vice-versa. However, most of them don't work for my needs for one particular reason: virtual hosts.

In my scenario, I can't redirect ALL requests to non-www subdomains to the www-subdomain, because I have other virtual hosts on specific sub-domains that should not be re-routed. However, I can't simply enumerate the ones that I want redirected, since that is meant to be a catch-all. Rather than try to maintain a regular expression that matches all subdomains except the ones I setup virtual hosts on, I decided to just test the hostname inside the default virtual host, as shown in the conclusion section below. This way, specific subdomains get routed to the right virtual host, anything else gets routed to the www.example.com virtual host, and within that one, anything that isn't www.example.com gets redirected to www.example.com.

Solution

The final solution: this nginx config mockup shows how I can have specific subdomains handled by their virtual hosts, and everything else redirected to the www subdomain.

Huge speed gain migrating large tables in Rails

Have you ever needed to make multiple column adds/removes/changes to a large table in Rails in a single migration? If so, you've probably noticed that each of these individual changes issues a single ALTER TABLE command to the database, which can take a long time on a large table. At least with MySQL, combining these all into a single ALTER TABLE dramatically reduces the amount of time it takes for your migration to run.

Rather than write custom SQL for this, the folks at XING have created a very cool Rails plugin called alter_table. Their blog post, Alter Table Rails Plugin, gives more background on this. Check it out, it's a good, quick read.

One thing they didn't mention in their post was what kind of performance gains they observed. I was curious, so I did a quick-and-dirty experiment.

Consider an existing table named "songs" that has 330K records in it. First, I tried just adding a column named "foo" using the normal migration style we're all accustomed to:

On my MacBook Pro, this is what the timings for this looked like:

Now, after rolling that back, installing the alter_table plugin (as per their instructions), I rewrote this migration to use the new alter_table method:

Now, on the same machine, this migration takes roughly half the time because it does all the alterations in a single pass. Here are the new timings:

In real life, I had a table with almost 10 million rows that I needed to do 10 alterations to. Using this plugin, I cut my migration time down by almost 10x! If you find yourself doing more than one alteration on the same table in a single migration, I highly recommend checking out alter_table. You can find their source code at:

http://github.com/xing/alter_table

Enumerate all Rails ActiveRecord::Base model classes

I couldn't quickly find an easy way to enumerate all ActiveRecord::Base-derived model classes in my Rails project (for some simple analysis scripts), so I whipped up this quick and dirty solution.

I extended ActiveRecord::Base just out of convenience, so I can call ActiveRecord::Base.each_model_class{|klass| ...} easily. You could apply the same technique in another class if you don't like extending Rails core classes.

The "rescue nil" is just a quick hack to catch the cases where you have tables that don't map to classes directly.

NOTE: This does NOT enumerate STI-based model classes that don't have their own tables.

Rails 2.3 ActiveSupport::Cache::MemoryStore Freezes Objects

Using MemoryStore for caching in Rails 2.3.X has what I would consider a show-stopper bug. When you write an object to the cache using Rails.cache.write, the original object gets frozen. When you read that object from the cache, it is also frozen. So, if you do something like read/write-through a cache for all your ActiveRecord models, you'll never be able to modify any instances of your models in your code.

This same behavior is not present in other cache storage classes such as ActiveSupport::Cache::MemCacheStore. I don't think I'd ever use MemoryStore in production...but it is great for testing. When running all of my rspecs and cucumber features, I have those environments set to use MemoryStore. The problem is, many of my tests break when they try to modify objects that were frozen by the cache.

Here's a simple example demonstrating this. Look at the WTFs on lines 10 and 14 of the output below:


The Attempted Fix
It turns out this bug was filed (and partially fixed) a long time ago. Rails ticket #2655 was filed back in May 2009 about this exact issue. Yehuda Katz and Carl Lerche from EngineYard committed a fix for this (to duplicate the object into the cache instead of freezing it) on July 1, 2009.

This is great. It fixes line 10 in the example output above. However, it is not quite enough if you are putting ActiveRecord::Base objects into the cache because the object's @attributes hash needs to be dup'd too. It turns out, the "carlhuda" pair was well aware of this and committed a fix for that too, just moments before the other one.

Another problem still remains though. Their fix still involves freezing the duplicate object in the cache and directly returning that frozen object. We need one more change to the read method to duplicate the object on the way out too. I made such a fix and updated the test_store_objects_should_be_immutable test for MemoryStore to be consistent with that of MemCacheStore

The final problem is that carlhuda's commits never got merged into the Rails 2.3 branch. A quick look at the github network graph for the rails project on July 1, 2009 tells the story.

Click Image To Enlarge

That yellow branch looks like wycats and carllerche were pairing on a bunch of bug fixes around this time. Notably the two short red arrows I drew here show that they cherry-picked two of those and merged them into the 2-3-stable branch. Then, shortly after that, they make a few more fixes that never got merged into the 2-3-stable branch. The two I circled in red are the two fixes I mentioned above.

Completing The Fix
I don't know if the Rails team is planning any more updates to the 2.3 line, but if they do release a 2.3.9, I vote for including these two commits, plus the one I applied. So, I forked the repo and created a topic branch off of 2-3-stable that cherry-picks the two carlhuda commits and includes my additional commit.

A Workaround
Since I'd rather not maintain my own patched version of Rails, for now I am just monkey-patching MemoryStore and ActiveRecord::Base like so:

Install a Rails App's dependencies where 'rake gems:install' fails.

Using rake gems:install to install the gem dependencies of your Rails 2 app just doesn't seem to work. There are a number of problems you may encounter:

Problem 1: It depends on Rails.

You have to manually install Rails before you can use rake gems:install. Ok, so this might not be that big of a deal, but consider this scenario:

• Your app is running in production on rails 2.3.4.
• You upgrade your app to rails 2.3.5, including the RAILS_GEM_VERSION.
• You deploy your app.
• Your deployment script calls rake gems:install to keep the dependencies up to date with each new release.
• That fails with something that looks like:

Problem 2: Sometimes, for some reason, it just doesn't work.

Sometimes (most times) you add a new config.gem command to environment.rb. The next deployment runs rake gems:install and it fails with the complaint that you are missing the very gem you are hoping it will install for you. For example, in a working Rails 2.3.5 project, add this line to environment.rb:

config.gem 'sanitize', :version => '1.2.1'

Then, run rake gems:install, and it fails with a "Missing these required gems:" message, e.g.:

WTF? It's telling me to run the command I just ran, to install the missing gem that's preventing that command from running.

Problem 3: It depends on your Rails environment.

This can create a circular dependency where some file (particularly vendored plugins/gems) can require a gem to be loaded before the task to install it runs. For example, consider that some file in your vendor directory does this:

require 'gdata'

Then naturally, you add this to environment.rb:

config.gem 'gdata'

The next time you run rake gems:install, it will fail with a "no such file to load" error, e.g.:

The Solution: rails_gem_install

I have created a new tool called rails_gem_install to help alleviate this problem. Use this instead of rake gems:install, and you should be much more successful at getting all your Rails app's dependencies installed without manual intervention.

To install:

gem install rails_gem_install

To use:

cd my_rails_app

RAILS_ENV=production rails_gem_install

This will install all the gems required to run your app in the production environment, including Rails. Replace rake gems:install in your deployment scripts with rails_gem_install, and as you change config.gem requirements, those gems will be properly installed.

See the github project page for more details.

How It Works

The main principle behind this tool is that it does not depend on Rails. It creates its own module named Rails that provides some of the functionality from the real Rails that it needs, up until the point that Rails is installed. Then, it only loads some very specific parts of Rails that implement the gem dependency and installation mechanisms, without actually running the app's Rails::Initializer and loading all of its environment, plugins, etc.

First, it runs a simple rake -T to see if it complains about Rails missing. It parses the output of this, and if necessary, installs the indicated version of Rails.

Next, it parses out the config.gem statements and uses those to ensure all the listed gems requirements are met, and installs those that aren't. It does this without actually loading Rails or the app environment. This carries us most of the way.

Finally, it runs the rake gems command and parses its output to detect all the kinds of errors described above and install the corresponding gems. This step is repeated until rake gems does not complain anymore.

Sharing a partial for both server-side and client-side HTML generation.

Say I have a page that shows a table of items (like a shopping cart), and I want to be able to generate the HTML for this table from the server-side (of course). Let's also say that I want to have client-side javascript be able to dynamically add items to this table without a round-trip to the server, based on data gathered on the client. What I end up with is some HTML code (actually, ERb or Haml) that defines the layout of a row in this table, and a duplicate piece of code in javascript that generates identical HTML. The server-side code uses a nice templating language like Haml to fill in the variable pieces, while the client-side code pieces together HTML embedded in strings concatenated with the dynamic data.

Not very DRY. Not easy to maintain. Not pretty to look at.

So...What's a good solution that allows me to write the HTML once, and have one place to maintain it, while still being able to use it to dynamically evaluate the template based on parameterized data on both the server and client sides?

Below is a solution that I came up with. (And I would love to hear any better ideas).

For brevity, I'll use a very simple HTML structure in these examples, but in practice I have used this with much more complex structure and logic. Here's what our example table looks like:

<table id="items">
  <tr>
    <td class="name">Shirt</td>
    <td class="desc">White short-sleeved shirt</td>
    <td class="price">$5.99</td>
  </tr>

  <tr>
    <td class="name">Pants</td>
    <td class="desc">Blue pants with zipper</td>
    <td class="price">$25.99</td>
  </tr>
</table>

Assume we have this built using Haml, and there are two partials: one for the top-level table, and one for an individual row. E.g.:

items_table.haml:

%table#items
  - items.each do |item|
    = render :partial => 'item_row', :locals => {:item => item}

item_row.haml:

%tr
  %td.name= item.name
  %td.desc= item.desc
  %td.price= item.price

In the client-side javascript in our top-level Haml partial, assume we have some function that gets called from some user input action (like filling out a form and clicking a button). E.g.:

:javascript
  var addItem = function(name, desc, price) {
    var html =
      "<tr>" +
      "  <td class=\"name\">" + name + "</td>" +
      "  <td class=\"desc\">" + desc + "</td>" +
      "  <td class=\"price\">" + price + "</td>" +
      "</tr>";
    $('items').insert(html); // using prototype.js
  };

OK...so this doesn't seem so bad with such a simple example, but imagine that the HTML structure for a table row is something much more complex, with conditionals based on the item, and maybe even usage of additional sub-partials. This maintenance for this becomes a nightmare.

This first thing I am going to do is change the item_rows.haml partial. Instead of having it actually render the item's values, it's going to render template replacement variables using #{...} syntax. Note, these are NOT interpolated in the partial. We'll do that in a separate phase from rendering:

item_row.haml:

%tr
  %td.name #{item_name}
  %td.desc #{item_desc}
  %td.price #{item_price}

Next, let's make a couple helper methods. The first one returns this partial rendered as a string. (I do this because in the real world, this takes a number of parameters, and adding a helper layer makes things easier to manage than always calling render directly). The second one renders this partial and then interpolates the #{...} strings inside it, based on the properties of the item we want to render -- effectively making this equivalent to rendering the original version of this template.

item_helper.rb:

module ItemHelper
  # Make ActionController::Base#render_to_string available.
  helper_method :render_to_string

  def item_row_template
    render_to_string :partial => item_row
  end

  def item_row(item)
    # Get the template.
    t = item_row_template

    # Define the template replacement variables.
    item_name = item.name
    item_desc = item.desc
    item_price = item.price

    # Evaluate the template
    eval(t.inspect.gsub("\\#", "#"))
  end
end

This, of course, means we change the top-level partial to use this new item_row helper, like so:

items_table.haml:

%table#items
  - items.each do |item|
    = item_row(item)

At this point, we've got a functionally equivalent server-side. But the big benefit is that now the HTML template returned by the item_row_template helper method can also be used to do the same substitutions on the client side, thanks to the handy Template class in prototype.js. The Haml that defines the javascript addItem method now looks like this:

:javascript
  var addItem = function(name, desc, price) {
    var html = new Template(
      '#{escape_javascript(item_row_template)}'
    ).evaluate({
      item_name : name,
      item_desc : desc,
      item_price : price
    });
    $('items').insert(html);
  };

Now, when the markup for the item row changes, it only needs to change in one place, and both the server-side and client-side benefit from it. This certainly feels more DRY...but also seems like a hefty price to pay in terms of complexity...

Bug in Haml Partial Rendering?

I have stumbled across what I think is a bug in rendering a Haml partial in Ruby on Rails. The issue is that when I use render(:partial => '...', :locals => {...}), and one of the locals I define is a Boolean, it seems that if I pass a value of false, that value erroneously gets converted to nil. This causes a problem with a partial that wants to treat false, true, and nil as meaning different things.

Consider the following example:

I have a simple TestController:

class TestController < ApplicationController
  def index
  end
end

Then, the haml template for that controller:

!!!
%html
  %head
    %title haml partial test
  %body
    %b Rendering a Haml Partial
    %br
    = render :partial => 'test_haml', :locals => {:flag1 => true, :flag2 => false}

    %hr
    %b Rendering an RHTML Partial
    %br
    = render :partial => 'test_rhtml', :locals => {:flag1 => true, :flag2 => false}

Notice that partial renders two partials. One is a haml partial, one is an rhtml partial. The rthml one behaves as I would expect, the haml one does not.

Here is what the haml partial looks like:

- if flag1 == false
  flag1 is false
- elsif flag1 == true
  flag1 is true
- elsif flag1 == nil
  flag1 is nil
- else
  = "flag1 is #{flag1.inspect}"

%br
%br

- if flag2 == false
  flag2 is false
- elsif flag2 == true
  flag2 is true
- elsif flag2 == nil
  flag2 is nil
- else
  = "flag2 is #{flag2.inspect}"

Here is what the rhtml partial looks like:

<% if flag1 == false %>
  flag1 is false
<% elsif flag1 == true %>
  flag1 is true
<% elsif flag1 == nil %>
  flag1 is nil
<% else %>
  flag1 is <%= flag1.inspect %>
<% end %>






<% if flag2 == false %>
  flag2 is false
<% elsif flag2 == true %>
  flag2 is true
<% elsif flag2 == nil %>
  flag2 is nil
<% else %>
  flag2 is <%= flag2.inspect %>
<% end %>

Here is what the output looks like:



Notice the discrepancy between the output of the haml partial versus the rhtml partial, with respect to flag2.

This is gonna turn out to be a real drag on a project that I am converting from rhtml to haml that has a number of partials that assume that false means false and not nil.

Anybody out there have any thoughts on this?