Rackspace Cloud Files SSL Failures Over ServiceNet

I recently switched from using the public interface on my Rackspace Cloud Server to using the private interface (a.k.a. ServiceNet) for uploading files from my instance to Cloud Files. When I made this switch, I started noticing a number of intermittent "Connection reset by peer" errors.

I have reproduced this problem with both the `Fog` and `ruby-cloudfiles` gems.

To upload to Cloud Files, I'm using CarrierWave, which uses Fog, which uses Excon. With those, the error looks something like this:

Excon::Errors::SocketError: Connection reset by peer - SSL_connect



[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `open_ssl_socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:243:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:282:in `socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:156:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/connection.rb:20:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/rackspace.rb:102:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/requests/rackspace/put_object.rb:19:in `put_object'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/models/rackspace/file.rb:59:in `save'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/collection.rb:50:in `create'

[GEM_ROOT]/gems/carrierwave-0.5.6/lib/carrierwave/storage/fog.rb:229:in `store'

I never really narrowed down why this is happening, but it seems to happen only after a period of inactivity. I.e.: it fails once, then works fine as long as I keep making requests, but after 10+ minutes, the next request fails again.

Since I am only storing files to Rackspace Cloud Files via CarrierWave, the most expedient thing for me to do was to wrap the storage call in CarrierWave with some exception handling to retry these failures. It's a hack, but it works like a charm and saved me at crunch time:



Even if you're not using CarrierWave or Fog, this concept could work for you. Of course, you may use more operations that just storing files, and you'll probably have to wrap those too.

Rails incorrectly parses HTTP Accept header for single media with range

Try hitting your Rails app with an HTTP header like this:

Accept: text/html;q=0.7

You will most likely get an exception that looks something like this:

Missing template home/index with {:locale=>[:en, :en],  :formats=>["text/html;q=0.7"], :handlers=>[:rhtml, :erb, :haml,  :rxml, :builder, :rjs]} in view paths  "/path/to/my_app/app/views"

This is because the ActionPack Mime::Type class is buggy in the way it parses this header when there is only one media type specified, and that media type has a range specified (the ";q=0.7"). This is perfectly valid HTML, and ActionPack actually handles the ranges correctly when there are more than one type specified.

An issue has been open for a while on this: https://github.com/rails/rails/issues/736 -- working patches have been proposed but not applied, yet the issue got closed for some reason.

I prefer not to patch my own version of Rails, so instead I am using a small monkey patch that works around this problem. I've created a file named config/initializers/rails_issue_736.rb that fixes the Mime::Type.lookup method to only use the part that comes before the semi-colon:

YUI Compressor breaks CSS3 Media Query Syntax

If you use the CSS3 Media Query Syntax for something like this:

@media screen and (max-device-width: 480px) {
  body {font-size: 18px;}
}

You will find that when you compress this with YUI Compressor (as is done when packaging your assets with Jammit), that the output breaks this syntax. YUI Compressor removes the whitespace between the "and" and the opening parenthesis, which does not conform to the CSS3 standard, and renders those styles unusable in most browsers (tested in Chrome and Safari, read about others hitting this in Firefox).

There's a YUI Compressor bug filed here that covers this: http://yuilibrary.com/projects/yuicompressor/ticket/2528053

My Workaround

I am invoking YUI Compressor via an asset-packager called Jammit. In a blog post yesterday, I noted some problems I encountered with Jammit not handling duplicate @charset directives generated by Sass/Compass, and outlined a quick hack workaround I used to deal with it.

Today, encountering this @media-related bug leads to an additional workaround I had to add to that. So, I figured I'd share a bit of my resulting rake task that combines all of these.

• Use compass to compile my Sass to CSS without compressing it (this is important)


• Use sed to remove all @charset directives (dangerous, but I know I don't need them in this case). Take care to make sure the differences in how the -i parameter to sed works between OS X and Linux.


• Use jammit to package up everything.


• For each permutation (e.g.: normal, datauri) CSS package jammit generated, use sed to add the missing space back in to offending @media directives.


• For each of those CSS files we sed-edited, re-gzip them. (Jammit made gzipped versions originally, but we have to regenerate them since we changed the contents.)

Here's the final rake task:

Jammit, Compass, and Safari: Can't we all just get along?

When Jammit is used with CSS files generated by Compass, the resulting CSS file may not work with Safari. This is because every Sass file that includes Compass will add the following line to the top of the generated CSS file:

    @charset "UTF-8";

That happens because something in Compass requires UTF-8, and Sass will generate that line if anything it is processing requires UTF-8. This is all fine...until you try to combine more than one of these standalone-css files with an asset packager such as Jammit.

Once Jammit combines all these CSS files, you end up with one CSS file that has multiple @charset directives in it, which according to the W3, is a no-no. Generating a CSS file with more than one @charset directive (and anywhere other than the very first line) is clearly against the W3's spec...but the only browser it seems to break is Safari.

But here's the thing: why do my CSS files need to have the @charset added by Sass when they don't actually get output with any non-ASCII characters?

In the end, my quick stop-gap solution was to remove them myself. I used to let Jammit do all the driving. During deployment it would force the regeneration of the CSS files and package them. Now I have to break out the Compass step and add my own sed command in order to remove all the @charset directives:

I think each one of these components has something to improve:

Safari:

If you don't support multiple @charset directives, I support you on that front. The spec says you're right. However...can't you just raise an error that says "ERROR: multiple @charset directives" or something? That would be much better than just randomly disabling half the CSS styles.

Sass:

Sass shouldn't add a @charset line if the generated CSS does not need it. If a mixin that uses UTF-8 is included but not used, and is not represented in the generated CSS, there is no need to add a @charset directive on its behalf.

Compass:

Compass is probably doing nothing wrong here. Though I do question why it needs to use UTF-8 characters. Maybe that's for a feature I haven't used yet.

Jammit:

In the end, I think Jammit is the real culprit here. It takes in N perfectly-valid CSS files and outputs one invalid CSS file. Jammit should remove duplicate @charset directives, and if they are conflicting, it should convert the charset of the conflicting stylesheet and into the charset being used by the output file.

Workaround an IE bug: Redirecting subdomains to www while preserving virtual hosts (Nginx, Rails)

I ran into an interesting redirect configuration problem that was exacerbated by Internet Explorer's handing of cookies, which led me to move all subdomain redirection logic out of my Rails app and into Nginx. In the end, this feels like a good thing anyway. If you just want the conclusion, skip to the solution section.

Background

My original setup was to have nginx have virtual hosts for www.example.com, test.example.com, and staging.example.com, where the www.example.com was the default one that would catch anything else such as foo.example.com, or just example.com. Then the Rails app at www.example.com would check the URI in a before_filter and redirect to www.example.com, using a solution similar to the one described here: http://stackoverflow.com/questions/327122/redirect-myapp-com-to-www-myapp-com-in-rails-without-using-htaccess.

Problem

The problem arose when an Internet Explorer user hit the site via http://example.com (i.e.: without the "www." prefix). Here is what happens when someone hits example.com in this setup:

1. The default virtual host for www.example.com picks up the request and hands it to the Rails app.
2. The Rails app's before_filter sees that the request.host does not start with 'www', and does a redirect.
3. Somewhere the ActionController internals set a session cookie on the domain "example.com" -- even though this before_filter redirects BEFORE any code tries to touch the session. (Perhaps this is an artifact of using CookieStore.)
4. The browser stores the session cookie for example.com, the follows the redirect to www.example.com, sending that cookie (which is basically an empty session at this point).
5. The user logs in. The Rails app sets stuff in the session and returns a response that sets the new session cookie with the new session, for www.example.com.
6. The browser now has two session cookies: One not-logged-in one for example.com, and one logged-in one for www.example.com.
7. The next page load is where the problems start...

On Safari, Chrome, and Firefox, the next page load, on www.example.com, only sends the session cookie that is for www.example.com...and all is well. However, on Internet Explorer (tested on IE8 and IE9 beta), the browser sends BOTH session cookies, which both have the same name. In my observation, Rails always choose the first one, which is always the LEAST specific one - the one for example.com. Now, even though the user just logged in, he is not logged in.

I don't know what the HTTP RFC for user agents says about this, but I am just gonna go ahead and call this in IE bug. According to an MSDN FAQ about IE's cookie-handling, IE is different than other browsers, and they're just fine with that (see Q3 on that page). Also see Q1 where they say they don't even try to support the RFC for cookies.

Similar problems have been documented on the web. I read somewhere that if your Rails app redirects before touching the session, that you won't have this problem. That's not the case for me, but I suspect that may have to do with using CookieStore. I can imagine that CookieStore touches the session by reading it when it comes time to generate the cookie, and that this gets triggered even after a redirect.

Redirect with Nginx

Since I could not get Rails to NOT set a cookie for example.com, I decided I would do this redirect in nginx. There are plenty of examples on the web about redirecting example.com to www.example.com, or vice-versa. However, most of them don't work for my needs for one particular reason: virtual hosts.

In my scenario, I can't redirect ALL requests to non-www subdomains to the www-subdomain, because I have other virtual hosts on specific sub-domains that should not be re-routed. However, I can't simply enumerate the ones that I want redirected, since that is meant to be a catch-all. Rather than try to maintain a regular expression that matches all subdomains except the ones I setup virtual hosts on, I decided to just test the hostname inside the default virtual host, as shown in the conclusion section below. This way, specific subdomains get routed to the right virtual host, anything else gets routed to the www.example.com virtual host, and within that one, anything that isn't www.example.com gets redirected to www.example.com.

Solution

The final solution: this nginx config mockup shows how I can have specific subdomains handled by their virtual hosts, and everything else redirected to the www subdomain.

Rails 2.3 ActiveSupport::Cache::MemoryStore Freezes Objects

Using MemoryStore for caching in Rails 2.3.X has what I would consider a show-stopper bug. When you write an object to the cache using Rails.cache.write, the original object gets frozen. When you read that object from the cache, it is also frozen. So, if you do something like read/write-through a cache for all your ActiveRecord models, you'll never be able to modify any instances of your models in your code.

This same behavior is not present in other cache storage classes such as ActiveSupport::Cache::MemCacheStore. I don't think I'd ever use MemoryStore in production...but it is great for testing. When running all of my rspecs and cucumber features, I have those environments set to use MemoryStore. The problem is, many of my tests break when they try to modify objects that were frozen by the cache.

Here's a simple example demonstrating this. Look at the WTFs on lines 10 and 14 of the output below:


The Attempted Fix
It turns out this bug was filed (and partially fixed) a long time ago. Rails ticket #2655 was filed back in May 2009 about this exact issue. Yehuda Katz and Carl Lerche from EngineYard committed a fix for this (to duplicate the object into the cache instead of freezing it) on July 1, 2009.

This is great. It fixes line 10 in the example output above. However, it is not quite enough if you are putting ActiveRecord::Base objects into the cache because the object's @attributes hash needs to be dup'd too. It turns out, the "carlhuda" pair was well aware of this and committed a fix for that too, just moments before the other one.

Another problem still remains though. Their fix still involves freezing the duplicate object in the cache and directly returning that frozen object. We need one more change to the read method to duplicate the object on the way out too. I made such a fix and updated the test_store_objects_should_be_immutable test for MemoryStore to be consistent with that of MemCacheStore

The final problem is that carlhuda's commits never got merged into the Rails 2.3 branch. A quick look at the github network graph for the rails project on July 1, 2009 tells the story.

Click Image To Enlarge

That yellow branch looks like wycats and carllerche were pairing on a bunch of bug fixes around this time. Notably the two short red arrows I drew here show that they cherry-picked two of those and merged them into the 2-3-stable branch. Then, shortly after that, they make a few more fixes that never got merged into the 2-3-stable branch. The two I circled in red are the two fixes I mentioned above.

Completing The Fix
I don't know if the Rails team is planning any more updates to the 2.3 line, but if they do release a 2.3.9, I vote for including these two commits, plus the one I applied. So, I forked the repo and created a topic branch off of 2-3-stable that cherry-picks the two carlhuda commits and includes my additional commit.

A Workaround
Since I'd rather not maintain my own patched version of Rails, for now I am just monkey-patching MemoryStore and ActiveRecord::Base like so:

Fixing IE onChange event for checkboxes and radio buttons

In Internet Explorer (up to at least IE8) clicking a radio button or checkbox to change its value does not actually trigger the onChange event until the the input loses focus. On every other browser, when the value of the input changes, onChange is triggered.

This is quite annoying, for example, if you want to trigger some other page changes off of the selection of a radio button in a radio button group changing. Some may suggest using onClick instead, and while that may be sufficient for a checkbox, for radio buttons, this is not a good solution. The problem is that onClick will fire when you click the radio button that is already selected, so you'll get your code triggered even when the value doesn't change.

Here's a quick hack that worked for me. On IE, handle the onClick event of radio and checkboxes and have it blur and re-focus the input element. This will trigger it to signal the onChange event like you'd expect from a decent browser. Then, you can go about your business using the onChange event like you should, and it will work correctly for you across the popular browsers. (WARNING: Of course, if you have other code triggered off of the blur and focus events, your mileage may vary.)

Here's an example of how you might do this using jQuery:

  if ($.browser.msie) {
    $(function() {
      $('input:radio, input:checkbox').click(function() {
        this.blur();
        this.focus();
      });
    });
  }

Bug in Haml Partial Rendering?

I have stumbled across what I think is a bug in rendering a Haml partial in Ruby on Rails. The issue is that when I use render(:partial => '...', :locals => {...}), and one of the locals I define is a Boolean, it seems that if I pass a value of false, that value erroneously gets converted to nil. This causes a problem with a partial that wants to treat false, true, and nil as meaning different things.

Consider the following example:

I have a simple TestController:

class TestController < ApplicationController
  def index
  end
end

Then, the haml template for that controller:

!!!
%html
  %head
    %title haml partial test
  %body
    %b Rendering a Haml Partial
    %br
    = render :partial => 'test_haml', :locals => {:flag1 => true, :flag2 => false}

    %hr
    %b Rendering an RHTML Partial
    %br
    = render :partial => 'test_rhtml', :locals => {:flag1 => true, :flag2 => false}

Notice that partial renders two partials. One is a haml partial, one is an rhtml partial. The rthml one behaves as I would expect, the haml one does not.

Here is what the haml partial looks like:

- if flag1 == false
  flag1 is false
- elsif flag1 == true
  flag1 is true
- elsif flag1 == nil
  flag1 is nil
- else
  = "flag1 is #{flag1.inspect}"

%br
%br

- if flag2 == false
  flag2 is false
- elsif flag2 == true
  flag2 is true
- elsif flag2 == nil
  flag2 is nil
- else
  = "flag2 is #{flag2.inspect}"

Here is what the rhtml partial looks like:

<% if flag1 == false %>
  flag1 is false
<% elsif flag1 == true %>
  flag1 is true
<% elsif flag1 == nil %>
  flag1 is nil
<% else %>
  flag1 is <%= flag1.inspect %>
<% end %>






<% if flag2 == false %>
  flag2 is false
<% elsif flag2 == true %>
  flag2 is true
<% elsif flag2 == nil %>
  flag2 is nil
<% else %>
  flag2 is <%= flag2.inspect %>
<% end %>

Here is what the output looks like:



Notice the discrepancy between the output of the haml partial versus the rhtml partial, with respect to flag2.

This is gonna turn out to be a real drag on a project that I am converting from rhtml to haml that has a number of partials that assume that false means false and not nil.

Anybody out there have any thoughts on this?