Rackspace Cloud Files SSL Failures Over ServiceNet

I recently switched from using the public interface on my Rackspace Cloud Server to using the private interface (a.k.a. ServiceNet) for uploading files from my instance to Cloud Files. When I made this switch, I started noticing a number of intermittent "Connection reset by peer" errors.

I have reproduced this problem with both the `Fog` and `ruby-cloudfiles` gems.

To upload to Cloud Files, I'm using CarrierWave, which uses Fog, which uses Excon. With those, the error looks something like this:

Excon::Errors::SocketError: Connection reset by peer - SSL_connect



[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `open_ssl_socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:243:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:282:in `socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:156:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/connection.rb:20:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/rackspace.rb:102:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/requests/rackspace/put_object.rb:19:in `put_object'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/models/rackspace/file.rb:59:in `save'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/collection.rb:50:in `create'

[GEM_ROOT]/gems/carrierwave-0.5.6/lib/carrierwave/storage/fog.rb:229:in `store'

I never really narrowed down why this is happening, but it seems to happen only after a period of inactivity. I.e.: it fails once, then works fine as long as I keep making requests, but after 10+ minutes, the next request fails again.

Since I am only storing files to Rackspace Cloud Files via CarrierWave, the most expedient thing for me to do was to wrap the storage call in CarrierWave with some exception handling to retry these failures. It's a hack, but it works like a charm and saved me at crunch time:



Even if you're not using CarrierWave or Fog, this concept could work for you. Of course, you may use more operations that just storing files, and you'll probably have to wrap those too.

Performance of Rackspace Cloud Files Upload Over Private Network

Does using the private network interface on a Rackspace Cloud Server instance to transfer data into Rackspace Cloud files save you anything?

I am certain it saves you money. You pay for outgoing bandwidth on your Cloud Server instance's public IP address, but not on the private one. But, I've also read that it is much faster to use the private IP, or "ServiceNet" as they call it. "Lightning fast" even. So, I tried make a quick-and-dirty benchmark to see for myself.

In short, I don't think it makes much difference, considering the margin of error. Here are the results of a typical test run that shows the timings of uploads of a 1MB file alternating over the public and private interfaces, with a cumulative time printed out at the bottom:

https://gist.github.com/1155335

That example is the most sane run of many test runs I did with this script. What is more disconcerting is that both public and private have wild outliers. Sometimes a transfer that normally takes a tenth of a second can take 30 seconds or even two minutes!

Check out this test run: https://gist.github.com/1155349

Whoa! The 31st upload took 132 seconds! In my experience over multiple runs, the private snet almost always comes out way ahead. It seems like it is just as likely to have 10-second outliers, but less likely to have 30+ second outliers.

If you are interested in reproducing my tests, you can get the code at: https://github.com/scottwb/cloudfiles-snet-bench

On caveat to this test, other than generally not being very scientific, is that I am not using large files (which is the use case I am actually interested in). It is possible that large transfers may get big speed gains over the private servicenet.

Rails incorrectly parses HTTP Accept header for single media with range

Try hitting your Rails app with an HTTP header like this:

Accept: text/html;q=0.7

You will most likely get an exception that looks something like this:

Missing template home/index with {:locale=>[:en, :en],  :formats=>["text/html;q=0.7"], :handlers=>[:rhtml, :erb, :haml,  :rxml, :builder, :rjs]} in view paths  "/path/to/my_app/app/views"

This is because the ActionPack Mime::Type class is buggy in the way it parses this header when there is only one media type specified, and that media type has a range specified (the ";q=0.7"). This is perfectly valid HTML, and ActionPack actually handles the ranges correctly when there are more than one type specified.

An issue has been open for a while on this: https://github.com/rails/rails/issues/736 -- working patches have been proposed but not applied, yet the issue got closed for some reason.

I prefer not to patch my own version of Rails, so instead I am using a small monkey patch that works around this problem. I've created a file named config/initializers/rails_issue_736.rb that fixes the Mime::Type.lookup method to only use the part that comes before the semi-colon: