Sunday, August 21, 2011

Rackspace Cloud Files SSL Failures Over ServiceNet

I recently switched from using the public interface on my Rackspace Cloud Server to using the private interface (a.k.a. ServiceNet) for uploading files from my instance to Cloud Files. When I made this switch, I started noticing a number of intermittent "Connection reset by peer" errors.

I have reproduced this problem with both the `Fog` and `ruby-cloudfiles` gems.

To upload to Cloud Files, I'm using CarrierWave, which uses Fog, which uses Excon. With those, the error looks something like this:
Excon::Errors::SocketError: Connection reset by peer - SSL_connect


[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `connect'
[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `open_ssl_socket'
[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:243:in `connect'
[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:282:in `socket'
[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:156:in `request'
[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/connection.rb:20:in `request'
[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/rackspace.rb:102:in `request'
[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/requests/rackspace/put_object.rb:19:in `put_object'
[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/models/rackspace/file.rb:59:in `save'
[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/collection.rb:50:in `create'
[GEM_ROOT]/gems/carrierwave-0.5.6/lib/carrierwave/storage/fog.rb:229:in `store'

I never really narrowed down why this is happening, but it seems to happen only after a period of inactivity. I.e.: it fails once, then works fine as long as I keep making requests, but after 10+ minutes, the next request fails again.

Since I am only storing files to Rackspace Cloud Files via CarrierWave, the most expedient thing for me to do was to wrap the storage call in CarrierWave with some exception handling to retry these failures. It's a hack, but it works like a charm and saved me at crunch time:



Even if you're not using CarrierWave or Fog, this concept could work for you. Of course, you may use more operations that just storing files, and you'll probably have to wrap those too.

1 comment:

Rich said...

Good jjob