Sleepless Coding

My Blog Has Moved

2012-01-15T18:22:00.000-08:00

I will no longer be posting at this blog. The posts and comments will remain intact, but all new posts will happen at by new site: http://scottwb.com

Recreate a single version of a CarrierWave::Uploader file

2011-09-04T11:33:00.000-07:00

CarrierWave supports a simple way to recreate all the versions of an uploaded file. This comes in handy when you add a new version to your uploader, or when you change the parameters of one of them. For example, assume you have a Post class, with a CarrierWave mounted uploader on the photo column, and you have version called thumb. If you decide to change the resolution of the thumb version, and need to go back and regenerate all existing thumbs, you can do something like this:

Post.all.each do |post|
  post.photo.recreate_versions!
end

What if you have 5 different versions on this Uploader, and you only changed one of them? The above method will download all the versions of every Post#photo from storage (think S3 or Cloud Files), regenerate all of them - even the ones that haven't changed - and re-upload all of them to storage.

That's a lot of wasted time, CPU, and bandwidth.

To address this, I've created a simple monkey patch for CarrierWave to provide a new method called recreate_version!. This method works just like recreate_versions!, except that it takes a parameter specifying a single version to recreate, rather than recreating all of them.

To use this, first drop this monkey-patch in somewhere that it will get loaded. Above your Uploader class will suffice.

Now your script to recompute all the thumbs can look like this:

Post.all.each do |post|
  post.photo.recreate_version!(:thumb)
end

This will recreate and re-upload only the thumb versions of your Post#photo uploads.

As an added side benefit, the recreate_version! method cleans up after itself. The recreate_versions! method downloads into a temporary cache dir, but leaves it up to the caller to clean that up. This new method assumes that if it had to download and cache the uploaded file and its versions, then it should clean them up immediately.

This method has turned out to save me a lot of time in regenerating large numbers of uploaded file versions. If there's any interest in this, I might work up a pull request with test cases and try to get it into the mainline CarrierWave gem release.

Adding filters to stock Devise controllers

2011-08-23T22:04:00.000-07:00

Have some before_filters or after_filters you want to add to some or all of the Devise controller actions without having to subclass all the Devise controllers?

Here's a little trick I use: directly call the before_filter or after_filter class methods in the devise.rb initializer. For example, I have a prepare_for_mobile method that I want to apply as a before_filter to all of the Devise controller actions. This is what I have at the bottom of config/initializers/devise.rb:

Note that this is exactly the same as if you had added the before_filter call directly inside these controllers. That means if you wanted to be more selective, you could just as easily do something like:

IMPORTANT NOTE: This tactic only works reliably in the production environment. That is because in development mode, the Devise controller classes get reloaded on each request, but this config file that we've edited does not.

Rackspace Cloud Files SSL Failures Over ServiceNet

2011-08-21T14:21:00.000-07:00

I recently switched from using the public interface on my Rackspace Cloud Server to using the private interface (a.k.a. ServiceNet) for uploading files from my instance to Cloud Files. When I made this switch, I started noticing a number of intermittent "Connection reset by peer" errors.

I have reproduced this problem with both the `Fog` and `ruby-cloudfiles` gems.

To upload to Cloud Files, I'm using CarrierWave, which uses Fog, which uses Excon. With those, the error looks something like this:

Excon::Errors::SocketError: Connection reset by peer - SSL_connect



[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:264:in `open_ssl_socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:243:in `connect'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:282:in `socket'

[GEM_ROOT]/gems/excon-0.6.4/lib/excon/connection.rb:156:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/connection.rb:20:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/rackspace.rb:102:in `request'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/requests/rackspace/put_object.rb:19:in `put_object'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/storage/models/rackspace/file.rb:59:in `save'

[GEM_ROOT]/gems/fog-0.9.0/lib/fog/core/collection.rb:50:in `create'

[GEM_ROOT]/gems/carrierwave-0.5.6/lib/carrierwave/storage/fog.rb:229:in `store'

I never really narrowed down why this is happening, but it seems to happen only after a period of inactivity. I.e.: it fails once, then works fine as long as I keep making requests, but after 10+ minutes, the next request fails again.

Since I am only storing files to Rackspace Cloud Files via CarrierWave, the most expedient thing for me to do was to wrap the storage call in CarrierWave with some exception handling to retry these failures. It's a hack, but it works like a charm and saved me at crunch time:

Even if you're not using CarrierWave or Fog, this concept could work for you. Of course, you may use more operations that just storing files, and you'll probably have to wrap those too.

Performance of Rackspace Cloud Files Upload Over Private Network

2011-08-18T14:33:00.000-07:00

Does using the private network interface on a Rackspace Cloud Server instance to transfer data into Rackspace Cloud files save you anything?

I am certain it saves you money. You pay for outgoing bandwidth on your Cloud Server instance's public IP address, but not on the private one. But, I've also read that it is much faster to use the private IP, or "ServiceNet" as they call it. "Lightning fast" even. So, I tried make a quick-and-dirty benchmark to see for myself.

In short, I don't think it makes much difference, considering the margin of error. Here are the results of a typical test run that shows the timings of uploads of a 1MB file alternating over the public and private interfaces, with a cumulative time printed out at the bottom:

https://gist.github.com/1155335

That example is the most sane run of many test runs I did with this script. What is more disconcerting is that both public and private have wild outliers. Sometimes a transfer that normally takes a tenth of a second can take 30 seconds or even two minutes!

Check out this test run: https://gist.github.com/1155349

Whoa! The 31st upload took 132 seconds! In my experience over multiple runs, the private snet almost always comes out way ahead. It seems like it is just as likely to have 10-second outliers, but less likely to have 30+ second outliers.

If you are interested in reproducing my tests, you can get the code at: https://github.com/scottwb/cloudfiles-snet-bench

On caveat to this test, other than generally not being very scientific, is that I am not using large files (which is the use case I am actually interested in). It is possible that large transfers may get big speed gains over the private servicenet.

Rails incorrectly parses HTTP Accept header for single media with range

2011-08-16T23:35:00.000-07:00

Try hitting your Rails app with an HTTP header like this:

Accept: text/html;q=0.7

You will most likely get an exception that looks something like this:

Missing template home/index with {:locale=>[:en, :en],  :formats=>["text/html;q=0.7"], :handlers=>[:rhtml, :erb, :haml,  :rxml, :builder, :rjs]} in view paths  "/path/to/my_app/app/views"

This is because the ActionPack Mime::Type class is buggy in the way it parses this header when there is only one media type specified, and that media type has a range specified (the ";q=0.7"). This is perfectly valid HTML, and ActionPack actually handles the ranges correctly when there are more than one type specified.

An issue has been open for a while on this: https://github.com/rails/rails/issues/736 -- working patches have been proposed but not applied, yet the issue got closed for some reason.

I prefer not to patch my own version of Rails, so instead I am using a small monkey patch that works around this problem. I've created a file named config/initializers/rails_issue_736.rb that fixes the Mime::Type.lookup method to only use the part that comes before the semi-colon:

MailMatic: HTML emails using Haml and Sass

2011-08-15T14:42:00.001-07:00

Developing static HTML emails is a pain in the butt. For best results, you need to inline all of your styles at HTML attributes instead of using real CSS (externally or inline). You also typically find yourself being very repetitive with your markup if you have a bunch of static emails that share common elements.

MailMatic to the rescue.

MailMatic is a Ruby gem, that is really just a mashup of tools, that aims to ease the burden of creating and maintaining static HTML emails. It uses StaticMatic, Haml, Sass, and Compass to let you author your HTML emails with all of the facilities you enjoy while creating your dynamic site: templates, partials, helper functions, inline Ruby code, reuse of markup and styles, etc. It uses Premailer to convert the HTML/CSS generated from Haml/Sass into self-contained HTML files that inline all of their styles. Premailer even gives you warnings about styles that may not work nicely in certain email readers.

Armed with this, you can accelerate your ability to crank out static HTML emails, and lessen the burden of maintaining them.

To get started with MailMatic, check out its installation and usage instructions on github at: https://github.com/scottwb/mailmatic.

The ask_geo gem: find the time zone for a given latitude and longitude

2011-07-19T21:18:00.000-07:00

The Google Maps API has all kinds of great services like geocoding and reverse geocoding, but it lacks an API to determine a timezone from a latitude and longitude. This kind of service would come in handy, for example, when you are picking a point on Earth and scheduling an event there in the point's local time. It could also be useful for automatically determining a website visitor's current local time based on the geolocation obtained from their browser or IP address.

There is a service called GeoNames, which comes close to serving this need. However, its look-ups are based on "closest point of interest", which means that you can get errors when you are near borders, and some locations may not return any results at all.

There is another service called EarthTools which doesn't have this problem, but it only returns the current offset from UTC at that point. That is great if you only care about the current local time, but leaves you stranded if you need to account for Daylight Savings Time on some future or past date.

After a bit of searching, I came across a great free service called AskGeo. AskGeo provides a very simple JSON-over-HTTP API (or XML if you prefer to party like it's 1999). AskGeo uses a real time zone map, so every point on earth returns a result, and returns real Olson IDs for time zones, so you can look up DST transitions, leap seconds, etc.

I gave AskGeo a try and it works great...and it's fast! Since there was no ready-made Ruby gem that provided a client API, I decided to create one.

Introducing the ask_geo gem: https://github.com/scottwb/ask_geo

It can be installed as a ruby gem. See the github page for installation and usage instructions.

Custom RSpec be_square matcher for CarrierWave

2011-07-10T18:10:00.000-07:00

It seems to me that when using CarrierWave for image uploads, particularly user avatars, it would be a pretty common thing to generate at least one square-cropped version of the image. CarrierWave ships with some nice custom matchers to test exact dimensions and such, but I found I wanted a way to just test that an image was square.

I did this using by building a custom matcher on top of the ones CarrierWave provides. If you have the conventional spec_helper.rb that requires support/**/*.rb, then you can just drop this somewhere like support/matchers/carrierwave_matchers.rb:

Then, you can use this like:

Encrypted Cookie Store that works with Rails 3.0

2011-05-31T11:48:00.000-07:00

If you like the benefits using Rails's CookieStore, but want to store possibly sensitive data in the cookie, then encrypted_cookie_store is for you. It works like CookieStore, but encrypts the cookie payload so that it cannot be read by the client.

If you are using Ruby on Rails 3.0.0 or higher (but not yet Rails 3.1), then you'll need my fork of encrypted_cookie_store that fixes it for Rails 3.

To install it:

gem install scottwb-encrypted_cookie_store

Then add to your bundler Gemfile:

gem 'scottwb-encrypted_cookie_store', :require => 'encrypted_cookie_store'

Then read the rest of the installation/configuration instructions.

Background

For Rails 2.3, the folks at Phusion created encrypted_cookie_store, which you'd install as a plugin, and it worked great. I used it for a long time.

However, they never updated it for Rails 3. That's where Ben Sales came in. He forked this project and made it work for Rails 3...in the pre-release days, that is. Ben did all the work to get it packaged up as a gem and updated it to work with Rails 3 railties and initializers.

Unfortunately, sometime between Rails 3.0.0.beta3 and 3.0.0.beta4, the layout of AbstractStore and CookieStore changed quite a bit, pushing a lot of the functionality out to Rack, and breaking the encrypted_cookie_store gem.

That's where I come in. I basically did the minimal amount of work required to get it to work with Rails 3.0 (tested on 3.0.0, 3.0.7, and 3.0.8.rc4), got all the specs working again, and created a new gem called 'scottwb-encrypted_session_cookie'.

It doesn't work in Rails 3.1, but I'll probably remedy that once Rails 3.1 officially releases. I'm also happy to accept patches if anyone else onces to tackle that.

This is a nice gem. Maybe some day I'll make a push to clean it up and lobby to have it as one of the packaged options that ships with Rails...

How to delete a remote git tag

2011-03-30T07:57:00.000-07:00

I suppose this is probably taboo. You're not supposed to mess with tags once you've pushed them upstream. Still, there are times when it is necessary, and unfortunately `git push --tags` does not push the local removal of tags upstream. It always takes me forever to figure this out, so I figured I'd write it down. Maybe someone else will find it useful.


  git tag -d tag-to-delete
  git push origin :refs/tags/tag-to-delete

This will remove that tag from the remote repository. You can verify this by pulling from the remote again and seeing that it does not get pulled back down.

Rails-like number_with_delimiter in javascript

2011-02-10T20:09:00.000-08:00

The number_with_delimiter view helper is Rails is nice. I wanted to do the same thing in Javascript. I went to just grab it from Rails and port it to js. But first I googled and found someone had already done exactly that. I took his code and made it extend Number.protoype so you can do something like:


var num = 12345678;
var str = num.number_with_delimiter();
alert(str);

This will show that str is "12,345,678".

Her's my version of the code:

Based on code from: http://kevinvaldek.com/number-with-delimiter-in-javascript

YUI Compressor breaks CSS3 Media Query Syntax

2011-01-27T21:48:00.000-08:00

If you use the CSS3 Media Query Syntax for something like this:


@media screen and (max-device-width: 480px) {
  body {font-size: 18px;}
}

You will find that when you compress this with YUI Compressor (as is done when packaging your assets with Jammit), that the output breaks this syntax. YUI Compressor removes the whitespace between the "and" and the opening parenthesis, which does not conform to the CSS3 standard, and renders those styles unusable in most browsers (tested in Chrome and Safari, read about others hitting this in Firefox).

There's a YUI Compressor bug filed here that covers this: http://yuilibrary.com/projects/yuicompressor/ticket/2528053

My Workaround

I am invoking YUI Compressor via an asset-packager called Jammit. In a blog post yesterday, I noted some problems I encountered with Jammit not handling duplicate @charset directives generated by Sass/Compass, and outlined a quick hack workaround I used to deal with it.

Today, encountering this @media-related bug leads to an additional workaround I had to add to that. So, I figured I'd share a bit of my resulting rake task that combines all of these.

Use compass to compile my Sass to CSS without compressing it (this is important)

Use sed to remove all @charset directives (dangerous, but I know I don't need them in this case). Take care to make sure the differences in how the -i parameter to sed works between OS X and Linux.

Use jammit to package up everything.

For each permutation (e.g.: normal, datauri) CSS package jammit generated, use sed to add the missing space back in to offending @media directives.

For each of those CSS files we sed-edited, re-gzip them. (Jammit made gzipped versions originally, but we have to regenerate them since we changed the contents.)

Here's the final rake task:

Jammit, Compass, and Safari: Can't we all just get along?

2011-01-26T15:17:00.000-08:00

When Jammit is used with CSS files generated by Compass, the resulting CSS file may not work with Safari. This is because every Sass file that includes Compass will add the following line to the top of the generated CSS file:

    @charset "UTF-8";

That happens because something in Compass requires UTF-8, and Sass will generate that line if anything it is processing requires UTF-8. This is all fine...until you try to combine more than one of these standalone-css files with an asset packager such as Jammit.

Once Jammit combines all these CSS files, you end up with one CSS file that has multiple @charset directives in it, which according to the W3, is a no-no. Generating a CSS file with more than one @charset directive (and anywhere other than the very first line) is clearly against the W3's spec...but the only browser it seems to break is Safari.

But here's the thing: why do my CSS files need to have the @charset added by Sass when they don't actually get output with any non-ASCII characters?

In the end, my quick stop-gap solution was to remove them myself. I used to let Jammit do all the driving. During deployment it would force the regeneration of the CSS files and package them. Now I have to break out the Compass step and add my own sed command in order to remove all the @charset directives:

I think each one of these components has something to improve:

Safari:

If you don't support multiple @charset directives, I support you on that front. The spec says you're right. However...can't you just raise an error that says "ERROR: multiple @charset directives" or something? That would be much better than just randomly disabling half the CSS styles.

Sass:

Sass shouldn't add a @charset line if the generated CSS does not need it. If a mixin that uses UTF-8 is included but not used, and is not represented in the generated CSS, there is no need to add a @charset directive on its behalf.

Compass:

Compass is probably doing nothing wrong here. Though I do question why it needs to use UTF-8 characters. Maybe that's for a feature I haven't used yet.

Jammit:

In the end, I think Jammit is the real culprit here. It takes in N perfectly-valid CSS files and outputs one invalid CSS file. Jammit should remove duplicate @charset directives, and if they are conflicting, it should convert the charset of the conflicting stylesheet and into the charset being used by the output file.

Workaround an IE bug: Redirecting subdomains to www while preserving virtual hosts (Nginx, Rails)

2011-01-25T08:38:00.000-08:00

I ran into an interesting redirect configuration problem that was exacerbated by Internet Explorer's handing of cookies, which led me to move all subdomain redirection logic out of my Rails app and into Nginx. In the end, this feels like a good thing anyway. If you just want the conclusion, skip to the solution section.

Background

My original setup was to have nginx have virtual hosts for www.example.com, test.example.com, and staging.example.com, where the www.example.com was the default one that would catch anything else such as foo.example.com, or just example.com. Then the Rails app at www.example.com would check the URI in a before_filter and redirect to www.example.com, using a solution similar to the one described here: http://stackoverflow.com/questions/327122/redirect-myapp-com-to-www-myapp-com-in-rails-without-using-htaccess.

Problem

The problem arose when an Internet Explorer user hit the site via http://example.com (i.e.: without the "www." prefix). Here is what happens when someone hits example.com in this setup:

The default virtual host for www.example.com picks up the request and hands it to the Rails app.
The Rails app's before_filter sees that the request.host does not start with 'www', and does a redirect.
Somewhere the ActionController internals set a session cookie on the domain "example.com" -- even though this before_filter redirects BEFORE any code tries to touch the session. (Perhaps this is an artifact of using CookieStore.)
The browser stores the session cookie for example.com, the follows the redirect to www.example.com, sending that cookie (which is basically an empty session at this point).
The user logs in. The Rails app sets stuff in the session and returns a response that sets the new session cookie with the new session, for www.example.com.
The browser now has two session cookies: One not-logged-in one for example.com, and one logged-in one for www.example.com.
The next page load is where the problems start...

On Safari, Chrome, and Firefox, the next page load, on www.example.com, only sends the session cookie that is for www.example.com...and all is well. However, on Internet Explorer (tested on IE8 and IE9 beta), the browser sends BOTH session cookies, which both have the same name. In my observation, Rails always choose the first one, which is always the LEAST specific one - the one for example.com. Now, even though the user just logged in, he is not logged in.

I don't know what the HTTP RFC for user agents says about this, but I am just gonna go ahead and call this in IE bug. According to an MSDN FAQ about IE's cookie-handling, IE is different than other browsers, and they're just fine with that (see Q3 on that page). Also see Q1 where they say they don't even try to support the RFC for cookies.

Similar problems have been documented on the web. I read somewhere that if your Rails app redirects before touching the session, that you won't have this problem. That's not the case for me, but I suspect that may have to do with using CookieStore. I can imagine that CookieStore touches the session by reading it when it comes time to generate the cookie, and that this gets triggered even after a redirect.

Redirect with Nginx

Since I could not get Rails to NOT set a cookie for example.com, I decided I would do this redirect in nginx. There are plenty of examples on the web about redirecting example.com to www.example.com, or vice-versa. However, most of them don't work for my needs for one particular reason: virtual hosts.

In my scenario, I can't redirect ALL requests to non-www subdomains to the www-subdomain, because I have other virtual hosts on specific sub-domains that should not be re-routed. However, I can't simply enumerate the ones that I want redirected, since that is meant to be a catch-all. Rather than try to maintain a regular expression that matches all subdomains except the ones I setup virtual hosts on, I decided to just test the hostname inside the default virtual host, as shown in the conclusion section below. This way, specific subdomains get routed to the right virtual host, anything else gets routed to the www.example.com virtual host, and within that one, anything that isn't www.example.com gets redirected to www.example.com.

Solution

The final solution: this nginx config mockup shows how I can have specific subdomains handled by their virtual hosts, and everything else redirected to the www subdomain.

Using tcpdump to sniff HTTP traffic from a specific host

2011-01-25T08:10:00.000-08:00

This is mostly just a reminder to myself about my preferred parameters to tcpdump on linux, so that I don't have to keep reading the man page.

tcpdump -c 20 -s 0 -i eth1 -A host 192.168.1.1 and tcp port http

The parameter breakdown:

-c 20: Exit after capturing 20 packets.
-s 0: Don't limit the amount of payload data that is printed out. Print it all.
-i eth1: Capture packets on interface eth1
-A: Print packets in ASCII.
host 192.168.1.1: Only capture packets coming to or from 192.168.1.1.
and tcp port http: Only capture HTTP packets.

Hate the emacs tilde backup files?

2010-12-10T23:10:00.001-08:00

I do...but for decades I have been afraid to turn them off in case I ever needed the backup file, even though I have never used them once. Finally I snapped and tried to figure out how to turn them off. While looking this up, I came across a better option: store them all in a single hidden directory, instead of littering the whole filesystem with them. There is a nice package called "backup-dir" that even provides the ability to keep multiple versions of your backup files, while keeping them all out of sight and out of mind, and automatically cleaning up old versions. Learn more at:

http://www.emacswiki.org/emacs/BackupDirectory

To see how I pulled in the backup-dir package and set it up in my init.el file, see the change to my dotfiles project.

Creating/Deleting local and remote branches with git

2010-12-08T18:21:00.000-08:00

In the course of my git-based workflow, more often than not, I use branches like this:

Create a new local branch starting from the HEAD of my current working branch.
Immediately push it upstream to the remote origin so that it exists in both places.
Set the local branch to track the remote branch.
Commit/push/repeat
Merge the branch back into master (or wherever it started from)
Delete the local branch.
Push to delete the remote branch.

Since this seems pretty common, and some of the commands to do this are a bit difficult to remember sometimes, I created two simple scripts called git-branch-create and git-branch-delete to help with this. Now my workflow looks like:

git branch-create feature_branch
Commit/push/repeat.
Merge branch back into parent branch.
git branch-delete feature_branch

To use these scripts via the above commands, download the scripts above and place them somewhere in your path. I even created short aliases for them like so:

Installing curb gem on Mac OS X 10.6 (Snow Leopard)

2010-12-04T11:35:00.000-08:00

If you have previously installed curl with something like:

    sudo port install curl +ssl

and then try to install the curb gem like:

    sudo gem install curb

and you find yourself getting an error that looks something like this:


    In file included from /opt/local/include/curl/curl.h:44,
                     from curb.h:12,
                     from curb.c:8:
    /opt/local/include/curl/curlrules.h:144: error: size of array ‘__curl_rule_01__’ is negative
    /opt/local/include/curl/curlrules.h:154: error: size of array ‘__curl_rule_02__’ is negative
    lipo: can't open input file: /var/folders/wX/wX64Cb+PGjG-EXuklO+I+k+++TI/-Tmp-//ccKIrqTY.out (No such file or directory)
    make: *** [curb.o] Error 1

Then, I think I have a solution for you. Change your curb install command to this instead:

    sudo env ARCHFLAGS="-Os -arch x86_64 -fno-common" gem install curb

It seems a number of people on the web suggest re-installing curl as a universal binary using something like:

    sudo port install zlib +universal
    sudo port upgrade --enforce-variants openssl +universal
    sudo port install curl +universal

I do not like this approach. The original problem is that you installed an x86_64 version of curl (which is the default, as it should be) but you are trying to install a universal version of curb (which is unfortunately the default). Instead of re-installing curl to be universal, which is wasteful and takes forever, install curb using the x86_64 flags to build it. This is what you want, it works, and it only takes a few seconds.

Huge speed gain migrating large tables in Rails

2010-09-21T12:08:00.000-07:00

Have you ever needed to make multiple column adds/removes/changes to a large table in Rails in a single migration? If so, you've probably noticed that each of these individual changes issues a single ALTER TABLE command to the database, which can take a long time on a large table. At least with MySQL, combining these all into a single ALTER TABLE dramatically reduces the amount of time it takes for your migration to run.

Rather than write custom SQL for this, the folks at XING have created a very cool Rails plugin called alter_table. Their blog post, Alter Table Rails Plugin, gives more background on this. Check it out, it's a good, quick read.

One thing they didn't mention in their post was what kind of performance gains they observed. I was curious, so I did a quick-and-dirty experiment.

Consider an existing table named "songs" that has 330K records in it. First, I tried just adding a column named "foo" using the normal migration style we're all accustomed to:

On my MacBook Pro, this is what the timings for this looked like:

Now, after rolling that back, installing the alter_table plugin (as per their instructions), I rewrote this migration to use the new alter_table method:

Now, on the same machine, this migration takes roughly half the time because it does all the alterations in a single pass. Here are the new timings:

In real life, I had a table with almost 10 million rows that I needed to do 10 alterations to. Using this plugin, I cut my migration time down by almost 10x! If you find yourself doing more than one alteration on the same table in a single migration, I highly recommend checking out alter_table. You can find their source code at:

http://github.com/xing/alter_table

Enumerate all Rails ActiveRecord::Base model classes

2010-08-23T13:39:00.001-07:00

I couldn't quickly find an easy way to enumerate all ActiveRecord::Base-derived model classes in my Rails project (for some simple analysis scripts), so I whipped up this quick and dirty solution.

I extended ActiveRecord::Base just out of convenience, so I can call ActiveRecord::Base.each_model_class{|klass| ...} easily. You could apply the same technique in another class if you don't like extending Rails core classes.

The "rescue nil" is just a quick hack to catch the cases where you have tables that don't map to classes directly.

NOTE: This does NOT enumerate STI-based model classes that don't have their own tables.

Rails 2.3 ActiveSupport::Cache::MemoryStore Freezes Objects

2010-08-07T11:44:00.000-07:00

Using MemoryStore for caching in Rails 2.3.X has what I would consider a show-stopper bug. When you write an object to the cache using Rails.cache.write, the original object gets frozen. When you read that object from the cache, it is also frozen. So, if you do something like read/write-through a cache for all your ActiveRecord models, you'll never be able to modify any instances of your models in your code.

This same behavior is not present in other cache storage classes such as ActiveSupport::Cache::MemCacheStore. I don't think I'd ever use MemoryStore in production...but it is great for testing. When running all of my rspecs and cucumber features, I have those environments set to use MemoryStore. The problem is, many of my tests break when they try to modify objects that were frozen by the cache.

Here's a simple example demonstrating this. Look at the WTFs on lines 10 and 14 of the output below:

The Attempted Fix
It turns out this bug was filed (and partially fixed) a long time ago. Rails ticket #2655 was filed back in May 2009 about this exact issue. Yehuda Katz and Carl Lerche from EngineYard committed a fix for this (to duplicate the object into the cache instead of freezing it) on July 1, 2009.

This is great. It fixes line 10 in the example output above. However, it is not quite enough if you are putting ActiveRecord::Base objects into the cache because the object's @attributes hash needs to be dup'd too. It turns out, the "carlhuda" pair was well aware of this and committed a fix for that too, just moments before the other one.

Another problem still remains though. Their fix still involves freezing the duplicate object in the cache and directly returning that frozen object. We need one more change to the read method to duplicate the object on the way out too. I made such a fix and updated the test_store_objects_should_be_immutable test for MemoryStore to be consistent with that of MemCacheStore

The final problem is that carlhuda's commits never got merged into the Rails 2.3 branch. A quick look at the github network graph for the rails project on July 1, 2009 tells the story.

Click Image To Enlarge

That yellow branch looks like wycats and carllerche were pairing on a bunch of bug fixes around this time. Notably the two short red arrows I drew here show that they cherry-picked two of those and merged them into the 2-3-stable branch. Then, shortly after that, they make a few more fixes that never got merged into the 2-3-stable branch. The two I circled in red are the two fixes I mentioned above.

Completing The Fix
I don't know if the Rails team is planning any more updates to the 2.3 line, but if they do release a 2.3.9, I vote for including these two commits, plus the one I applied. So, I forked the repo and created a topic branch off of 2-3-stable that cherry-picks the two carlhuda commits and includes my additional commit.

A Workaround
Since I'd rather not maintain my own patched version of Rails, for now I am just monkey-patching MemoryStore and ActiveRecord::Base like so:

Install a Rails App's dependencies where 'rake gems:install' fails.

2010-07-05T14:02:00.000-07:00

Using rake gems:install to install the gem dependencies of your Rails 2 app just doesn't seem to work. There are a number of problems you may encounter:

Problem 1: It depends on Rails.

You have to manually install Rails before you can use rake gems:install. Ok, so this might not be that big of a deal, but consider this scenario:

Your app is running in production on rails 2.3.4.
You upgrade your app to rails 2.3.5, including the RAILS_GEM_VERSION.
You deploy your app.
Your deployment script calls rake gems:install to keep the dependencies up to date with each new release.
That fails with something that looks like:

Problem 2: Sometimes, for some reason, it just doesn't work.

Sometimes (most times) you add a new config.gem command to environment.rb. The next deployment runs rake gems:install and it fails with the complaint that you are missing the very gem you are hoping it will install for you. For example, in a working Rails 2.3.5 project, add this line to environment.rb:

config.gem 'sanitize', :version => '1.2.1'

Then, run rake gems:install, and it fails with a "Missing these required gems:" message, e.g.:

WTF? It's telling me to run the command I just ran, to install the missing gem that's preventing that command from running.

Problem 3: It depends on your Rails environment.

This can create a circular dependency where some file (particularly vendored plugins/gems) can require a gem to be loaded before the task to install it runs. For example, consider that some file in your vendor directory does this:

require 'gdata'

Then naturally, you add this to environment.rb:

config.gem 'gdata'

The next time you run rake gems:install, it will fail with a "no such file to load" error, e.g.:

The Solution: rails_gem_install

I have created a new tool called rails_gem_install to help alleviate this problem. Use this instead of rake gems:install, and you should be much more successful at getting all your Rails app's dependencies installed without manual intervention.

To install:

gem install rails_gem_install

To use:

cd my_rails_app
RAILS_ENV=production rails_gem_install

This will install all the gems required to run your app in the production environment, including Rails. Replace rake gems:install in your deployment scripts with rails_gem_install, and as you change config.gem requirements, those gems will be properly installed.

See the github project page for more details.

How It Works

The main principle behind this tool is that it does not depend on Rails. It creates its own module named Rails that provides some of the functionality from the real Rails that it needs, up until the point that Rails is installed. Then, it only loads some very specific parts of Rails that implement the gem dependency and installation mechanisms, without actually running the app's Rails::Initializer and loading all of its environment, plugins, etc.

First, it runs a simple rake -T to see if it complains about Rails missing. It parses the output of this, and if necessary, installs the indicated version of Rails.

Next, it parses out the config.gem statements and uses those to ensure all the listed gems requirements are met, and installs those that aren't. It does this without actually loading Rails or the app environment. This carries us most of the way.

Finally, it runs the rake gems command and parses its output to detect all the kinds of errors described above and install the corresponding gems. This step is repeated until rake gems does not complain anymore.

Hudson CI behind an Nginx Reverse Proxy with SSL

2010-07-01T20:37:00.000-07:00

Here is a quick example nginx configuration to reverse proxy on an HTTPS virtual host to a Hudson CI server running on localhost. When I first tried to do this, the management page displays an error about the configuration being wrong. There are instructions for Running Hudson behind Apache that were helpful, and this email thread that seems to suggest terminating SSL at Hudson, not at the reverse proxy. Well, after a bit of tinkering, I worked out this configuration for nginx that worked out great:

Find the SHA1 of the latest commit in a git branch

2010-04-22T01:37:00.000-07:00

Here's a way to get the SHA1 of the last commit in a given branch in a way that can be used in a script easily. For humans `git log branchname` is probably fine - just read the first line. For a shell script that wants to get that value into a variable, here's a quick way to do that:

Seems like there ought to be an easier way...but not that I could find in less time than it took to write this command.

GitHub Redemption

2010-04-17T06:23:00.000-07:00

Moments after posting that last blog post about full directory compare of a git branch, I found out GitHub supports this already!

My post didn't explicitly call out GitHub, because I really think of this mostly as a local working tree issue for me. But, as I went to tweet about it, I realized I would probably be happy enough if GitHub supported something like this, so I called them out in the tweet...and they answered!

Many thanks to Tom Preston-Werner (@mojombo) for correcting me, and pointing out probably the one link on GitHub I have never clicked: The "Branch List" sub-tab:

I guess I just assumed that it was going to be just a list of all the branches. You know, like `git branch -a`. So I never clicked on it. Stupid. It turns out it's quite useful.

The stats are pretty cool, but the "Compare" button is what I've been looking for. Clicking that takes you to a page that shows the full directory diff between the head of the branch relative to the "base branch", which in this case is "master". Better yet, it gives you three optio

ns on how to view this diff:

You can see the diffs as a series of commits, the overall file changes (the one I was pining for), or the set of commit comments. This is great. Other than the fact that it doesn't cover

un-pushed commits and uncommitted changes in my local clone, and the fact that I really prefer side-by-side diff format ala SmartGit or Trac (but that's a story for another day), this is pretty much exactly what I wanted -- and if I had known about this 8 hours ago, I probably wouldn't have built the stuff I did.

But it gets even better. These buttons that show the two branches being compared is clickable.

Clicking one of these buttons bring up a modal that allows you to specify a specific commit. This lets you do a full tree diff between any two commits. Awesome!

In the end, I still think my solution is useful for the case where you want to deal with branches/commits that have not been pushed to GitHub, or you are working in a repository that is not on GitHub. (But who in their right mind does that, right?)

But, as soon as SmartGit covers this territory, I see no need for my solution...unless you don't use SmartGit (even though you should).

See a visual diff of work done in a git branch with git-branch-diff

2010-04-17T04:19:00.000-07:00

I have developed a Ruby script called git-branch-diff that allows you to use a graphical directory compare tool to compare the working tree of a branch with the state of the tree from which it was originally branched. For download, setup, and usage instructons, see:

http://github.com/scottwb/git-branch-diff

Background - My Workflow

I don't know about you, but a big part of my development workflow involves constantly diff'ing the changes I have made and am in the progress of making, as a way to constantly keep in touch with the state of what I am working on, and even as a reminder of what I am supposed to be doing. I like to write notes in contextual comments that I never intend to push out to the master branch, to help remind me of TODOs that need to be done before calling a feature sprint done.

In my world of constant multi-tasking and task-switching, I frequently have numerous topic branches in progress on the same repository. I like to end the day with no uncommitted changes, and even no un-pushed changes (i.e.: let github handle backups). So, after a task-switch, it's an integral part of my workflow to be able to see a good diff of the cumulative work I've done in a branch that hasn't been merged back in the master yet. This is usually pretty easy when we're talking about uncommitted changes. But, when I want to see the set of changes in that branch that would be merged if I were to do it now, things get a little more difficult.

The Problem

Git is a powerful tool with a crazy number of features. I probably only know 1% of them...and everywhere I searched, I could not find a way to do a visual directory compare of what has changed in my branch. Perhaps I've been spoiled by a great line of tools from syntevo GmbH that I've been using for years. I started with SmartCVS, then moved on to SmartSVN, and now to SmartGit. I am, always have been, and always will be a command-line guy. I'll admit that most of my cvs/svn/git usage happens from the keyboard. But I have come to rely on these GUI tools as an excellent addition to the command-line when it comes to visualizing my working tree status, performing directory and file diffs, editing line-by-line diffs, 3-way-merge conflict resolution, and easy selective add/commit/revert operations. Over the years I have come to rely on them as a key part of my routine - especially the directory diff view of all the changes in my working tree.

SmartGit has that capability for comparing the index, HEAD, and working tree, but not (yet) for arbitrary commits...or what I want most: comparing the HEAD of a topic-branch to the commit from which it was branched. The folks at syntevo assure me this will be included in a forthcoming point release, but what am I to do in the meantime?

Attempted Solution

I can get the data I want with a command like:

git checkout my_topic_branch
git diff `git merge-base master HEAD`

That shows me exactly the diffs I am interested in, but in a primitive textual (albeit colorized) context diff that I have to navigate with less.

Well, the folks at syntevo have another awesome tool called SmartSynchronize. This is their directory/file diff/sync/merge tool from their source control products as a standalone diff tool. So...why not just pump this diff data into that? To do that, you need to edit your ~/.gitconfig file to specify the custom difftool. For example, on Mac OS X, these additions look like:

The problem with this is that git invokes the difftool once per file being diff'ed. That's no good. I don't want to open my external visual diff tool 100 times. I want to open it once with a list of all the files added/modified/removed and be able to click on them individually to see their diffs, and ideally even be able to edit, synchronize, revert, etc. SmartSynchronize gives me all this...but git doesn't give it two directories to compare, it gives it two files to compare, one file at a time.

The Solution

So, I developed a tool called git-branch-diff that does all this for me. It works by finding the names and statuses of the differing files and constructing two temporary directories that match the original directory structure with the old and new versions, respectively. Then it passes those directories to SmartSynchronize, and voila! I've got a beautiful directory compare that represents the state of my topic-branch:

It only takes a few seconds to download and setup, and using it is as simple as:

git checkout my_topic_branch
git branch-diff

This script would be pretty easy to modify to handle comparing arbitrary commits...but once syntevo adds this to SmartGit, I won't need this anymore anyway! But, for those of you that won't be using SmartGit, this script can still be useful to you, and you can hook it up to whatever directory compare tool you want (e.g.: opendiff or BeyondCompare).

If you think this tool would be a valuable addition to your workflow, you can get the code and instructions for this tool at:

http://github.com/scottwb/git-branch-diff

It can easily plug in any directory compare tool, but I highly recommend you give SmartSynchronize a spin with its 30-day free trial.

NOTE: One current limitation in this script is that after making and saving modifications in SmartSynchronize, this script does not apply those back to your working tree. I'll probably implement that soon, so if you're interested, stay tuned...

How to detect what other sites your visitors have been to

2010-04-17T01:21:00.001-07:00

When someone visits your site, do you want to know where else they've been? Maybe you want to know if they've been to a competitor or affiliate site so you can make them special offers. Maybe you want to intelligently suggest they signup for your site with Facebook Connect, Twitter, Google OpenID, or Yahoo! ID, based on which of those sites you know they frequent. Maybe you're just nosy.

Regardless, there are pretty good controls in place with most browser to prevent you from developing javascript on your site that can sniff the user's browser history. There's one small loophole though...you can stick all the URLs you want to know about in your page, and test to see if they have been visited, using the CSS selector features of javascript frameworks such as jQuery and Prototype.js. This is not a fool-proof way to tell where they've been, but it might just be good enough for the kinds of suggestions you want to make to your user.

Based on ideas from Spyjax (described in this blog post), I whipped up a simpler demonstration of this using Haml to make a hidden list of URLs and, instead of trying to sniff links by their colors, using Prototype.js to detect which of them have been visited. This could have just as easily been done in jQuery:

One extension to this technique, if you have lots of URLs to test, might be to dynamically load 100 at a time via AJAX periodically, and asynchronously trigger actions in your page when visited URLs are detected.

Isn't that a little creepy? What if I don't want sites to get this info about me?

The Spyjax blog post linked to earlier has a decent answer for this:

The less browser history you store the fewer URLs someone can steal from that history. In Firefox you can change the amount of browser history by going to Tools -> Options -> Privacy and then either uncheck the “Remember visited pages” checkbox or change the number of days that history is stored for.

Fixing IE onChange event for checkboxes and radio buttons

2010-01-27T00:33:00.000-08:00

In Internet Explorer (up to at least IE8) clicking a radio button or checkbox to change its value does not actually trigger the onChange event until the the input loses focus. On every other browser, when the value of the input changes, onChange is triggered.

This is quite annoying, for example, if you want to trigger some other page changes off of the selection of a radio button in a radio button group changing. Some may suggest using onClick instead, and while that may be sufficient for a checkbox, for radio buttons, this is not a good solution. The problem is that onClick will fire when you click the radio button that is already selected, so you'll get your code triggered even when the value doesn't change.

Here's a quick hack that worked for me. On IE, handle the onClick event of radio and checkboxes and have it blur and re-focus the input element. This will trigger it to signal the onChange event like you'd expect from a decent browser. Then, you can go about your business using the onChange event like you should, and it will work correctly for you across the popular browsers. (WARNING: Of course, if you have other code triggered off of the blur and focus events, your mileage may vary.)

Here's an example of how you might do this using jQuery:


  if ($.browser.msie) {
    $(function() {
      $('input:radio, input:checkbox').click(function() {
        this.blur();
        this.focus();
      });
    });
  }

Sharing a partial for both server-side and client-side HTML generation.

2008-11-30T15:20:00.000-08:00

Say I have a page that shows a table of items (like a shopping cart), and I want to be able to generate the HTML for this table from the server-side (of course). Let's also say that I want to have client-side javascript be able to dynamically add items to this table without a round-trip to the server, based on data gathered on the client. What I end up with is some HTML code (actually, ERb or Haml) that defines the layout of a row in this table, and a duplicate piece of code in javascript that generates identical HTML. The server-side code uses a nice templating language like Haml to fill in the variable pieces, while the client-side code pieces together HTML embedded in strings concatenated with the dynamic data.

Not very DRY. Not easy to maintain. Not pretty to look at.

So...What's a good solution that allows me to write the HTML once, and have one place to maintain it, while still being able to use it to dynamically evaluate the template based on parameterized data on both the server and client sides?

Below is a solution that I came up with. (And I would love to hear any better ideas).

For brevity, I'll use a very simple HTML structure in these examples, but in practice I have used this with much more complex structure and logic. Here's what our example table looks like:


<table id="items">
  <tr>
    <td class="name">Shirt</td>
    <td class="desc">White short-sleeved shirt</td>
    <td class="price">$5.99</td>
  </tr>

  <tr>
    <td class="name">Pants</td>
    <td class="desc">Blue pants with zipper</td>
    <td class="price">$25.99</td>
  </tr>
</table>

Assume we have this built using Haml, and there are two partials: one for the top-level table, and one for an individual row. E.g.:

items_table.haml:


%table#items
  - items.each do |item|
    = render :partial => 'item_row', :locals => {:item => item}

item_row.haml:


%tr
  %td.name= item.name
  %td.desc= item.desc
  %td.price= item.price

In the client-side javascript in our top-level Haml partial, assume we have some function that gets called from some user input action (like filling out a form and clicking a button). E.g.:


:javascript
  var addItem = function(name, desc, price) {
    var html =
      "<tr>" +
      "  <td class=\"name\">" + name + "</td>" +
      "  <td class=\"desc\">" + desc + "</td>" +
      "  <td class=\"price\">" + price + "</td>" +
      "</tr>";
    $('items').insert(html); // using prototype.js
  };

OK...so this doesn't seem so bad with such a simple example, but imagine that the HTML structure for a table row is something much more complex, with conditionals based on the item, and maybe even usage of additional sub-partials. This maintenance for this becomes a nightmare.

This first thing I am going to do is change the item_rows.haml partial. Instead of having it actually render the item's values, it's going to render template replacement variables using #{...} syntax. Note, these are NOT interpolated in the partial. We'll do that in a separate phase from rendering:

item_row.haml:


%tr
  %td.name #{item_name}
  %td.desc #{item_desc}
  %td.price #{item_price}

Next, let's make a couple helper methods. The first one returns this partial rendered as a string. (I do this because in the real world, this takes a number of parameters, and adding a helper layer makes things easier to manage than always calling render directly). The second one renders this partial and then interpolates the #{...} strings inside it, based on the properties of the item we want to render -- effectively making this equivalent to rendering the original version of this template.

item_helper.rb:


module ItemHelper
  # Make ActionController::Base#render_to_string available.
  helper_method :render_to_string

  def item_row_template
    render_to_string :partial => item_row
  end

  def item_row(item)
    # Get the template.
    t = item_row_template

    # Define the template replacement variables.
    item_name = item.name
    item_desc = item.desc
    item_price = item.price

    # Evaluate the template
    eval(t.inspect.gsub("\\#", "#"))
  end
end

This, of course, means we change the top-level partial to use this new item_row helper, like so:

items_table.haml:


%table#items
  - items.each do |item|
    = item_row(item)

At this point, we've got a functionally equivalent server-side. But the big benefit is that now the HTML template returned by the item_row_template helper method can also be used to do the same substitutions on the client side, thanks to the handy Template class in prototype.js. The Haml that defines the javascript addItem method now looks like this:


:javascript
  var addItem = function(name, desc, price) {
    var html = new Template(
      '#{escape_javascript(item_row_template)}'
    ).evaluate({
      item_name : name,
      item_desc : desc,
      item_price : price
    });
    $('items').insert(html);
  };

Now, when the markup for the item row changes, it only needs to change in one place, and both the server-side and client-side benefit from it. This certainly feels more DRY...but also seems like a hefty price to pay in terms of complexity...

Bug in Haml Partial Rendering?

2008-11-17T15:45:00.000-08:00

I have stumbled across what I think is a bug in rendering a Haml partial in Ruby on Rails. The issue is that when I use render(:partial => '...', :locals => {...}), and one of the locals I define is a Boolean, it seems that if I pass a value of false, that value erroneously gets converted to nil. This causes a problem with a partial that wants to treat false, true, and nil as meaning different things.

Consider the following example:

I have a simple TestController:


class TestController < ApplicationController
  def index
  end
end

Then, the haml template for that controller:


!!!
%html
  %head
    %title haml partial test
  %body
    %b Rendering a Haml Partial
    %br
    = render :partial => 'test_haml', :locals => {:flag1 => true, :flag2 => false}

    %hr
    %b Rendering an RHTML Partial
    %br
    = render :partial => 'test_rhtml', :locals => {:flag1 => true, :flag2 => false}

Notice that partial renders two partials. One is a haml partial, one is an rhtml partial. The rthml one behaves as I would expect, the haml one does not.

Here is what the haml partial looks like:


- if flag1 == false
  flag1 is false
- elsif flag1 == true
  flag1 is true
- elsif flag1 == nil
  flag1 is nil
- else
  = "flag1 is #{flag1.inspect}"

%br
%br

- if flag2 == false
  flag2 is false
- elsif flag2 == true
  flag2 is true
- elsif flag2 == nil
  flag2 is nil
- else
  = "flag2 is #{flag2.inspect}"

Here is what the rhtml partial looks like:


<% if flag1 == false %>
  flag1 is false
<% elsif flag1 == true %>
  flag1 is true
<% elsif flag1 == nil %>
  flag1 is nil
<% else %>
  flag1 is <%= flag1.inspect %>
<% end %>






<% if flag2 == false %>
  flag2 is false
<% elsif flag2 == true %>
  flag2 is true
<% elsif flag2 == nil %>
  flag2 is nil
<% else %>
  flag2 is <%= flag2.inspect %>
<% end %>

Here is what the output looks like:

Notice the discrepancy between the output of the haml partial versus the rhtml partial, with respect to flag2.

This is gonna turn out to be a real drag on a project that I am converting from rhtml to haml that has a number of partials that assume that false means false and not nil.

Anybody out there have any thoughts on this?

Blogger Syntax Highlighting

2008-10-12T01:14:00.000-07:00

Since I intend to post code snippets here, I guess I had better figure out how to format code nicely in my blog. I am using Blogger, and from what I can tell in my vast 2 hours of Blogger experience, they don't have any simple built-in way to do nice code formatting and syntax highlighting.

I found a lot of references on how to use the very cool SyntaxHighlighter. Check out blogs on this topic from developertips and IdeaMonk -- that stuff helped a lot.

However, I think since these posts, the javascript code they describe to fix Blogger-specific issues with automatically-inserted <br/> tags has been rolled into the SyntaxHighlighter code, or is otherwise no longer necessary. Including it seems to break the formatting on my blog -- it was removing too many line breaks and putting all my code on one line. I didn't really dig into why that was the case. I just noticed that by removing the extra javascript code and adding the BloggerMode() call, everything worked out nicely.

So...here's a step-by-step walk-through of what I did to get this working:

Step 1: Get SyntaxHighlighter

Download the latest version of SyntaxHighlighter (1.5.1 as of this writing).
Uncompress the RAR file (download WinRar if necessary).
Place the contents of the scripts and styles directories onto a web server that can serve these files for your blog (more on that later). I put these files all in a flat directory structure in my example.

Step 2: Add SyntaxHighlighter to your template

On Blogger, go to Layout->Edit HTML and add the following code to your template, just before the </head> closing tag:


<link href="http://[YOURSITE]/SyntaxHighlighter.css" rel="stylesheet" type="text/css"/>
<script src="http://[YOURSITE]/shCore.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushCpp.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushCSharp.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushCss.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushDelphi.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushJava.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushJScript.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushPhp.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushPython.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushRuby.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushSql.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushVb.js" type="text/javascript"/>
<script src="http://[YOURSITE]/shBrushXml.js" type="text/javascript"/>

(You can delete the lines for languages you aren't going to use.)

Then, at the bottom of your template, just before the </body> closing tag, add the following code:


<script type="text/javascript">
//<![CDATA[
dp.SyntaxHighlighter.ClipboardSwf = "http://[YOURSITE]/clipboard.swf";
dp.SyntaxHighlighter.BloggerMode();
dp.SyntaxHighlighter.HighlightAll("code");
//]]>
</script>

Step 3: Markup your code snippets

Anywhere you want to stick a piece of code in your blog, put code like:


<pre name="code" class="ruby">
  class Foo
    def bar
      puts "foo bar"
    end
  end
</pre>

That should give you something that looks like:


class Foo
  def bar
    puts "foo bar"
  end
end

The name of the class you use specifies what language to format the code as, and directly corresponds to the names of those javascript files you included in your template.

What if I don't have a web server for these files?

Unfortunately, as far as I can tell, Blogger doesn't provide any static file storage that you can use to serve up these files. (It sure would be nice if they had them already installed somewhere we could all pull them from.) For a quick-and-dirty, free solution that I am not really happy with, try Google Sites. There is one catch: they don't let you store .js files. Here's what I did with Google Sites:

Create a new account and a new site.
Once you're in, create a new page called "SyntaxHighlighter", that is a "File Cabinet".
Rename all the .js files to end in .sj instead.
Update the .css file and all the .sj files to this new file cabinet.
Modify all the code in your Blogger template that points to these .js files to end in .sj.
From the Google Sites file cabinet, copy the link location to one of the files and use the relevant part of that to fix up all the css and js links in your template.

The main caveat to this approach is that it seems to be really slow. I am using the nice friendly URLs to the files on Google Sites, which all end up using HTTP redirects. It's less than ideal, but it's also quick to setup and free. Once you like the way this stuff works on your blog, you can probably find a better place to host those css and js files.

Caveats

I still have a couple issues that I didn't spend any time figuring out yet:

I think if you go back and forth between "Edit Html" mode and "Compose" mode, you may get extra <br/> tags inserted into your code.
This seemed to mess up the drag-n-drop layout editor. As far as I can tell, it only messed up the placement of the gadgets in the layout editor, but not the actual functionality, and the gadgets still appear on the blog posts in the correct place.
The SyntaxHighlighter, or at least the way I added it to my template, seems to break on IE6. It's way too late to care about IE6 users. Those people don't deserve to suck any more time from web developers than they already have.
Shit. I just found out it's broken on Safari too. Looks like its "too many redirects" for the ".sj" files. I am thinking that hosting these files on Google Sites is not such a good idea...

I Can't Sleep

2008-10-11T12:01:00.000-07:00

I can't sleep. Granted, it's noon on a Saturday and I am not supposed to be sleeping, but I don't mean right this minute...I mean most nights. I have hundreds of thoughts running through my head laying in bed...usually about code of some sort. Most times when this happens, I get up and start coding. Usually for work, but often just messing around for fun. I have been told many times that I should start a blog, post code snippets, talk about interesting new things I am playing around with, etc. Yeah...I am years and years behind the curve on this -- mainly because I prefer coding to talking about coding. But alas, I have decided to try my hand at blogging a little bit about the stuff I am toying around with, so here goes...