pingles

D’oh. Looks like I can’t use `script/console` just at the moment on my VPS’s install of Ruby, which I seem to remember just installing via my host’s `apt` repository for Red Hat Enterprise 4.

Anyway, here’s the output of the command::

Loading development environment.  /usr/local/lib/ruby/1.8/irb/completion.rb:10:in `require':  no such file to load -- readline (LoadError)    from /usr/local/lib/ruby/1.8/irb/completion.rb:10    from /usr/local/lib/ruby/1.8/irb/init.rb:252:in `load_modules'    from /usr/local/lib/ruby/1.8/irb/init.rb:250:in `load_modules'    from /usr/local/lib/ruby/1.8/irb/init.rb:21:in `setup'    from /usr/local/lib/ruby/1.8/irb.rb:54:in `start'    from /usr/local/bin/irb:13

Do I have to rebuild Ruby for `readline` to be included? Or is anyone aware of any clever hackery to avoid it? I’d just rather not be tinkering with Ruby on my server which would surely kill a few things whilst I do.

Incidentally I need this because I’m trying to get a clever way of migrating data from MySQL to Postgresql and rather than hacking away at SQL statements I thought it’d be much nicer to bend Chad Fowler’s suggestion in Rails Recipes to extract data to YAML from MySQL and then back from YAML to Postgresql and let Ruby and ActiveRecord do the legwork.

If you’re viewing this through your web-browser then you will (hopefully) notice the new theme that I’m trialling. I was previously using Hemingway a brilliant theme produced for the original Typo theme contest at TypoGarden.

Although I loved its two column layout, it could become a little difficult to read the home-page, and for the usual sidebar stuff (links to archives, searching and the like) you had to scroll to the bottom of the page, not that great a usability thing.

Then today I noticed Brighthouse on the TypoGarden site. It’s a much lighter, brighter looking theme, and hopefully just as clear and perhaps even a little more usable?

So, to all those who have seen the site both pre and post Hemingway, how do you feel? Think it’s worth the swap? Or am I to expect a great swathe of unapologetic apathy :)

I’ve been using ReSharper with Visual Studio 2003 for probably 15 months or so now, and have found it to be one of the most invaluable tools I use (along with Reflector, CruiseControl.net, NUnit, TestDriven.net, Clover.NET, the list goes on :). But ReSharper just makes the development process nicer.

I can add references and not worry about having to add the using statement myself, ReSharper pops up the red for me automatically. It makes writing code test-first easier, I can write a statement then have ReSharper write the new class/property/method when it recognises it doesn’t already exist. And now, with ReSharper 2 (via. their EAP program) I’ve been using their built-in unit testing runner.

Previous EAP builds have been a little buggy, and the unit test runner has been pretty darn slow, but this seems to have improved with the latest build. At least to the point where I can use it and tolerate the occasional unhandled exception message appear. I just can’t wait for it to be released, which I believe is very soon!

In fact, I find their unit test runner experience to be superior to TDD.net’s, and NUnit GUI’s.

1. It sits in a window giving me a constant green/red bar. With TDD.net it kind of hides away after I run the tests and go back to the code. It’s a good constant reminder of progress. A positive psychological thing.
2. When I then update the code and rebuild, it will colour the tests that existed and ran previously a slightly different colour. Obviously these should all be green, but, its nice to see which tests I’d just run and just added etc.
3. It has a very nice assertion message viewer that is formatted. It’s surprising just how much nicer this is to use. Both NUnit’s GUI and TDD.net just display regular text. It can be difficult to separate messages from the stack trace, and messages can even be hard to read.
4. For `Assert.AreEqual(string, string)` assertions, it’s posssible to get a diff view of the comparison. This would’ve been extremely useful where we’ve previously done stuff in the past to compare generated HTML and templates for mail merges, where its often just an `Environment.NewLine` out, or a space misplaced in a sea of markup.

In fact, it makes me wish that Ruby’s Test::Unit unit test framework had a way of colouring the messages it displayed so my Rails’ `rake test_units` and `test_functional` commands would be easier to see at a glance in the terminal window.

Don’t worry, despite my happiness with a ReSharper + Visual Studio 2005, I still have hankerings to get back to the Ruby on Rails development later tonight and tomorrow! Let the fun begin! I’ve got some great projects lined up, including one or two multiplayer online games! More on those soon!

Oh, and apologies for the lack of regular updates, it won’t happen again. I promise.

One of the quickest and simplest ways is to use Photoshop’s Levels through an adjustment layer, allowing you to leave the original image intact. To accomplish this, click the half-black and half-white circle icon in your layer panel (shown on the right here). Then, in the pop-up menu, select “Levels…”. This will pop up an editor which will look something like the shot below.

This window displays the distribution of information from dark to light across the image. Using the sliders across the bottom it’s possible to adjust the dark point, mid point, and white point through the sliders.

So, to improve the contrast and stop the image feeling a little flat, we’ll move the dark slider across (telling Photoshop to consider that the new position is the dark point—making lighter areas appear darker). We’ll also slide the mid-tone slider across to the right to make the whole image appear darker, and move the left slider in slightly to lighten the top-end. The new positions look as in the image to the right.

They’re relatively small changes, but they make quite a large improvement as you’ll see from the adjusted image to the left. Now, although this image looks far more vibrant, and less flat, it also looks markedly red. So, time to adjust things a little more to remove that redness so it looks a little more natural.

To do that, click that black/white semicircle thing again, but this time we'll insert a "Color Balance" adjustment layer in. Once that pops up, we just move the top slider a little to the left until the image looks a little better. We just want to tone the over-bright redness a little.

This leaves us with a final image as follows:

And for comparison to the original:

As you can see, the image is bright and vibrant, the blues, reds, greens and blacks are solid, and it looks much more lively. Overall, much more pleasing. The Levels adjuster is one of the quickest ways I’ve found to help a photo jump out at you, although there are more advanced ways to adjust more precisely—using the Curves adjuster for instance, maybe I’ll try and find an example to show that in the future. My next job is to remove some of the dust and other bits ‘n pieces in the mesh that distract the eye in the background.

Incidentally, I picked up this (and my other retouching ‘skillz’) through Katrin Eismann’s excellent book “Photoshop Retouching & Restoration”. It has some great walkthroughs, including some amazing work with restoring old destroyed photos. I’d thoroughly, thoroughly recommend it to anyone looking to improve their photos with Photoshop!

You read it here first (well, you didn’t really—it’s Visual Studio Magazine that’s making the rather grandiose assertion in this article).

A message was posted to the XP mailing list earlier today that thanked the article for not calling it Test-Driven Development (due to the last fiasco where Microsoft seemed to have bungled their intepretation of test driven development with regard to their integrated testing tools).

However, in the first paragraph of the article is the following nugget:

“To be fair, it isn’t just the Visual Studio IDE that lacked integrated unit testing. None of the major development environments—and this includes Java IDEs—have integrated unit testing directly into the development IDEs. The Visual Studio Team System (VSTS) edition of Visual Studio 2005 corrects this …”

Now, admittedly my exposure to other IDEs is pretty limited (I’m primarily a .NET developer) but both Eclipse and IDEA when I used them had JUnit integrated nicely. Victor Goldberg followed up later mentioning a Smalltalk IDE that had a TDD framework built-in. So there are definitely other IDEs that have unit testing integrated.

Now, it’s true that this is the first time unit testing has been directly integrated into Visual Studio by Microsoft, but there have been add-ins for doing the same thing for some time (TestDriven.NET is the shining example). So part of the assertion is true. But, it’s just plain wrong to state that “[none] of the major development environments—and this includes Java IDEs—have integrated unit testing”.

Of course it’s possible my interpretation is a little different to what was intended, but I’m not sure it is. A more realistic explanation is that the language is a little looser than it should be and that Visual Studio is the first to provide an integrated, out of the box unit testing framework, which, is still something I’m a little unsure is true—my recollection is both IDEA and Eclipse had it out of the box, but maybe that’s just my memory?

Either way, it’s this definition of integration that bothers me. Why is it so much better that it’s provided by Microsoft straight away? It doesn’t take much to install the add-ins for the various IDEs, and more importantly, they appear to be far better (at least in Visual Studio’s case anyway). Visual Studio’s integrated unit testing appears to be entirely inadequate for those wanting to work test first with Test Driven Development, so I think I’ll stick with the free tools that work better thanks!

So it looks like during the past week or so whilst I’ve been away has been a relatively unstable one for the site. It boiled down to MySQL disappearing, resulting in Typo (Rails) dying.

I checked out `/var/mysqld.log` and there were a few entries with the following snippets:

051224  7:06:35  InnoDB: Fatal error: cannot allocate 8404992 bytes ofInnoDB: memory with malloc! Total allocated memoryInnoDB: by InnoDB 6319072 bytes. Operating system errno: 12InnoDB: Cannot continue operation!InnoDB: Check if you should increase the swap file orInnoDB: ulimits of your operating system.InnoDB: On FreeBSD check you have compiled the OS withInnoDB: a big enough maximum process size.InnoDB: Fatal error: cannot allocate the memory for the buffer pool051224  7:06:57 [ERROR] Can't init databases051224  7:07:00 [ERROR] Aborting051224  7:07:55 [Note] /usr/libexec/mysqld: Shutdown complete051224 07:08:02  mysqld ended

It looks like something is gobbling up memory to the point where MySQL can’t allocate enough. What’s weird is I can’t see what could possibly be trying to get MySQL to allocate 8.4MB? Any MySQL gurus know anything? Does MySQL just expand the data files by a certain margin every now and then, or is this allocation as a direct result of data manipulation/querying?

I was quite happily taking around 200+ hits per day a few weeks ago (although this seems to have dropped significantly as a result of my Google ranking for a few things dropping, related to outages perhaps?) so I’m guessing this is related to my (mis)configuration of Lighty and PHP, although I can’t quite see where.

I’ve changed my PHP FCGI socket configuration inside Lighty to recycle PHP processes sooner to see whether that helps:

"bin-environment" => (  "PHP_FCGI_CHILDREN" => "4",   "PHP_FCGI_MAX_REQUESTS" => "100" )

Failing that, any suggestions from any Linux gurus? I’ve noticed that when it does die, it either completely locks-up the VPS, or, it seems to look absolutely fine without any process hogs. I’ve noticed that there’s been a couple of defunct PHP processes listed, is that indicative of a problem?

Anyway, hopefully this’ll sort itself following my tweaks, if not, any MySQL configuration changes people can suggest?

UPDATE: Seems like things seem to be running a little better now. I also compiled and deployed the latest release of Lightty to 1.4.11. So far it’s looking quite stable, no defunct processes to be found, and so far no hangs or crashes! However, following Robby Russell’s comments below, I think I’ll check out Postgres anyway!

What better way to start than with the things I really like:

1. Freshness. The fact I can see results in realtime is great, I used to check Google each morning to see how I was doing (egotistical or what? :p), and this satisfies my urge for more frequent updates.
2. Extensible. I like that it’s extensible and largely open-source (albeit under a commercial license). For instance, one of my favourite add-ins (in Mint speak—pepper) is Session Tracker which shows me recent sessions, and how they got here! I can also have a panel showing info from FeedBurner, so I don’t need to visit 2 sites.
3. Clean. Everything is visible through the one page, and updates are performed in a behind-the-scenes-AJAX-goodness kind of way. Google was a bit too clunky for my liking—it seemed to jump and move around everywhere.

However, there are a few things that I do prefer Google for:

1. Exit ratio’s. These let me know the number of users that would head off elsewhere from a page in my site, this gave me an idea about the stickiness of content—whether I had posts of a high enough value to keep people looking around.
2. Pages per visit. As far as I’m aware this isn’t supported directly in Mint (perhaps there’s some pepper to do this?) and it was again useful to determine how active users were once they came to my site. All helpful in tuning content. For instance, I found that with a few of my Rails articles, over 50% of visitors then visited other pages on my site. A good indicator I’m getting some good traffic of people interested in Rails.
3. Depth of data. Google Analytics also had a number of extremely useful widgets to let me see various bits of info about my top visited content—compare visits over the last month for instance, or show the top referrers for any given page.
4. Length of Visit. I can see how long people spent on any given page, for instance on average people spend around 2mins looking at my homepage, and approximately 5mins for my article on Rails’ validation. Again, this is very useful to determine whether someone is actually reading the whole piece, whether they see it’s not for them soon (or just give up with the incoherence ;).

So, if you like your stats simple, clean, and fresh I’d highly recommend it, especially at just $30. The difficulty I had with installation was largely down to a slightly non-standard configuration. However, I don’t think it provides (nor really intended to provide) statistics as detailed and deep as Google’s.

Anyone have any great Mint add-ons they’ve found and could recommend to fill the stats holes I have?

So, for instance, the first test I have is that when I enter a valid postcode, I get a result back:

12345678910111213

def test_can_retrieve_postcodes_for_ajax_finder      xhr :post, :lookup_postcode, :postcode => "SW1A 1AA"          assert_response :success      assert_select_options "selected_address", 1    end    def lookup_postcode      @results = PostcodeLookup.find(        params[:postcode]        )      render_partial("postcode_results")    end

In this test, all I’m looking for is that we have a successful response and that we have a `select` html tag that contains 1 option (the address for our postcode). The helper assertion method looks as follows:

def assert_select_options(name, option_count)  assert_tag :tag => "select",    :attributes => {:id => name},    :children => {      :count => option_count,      :only => {:tag => "option"}      }end

This raises an important question—one of determinism and roles. In our tests, we’re not so much interested in the actual postcode results, we’re just interested that given a particular postcode, we have a single result. We want to test that our controller uses the postcode lookup code correctly (encapsulated through a separate set of classes I’ve written).

Once again, a fine example of where to mock. Not only do we prevent dependency on the web-service being accessible and running, we improve unit test performance, and we can guarantee results.

I’ll be writing another post on how I went about developing the postcode lookup aspect of the site (since I think mocking and test-first is something that Rails makes so easy it’s an incredibly valuable part of the framework).

So, our postcode lookup class guarantees us that with certain postcodes we will have guaranteed results. Which lets us also test with a bad postcode:

def test_not_filled_when_bad_address_selected  xhr :post, :fill_address, :selected_address => 1101010end

assert_response :successassert_equal "text/javascript", @response.headers["Content-Type"]assert !/Buckingham Palace/.match(@response.body)

This covers the display of possible addresses for a postcode. Now, to ensure that when we select an address we are given code that will change our input fields to contain the correct information.

Since we’re using inline RJS for this aspect of the application, we want to check that when we perform the lookup, our address entry `div` is replaced with the updated markup, containing our address.

RJS is rendered as JavaScript that will be interpreted and executed by the browser, so we want to assert that our response is marked as `text/javascript`. We then want to test that we can find our input tag that contains the first line of the address.

def test_when_selecting_address_address_fields_populated  xhr :post, :fill_address, :selected_address => 1end

assert_response :successassert_equal "text/javascript",   @response.headers["Content-Type"]assert /Buckingham Palace/.match(  @response.body)

As you can see we use a rather ugly regular expression to match on the address, rather than re-use the lovely `assert_tag` helper. Although it works, it’s not especially pretty and is something I’m looking to change. Has anyone tried doing this already? If not, I’ll let you know how I get on!

Within the main address editing form I have the following:

<%= text_field_tag :postcode %><%= submit_to_remote "addressSearch", "Search",  :url => {:action => 'lookup_postcode'},  :with => "'postcode='+postcode.value",  :update => :addressLookupResults%>

Results are then rendered via a partial as follows:

def lookup_postcode  @results = PostcodeLookup.find("SW1A 1AA")  render_partial("postcode_results")end

The results partial looks as follows:

<%= select_tag(  :selected_address,  options_for_select(    @results.map{|a| [a.description, a.address_id]},  '')  )%><%= submit_to_remote "selectAddress",  "Use",  :url => {:action => 'fill_address'},  :with => "'selected_address='+selected_address.value",%>

Upon a user pressing the button, the `fill_address` action will be invoked, and selected_address will be passed through. Now, for the magic (courtesy of RJS templates):

def fill_address  address = PostcodeLookup.get_address(    params[:selected_address]  )end

@address = Address.new(  :line_1 => address.line_1,  :line_2 => address.line_2,  :town => address.town,  :county => address.county,  :postcode => address.postcode  )

render :update do |page|  page.hide('addressLookupResults')  page.replace_html(    'address',    :partial => 'address'  )end

The significant stuff happens during the render method call. This generates JavaScript which is emitted to the browser, which when executed will replace the address entry section of the HTML with updated HTML containing the newly filled address fields.

So far it works quite well, but, as mentioned in a few places it does lead to relatively unsightly markup. Not that much of a problem in this situation as the physical impact on the rendered code isn’t that large. In a results page, it would perhaps be more significant.

The complaint seems to be a more theoretical one—that JavaScript shouldn’t belong in the markup, which is something a commenter (Jonathan Conway) on my previous post also mentioned.

I think there’s a tendency towards some people becoming quite millitant over these kinds of things, but I can fully appreciate that one bad step at the beginning can easily evolve into a messy, inelegant codebase that you don’t want to work with, and users don’t like having to consume.

So, my next step is to wrap this stuff up with Behaviour.js and improve my CSS knowledge a little and improve my code quality before I get too far into Ajax land!

I have the following form allowing users to enter their postcode:

<%= form_remote_tag(  :action => 'lookup_postcode',  :update => 'results',  :complete => "new Effect.SlideDown('lookupResults')" )%>...

To display the matching addresses, I then render a partial that contains the drop-down list and select button as follows (inside the `lookup_postcode` action):

if params[:selected_address].nil?  @results = PostcodeLookup.find(params[:postcode])  render_partial("postcode_results")

To avoid having many nested forms, I check to see whether the posted information included a selected address. If they have selected an address, I then use inline RJS to render the JavaScript necessary to get the address_fields filled:

render :update do |page|  page.hide 'originAddressLookupResults'  page.visual_effect :highlight,    'originAddress',    :duration => 3  page.replace_html('originAddressEntry',    :partial => 'address')end

Since the model fields the address partial is bound to are populated with the data from the lookup, the address does display with the selected values.

However, I’ve noticed that in FireFox I see the generated JavaScript from the inline RJS code—not particularly nice.

I originally thought it might be something to do with how WEBrick and FireFox were interpreting the generated script, but having also run the site through Lighttpd it also seems to show the code. Has anyone else had a similar problem? Or is it abundantly clear that I’ve missed something (again) :)

Anyway, it’s been fun getting Rails onto Edge and then playing with RJS, think I just need to find some good form patterns to determine the best way of having multiple tasks within a wider process performed through a single form—I’ll post my thoughts as I get further into it!