1..2..3.. you're back in the room!

After a few woes over the past few days all seems well again.

First, my email died—re-sending old emails, which, even after deleting boatloads just seemed to keep filling up my inbox. In an attempt to keep better track of things and provide a far better service I’ve switched to a new (and separate) provider. I’m going to see how it goes for a month and go from there. I ended up choosing one in the UK to provide a snappier IMAP experience.

If that wasn’t enough, the hostserver my VPS is on hit a ‘kernel bug’ earlier this morning and wiped me out for an hour or so, just as well I don’t host my own email!

So, if you’ve been trying to download code, read posts or contact me, that’s why I’d ‘gone dark’, but the LEDs are back on and blinking now!

Mint Follow-up

Firstly, one of my concerns was that I no longer had a way of viewing Pages per Visit, this was a great way of determining whether people navigated around my site having got here. For me this is important as if other people link to my posts, or are referred here through my comments to other posts/pages, they’re interested in what I have covered.

Mint has something similar to this (Total vs. Unique in it’s Pages panel), but the excellent Fresh View pepper let’s me visualise it. I can tell from the spikes (directly correlated to me making a posting (and receiving hits from referrers such as Planet Ruby on Rails and Developer Fusion) that there are more views per visit for that day, than a non-posting day. Leading me to believe that when other people find their way here (by way of Google, for instance) that they stick with the article they found. Perhaps making it easier to find related posts may encourage people to stay?

The brilliant Outclicks pepper also lets me see where people go afterwards, and the page they were on beforehand. It doesn’t really help me target posts better, but it does let me know that people are finding things useful, if not the things I directly write.

lets me know whether I'm doing relatively better, or worse, between two time periods. For me, this is weekly. I know that because I posted more times this week, I've served markedly more requests. A great encouragement to keep posting. However, what's more important is that I can see which posts are retaining popularity through being referenced elsewhere, or found by Google. If I can keep on finding interesting things to post about, perhaps people will stick around.

Overall, it’s feeling much better. I still miss a way of seeing the amount of time a user spends on a particular page. Google Analytics had a great page that could show you a breakdown of pages, based on pages people spent most time at, the average time a user spent on any given page etc. Very useful stuff. Perhaps I should look at the Pepper API and see whether I could do it. Unless someone’s seen something else to do it?

Have a good weekend!

Great Hosted Email Needed

My email seems to be going a little screwy right now (currently hosted by my old ASP.NET provider – CrystalTech). Their webmail has never been that great to use, and anti-spam/virus has also been a little flaky—I ended up just rejecting any emails with any attachments.

So, I’m now looking to find somewhere else to host my emails. I could move it onto my VPS, but I’d rather not have something as essential as email at the will of my tinkering! It’d also be nice to have really, really solid anti-virus and anti-spam protection that I just don’t have time to administer.

Essentially, I’m looking for something along the lines of MessageLabs, but usable (and affordable) for an individual. I also need IMAP (rather than POP3—I access emails all over the place), and a good webmail app would be really nice too. Finally, it has to just be a hosted email service, I don’t really need any webhosting as part of the package.

So, does anyone have any good recommendations?

Ruby and Readline

D’oh. Looks like I can’t use `script/console` just at the moment on my VPS’s install of Ruby, which I seem to remember just installing via my host’s `apt` repository for Red Hat Enterprise 4.

Anyway, here’s the output of the command::

Loading development environment.  /usr/local/lib/ruby/1.8/irb/completion.rb:10:in `require':  no such file to load -- readline (LoadError)    from /usr/local/lib/ruby/1.8/irb/completion.rb:10    from /usr/local/lib/ruby/1.8/irb/init.rb:252:in `load_modules'    from /usr/local/lib/ruby/1.8/irb/init.rb:250:in `load_modules'    from /usr/local/lib/ruby/1.8/irb/init.rb:21:in `setup'    from /usr/local/lib/ruby/1.8/irb.rb:54:in `start'    from /usr/local/bin/irb:13

Do I have to rebuild Ruby for `readline` to be included? Or is anyone aware of any clever hackery to avoid it? I’d just rather not be tinkering with Ruby on my server which would surely kill a few things whilst I do.

Incidentally I need this because I’m trying to get a clever way of migrating data from MySQL to Postgresql and rather than hacking away at SQL statements I thought it’d be much nicer to bend Chad Fowler’s suggestion in Rails Recipes to extract data to YAML from MySQL and then back from YAML to Postgresql and let Ruby and ActiveRecord do the legwork.

New Typo Theme

If you’re viewing this through your web-browser then you will (hopefully) notice the new theme that I’m trialling. I was previously using Hemingway a brilliant theme produced for the original Typo theme contest at TypoGarden.

Although I loved its two column layout, it could become a little difficult to read the home-page, and for the usual sidebar stuff (links to archives, searching and the like) you had to scroll to the bottom of the page, not that great a usability thing.

Then today I noticed Brighthouse on the TypoGarden site. It’s a much lighter, brighter looking theme, and hopefully just as clear and perhaps even a little more usable?

So, to all those who have seen the site both pre and post Hemingway, how do you feel? Think it’s worth the swap? Or am I to expect a great swathe of unapologetic apathy :)

ReSharper 2 is shaping up!

I’ve been using ReSharper with Visual Studio 2003 for probably 15 months or so now, and have found it to be one of the most invaluable tools I use (along with Reflector, CruiseControl.net, NUnit, TestDriven.net, Clover.NET, the list goes on :). But ReSharper just makes the development process nicer.

I can add references and not worry about having to add the using statement myself, ReSharper pops up the red for me automatically. It makes writing code test-first easier, I can write a statement then have ReSharper write the new class/property/method when it recognises it doesn’t already exist. And now, with ReSharper 2 (via. their EAP program) I’ve been using their built-in unit testing runner.

Previous EAP builds have been a little buggy, and the unit test runner has been pretty darn slow, but this seems to have improved with the latest build. At least to the point where I can use it and tolerate the occasional unhandled exception message appear. I just can’t wait for it to be released, which I believe is very soon!

In fact, I find their unit test runner experience to be superior to TDD.net’s, and NUnit GUI’s.

1. It sits in a window giving me a constant green/red bar. With TDD.net it kind of hides away after I run the tests and go back to the code. It’s a good constant reminder of progress. A positive psychological thing.
2. When I then update the code and rebuild, it will colour the tests that existed and ran previously a slightly different colour. Obviously these should all be green, but, its nice to see which tests I’d just run and just added etc.
3. It has a very nice assertion message viewer that is formatted. It’s surprising just how much nicer this is to use. Both NUnit’s GUI and TDD.net just display regular text. It can be difficult to separate messages from the stack trace, and messages can even be hard to read.
4. For `Assert.AreEqual(string, string)` assertions, it’s posssible to get a diff view of the comparison. This would’ve been extremely useful where we’ve previously done stuff in the past to compare generated HTML and templates for mail merges, where its often just an `Environment.NewLine` out, or a space misplaced in a sea of markup.

In fact, it makes me wish that Ruby’s Test::Unit unit test framework had a way of colouring the messages it displayed so my Rails’ `rake test_units` and `test_functional` commands would be easier to see at a glance in the terminal window.

Don’t worry, despite my happiness with a ReSharper + Visual Studio 2005, I still have hankerings to get back to the Ruby on Rails development later tonight and tomorrow! Let the fun begin! I’ve got some great projects lined up, including one or two multiplayer online games! More on those soon!

Oh, and apologies for the lack of regular updates, it won’t happen again. I promise.

How to improve your photos in Photoshop

One of the quickest and simplest ways is to use Photoshop’s Levels through an adjustment layer, allowing you to leave the original image intact. To accomplish this, click the half-black and half-white circle icon in your layer panel (shown on the right here). Then, in the pop-up menu, select “Levels…”. This will pop up an editor which will look something like the shot below.

This window displays the distribution of information from dark to light across the image. Using the sliders across the bottom it’s possible to adjust the dark point, mid point, and white point through the sliders.

So, to improve the contrast and stop the image feeling a little flat, we’ll move the dark slider across (telling Photoshop to consider that the new position is the dark point—making lighter areas appear darker). We’ll also slide the mid-tone slider across to the right to make the whole image appear darker, and move the left slider in slightly to lighten the top-end. The new positions look as in the image to the right.

They’re relatively small changes, but they make quite a large improvement as you’ll see from the adjusted image to the left. Now, although this image looks far more vibrant, and less flat, it also looks markedly red. So, time to adjust things a little more to remove that redness so it looks a little more natural.

To do that, click that black/white semicircle thing again, but this time we'll insert a "Color Balance" adjustment layer in. Once that pops up, we just move the top slider a little to the left until the image looks a little better. We just want to tone the over-bright redness a little.

This leaves us with a final image as follows:

And for comparison to the original:

As you can see, the image is bright and vibrant, the blues, reds, greens and blacks are solid, and it looks much more lively. Overall, much more pleasing. The Levels adjuster is one of the quickest ways I’ve found to help a photo jump out at you, although there are more advanced ways to adjust more precisely—using the Curves adjuster for instance, maybe I’ll try and find an example to show that in the future. My next job is to remove some of the dust and other bits ‘n pieces in the mesh that distract the eye in the background.

Incidentally, I picked up this (and my other retouching ‘skillz’) through Katrin Eismann’s excellent book “Photoshop Retouching & Restoration”. It has some great walkthroughs, including some amazing work with restoring old destroyed photos. I’d thoroughly, thoroughly recommend it to anyone looking to improve their photos with Photoshop!

Visual Studio the first to offer integrated Unit Testing

You read it here first (well, you didn’t really—it’s Visual Studio Magazine that’s making the rather grandiose assertion in this article).

A message was posted to the XP mailing list earlier today that thanked the article for not calling it Test-Driven Development (due to the last fiasco where Microsoft seemed to have bungled their intepretation of test driven development with regard to their integrated testing tools).

However, in the first paragraph of the article is the following nugget:


“To be fair, it isn’t just the Visual Studio IDE that lacked integrated unit testing. None of the major development environments—and this includes Java IDEs—have integrated unit testing directly into the development IDEs. The Visual Studio Team System (VSTS) edition of Visual Studio 2005 corrects this …”

Now, admittedly my exposure to other IDEs is pretty limited (I’m primarily a .NET developer) but both Eclipse and IDEA when I used them had JUnit integrated nicely. Victor Goldberg followed up later mentioning a Smalltalk IDE that had a TDD framework built-in. So there are definitely other IDEs that have unit testing integrated.

Now, it’s true that this is the first time unit testing has been directly integrated into Visual Studio by Microsoft, but there have been add-ins for doing the same thing for some time (TestDriven.NET is the shining example). So part of the assertion is true. But, it’s just plain wrong to state that “[none] of the major development environments—and this includes Java IDEs—have integrated unit testing”.

Of course it’s possible my interpretation is a little different to what was intended, but I’m not sure it is. A more realistic explanation is that the language is a little looser than it should be and that Visual Studio is the first to provide an integrated, out of the box unit testing framework, which, is still something I’m a little unsure is true—my recollection is both IDEA and Eclipse had it out of the box, but maybe that’s just my memory?

Either way, it’s this definition of integration that bothers me. Why is it so much better that it’s provided by Microsoft straight away? It doesn’t take much to install the add-ins for the various IDEs, and more importantly, they appear to be far better (at least in Visual Studio’s case anyway). Visual Studio’s integrated unit testing appears to be entirely inadequate for those wanting to work test first with Test Driven Development, so I think I’ll stick with the free tools that work better thanks!

Stability Problems

So it looks like during the past week or so whilst I’ve been away has been a relatively unstable one for the site. It boiled down to MySQL disappearing, resulting in Typo (Rails) dying.

I checked out `/var/mysqld.log` and there were a few entries with the following snippets:

051224  7:06:35  InnoDB: Fatal error: cannot allocate 8404992 bytes ofInnoDB: memory with malloc! Total allocated memoryInnoDB: by InnoDB 6319072 bytes. Operating system errno: 12InnoDB: Cannot continue operation!InnoDB: Check if you should increase the swap file orInnoDB: ulimits of your operating system.InnoDB: On FreeBSD check you have compiled the OS withInnoDB: a big enough maximum process size.InnoDB: Fatal error: cannot allocate the memory for the buffer pool051224  7:06:57 [ERROR] Can't init databases051224  7:07:00 [ERROR] Aborting051224  7:07:55 [Note] /usr/libexec/mysqld: Shutdown complete051224 07:08:02  mysqld ended

It looks like something is gobbling up memory to the point where MySQL can’t allocate enough. What’s weird is I can’t see what could possibly be trying to get MySQL to allocate 8.4MB? Any MySQL gurus know anything? Does MySQL just expand the data files by a certain margin every now and then, or is this allocation as a direct result of data manipulation/querying?

I was quite happily taking around 200+ hits per day a few weeks ago (although this seems to have dropped significantly as a result of my Google ranking for a few things dropping, related to outages perhaps?) so I’m guessing this is related to my (mis)configuration of Lighty and PHP, although I can’t quite see where.

I’ve changed my PHP FCGI socket configuration inside Lighty to recycle PHP processes sooner to see whether that helps:

"bin-environment" => (  "PHP_FCGI_CHILDREN" => "4",   "PHP_FCGI_MAX_REQUESTS" => "100" )

Failing that, any suggestions from any Linux gurus? I’ve noticed that when it does die, it either completely locks-up the VPS, or, it seems to look absolutely fine without any process hogs. I’ve noticed that there’s been a couple of defunct PHP processes listed, is that indicative of a problem?

Anyway, hopefully this’ll sort itself following my tweaks, if not, any MySQL configuration changes people can suggest?

UPDATE: Seems like things seem to be running a little better now. I also compiled and deployed the latest release of Lightty to 1.4.11. So far it’s looking quite stable, no defunct processes to be found, and so far no hangs or crashes! However, following Robby Russell’s comments below, I think I’ll check out Postgres anyway!

Mmmm Minty Fresh

What better way to start than with the things I really like:

1. Freshness. The fact I can see results in realtime is great, I used to check Google each morning to see how I was doing (egotistical or what? :p), and this satisfies my urge for more frequent updates.
2. Extensible. I like that it’s extensible and largely open-source (albeit under a commercial license). For instance, one of my favourite add-ins (in Mint speak—pepper) is Session Tracker which shows me recent sessions, and how they got here! I can also have a panel showing info from FeedBurner, so I don’t need to visit 2 sites.
3. Clean. Everything is visible through the one page, and updates are performed in a behind-the-scenes-AJAX-goodness kind of way. Google was a bit too clunky for my liking—it seemed to jump and move around everywhere.

However, there are a few things that I do prefer Google for:

1. Exit ratio’s. These let me know the number of users that would head off elsewhere from a page in my site, this gave me an idea about the stickiness of content—whether I had posts of a high enough value to keep people looking around.
2. Pages per visit. As far as I’m aware this isn’t supported directly in Mint (perhaps there’s some pepper to do this?) and it was again useful to determine how active users were once they came to my site. All helpful in tuning content. For instance, I found that with a few of my Rails articles, over 50% of visitors then visited other pages on my site. A good indicator I’m getting some good traffic of people interested in Rails.
3. Depth of data. Google Analytics also had a number of extremely useful widgets to let me see various bits of info about my top visited content—compare visits over the last month for instance, or show the top referrers for any given page.
4. Length of Visit. I can see how long people spent on any given page, for instance on average people spend around 2mins looking at my homepage, and approximately 5mins for my article on Rails’ validation. Again, this is very useful to determine whether someone is actually reading the whole piece, whether they see it’s not for them soon (or just give up with the incoherence ;).

So, if you like your stats simple, clean, and fresh I’d highly recommend it, especially at just $30. The difficulty I had with installation was largely down to a slightly non-standard configuration. However, I don’t think it provides (nor really intended to provide) statistics as detailed and deep as Google’s.

Anyone have any great Mint add-ons they’ve found and could recommend to fill the stats holes I have?