Essentially the data provides 1.7 million UK postcodes, with their British National Grid Reference, local authority and ward listed for each. The most interesting data is the grid reference, supplied as a full Northing and Easting value. These can be used to calculate distances between postcodes, as well as being able to convert them into Latitude and Longitude data for displaying on maps such as Google’s offering.

The data does lack Northern Ireland and the Channel Island data at present, which can be a problem if you wish to use the data to cover the whole of the UK. The Northern Ireland grid system has a different origin and projection, so care must be taken if those postcodes and data are merged in from other sources. See here for details of The Irish Grid system.

There is a wiki that covers this data, but it is incomplete and has not been updated in nearly half a year, which is a bit strange. I suspect updates are being lost somewhere, because I cannot believe nobody wants to update it (there are technical inaccuracies in the description of the data that someone should have noticed by now). I have made a few additions to the wiki page, so we will see what happens next. If it sticks, I would encourage others to go in and add what you know and discover in the data.

So, what does this data look like? In short, taking into account the areas it does not cover, pretty much complete. I have loaded the current dataset into MySQL, then plotted each postcode as a pixel in an image. Some pixels contain more than one closely positioned postcodes, and a more sophisticated plot would show this, but a simple plot of pixels gives us an idea of what it looks like.

OpenData Postcodes - Great Britain - Click for larger image

Click the image to look at a much larger version with lots of detail. I particularly love how you can see the valleys in South Wales.

This simple PHP script generated the above image. All the table needs to contain are the Eastings and Northings from the Ordnance Survey data. It is surprising how it manages to get through 1.7 million postcodes in about ten seconds:

<?php
$width = 3200;
$height = $width * 1.5;

$db_name = 'opendata_postcodes';
$db_user = 'username';
$db_pass = 'password';

$easting_max = 655448;
$northing_max = 1213660;

$max_points = 2000000;

$db = new PDO('mysql:host=localhost;dbname='.$db_name.';charset=UTF-8', $db_user, $db_pass);

$sql = 'SELECT easting, northing FROM postcode LIMIT :limit';

$stmt = $db->prepare($sql);
$stmt->bindParam(':limit', $max_points, PDO::PARAM_INT);
$stmt->execute();

$img = imagecreate($width, $height);
$background = imagecolorallocate($img, 230, 230, 255);

while ($row = $stmt->fetch(PDO::FETCH_ASSOC, PDO::FETCH_ORI_NEXT)) {
  $x = round(($width*$row['easting'])/$easting_max);
  $y = $height - round(($height*$row['northing'])/$northing_max);
  imagesetpixel($img, $x, $y, 300);
}

header("Content-type: image/png");
imagepng($img);
imagedestroy($img);

Here is the same image with some crude colours introduced to show the density of postcodes.

Colours to show the density of postcodes

Step 1 was to create a list of URLs that needed screengrabs. I wrote an small PHP that printed a list of URLs (one per line) which I then saved to a text file:

http://www.siteroom.co.uk/

http://www.consil.co.uk/

The first tool I came across was WebShot. The GUI interface is simple, but there’s far more flexibility using the command line. This is the command I used to run a batch of screenshots listed in the text file I created earlier

webshotcmd.exe /in "C:screenshots.txt"

The tool worked well and uses Internet Explorer to render the web pages (so make sure you have the latest version!). You can also add parameters to your batch list too. I wanted the width of the image to be 1280px wide, rather than defaulting to the width of the web page, and also to use custom file names.

/url "http://www.siteroom.co.uk/" /bwidth "1280" /out "C:screenshotssiteroom.jpg"
/url "http://www.consil.co.uk/" /bwidth "1280" /out "C:screenshotsconsil.jpg"

Unfortunately the free version only renders screengrabs in black and white.

I settled on Grab them all – a Firefox addon. Although it doesn’t have advanced features like watermarking, it does boast enough options for my needs. You’ve also got the benefit of Firefox’s web page rendering engine, which means screengrabbing the page with all the good-looking CSS3 features.

Options can be tweaked by going:

Tools > Add-ons > Extensions > Grab them all ‘Options’ button

I was able to change the filename from Base64 to the URL and set the width of the window. To run the program go to:

Tools >> Grab them all

Pass it the simple list of URLs (one URL per line) and let it run its magic!

I’ve seen similar tools for slip-streaming service packs into disks, but this one goes a step further and makes it dead easy to include drivers and various other Windows tweaks. It is all very straight-forward to use:

1. Point it at the original Windows disk.
2. Tell it what you want to change (point it at service packs, drivers, etc.)
3. Tell it to generate an ISO.
4. Press GO.

The application will then spit out an ISO image that you burn to CDROM. I’m sure there are ways to install it over the network without burning a CD too.

I found it while looking for a way to reinstall XP onto Dell machine with a SATA device that the default Windows disk did not recognise.

nlite – http://www.nliteos.com/ – a great application for your toolbox.

This was an old WIndows 98 system, and I could not simply plug in a USB drive, because there were no Windows 98 drivers around for the external USB drives that I had. In the end the disk had to be removed, and connected up to a working machine with a flying lead.

Since then, I have come across Puppy Linux. It seems to be ideal for this type of thing. By booting it directly from CDROM or USB key, it is possible to have a working Linux box in 60 seconds flat. It is then a simple matter of mounting the local drive, plugging in a USB drive, and transferring the files across.

Puppy Linux is not something I have ever come across before, but it is now a part of my standard toolbox. It also looks like the ideal platform for building custom appliances, such as video players and thin clients.

The most important thing is that once configired, I need to be able to hand over the keys to a client so that they can manage it from that point on. That is the hardest part, because no matter what I try, there are always highly technical aspects to sending out a newsletter that really stump the non-technical clients.

This is what I have come up with (and tried) so far:

PHPlist (Tincan)

PHPlist Logo

This has some lovely features, but some pretty dire ‘gotchas’ that can trap an unwary user. For a start the admin screens are as ugly as hell, and the terminology can make it hard to fathom out what each option does (for example, to create a draft newsletter, you need to use the ‘send a message’ option). It is just not intuitive.

Problems for end users involve its uncanny ability to drop stylesheets when editing templates. Another is signing lists of imported users up to text-only e-mails even when ‘HTML’ is selected. Yes these can be worked around, but only if you know the problems exist.

On the plus side, the e-mail sending system is brilliant. You queue messages to send, and leave it to do its job. If it fails (and anything that involves a browser window staying open will fail) then you just start it again and it carries on from where it left off. It keeps track of who a newsletter has been sent to, so you can requeue a newsletter every day until the next newsletter is ready, and it will send just to new subscribers.

The ability to pick of bounces from an IMAP or POP3 mail box is also great. Each bounce will increment a counter, and subscribers can be disabled when a preset number of bounces is reached.

This system also supports tracking, counting each individual e-mail as it is opened, using an image embedded in HTML e-mails.

Another downside is the lack of APIs, making integration difficult. Signing users up to a list automatically, while they are signing up to a CMS, is impossible without some major hacking.

The problems with PHPlist stem from it being around a long time. It has grown and grown over the years, but is creaking at the seams. Everything it does is great, but the way it does it is not so hot. It is time, IMO, to throw away the old code and start again, with all the same features, an established framework (e.g. Zend or CakePHP) and it will be a killer application.

I must also add that I believe PHPlist got the data schema right from the start. Most other newsletter systems I have looked at have gone for a much more simplistic approach, that makes it much harder for them to advance their products beyond very simple functionality.

With all that it is, I do use it, and find it great at what it does. What I can’t do, howver, is roll it out to clients, because it results in nothing but hassle from the non-technical users who just can’t get to grips with its mix of high-level features, and low-level ‘black art’ knowledge that is needed.

poMMo

poMMo logo

This one has to get the award for ‘hardest to find mailing list manager’. It was formally the bMail project and is hosted on SourceForge. Trying to find it when you can’t quite remember its ‘web 2.0′ name is quite an effort, but the result is well worth it.

Checking the project’s subversion repository, it does not seem there has been any activity on this project since August 2008, which is not very encouraging.

First of all, the admin screens are lovely. They are smooth, AJAX/jQuery-based, clear and well designed. The whole thing feels smooth – it works with you.

The first thing to note is that it does not support multiple mailing lists. In fact, it does not support any ‘mailing lists’ at all. However, what it does provide is a means to mail out to groups of users depending on attributes of those users.

For example, you could provide a series of checkboxes for a subscriber to select what subjects they are interested in. When sending out a newsletter, the author would send it to all subscribers who have expressed an interest in the subject of that newsletter. Now, that does sound rather like a multiple newslist system, but the subtle difference is that there is no ‘newslist’ object in the database – anything that looks like a newslist subscription is specified by the administrator, and not enforced as any kind of fundamental part of the system’s structure.

I think that approach works well. The system is simpler, and the flexibility is increased. There are a few downsides though.

The first downside is that without a newslist for a user to subscribe to, there is no place in which you can find out exactly when a user subscribed to that newslist. That means sending out additional copies of a newsletter to late subscribers simply cannot be done. Remember PHPlist knows who subscribed to what and when, and so is able to send out ‘catchup’ newsletters so a subscriber will always get the latest newsletter very soon after subscribing.

The other downside is in organisation. Without being able to group newsletters into newslists, it is much harder to provide archives organised by subject. On the other hand, the archives are there, with very easy access to the newsletters from a web-based front end.

The e-mail sending backend is both genius and a little frightening for a control-freak such as myself. It also misses a few tricks, I think.

In order to send e-mails, the system first creates a list of who to send to (in a text file, I believe). It then spawns a process through the HTTP protocal to send as many e-mails as it can in the PHP timeout period (you would set this to as long as possible). Before that script times out, it spawns another process to carry on where it left off, and so on until all the e-mails have been sent, whereupon a lock file created right at the start is released.

This is genious in that the developers have worked out a way to run a background script that can keep going even when the browser window is closed. It is frightening for much the same reason. I just don’t feel comfortable letting something like that loose on my server. It could be running all night in some kind of endless loop, and I would never know until I get irate e-mails from subscribers saying that I have filled up their inboxes.

Another minor flaw in the e-mail sending is that no record is kept of who each newsletter has been sent to. Doing that would make it very easy to sort out who has received a copy and who needs a catch-up copy sent. Following from that, without records of e-mails sent, there is no place to hang any flags to say whether that e-mail has been opened and read, so e-mail tracking is out. Clients need to know these things: how well read are those newsletters? Everyone has a master up the chain to report to, and someone up that line, often holding the purse-strings, likes simple measures of performance.

My ideal newsletter system would contain the database and functional features of PHPlist, with the administration front-end of poMMo.

Other Systems

I’ll add a few more when I get the time. I’m mainly looking at Open Source newsletter systems, aimed at sending to self-subscribed users (i.e. not bulk spam systems). Integration with an existing site or CMS is high on the priority list, along with ease of use for non-technical users. Other systems we will be evaluating are:

• ListMessenger – Free light version, but dirt cheap Pro version with all the features you would want. This one seems to include captchas for registration.
• Dada Mail – Old, well known, Perl-based, and it looks like a pain to install.
• Sympa – Again, Perl. This one seems to have been designed by engineers. That is to say the system looks very robust and complete, but there is little gloss to the system. Like Data Mail, you need root access to install it, so you have to be careful about dependances with other modules on your server.

Please note that I won’t be publishing every comment that starts “I work for XYZ and we have a PDF product…” (especially if I see the same posts listed on a hundred other sites with ‘PDF’ somewhere in the title). These are mainly the open source, free or cheap tools that I have personally found useful in projects. I still appreciate any tips or recommendatios.

Most of these tools fall into the data conversion category – they convert to- or from- PDF formats.

pdf2svg (PDFTron)

Convert a PDF document to an SVG document. The command line tool pdf2svg.exe is a WIndows-only tool, but is useful for off-line conversions. It is not free, but significantly cheaper than the Adobe CS3 suite that you would otherwise have to use.

Using the tool without license inserts several watermark layers. The resulting SVG can be manipulated easily using Inkscape (a free SVG editor that every developer should have in their toolbox). The thumbnails generated from each page contain the same watermarks.

This tool will also extract all images (as PNGs) from the PDF document, which is very handy.

iText

This free Java tool is platform-idependant, and provides a few features that make it very suited to batch PDF manipulation. Features include the ability to:

• Split up a document and manipulate pages.
• Generate PDF content on-the-fly.
• Fill out PDF forms.
• Add digital signatures.
• Create PDFs from scratch, including barcodes.

iText can also output Rich Text Format (RTF) documents. I have used it in the past to extract text from PDFs for indexing on a website, though there are lighter tools for doing this.

pdftk (PDF Tool Kit, by AccessPDF)

This is surely the lightest Swiss Army Knife of PDF tools. Features include:

• Bursting into single-page PDFs and recombining.
• Encrypting/decrypting.
• Inserting and extracting form data (for older style forms, though Adobe seems to have changed the way forms work in later versions of the PDF format, so that extracting from filled forms is no longer straight-forward).
• Extract and manipulate metadata.

PDFBox

This is another free java library, with features including:

• Extracting text from a PDF (for indexing, e.g. with Lucene or mnoGoSearch).
• Manipulating pages (inserting, extracting, reordering).
• Filling and extracting form data (PDF version below 1.6) using FDF and XFDF data files.
• Creating images from PDF files – good for thumbnails and creating ‘page flipper’ applications.

To use the text extraction in a search engine, I use the following shell script to wrap it all up:

[sourcecode language="cpp"]#!/bin/sh
# Convert a PDF document to text
# Usage: $0 [OPTIONS] [Text File]
#   -password      Password to decrypt document
#   -encoding      (ISO-8859-1,UTF-16BE,UTF-16LE,…)
#   -console    Send text to console instead of file
#   -html    Output in HTML format instead of raw text
#   -sort    Sort the text before writing
#   -startPage     The first page to start extraction(1 based)
#   -endPage     The last page to extract(inclusive)
#       The PDF document to use
#   [Text File]    The file to write the text to

PDFBOX_BASE=/usr/local/lib/pdfbox

export CLASSPATH=$PDFBOX_BASE/external/FontBox-0.1.0-dev.jar:$PDFBOX_BASE/lib/PDFBox-0.7.3.jar
java org.pdfbox.ExtractText “$@”[/sourcecode]

This assumes that PDFBox has been installed under /usr/local/lib.

SWFtools

Although this toolkit is primarily about SWF files, it does have some neat PDF to SFW conversion scripts. Versions are available for Windows and Linux under an Open Source licence.

Notes

One of these tools also extracts images from PDFs, which can b every useful when converting PDF to HTML formats.

There are a number of PHP tools, other libraries (e.g. Image Magick) and more heavy-weight tools (e.g. Ghostscript) that I will cover later. Hopefully this selection will help in the meantime. If you have any further suggestions, I would love to hear of them. Even if a tool duplicates much of what these do, it only needs have do one extra feature that the others don’t cover to be worthwhile using.

KeePass: http://keepass.info/

All very self-explanatory. A master password locks the database. Plenty of plugins to make life easier for you.

So far I’ve only tryed using the Firefox >> KeyPass importer which in turn requires you to download the KeyPass XML importer. I experienced problems when trying to directly import from Firefox so I downloaded the Password Exporter addon. This exports your Firefox passwords into an XML file which the Firefox >> KeyPass importer can then import.

The next plugin I want to try is KeyPassSync. This will allow me to syncronise my passwords on my home desktop, laptop and work desktop computers. Unfortunately this is only available with version 2 that’s currently in alpha testing. I’d rather wait until the BETA before I trust it with my password database!