I’ve been doing a fair bit of coding in recent weeks and I thought I’d better write a few details down before I forget about them.

As previously noted, I’ve been gathering together various historical data sets for a project at the National Museum of Australia. One resource that I was keen on including was the fantastic Australian Newspapers project at the National Library of Australia. What I had in mind was being able to give a sense of context to any historical event by calling up the headlines for that particular time.

Unfortunately there’s no API for the newspapers project (or Trove in general), though apparently it’s in the works. So I had to reverse engineer the advanced search page to work out the various query options, and then build a screen scraper to harvest the results. I played around with the search options a bit to fine tune the results, finally deciding to limit them to ‘news’ articles with more than 1000 words. Annoyingly, only 10 results are returned at a time.

I had hoped to parse the results as xml, but a rogue <br> tag broke the XHTML, so I fell back on Beautiful Soup – a Python module that makes screen scraping considerably easier by tidying up HTML structures. After than it was pretty straightforward. Soon I had my own Python module to query the newspapers database and process the results.

The next step was to use the module to build a simple API that would let us quickly grab a set of headlines for a particular date and place. Django and Piston made this easy. To see headlines from Victoria on 1 January 1901, for example:

http://wraggelabs.com/api/newspapers/1901-01-01/nsw/

That was pretty cool and it started me thinking about what else I might do with the data. At first I was planning some sort of browser, like my Population Browser, but that seemed a bit boring. So I decided to create a simple game that grabbed a random headline and asked you to try and guess the date. After further refinement I decided to impose a limit of 10 guesses, with ‘higher’ or ‘lower’ prompts to get you moving in the right direction. Yes, basically it was a rip-off of The Price is Right – but an interesting, ironic and historically engaged rip-off…

This required me to make a change to the API and Python module so that I could retrieve a random headline. Basically it just meant generating a query based on random values for the day, month, year and state. For the interface I once again delved into JQuery’s box of tricks. With all the kerfuffle about ChatRoulette in the media, the name seemed obvious – Wragge’s Headline Roulette was born.

Test your historical nous with Headline Roulette!

It’s a very simple little app, but a number of people have said how much fun it is. The bad news is that imminent changes to the NLA newspapers site are probably going to break it (at least in its current form). So enjoy it while you can. When the NLA makes an API available I might work on something a little more sophisticated.

Of course, the broader point is that there are a whole range of cultural materials out there waiting to be remixed and re-used in various forms. Get hacking…

For a project that I’m working on at the National Museum of Australia, I’ve started collecting various sources of date-identified data. Most recently I had a go at extracting historical population data from the Australian Bureau of Statistics.

The data can all be downloaded as .xls files, but they’re not simple, flat spreadsheets – they’re data cubes. As the name suggests, data cubes are organised along a number of dimensions. In the case of the population data it’s year, state and gender.

This means that you can’t just export the data to CSV and suck it into your database – first you’ve got to flatten the cube. No doubt there are other ways to do this, but I just wrote a simple python script. It uses xlrd to read from the spreadsheet, does a bit or reorganisation, then writes the output to a CSV file. The code, for what it’s worth, is available at Bitbucket.

Once I had the CSV file I just imported it into MySQL and used Django and Piston to build a basic API. So if you want to know the population of NSW in 1856, you just go to:

http://wraggelabs.com/api/json/population/nsw/1856/

The number of infant deaths in Tasmania in 1932:

http://wraggelabs.com/api/json/infantdeaths/tas/1932/

The number of female births in Australia in 1959:

http://wraggelabs.com/api/json/births/australia/females/1959/

I’m sure you get the picture. You can change the ‘json’ to ‘xml’ if you’d like another flavour of data.

The API in action - a simple population browser

With an API delivering JSON you can start playing around with all sorts of fun AJAX-y stuff. To demonstrate I built a simple population browser using JQuery. Just drag the slider!