Letter from CKR Kilby (‘Parkwood’, Hall FCT)

One of my most exciting archival discoveries was a cache of letters written by farmers from across NSW in 1938. Seeking to marshal support for the long-range weather forecaster Inigo Jones, The Land newspaper asked it's readers to send their opinions of the 'weather prophet'. And they did. One hundred and two letters were received, and duly forwarded to the Minister for the Interior. These 102 letters now reside in the National Archives of Australia.

I tell the full story of the newspaper’s campaign in Inigo Jones: The Weather Prophet. but I’ve always wanted to do something more with the letters. Inspiration finally arrived this year when a conference on rural media was organised to mark the The Land‘s centenary. I decided to create a little exhibition using the letters and, in doing so, provide myself with a platform for further research.

Step one: assembling the letters with Zotero

I’d already asked for the file containing the letters to be digitised, so images of all of the letters were available through RecordSearch. Zotero ships with a RecordSearch translator that I wrote some years ago, so 102 clicks later, I’d captured both the metadata and all of the images into my own database.

The Zotero translator saves individual pages with a generic title, so I then had to do some manual editing. I also had partial transcripts of the letters which I’d created during my original research. I simply cut and pasted the transcripts into a note field in Zotero and tidied them up. Pretty soon my collection was complete.

Step two: creating an exhibition with Omeka

The Weather Prophets

I actually started building the exhibition during an Omeka Bootcamp session I ran at THATCamp Melbourne. Starting with nothing but a LAMP server and my Zotero library, I had the basics of site up within the hour.

It was simply a matter of installing Omeka, adding the Zotero import plugin, creating an API key in my Zotero profile, and then pointing the Zotero import plugin to my collection. (The API key is necessary if you want to download files, such as my images.)  With a press of a button the internets started chugging away and pretty soon all the letters were in Omeka. Hey presto — instant exhibition!

With the framework built I could start to play a bit. First I installed Omeka’s geolocation plugin and started mapping where the letters had come from. Google’s geocoder did a pretty good job of finding many of the small country towns, but I also made use of Geoscience Australia’s gazetteer. In the end, I managed to geolocate all but one letter. Hey presto — a map!

I thought the pre-installed ‘Summer’ theme suited my exhibition, but I did make a few minor tweaks here and there. This included adding some RDFa into the list of references (which were also imported from Zotero).

Step three: going deeper with Voyeur Tools

Word cloud of the letters created by Voyeur Tools

The transcripts of the letters were all available on the site, but I thought it would be interesting to analyse their content a bit more systematically. To do this, I exported the collection from Zotero and wrote a few lines of Python code to save each transcript in a separate text file, named after the letter’s author. I could then zip up the transcripts and feed them to Voyeur Tools.

Most of the letters are pretty short, so there’s a limited amount to be gleaned from them in isolation. Voyeur Tools did, however, make it easy for me to explore further the prevalence of the phrase ‘on the right track‘.

What will be more interesting will be to compare the letters to other bodies of text about Inigo Jones, weather forecasting and rural life. I’m hoping to start mining some of this sort of material from the Trove newspapers database.

Next steps?

Using Zotero , Omeka and Voyeur Tools it was quick and easy to build my own little research hub. It’s still a work-in-progress of course — that’s the point! Now I can continue to assemble and analyse my sources, while giving my work a public face. Magic!

Here’s the slides from the talk I gave recently in the Scholars’ Lab at the University of Virginia. Thanks to everyone who came and to the Scholars’ Lab for their hospitality.

A podcast version is now available.

Some of you may have noticed that my Hacking a research project post featured a file from the National Archives of Australia embedded as a Cooliris widget. Huh? To jog your memory, here it is again:

These certificates allowed non-white Australians travelling overseas to re-enter the country. NAA: ST84/1, 1906/21-30

No, it’s not just an image, it’s a little 3D wall. You can pan and zoom to your heart’s content. You can enlarge an image, view fullscreen — you can even share an image via Twitter. Fun for all the family!

Regular viewers will recall my previous encounters with CoolIris — Archives in 3D and CoolIris enabled scrapbook — but these relied on having the CoolIris plugin installed. The embeddable Flash version wouldn’t work when the images were coming from the NAA because it upset Flash’s cross-domain settings.

So how did I get it to work? For various other projects I’ve been playing with simple image proxies using Python and Django, so I just applied the same principles. The image proxy makes it seem as if the images are coming from a local source, thus keeping Flash happy. Hurrah!

I’ve added a few little tweaks, so you can now view any digitised file in the National Archives of Australia in a CoolIris wall. Just go the the file browser page and enter a barcode. Even better you can install a bookmarklet. Just drag this link to your bookmarks bar (or save as a favourite) — View on wall. Then go to an item page in RecordSearch and click on the bookmarklet for 3D magic.

If you want to share a link to a file displayed in the 3D file browser, just use a url of the form:

http://wraggelabs.com/recordsearch/wall/[barcode]

— where [barcode] is fairly obviously the barcode of the file you want to view. For example:

• http://wraggelabs.com/recordsearch/wall/3445411/

If you want to embed one of the mini-walls in your blog post it’s easy. Just go to the CoolIris Express site and create your own wall. When it asks you for content source, click on ‘Media RSS’ and then in the ‘Feed URL’ box put:

http://wraggelabs.com/recordsearch/rss/[barcode]

– where [barcode] is… well, you know…

I think this a pretty interesting way to view, browse and navigate digitised files. Using Flash, rather than a browser plugin makes it more accessible, but I’d still rather have something based on open software and standards. I think it won’t be too long before we see something similar using Canvas and Javascript. That’ll be really exciting.

I was doing some research using the National Archives of Australia’s RecordSearch database the other day and became frustrated that there is no way of seeing how many pages are in a digitised file without clicking on the ‘Display digital copy’ link. So I fixed it.

As a userscript it’s hardly worthy of a blog post. All it does it find out how many pages are in the file and insert the number in the link text. It’s very simple. But I think it’s also a useful illustration of the changing balance of power between archives and their users.

William E Landis argued that archivists were ‘guilty as a profession of fetishising the outputs of our descriptive systems’. The design of finding aids have often been determined not by the needs of users but by a desire to faithfully represent the underlying archival architecture. But now users don’t have to just take what they’re given.

Technologies such as Greasemonkey are useful for sketching out alternatives. For organisations with IT systems that inhibit experimentation, Greasemonkey (or Mozilla’s Jetpack) provides a way of playing with interfaces without touching any of the underlying code. My rewrite of the way RecordSearch displays digitised files is an example of this.

But no one interface is ever going to meet the needs of all archive users. Fortunately, there are a growing number of ways in which archives can work in partnership with their users to help them create the interfaces they want and need.

Archives are starting to expose their data directly using APIs and linked open data. This gives users the power to create whole new applications. But I still think there’ll be a place for the little tweak – a simple hack that meets some small but specific need. I can imagine communities of interest building and sharing a range of tools, hacks, applications and interfaces specifically tailored to their research habits.

So if you don’t like it, fix it.

It’s great to see that the National Archives of Australia has released a large swag of data through the new data.australia.gov.au site. In the Commonwealth Agencies zip file you can find xml dumps of all the publicly accessible agency and series data in RecordSearch, as well as item data for series A1. This is the same data that Mitchell Whitelaw visualised so brilliantly in his Visible Archive project. There’s also item data and images from series A3560 – the Mildenhall photographs of early Canberra.

What’s even more exciting is that people are already using this data. At the recent GovHack event in Canberra the What The Federal Government Does team worked on visualising the activities of government by using functions data pulled from the agencies file. Another group has generated a really nice tag cloud and photo gallery from the Mildenhall data. With further GovHack sessions to follow and the MashupAustralia contest open until 13 November, let’s hope for some more inspired archives hacking.

Seeing RecordSearch data out in the world like this reminded me of a little project I started a while back and then set aside. It was a simple PHP script that scraped data from RecordSearch and spat it out either as XML or JSON. Mitchell used a version of this script in his A1 Explorer in order to find out the number of pages in each digitised file.

I’ve now expanded and improved the script so that it provides data on items, series, agencies and persons. The output includes all the basic fields as well as links between entities – such as related series, controlling agencies etc. As an added bonus you also get some useful totals (where they’re available): items include the number of pages, series include the number of items described on RecordSearch, and agencies include the number of series recorded. I’ve also fiddled with mod_rewrite to provide a more rest-ful interface.

For XML output use the url http://discontents.com.au/shed/rs/xml/ followed by the appropriate identifier – a barcode for an item, a CA number for an agency, a CP number for a person or a series number.

Some examples:

• Series A1 – http://discontents.com.au/shed/rs/xml/a1
• Item B2455, WRAGGE C L E – http://discontents.com.au/shed/rs/xml/3445411
• CSIR Head Office – http://discontents.com.au/shed/rs/xml/CA+486
• Alfred Deakin – http://discontents.com.au/shed/rs/xml/CP+9

As you might have guessed, to get JSON output you just substitute ‘json’ for ‘xml’ in the url.

Being dependent on screen scraping, it’s inherently a bit fragile, but I’m hoping it might be of some use. My intention was to use it to start exploring some new ways of using and interacting with the data. The code itself is available at BitBucket. It’s not very elegant, but I don’t want to spend much time cleaning it up at the moment. If it seems like it might be useful, I’ll probably rewrite the whole thing in python and publish it through Google’s AppEngine.

Mapping our Anzacs, in case you don’t know, provides a Google map interface to the 375,000+ WWI service records held by the National Archives of Australia. Amongst other other things, you can add scrapbook posts to individual entries and create tributes. It’s meant to encourage exploration, so go on… explore!

If you’ll do, you’ll notice that there are direct links into the National Archives’ database RecordSearch. However, there are currently no links going to other way. Why does this matter? Well perhaps you’d like to use NameSearch to find an individual record, but then add a scrapbook post in Mapping our Anzacs. Up until now you had to find them all over again. But not any more…

Introducing our new range of ‘View in Mapping our Anzacs’ buttons:

• For the discerning Firefox devotee we have a Greasemonkey userscript which adds a button to the RecordSearch item details page.
• For fashion-challenged IE user we have a bookmarklet. Just right click on this link – View in Mapping our Anzacs – and save it as a favourite in your ‘Links’ folder (you may need to enable the ‘Links’ toolbar first by checking Tools > Toolbars > Links.)

Yes, it’s true… you could use the Bookmarklet with Firefox (just drag it to your bookmarks toolbar), but Greasemonkey is so much more chic.

Once you’re fully button-enabled just head into RecordSearch, find an item in series B2455 (the WWI service records) and click! Hurrah! You will be instantly transported to Mapping our Anzacs.

You can test out your new button by heading here:

• B2455, WRAGGE C L E

All dressed up – RecordSearch has a new look

The new version of my Greasemonkey userscript, RecordSearch Image Tools, gives RecordSearch’s digital image pages a rather new look. My previous version had done away with the tired ol ‘lemon-chiffon’ background colour, but I decided it was time to get a bit more adventurous, so I blitzed the old design and rebuilt the page from the beginning.

As you can see from the screenshot, I’ve tried to give the images as much as the screen as possible. I’ve also created a consistent set of navigation buttons, and improved the functionality in various ways.

Archives in 3D – CEDTs from NAA: ST84/1, 1906/21-30

But the most exciting thing is that I’ve worked out how to feed the images to the fabulous CoolIris 3D wall. My previous version used the javascript version of CoolIris, which displayed the images as a flat (but still very nice) slideshow. But now, if you have the CoolIris plugin installed you can zoom, pan, fly through the file, dipping in and out as you so desire. It’s a new way of looking at archives.

You can zoom in and out, even see a complete file on a single screen – NAA: B2455, WRAGGE C L E

To try for yourself you need to have Firefox with the Cooliris plugin installed. Then you need to get the Greasemonkey extension and, finally, install my userscript. Then just dive into RecordSearch, find a digitised file and enjoy!

File links:

• NAA: ST84/1, 1906/21-30
• NAA: B2455, WRAGGE C L E

BREAKING NEWS (2.00pm, Monday, 8 December): RecordSearch seems to be back on the old subdomain, so now the userscript fix is not working! To be safe, I’ve updated the userscript again so that it will work on both the old and new subdomains. I’ll do the same with the Zotero translator, though for the time being it should be working. If you updated the userscript in the last few hours, you’d better do it again – sorry…

Just a quick update in regard to the tools I’ve developed for use with the National Archives of Australia’s RecordSearch database — the RecordSearch translator for Zotero, and the RecordSearch Image Tools userscript.

The subdomain under which RecordSearch operates was changed over the weekend to recordsearch.naa.gov.au. As a result of this change neither the Zotero translator nor the userscript will work.

I’ve updated the the RecordSearch Image Tools userscript to fix the problem. Just go to:

http://userscripts.org/scripts/show/33485

and click on the ‘Install’ button. This will overwrite the old version with the new one. As a bonus extra, if you haven’t installed or updated the script recently, you’ll notice that I’ve improved the way it prints out files. Details are on the script page.

I will upload a new version of the Zotero translator to the developers asap. Once they add this to the repository your Zotero will automatically install the fix. However, I’m not sure how long it will take for it to be added. I’ll post an update once it’s been fixed.