One of the things that’s keeping me busy at the moment is THATCamp Canberra. Yes, I got sick of missing out on all the THATCamp fun happening elsewhere and decided we should have our own.

THATCamp Canberra is a user-generated unconference on the digital humanities. It’ll be held at the University of Canberra on 28–29 August. We’re getting a great mix of applications and I’m really looking forward to learning about what’s going on around Australia.

Applications close on 23 July, so get yours in soon!

Some of you may have noticed that my Hacking a research project post featured a file from the National Archives of Australia embedded as a Cooliris widget. Huh? To jog your memory, here it is again:

These certificates allowed non-white Australians travelling overseas to re-enter the country. NAA: ST84/1, 1906/21-30

No, it’s not just an image, it’s a little 3D wall. You can pan and zoom to your heart’s content. You can enlarge an image, view fullscreen — you can even share an image via Twitter. Fun for all the family!

Regular viewers will recall my previous encounters with CoolIris — Archives in 3D and CoolIris enabled scrapbook — but these relied on having the CoolIris plugin installed. The embeddable Flash version wouldn’t work when the images were coming from the NAA because it upset Flash’s cross-domain settings.

So how did I get it to work? For various other projects I’ve been playing with simple image proxies using Python and Django, so I just applied the same principles. The image proxy makes it seem as if the images are coming from a local source, thus keeping Flash happy. Hurrah!

I’ve added a few little tweaks, so you can now view any digitised file in the National Archives of Australia in a CoolIris wall. Just go the the file browser page and enter a barcode. Even better you can install a bookmarklet. Just drag this link to your bookmarks bar (or save as a favourite) — View on wall. Then go to an item page in RecordSearch and click on the bookmarklet for 3D magic.

If you want to share a link to a file displayed in the 3D file browser, just use a url of the form:

http://wraggelabs.com/recordsearch/wall/[barcode]

— where [barcode] is fairly obviously the barcode of the file you want to view. For example:

• http://wraggelabs.com/recordsearch/wall/3445411/

If you want to embed one of the mini-walls in your blog post it’s easy. Just go to the CoolIris Express site and create your own wall. When it asks you for content source, click on ‘Media RSS’ and then in the ‘Feed URL’ box put:

http://wraggelabs.com/recordsearch/rss/[barcode]

– where [barcode] is… well, you know…

I think this a pretty interesting way to view, browse and navigate digitised files. Using Flash, rather than a browser plugin makes it more accessible, but I’d still rather have something based on open software and standards. I think it won’t be too long before we see something similar using Canvas and Javascript. That’ll be really exciting.

Amongst the holdings of the National Archives of Australia are some of the most visually arresting documents you’ll see — thousands and thousands of forms from the early decades of the twentieth century, each with a portrait photograph and palm print, each documenting the movements of a non-white resident. Along with many other certificates, regulations, correspondence and case files, these forms are part of the massive bureaucratic legacy of the White Australia Policy.

These certificates allowed non-white Australians travelling overseas to re-enter the country. NAA: ST84/1, 1906/21-30

But these are more than just interesting looking pieces of paper, they are snapshots of people’s lives. The forms capture data about an individual’s place of birth, physical characteristics and more. Over time a person might have submitted several of these forms, so by bringing them together we could trace their history, we could map their journeys — we could even watch them age.

The system which sought to render non-whites invisible has captured and preserved the outlines of their lives. By extracting and linking this data we could build a picture of another Australia, an Australia in which non-white residents lived, loved, struggled and succeeded, despite the impositions of a repressive regime.

I talked about these records at the AAHC conference last year, inspired in part by Tim Hitchcock’s chapter in the Virtual Representation of the Past. Tim Hitchcock argues that technology can allow us to restructure archives, looking beyond institutional hierarchies to the lives of individuals contained within:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

I don’t know, but I’d like to find out.

During my AAHC talk, Dave Lester suggested that the extraction of data from these forms might make a good crowdsourcing project. It’s a great idea. As you can see, the data is generally well-structured and legible, it should be possible to construct a simple series of forms that would allow volunteers to transcribe the data. The next stage would be to try and match identities across forms. That’s more complicated, but projects such as Tim Hitchcock’s London Lives show how users can construct identities by connecting a range of historical documents.

Then there are connections to resources outside of the archives — photographs, local histories, newspapers, genealogies, cemetery registers and more. By keeping our system open and extensible, and by working with others to help them expose their information in standard ways, it should be possible to develop the framework for an evolving mesh of biographical data.

So, how do we get started? This is the point when you usually have to start thinking about money — how can I fund this? In Australia that generally means a journey into the arcane world of the Australian Research Council. The ARC suffers from all the problems of a peer-reviewed system, but added to this is a rather antiquated notion of what research is.

In the rules covering each of the main schemes it’s clearly stated that the ‘compilation of data’ and the ‘development of research aids or tools’ are not supported. I spend part of my life working for the Australian National Data Service, an organisation that seeks to highlight how the sharing and reuse of data can open up new research possibilities. The ARC, however, seems to think that data has little value beyond its original research context.

Of course you can still mount a case for such activities. Applicants for a ‘Discovery’ grant can argue that data creation is integral to their project and provide details of the ’specific research questions to be addressed’. But what if you don’t yet know what the questions are? Part of the point of a project such as this is to try and find out what questions we are able to ask. Until we start to compile, link and explore the data, the ’specific research questions’ will be little more than convenient fictions, dreamt up to satisfy the prodding of peer reviewers.

Tom Scheinfeldt wrote a fantastic blog post recently, responding to concerns about the failure of many digital humanities projects to make arguments or answer questions. Drawing examples from the history of science, Tom argues:

we need to make room for both kinds of digital humanities, the kind that seeks to make arguments and answer questions now and the kind that builds tools and resources with questions in mind, but only in the back of its mind and only for later. We need time to experiment and even… time to play.

The ARC does not fund play.

You might imagine that the ARC’s infrastructure funding scheme would offer more hope for a project such as this. And yes, there are many worthy projects involving databases and online tools that have been supported in this way (and I have benefited from some of them!). But it seems that in the minds of research funders infrastructure is always BIG. Grants start at $150,000, and applications are expected to involve multiple institutional partners. Projects have to be scaled up to fit the ARC’s definition of infrastructure, often resulting in complex, lumbering, long-term projects whose products are out of date by the time of their release.

There is no room in our current infrastructure models for agile, innovative, user-focused digital toolmakers seeking small amounts to experiment with apps, prototypes, datasets or visualisations. I often look with envy upon the US National Endowment for the Humanities Digital Humanities Start-Up Grants.

In any case, neither I nor my partner in this endeavour, Kate Bagnall (@baibi), are currently in academic positions, so our chances of gaining any sort of research funding are next to none. We have the expertise — Kate has spent many years researching Australian-Chinese families and knows the records back-to-front, while I just can’t help playing with biographical data — but is that enough? How can you mount an ongoing research project without institutional support, research funding and the various badges and signifiers of academic authority?

I don’t know that either, but I have some ideas.

Mrs Ah Yin Pak Chong. NAA: ST84/1, 1907/321-330

I didn’t manage to get a contribution together for Dan Cohen and Tom Scheinfeldt’s crowdsourced-in-a-week book, Hacking the Academy, but watching the process from afar I did begin to wonder about how we might hack the way we build and run major research projects. This is what I have in mind:

• To strip down the large, lumbering beasts and design projects that are modular and opportunistic — able to grow quickly when resources allow, to bolt on related projects, to absorb existing tools.
• To follow the data freely across technological and institutional boundaries, developing open networks that invite participation and use.
• To develop a floating pool of collaborators, both inside and outside of academia, who are able to come and go, contributing whatever and whenever they can.
• To make everything public, accessible and standards-compliant, so that even if the project stalls it could be picked up and developed by someone else.

Most of all I just want to be able to do it. I don’t want to second-guess the ARC. I don’t want to spend months negotiating with potential partners or begging for an institutional home. I want to build, experiment and play. I want to make a start.

So that’s what we’re going to do.

We have a topic, plenty of raw materials, some basic principles and the beginnings of a plan. We even have a name — Invisible Australians: Living under the White Australia Policy.

As the project develops, I’ll be blogging here about some of the technical stuff, while Kate will be exploring the content over at the tiger’s mouth. I hope to have a prototype of the transcription tool ready to demo at THATCamp Canberra, while Kate is already at work putting together guides on using the records and developing an Omeka site that follows a number of Chinese-Australian families through the archives.

Can we hack together a major research project? Let’s find out.

The trained guinea pigs in the Wragge Labs bunker have been churning out all sorts of stuff in the last few months, and I’m way behind in my attempts to document their activities. So this is a bit of a catch-up post to try and commit a few pertinent details to the collective memory bank before they are lost forever in the sleep-deprived fog of day-to-day existence.

Identity upgrades

There have been a number of major improvements to Wragge’s Identity Browser. Regular viewers will recall that the Identity Browser is built on top of the People Australia SRU interface. You might not realise, however, that People Australia contains details of many organisations as well as people. We can only be thankful that it wasn’t called Entity Australia.

The first version of my Identity Browser only searched for people, but now all your corporate-entity-identification needs are also met, with only a few minor changes to the interface so-beloved by numerous generations of identity seekers. To be specific, through the wonders of drop-down technology you can choose whether you want to search for a person or an organisation. Or not. You can also just ignore that and search for everything and get back sensible results anyway. It’s your choice. Or not.

Gaze in awe at the power of my dropdown

Ah pattern matching… there are few phrases so redolent of warm summer days, hidden pleasures, and the subtle delights of wildcard characters. The People Australia SRU interface was sadly lacking in the pattern matching department, but this has now been rectified. So now you mix your stems and asterixes with wild abandon. Searching for ‘Curtin, J*’ will now retrieve all those Curtins whose names begin with ‘J’. Amazing isn’t it?

Astonishing too is the fact that the accompanying ‘Identify me!’ bookmarklet continues to function with nary a murmur of protest. There is, however, a little bit of cleverness built-in to enhance your bookmarklet experience. If the text that you highlight has a comma in it, the Identity Browser will conclude that you’re feeding it the name of a person – ie Surname, Firstname – and will treat the Firstname as a stem. So if you highlight ‘Whitlam, G’ and click on the bookmarklet, the Identity Browser will be kick-started into life, searching for everything that matches surname equals ‘Whitlam’ and firstname is like ‘G*’. If there’s no comma – ie firstname secondname – then it heads off to look for either a person whose surname equals ’secondname’ and whose firstname is like ‘firstname*’, or an organisation whose name includes both ‘firstname’ and ’secondname’. Got all that?

Basically the idea was to try and provide some sensible defaults so you really don’t have to think about it too much.

I have it in my head to prepare a long and rapturous homage to the wonders of machine tags. With their sly semantic ways and easy-going nature, they offer some exciting possibilities not just for user-generated content, but user-generated meanings and user-generated relationships. But for the full, ripe pleasure of that post you will have to wait another day, for now I shall simply say that as well as RDFa, the Identity Browser provides automagically-generated machine tags.

Where might you use them? Flickr’s a good place to start. Try identifying the subjects and creators of Flickr photos. At the NSW Reference and Information Services Group Seminar the other day I challenged those in attendance to go forth and machine tag. Already more than 100 machine tags have been added to Flickr using my Identity Browser. Expect to hear more about the Great Flickr Machine Tag Challenge soon…

One more thing… try adding ‘.rdf’ on to the end of an identity record – eg http://wraggelabs.com/identities/person/612109.rdf. Just an experiment at the moment…

More machine tag love

One night on Twitter, @lifeasdaddy pointed out that someone had started using fragments of urls from the NLA newspapers site as tags in the Powerhouse Museum’s collection database. In the conversation that ensued with @sebchan and others, I suggested that the PHM could encourage this sort of rich tagging by supporting machine tags, with all their wonderful juicy semantic goodness The guinea pigs got excited as well, and before I knew it, they’d constructed a little Semweb Helper app.

The Semweb Helper comes with its very own custom-tailored bookmarklet. If you find an article on the NLA newspapers site that you’d like to point to, just click on the bookmarklet and marvel as a range of useful machine tags are automagically generated. Then you just pick the appropriate tag, copy and paste et voila – instant semantic gratification.

Try out the Semweb Helper

It’s a very simple little app, and really just a demonstration of how semantic web technologies might be made available to the masses. It was also the first time the guinea pigs had been allowed to play with the Google Apps Engine.

Who am I?

This short catch-up post has become something quite long and rambling. Did I mention that I’m sleep-deprived? Anyway, a recent addition to the Wragge Labs range of lifestyle accessories is ‘Who am I?’ – a simple little game that is something like a cross between hangman and Wheel of Fortune. Choosing a person at random from People Australia and the Australian Dictionary of Biography, ‘Who am I?’ tests your powers of logic, stamina and historical guesstimation.

Your challenge is to figure out the surname of the mystery historical personage. To help you there are a series of clues, such as their birthplace and known associates. With each guess you also see a little bit more of their portrait. But beware! For ten wrong guesses are all that are permitted to any so brave as to enter upon this quest. Not eleven or twelve, but ten and ten only. To ignore this limit is to invite ridicule and disdain – do so at your peril.

Play Who am I?

‘Who am I’ builds upon some work I’ve been doing for the National Museum of Australia – looking at ways of mashing together various types of date-identified data. As part of that project I’ve built a series of APIs and have scraped, pummelled and munged data from a variety of sources.

What’s the point? I wonder this myself sometimes, particularly after I fling such things off into the aethernet and hear naught but a rare retweet. I am, after all, only in it for the glory, oh and the money of course. (Hmmm, I must look again at that business plan.) The point is twofold: first to highlight possibilities for the re-use and remixing of cultural data; second, to play with game-based models for discovery and exploration of cultural resources; and… err… thirdly just to try building something a little different.

Of course, if you like ‘Who am I?’ you will probably also want to try Headline Roulette…

Headline Roulette Reprieve

At the end of our last instalment, the future of Headline Roulette seemed in dire peril. Changes to the National Library of Australia web site threatened its very existence. Did it have a future? Could it survive? And did anybody care?

As we pick up the story oblivion looms. The feared changes are confirmed, but just as all seems lost… is it? Could it be? Yes, an advanced search facility is added to the newspapers site within Trove. Sensing this may be their only opportunity, the guinea pigs leap into action, building a new screen-scraper, saving Headline Roulette from doom, and setting the world upon the path to a safer, happier future.

In short, Headline Roulette will live on… so enjoy.

Handing out some presents

My head is easily turned by flattery and praise. Yes, I really am so shallow and so vain. But this means that if people say nice things to me, I’m inclined to give them presents.

As well as doing exciting things in the web 2.0 realm for the PROV, @asaletourneau leaves nice comments on this blog. So he earned himself a present. It’s not much, but I built a userscript that displays photos from the PROV site in a neat little slideshow (it’s the non-3D javascript version of CoolIris). Install Greasemonkey, get the userscript and try it out (just do a search, then click on the ‘Browse as slideshow’ button’).

PROV transport photos in a pretty slideshow

The State Library of NSW, or more specifically @ellenforsyth, also earned my favour by inviting me to rave on about Linked Data at the afore-mentioned NSW RISG seminar. As a result, I added support for the SLNSW photo collections to my Flickr Context Harvester userscript. Well… it’s the thought that counts, right? Once again – install Greasemonkey, get the userscript and then try it out.

The Flickr Context Harvester in action

And coming up…

Stay tuned for more on the Great Flickr Machine Tag Challenge, screencasts demonstrating my Identity Browser, some playing with relationships, and much much more. But right now the squirming baby on my lap needs a nappy change…

Did I mention that I’m sleep deprived?

I’ve been doing a fair bit of coding in recent weeks and I thought I’d better write a few details down before I forget about them.

As previously noted, I’ve been gathering together various historical data sets for a project at the National Museum of Australia. One resource that I was keen on including was the fantastic Australian Newspapers project at the National Library of Australia. What I had in mind was being able to give a sense of context to any historical event by calling up the headlines for that particular time.

Unfortunately there’s no API for the newspapers project (or Trove in general), though apparently it’s in the works. So I had to reverse engineer the advanced search page to work out the various query options, and then build a screen scraper to harvest the results. I played around with the search options a bit to fine tune the results, finally deciding to limit them to ‘news’ articles with more than 1000 words. Annoyingly, only 10 results are returned at a time.

I had hoped to parse the results as xml, but a rogue <br> tag broke the XHTML, so I fell back on Beautiful Soup – a Python module that makes screen scraping considerably easier by tidying up HTML structures. After than it was pretty straightforward. Soon I had my own Python module to query the newspapers database and process the results.

The next step was to use the module to build a simple API that would let us quickly grab a set of headlines for a particular date and place. Django and Piston made this easy. To see headlines from Victoria on 1 January 1901, for example:

http://wraggelabs.com/api/newspapers/1901-01-01/nsw/

That was pretty cool and it started me thinking about what else I might do with the data. At first I was planning some sort of browser, like my Population Browser, but that seemed a bit boring. So I decided to create a simple game that grabbed a random headline and asked you to try and guess the date. After further refinement I decided to impose a limit of 10 guesses, with ‘higher’ or ‘lower’ prompts to get you moving in the right direction. Yes, basically it was a rip-off of The Price is Right – but an interesting, ironic and historically engaged rip-off…

This required me to make a change to the API and Python module so that I could retrieve a random headline. Basically it just meant generating a query based on random values for the day, month, year and state. For the interface I once again delved into JQuery’s box of tricks. With all the kerfuffle about ChatRoulette in the media, the name seemed obvious – Wragge’s Headline Roulette was born.

Test your historical nous with Headline Roulette!

It’s a very simple little app, but a number of people have said how much fun it is. The bad news is that imminent changes to the NLA newspapers site are probably going to break it (at least in its current form). So enjoy it while you can. When the NLA makes an API available I might work on something a little more sophisticated.

Of course, the broader point is that there are a whole range of cultural materials out there waiting to be remixed and re-used in various forms. Get hacking…

For a project that I’m working on at the National Museum of Australia, I’ve started collecting various sources of date-identified data. Most recently I had a go at extracting historical population data from the Australian Bureau of Statistics.

The data can all be downloaded as .xls files, but they’re not simple, flat spreadsheets – they’re data cubes. As the name suggests, data cubes are organised along a number of dimensions. In the case of the population data it’s year, state and gender.

This means that you can’t just export the data to CSV and suck it into your database – first you’ve got to flatten the cube. No doubt there are other ways to do this, but I just wrote a simple python script. It uses xlrd to read from the spreadsheet, does a bit or reorganisation, then writes the output to a CSV file. The code, for what it’s worth, is available at Bitbucket.

Once I had the CSV file I just imported it into MySQL and used Django and Piston to build a basic API. So if you want to know the population of NSW in 1856, you just go to:

http://wraggelabs.com/api/json/population/nsw/1856/

The number of infant deaths in Tasmania in 1932:

http://wraggelabs.com/api/json/infantdeaths/tas/1932/

The number of female births in Australia in 1959:

http://wraggelabs.com/api/json/births/australia/females/1959/

I’m sure you get the picture. You can change the ‘json’ to ‘xml’ if you’d like another flavour of data.

The API in action - a simple population browser

With an API delivering JSON you can start playing around with all sorts of fun AJAX-y stuff. To demonstrate I built a simple population browser using JQuery. Just drag the slider!

About a month ago I posted a copy of my report Emerging technologies for the provision of access to archives on Scribd. It’s already edging up towards a thousand reads, so I thought it was time I put a link in from here.

The basic message is we need to experiment and find the spaces both within and between our institutions to foster such experimentation. Is that asking too much? Anyway… read, enjoy, use!

Let me be clear. I am not Tim Sherratt the sound engineer. Nor, indeed, am I Timothy Sherratt, author of Saints as Citizens: A Guide to Public Responsibilities for Christians. We are three different people, spread across three continents, locked in a deadly battle for global supremacy via Google search rankings. There can be only one…

Of course you probably knew I wasn’t a British sound engineer or an American politics professor. There are plenty of contextual clues within this website, even on this page, to indicate that my interests lie elsewhere. But while we humans are pretty good at picking up such clues, it’s much harder for computers. When Google comes to index my site, how does it know I’m not a sound engineer who likes to dabble in history? Indeed, how does Google, or any computer know that the words ‘Tim Sherratt’ are actually a person’s name? These are questions of both identity and semantics.

Librarians have been dealing with questions of identity for many, many years developing detailed name authority records. Such records allow name variations to be cross-referenced and individuals to be uniquely identified. For example I have a control number of ‘n 2005043272′ in the Library of Congress authorities database, while Timothy R Sherratt, the politics professor has been assigned ‘n 94106739′.

The National Library of Australia has developed its own name authority file. However, the NLA has realised that reliable identity data has a much broader application that simply cataloguing, and is using its name authority data as the foundation of an exciting new resource – People Australia. People Australia will mesh its own records with biographical data from a variety of outside sources, creating a rich collection of interlinked identities. Already entries from the Australian Dictionary of Biography have been ingested.

So now, thanks to People Australia, if I ever get confused about who I am I just have to remember one little url – my very own persistent identifier – http://nla.gov.au/nla.party-479364. I’m going to get a t-shirt made up.

But that doesn’t help our new machine overlords very much. How can a computer tell that the words ‘Tim Sherratt’ describe a person and that more information about that person can be found at http://nla.gov.au/nla.party-479364? This is the sort of problem that the semantic web hopes to solve. The semantic web aims to expose the structures that are buried in our documents and databases, to make explicit the contextual clues that humans pick up, but computers ignore. As the slogan goes, it represents a change from a ‘web of documents to a web of data’.

The semantic web uses a variety of tools and standards to encode information in a form that means something to computers. FOAF (Friend of a Friend) is, for example, a machine-readable ontology that describes people and their relationships. A computer visiting this page can in fact find out a fair bit about me, including my NLA persistent identifier, because there is a link to a small XML file in which my details are expressed using FOAF.

But if this seems a little daunting, the semantic web offers another technology which is really just as easy as marking up a page in HTML – it’s called RDFa. This link – Tim Sherratt – is more than it seems. Here is what a computer sees:

<a typeof="foaf:Person" property="foaf:name" content="Sherratt, Tim" rel="foaf:isPrimaryTopicOf" href="http://nla.gov.au/nla.party-479364">Tim Sherratt</a>

This says that Tim Sherratt is a person whose name has the standard form ‘Sherratt, Tim’ and who is the primary topic of the page to be found at http://nla.gov.au/nla.party-479364. There’s a fair bit of semantic goodness in that one little link. If the NLA page also expressed its data in a machine-readable form, this link could send search engines and browsers into a whole new world of associations and inferences.

But I suppose you’re thinking that the code still looks a bit complicated. Well never fear, this long post is really just an introduction to a new project I’ve been working on – something that will help you generate markup like this with just a couple of clicks.

Introducing Wragge’s identity browser

I’ve been interested in publishing biographical data way back from the early days of Bright Sparcs and, sad as it may seem, I find the possibilities of People Australia pretty exciting. However, I don’t think we should expect the NLA to do all the work. People Australia provides a framework that we can all use to enrich our own documents, databases, finding aids, and applications.

You can easily access People Australia data through Trove. But to get a better idea of what’s in the database, I’d suggest you spend some time playing with its SRU interface. Using this you can query the database directly, retrieving results in XML – ready for your own application to suck up and use.

To make this even easier, I’ve written a People Australia client library in Python. This enables you to quickly extract and use identity information. Using it, your own web application can talk to People Australia directly. I won’t go into the details here – the code is farily heavily commented – but I welcome any feedback, suggestions or contributions. Copy it, change it, use it!

To try out my library and to provide a tool that might be of use to the average punter I’ve also built:

<TA-DA>Wragge’s identity browser!</TA-DA>

It’s pretty simple. Search for a surname, pick a name from the result list, and view their identity details. For example, here’s Clement Wragge’s details.

But there are a couple of extra features that I am rather smugly pleased with. First of all, there’s an 'Identify me!' bookmarklet. Just drag the link to your browser’s bookmarks or favourites toolbar (see below for some further notes).

Once you have the bookmarklet installed all you have to do to find the identity record for someone is to highlight their name on a webpage and click ‘Identify me!’. You could then grab the People Australia ID to store in your own application, allowing you (with the help of my client library) to automatically include links to relevant entries in the Australian Dictionary of Biography, for example.

Even better, Wragge’s identity browser will automagically generate the RDFa markup you need to semantically enrich your document. Whether you’re writing a blog post, publishing an article, drafting a caption, creating a database entry, or preparing a finding aid you can quickly and easily find an individual and then cut and paste the code you need.

To show this in action I used the bookmarklet to help me mark up many of the people named in one of my articles. We humans see a normal page with a few extra links. Computers, however, can extract the embedded RDFa to get at the structured information that’s hidden in the page.

Now I’ve got to go and semantify the rest of my articles…

Go forth and identify! And in the process help build a better web.

Notes on the bookmarklet

• Internet Explorer has ‘Favorites’, Firefox has ‘Bookmarks’ – whatever you’re using first make sure that your Bookmarks/Favourites toolbar is visible. Look under Tools->Toolbars in IE8, View->Toolbars in Firefox.
• Try dragging the ‘Identify me!’ link to your Bookmarks/Favourites toolbar. If it doesn’t work, try right clicking on the link and choose ‘Bookmark this link’ or ‘Add to Favourites’. Make sure you add it to the toolbar folder. IE will probably give you various warnings – ignore them.
• You should now have a working bookmarklet – highlight a name and click on it, a new window should open with results from Wragge’s identity browser. IE might complain about opening a pop-up – allow pop-ups and try again.
• The bookmarklet is pretty clever about working out which part of the highlighted text is the surname, so you can highlight names in a number of formats including:
  • Surname
  • Surname’s
  • Surname, Othernames
  • Othernames Surname
  • Othernames Surname’s
• For the moment this only works with ’straight’, ie non-curly, apostrophes – but I’ll fix this asap. Fixed!

Notes on RDFa markup

• You have a choice between visible (ie clickable) links or invisible ones. They look the same to computers, so it’s just a matter of whether you want your human visitors to see them. Click ‘change’ to toggle between the two options.
• You can just paste the RDFa markup straight into your document. If you’ve used the bookmarklet, the text you highlighted will be automatically inserted as the link text – so just copy and paste. If you haven’t used the bookmarklet you can insert the link text yourself.
• Somewhere in your document you need to tell computers what the FOAF in your RDFa markup means. You do this by inserting the text:
  xmlns:foaf="http://xmlns.com/foaf/0.1/" inside a tag that contains your marked up text. If you can edit the raw html of your page, you can just insert it in the <html> tag itself, so it becomes <html xmlns:foaf="http://xmlns.com/foaf/0.1/" >. Otherwise you can wrap your marked up text in a <div> tag and put the extra code in there.
• If you’re using something like Wordpress that strips out or converts any markup that it doesn’t expect, you need to be able to enter the RDFa as ‘raw’ html. In Wordpress you can do this using the Raw HTML plugin.
• For more on using RDFa have a look at: RDFa for HTML Authors.

I was doing some research using the National Archives of Australia’s RecordSearch database the other day and became frustrated that there is no way of seeing how many pages are in a digitised file without clicking on the ‘Display digital copy’ link. So I fixed it.

As a userscript it’s hardly worthy of a blog post. All it does it find out how many pages are in the file and insert the number in the link text. It’s very simple. But I think it’s also a useful illustration of the changing balance of power between archives and their users.

William E Landis argued that archivists were ‘guilty as a profession of fetishising the outputs of our descriptive systems’. The design of finding aids have often been determined not by the needs of users but by a desire to faithfully represent the underlying archival architecture. But now users don’t have to just take what they’re given.

Technologies such as Greasemonkey are useful for sketching out alternatives. For organisations with IT systems that inhibit experimentation, Greasemonkey (or Mozilla’s Jetpack) provides a way of playing with interfaces without touching any of the underlying code. My rewrite of the way RecordSearch displays digitised files is an example of this.

But no one interface is ever going to meet the needs of all archive users. Fortunately, there are a growing number of ways in which archives can work in partnership with their users to help them create the interfaces they want and need.

Archives are starting to expose their data directly using APIs and linked open data. This gives users the power to create whole new applications. But I still think there’ll be a place for the little tweak – a simple hack that meets some small but specific need. I can imagine communities of interest building and sharing a range of tools, hacks, applications and interfaces specifically tailored to their research habits.

So if you don’t like it, fix it.

It’s great to see that the National Archives of Australia has released a large swag of data through the new data.australia.gov.au site. In the Commonwealth Agencies zip file you can find xml dumps of all the publicly accessible agency and series data in RecordSearch, as well as item data for series A1. This is the same data that Mitchell Whitelaw visualised so brilliantly in his Visible Archive project. There’s also item data and images from series A3560 – the Mildenhall photographs of early Canberra.

What’s even more exciting is that people are already using this data. At the recent GovHack event in Canberra the What The Federal Government Does team worked on visualising the activities of government by using functions data pulled from the agencies file. Another group has generated a really nice tag cloud and photo gallery from the Mildenhall data. With further GovHack sessions to follow and the MashupAustralia contest open until 13 November, let’s hope for some more inspired archives hacking.

Seeing RecordSearch data out in the world like this reminded me of a little project I started a while back and then set aside. It was a simple PHP script that scraped data from RecordSearch and spat it out either as XML or JSON. Mitchell used a version of this script in his A1 Explorer in order to find out the number of pages in each digitised file.

I’ve now expanded and improved the script so that it provides data on items, series, agencies and persons. The output includes all the basic fields as well as links between entities – such as related series, controlling agencies etc. As an added bonus you also get some useful totals (where they’re available): items include the number of pages, series include the number of items described on RecordSearch, and agencies include the number of series recorded. I’ve also fiddled with mod_rewrite to provide a more rest-ful interface.

For XML output use the url http://discontents.com.au/shed/rs/xml/ followed by the appropriate identifier – a barcode for an item, a CA number for an agency, a CP number for a person or a series number.

Some examples:

• Series A1 – http://discontents.com.au/shed/rs/xml/a1
• Item B2455, WRAGGE C L E – http://discontents.com.au/shed/rs/xml/3445411
• CSIR Head Office – http://discontents.com.au/shed/rs/xml/CA+486
• Alfred Deakin – http://discontents.com.au/shed/rs/xml/CP+9

As you might have guessed, to get JSON output you just substitute ‘json’ for ‘xml’ in the url.

Being dependent on screen scraping, it’s inherently a bit fragile, but I’m hoping it might be of some use. My intention was to use it to start exploring some new ways of using and interacting with the data. The code itself is available at BitBucket. It’s not very elegant, but I don’t want to spend much time cleaning it up at the moment. If it seems like it might be useful, I’ll probably rewrite the whole thing in python and publish it through Google’s AppEngine.