Time to take stock of the year that was…

2017 certainly had its ups and downs. In the middle of the year, my university’s management decided that they could do without undergraduate teaching in cultural heritage. We managed to hold on, but it’s the sort of thing that makes you wonder why you bother.

On the flipside, the highlight of the year was working with my undergraduate class in Exploring Digital Heritage to create and promote a site for the transcription of records used in the administration of the White Australia Policy. It’s something Kate and I have been talking about for many years, so it was a rather emotional experience to see it finally happen. But it wasn’t just about the website — it was about the way my students responded to the records (many of them knew little about the White Australia Policy before we started); it was about seeing the records on the ABC news; and it was about seeing the faces of the people who lived under the weight of the White Australia Policy projected onto the walls of Old Parliament House. That’s why we bother.

Also in 2017 (and somewhat related to the above) I decided to adjust my working arrangements in attempt to find some balance. You can now find me over on Patreon.

The making

• Closed Access 2017 update! Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 9 January 2017.
• Wrestled with RecordSearch to extract summaries for all series. Along the way I found out some interesting stuff about the use of functions. Putting the two together, I tried to build a picture of what NAA holdings look like through the perspective of government functions.
• Played around with Twarc to create a collection of #invasionday tweets. Later in the year I captured and shared tweets with the tag #australianvalues.
• Have you ever wondered how the cut and thrust of parliaments past might translate to the world of social media? Wonder no longer, for here you can explore interjections in the Australian parliament from 1901 to 1980, reimagined as tweets. You might even find some emoji…
• Code and work-in-progress website exploring the language of Hansard.
• Went back to basics with LODBook — now developing a series of components using Jekyll and javascript.
• The Redaction Zoo — this collection of creatures was discovered amidst thousands of ASIO surveillance files held by the National Archives of Australia. While the practice of redaction is intended to withhold information from public view, an unknown archivist has used redactions to add an artistic flourish to the files. They are reminders that the processes that limit our access to information are human in their operation and design. There is nothing magical about the ‘secrets’ preserved in government archives. Video first exhibited as part of ‘Beauties and Beasts’, Belconnen Arts Centre, 6-28 May 2017.
• Here’s a repository of images in JPG and SVG format drawn from a collection of #redactionart discovered in ASIO surveillance files held by the National Archives of Australia. Use them to create your own #redactionart projects!
• I finally added a decent full-text search facility to Historic Hansard. Yay! Search for speeches or bills. Filter by date, house, and speaker. There’s even an option to save your complete result set as a CSV file for further analysis and exploration.
• The Tribune was a newspaper published by the Communist Party of Australia. The State Library of NSW holds more than 60,000 negatives and photos from the Tribune which document a wide range of political events and social issues from 1964 to 1991. This is a work-in-progress site documenting my exploration of the negatives as a DXLab Digital Drop-in.
• Help transcribe records that document the lives of ordinary people living under the restrictions of the White Australia Policy. The Real Face of White Australia site was built using the Scribe Framework with the help of students from my Exploring Digital Heritage class.
• Explore regular updates of data generated through the Real Face of White Australia project. Includes CSV files with transcribed fields, as well as photos and handprints.
• There’s lots of exciting new digitised content being added to Trove’s journals zone, but it’s not always easy to find and search. I made an app that lists journals that have been digitised by the NLA and have searchable records for individual articles. This means you can search inside the journal, just like you do in the newspapers zone. The code for the harvester and web app is on GitHub.
• Demonstrated how to harvest Parliamentary press releases from Trove. The repository includes a sample dataset of politicians talking about refugees.
• Kicked off my new 101 Digital Heritage Hacks site with a userscript to add a ‘copy permalink’ button to Trove.
• I made a little bot to post stories from the Real Face of White Australia data to the @invisibleaus Twitter account.
• Created two new DIY Trove bots using Glitch. The Trove Tag Bot tweets items with a specific tag, while the Trove List Bot tweets items from one or more lists. Both bots come with detailed instructions and, because they’re hosted on Glitch, it’s easy for anyone to modify the code and launch their own bots.
• Added a battle mode to Headline Roulette. It’s really just a way of generating a url to specific newspaper article in Trove. This means groups can compete on a level playing field, but it could have other uses. You could create a curated set of challenges for students, or use them in something like a scavenger hunt.

The talking

• 13 February 2017 – Digital humanities workshop at ALIAOnline.
• 24 February 2017 – Keynote presentation at HDR Summer School, Deakin University, The practice of play, <https://doi.org/10.6084/m9.figshare.4696258>
• 1-3 March 2017, attended ‘Always Already Computational: Library Collections as Data’ national forum in Santa Barbara. Read the position papers and the final statement.
• 20 April 2017 – Public lecture and workshop, University of Wollongong.
• 4 May 2017 – Presentation with Professor David Lowe, ‘The Political Language of the 1970s in Australia’, at The 1970s: Australian and Indian Perspectives on a Decade of Transition, Deakin University.
• 23 June 2017 – Workshop, Random acts of meaning: Digital skills for a post-truth world, NLS8.
• 4 August 2017, ‘The struggle for access’, presentation at DHPathways 2017, Canberra.
• 10 August 2017 — ‘Doing DH’ presentation for ACT teacher librarians.
• 13 October 2017 — ‘Trove tips and tricks’, at the Deniliquin Family History Expo.
• 8 November 2017 — ‘Making this happen’, presentation at Collaborating around Collections, ANU.
• 20 November 2017 – ‘”The badge of the outsider”: Open Access and Closed Boundaries’, invited presentation at Sharing is Caring 2017, Aarhus. Also put together a last minute ‘Hack the world’ workshop.
• 4 December 2017 — Digital methods workshop for the UTS Legal History group

Presented at Sharing is Caring 2017, 20 November 2017, in Aarhus, Denmark.
You can also watch the video.

In 1946, Britain decided that Australia would be the perfect place to test missiles. The Australian government, keen to play its part in the defence of the Empire, readily agreed. Ignoring, yet again, the presence of Australia’s Indigenous peoples, defence planners thought Australia was attractive to because it was ‘empty’, flat, and far from ‘prying eyes’.

The town of Woomera was built in the South Australian desert to house scientists, workers, and military personnel. It was a town where no housewife could go to the shop without her security pass; where curiosity was ’the badge of the outsider’.

But while Australia’s land seemed ideal for secret military operations, its people remained suspect. Britain’s plans were threatened by concerns about Communist infiltration of the Australian government. Under pressure from the UK and USA, Australia sought to lift its spy game through the establishment of a new agency to monitor such threats — the Australian Security Intelligence Organisation, or ASIO.

Legislation defined ASIO’s functions in very broad terms, ‘to obtain, correlate and evaluate intelligence relevant to security’. From the 1950s to the 1970s, this was used to justify surveillance of a wide range of potential ‘subversives’ — not just known Communists, but writers, artists, academics, scientists, Indigenous activists, and more. Many thousands of files were created to document their beliefs, activities, connections, and personal lives. Recordkeeping was critical to the practice of state surveillance.

We don’t know how many files were kept on ordinary Australians because ASIO is exempt from many of the key provisions of the government’s archives legislation. Unlike other agencies, ASIO does not routinely transfer records or indexes to the National Archives of Australia. Researchers have to go on a fishing expedition, asking the National Archives to ask ASIO whether they might have a file relating to a particular person or organisation. If ASIO admits it has a relevant file in the open period (more than twenty years old), the file goes through an ‘access examination’ process to determine whether it contains information that should be withheld for reasons of national security, or individual privacy. If anything is left, it is finally opened to public access.

Despite these hurdles, more than 12,000 ASIO surveillance files have been made public, though most include redactions — black boxes obscure words too sensitive to be read.

The files have been used in biographies, family histories, and studies of Australia’s literary community. One recent book invited the subjects of ASIO surveillance to reflect on the contents of their own files — to see their lives through a different set of eyes; to explore the intrusions and innuendo that passed for ‘intelligence’.

I’m currently working with a set of 60,000 photographs held by the State Library of New South Wales. These photos were taken for The Tribune, a Communist Party newspaper published in Sydney, and document protest and political activity in Australia from the 1960s to the 1990s. One of the things I’m interested in is finding overlaps between the Tribune photographs and ASIO surveillance files. For example, in February 1972 there was a demonstration on Indigenous rights held outside Parliament House. Because both sets of records have been digitised and made public, we can compare perspectives — spies versus ‘subversives’.

This is a reminder that the impact of digitisation is not simply easier, more immediate, access. We can also see the same things differently. We can interrogate the meaning of access itself.

RecordSearch, the National Archives of Australia’s online database, provides access to about 64,000 series descriptions, 11 million item descriptions, and 1.8 million digitised pages. There’s currently no API, or downloadable datasets at item level, so I make my own.

For six or seven years now I’ve relied on my own little library of screen scrapers to get data out of RecordSearch. They’re slow and they break easily, but they do the job.

Late last year I embarked upon what was probably my most ambitious data harvest. I gathered information about every series listed on RecordSearch and calculated, for each, the quantity of records (in linear metres), the number of individual items described, and the number of items digitised. I then aggregated the series by the top-level functions of agencies associated with them. Basically I grouped them by subject — defence, security, education etc.

Why?

Because digitisation shapes our perceptions of reality. The more we have in digital form, the easier cultural heritage collections are to find and use, the more likely we are to assume that everything (or at least everything important) is online. Ease of access bears an ontological weight — if we can’t find it online, does it exist?

Now that might not be a problem if what was digitised somehow provided a representative sample of the whole. But we all know how such decisions are shaped by political priorities, funding opportunities, user demand, public events, and happy accidents. There’s nothing necessarily wrong with that, it’s just the environment within which we work. There are never enough resources. We have to do what we can, when we can.

The problem is, we rarely expose the impact of these decisions to the users of our digital collections. We rarely give them the chance to reflect on how our decisions shape their assumptions.

The National Archives of Australia documents the workings of our democracy. If offers one important perspective on who we are as a nation. If we look at the quantity of records associated with each top-level function, we see a fairly even distribution. Nothing stands out.

But what happens when we view the activities of government through the number of files digitised in each subject area?

The prominence of defence is really no surprise. Service records are heavily used by family historians, and in 2007 the Australian government funded the digitisation of all 375,000 World War I service records in what was branded as ‘A Gift to the Nation’.

The National Archives is not alone. I often show people this graph of the number of digitised newspaper articles in Trove, pointing out the fairly dramatic peak around 1914. Did something happen in 1914? Were there more articles published, more newspapers? No, there’s just more money. In the lead up to the centenary of WWI, funding was directed towards the digitisation of newspapers from the wartime period.

Again, there’s nothing wrong with this. It’s just that these biases are not obvious to someone typing queries into a search box. In the context of Australian history, these decisions around digitisation help to reinforce the long-held belief that Australian national identity was somehow forged on the battlefields of WWI. It helps to put war at the centre of our history, at the centre of who we are.

But of course while digitisation can shape our assumptions, it also gives us new opportunities to critique them. I could only analyse the holdings of the National Archives because their collection data is online. We don’t have to take just what the search box delivers — we can ask our own questions. But this is only possible if people have the skills, the tools, and the confidence to poke around in the data. This too is access. Institutions should invite the public not to swoon at their digital delights, but to hack away at difficult questions — not to see collections, but to see them in unexpected and challenging ways.

I mentioned that ASIO files go through an process known as ‘access examination’ before they’re released to the public. This is the case for all records more than twenty years old, not just the super secret ones. The vast majority of files are simply opened without restriction. Some, including most of the ASIO files, are opened ‘with exceptions’ — pages can be withheld, and text redacted. A few are withheld from the public completely. They have entries in RecordSearch, but you can’t see them — their access status is officially ‘closed’.

But because the metadata about access decisions is available online, we can start to build a picture of what we’re not allowed to see.

At the start of 2016, I harvested the details of all files in the National Archives of Australia with the access status of ‘closed’. I’ve aggregated and sliced the data in a number of different ways, so you can explore the age of the files, what series they came from, and when decisions were made about their access status. At any point you can drill down to a list of the files you cannot see — making it perhaps the most frustrating search interface ever devised.

You can also examine the reasons why files have been withheld. Many of these exceptions are defined by the legislation that established the National Archives. Clause 33(1)(a), for example, relates to national security, 33(1)(g) is concerned with individual privacy. But the metadata reveals that files are withheld for a number of other reasons, such as ‘Pre Access Recorder’ and ‘Withheld Pending Advice’. There’s also, you might note, a category entitled ‘MAKE YOUR SELECTION’ — which reveals something about the limits of the data entry interface.

By poking around you in the data you can make some guesses as to how these additional categories are used. ‘Pre Access Recorder’ is used as a catch-all for records that were withheld from public access before the archives legislation was passed. ‘Withheld Pending Advice’ is used to label files that have been sent off to other government agencies for their assessment — they’re not yet finally closed, but as this process can take years, they’re sort of closed. Indeed, my interface shows that 1,467 files have been waiting more than three years for advice.

The point of this is not to embarrass the National Archives, nor the Department of Foreign Affairs and Trade which holds the most files in limbo. The point is to examine the ways in which access itself is constructed. Legislation defines an ideal, but the reality is more messy and human. By tracking patterns in the way access decisions are made we can explore the historical processes at work. Access is not allowed, it is made.

Remember those 12,000 ASIO files publicly available through the National Archives? You might not be surprised to know that I’ve harvested them all — both the metadata and the 300,000 digitised pages. There’s about 70gb of images.

Using these files we can dig a little deeper into the nature of access. I wrote a computer vision script to find redactions. It took a lot of trial and error, and I’m about to start work on a smarter version that incorporates machine learning, but it did the job. From one series of ASIO files, about 230,000 pages, I extracted 239,000 redactions — lots and lots of little black boxes. You’ll be pleased to know that not only can you download the complete set of redactions from the research repository Figshare, you can browse them. All of them! Hours of fun for all the family!

The interesting thing about this interface is that if you click on a redaction you can view the page that it was extracted from. So it’s sort of an inside out discovery interface. Instead of the redactions being a brick wall or a dead end, they’re a starting point. A practice intended to remove information, to limit access, becomes a gateway for exploration. Indeed, the redactions themselves provide an identifiable data point — something that can be analysed to turn the gaze of government surveillance upon itself.

But something else was hiding in those ASIO files. As I was reviewing the collection of redactions for false positives I discovered that someone tasked with the removal of information, decided to add a little creative flair.

I discovered #redactionart.

I assure you that these creations really are sitting inside ASIO files held by the National Archives. But since I’ve discovered them, they’ve developed a life of their own. Not only can you browse through them online, you can wear them.

I gave away about 80 of these badges at an exhibition earlier this year. To create the badges, I simply traced around the original images and saved the results as SVG files. These files themselves are shared through GitHub for anyone wanting to create their own #redactionart.

This amazing #redactionart dress was made by Bonnie Wildie, a librarian in NSW. My SVG files have also been turned into a set of 3D printable cookie cutters, as well as a range of t-shirts and stickers on RedBubble.

This escape from the archives is not only creative and fun, it’s important.

It’s important because it emphasises that the practices through which government information is controlled and withheld are profoundly human. People make decisions and they leave their marks. There is nothing mysterious or otherworldly in the secret — it is an exercise of power.

Archives are not just made of documents — there are people inside.

‘Surveillance’ is not included in the National Archives’ official thesaurus of government functions, yet the movements and activities of individuals are recorded in many thousands of files across an assortment of agencies.

A simple query of my harvested data reveals that the phrase ‘alien registration’ appears in the titles of only 29 series. But these series contain more than half a million files. 4.7% of digitised files in the National Archives document the movements of so-called ‘aliens’. While these registration systems were created during wartime, they lingered beyond. And they were not the only means of keeping track of potential threats. Just as at Woomera, boundaries were drawn, and outsiders marked for attention.

When I was last in this part of the world, I talked a bit about some work that Kate Bagnall and I had done with records of the White Australia Policy held by the National Archives.

A quick recap — when the Australian colonies federated in 1901, it was generally assumed that the new nation’s future could only be assured through strict racial homogeneity. A ’white’ Australia was a strong Australia. Legislation was quickly passed to restrict immigration and set the foundations for what became known as the White Australia Policy.

However, in 1901 there were around 40,000 people living in Australia whose background was neither European nor Indigenous — they were Chinese, Japanese, Syrian, Indian, and Malay. Some had been born in Australia, or had lived there for many years — raising families, building businesses; just living their lives.

If any of these people wanted to travel overseas they had to carry special documents, or they might not be allowed to return home. Customs officials at Australian ports would ask anyone who seemed not to be ‘white’ for identification. The badge of the outsider was the colour of their skin.

Many thousands of these documents, the remnants of a racist bureaucratic system, are preserved in the National Archives.

Back in 2011, I downloaded about 12,000 of these documents from RecordSearch and ran them through a facial detection script to create a seemingly endless scrolling wall of faces. We called it ‘The Real Face of White Australia’. It’s another inside-out interface — instead of showing the files, you see the people inside.

That was then, this is now! In the last few months, I’ve been working with a group of my digital heritage students to develop a website for the collaborative transcription of these same records. We want to put names to the faces. We want to chart their journeys. We want to document their lives.

Our project has no funding, and was only possible because Zooniverse and the New York Public Library created and shared Scribe, a framework for the transcription of structured documents — an easy way to get usable data out of forms, ledgers, and certificates.

The site was launched at a ‘transcribe-a-thon’ held at the Museum of Australian Democracy in Canberra, which just happens to be located in Australia’s first parliament house. The building didn’t exist when the Immigration Restriction Act was passed in 1901, but it was where the White Australia Policy was elaborated and maintained.

Transcription continues. There’s still much work to do on the documents, but data is already flowing. I’m making regular dumps available for download through a GitHub repository.

But it was never just about the data. Many more people now know that these records, this history, exists. Through the process of transcription you are confronted by the disturbing reality of the records — you’re surprised, puzzled, shocked, and often moved. Creating a space for these sorts of experiences is important in itself.

The Museum of Australian Democracy not only gave us their building for a weekend, they let us play with their data projectors. In some ways, I would have been happy if all we had achieved was this — to put these faces in this space.

Once again the gaze of surveillance is reversed. In the home of Australian democracy, people who lives were monitored under a racist system of exclusion and control were looking at us, asking questions of us.

Amongst those Tribune photos at the State Library of NSW, I recently found this. Believe it or not, I’m the spy on the right. This compelling piece of street theatre was performed at the gates of Pine Gap, a US electronic surveillance facility right in the centre of Australia. Pine Gap’s lease was due for renewal in 1987, so hundreds of protestors converged on the site, hoping that the Australian government might withdraw it’s support. Needless to say, it didn’t. Pine Gap remains, and in recent times has been implicated in US drone strikes

I found another photo of myself amongst the Tribune archives. A group of us climbed over the outer perimeter fence in the middle of the night and took up positions on a rocky outcrop that overlooked the main gate. At a predetermined time, we leapt out of our hiding places and lit smoke flares. I was arrested soon after, charged with trespass, and fined $100.

Another group of Pine Gap protestors are currently on trial in Australia. They made it through the protective fences and dared to play music and pray. For this they have been charged under the Defence (Special Undertakings) Act which carries a maximum sentence of seven years in prison. This Act was passed in 1952 when Britain decided to expand its weapons testing program in Australia to include atomic bombs. It expanded upon earlier legislation that had been intended to protect Woomera from Communist interference. This is one of the very few times anyone has been charged under the Act, despite there being hundreds of arrests like mine in the past.

As security services gain new powers, and electronic surveillance expands, it’s hard not to see the Pine Gap proceedings as an attempt to discourage criticism of the government’s tough on terrorism stance.

At a recent symposium on collaboration between researchers and collecting institutions, Seb Chan described some of the advances that had taken place in opening up collections, but then asked ‘So what?’.

I suppose that’s the question we’re hear to discuss. Why do we put all this effort into digitising collections, building interfaces, and sharing data? Easier access is great, beautiful interfaces are cool, but… so what? For me, as a historian, hacker, and sometime heritage professional, the answer is straightforward — it’s all about bringing the past into conversation with the present. It’s about mobilising our collections as critical resources in debates about who we are, what matters, and why we should care.

Those inky, black handprints on the White Australia records moved one of my students to reflect on her experience as a recent immigrant from Canada, required by the Australian government to supply a set of her fingerprints. She wrote a beautiful talk and presented it during the transcribe-a-thon in the original House of Representatives chamber at Old Parliament House. Another student noted in her final essay that the documents made non-white residents seem like criminals, pointing to parallels with the current treatment of refugees. On the flip-side, our efforts attracted the attention of a few racist trolls, one of whom referred to the White Australia Policy as ‘the good old days’.

Once again ’outsiders’ are being targeted as threats to our security. Boundaries are being reinforced, and efforts being made to define who belongs. We know this. We’ve seen this before. Europeana’s new project on the history of migration is an important initiative — we need to tell our stories, share our resources, grapple with our difficult and painful pasts. I don’t think this is a time to reassert the authority of our cultural institutions as reservoirs of truth. We are implicated in all of this. Our collections are built upon systems of surveillance, on attempts to put humans into categories. They are products of power and privilege. We are not the guardians of enlightenment, we are the keepers of horrors.

Just like #redactionart, the value of our collections lies in their complexity and contradictions — in their very humanity and all the confusion that entails. Digital collections lend themselves to an exploration of complexity. We can shift scales and perspectives, we can manipulate contexts, we can set collections loose in public spaces, we can turn them inside out. We can see differently, but perhaps more importantly, we can feel differently.

When you think about it, ‘impact’ is a pretty violent sort of word. There are perhaps a few people around the world we’d like to ‘wallop’ with our digital collections. But I suspect most of the time we’re after something more subtle — to expand possibilities, to undermine assumed certainties, maybe even to expose a glitch in the Matrix.

Perhaps we can offer a glimpse of an alternative reality, where we recognise the outsider as us.

Maybe in the end we will be able to see the outsider as us. Hear hear! #ShareCare17 pic.twitter.com/TWWYqyvCDg

— Johanna Berg (@johannaberg) November 20, 2017

Keynote presentation at the Deakin University Faculty of Arts and Education HDR Summer School, Geelong, 24 February 2017.

Cite as: Tim Sherratt, ‘The Practice of Play’, presented at Deakin University Faculty of Arts and Education HDR Summer School, Geelong, 24 February 2017, online at <https://doi.org/10.6084/m9.figshare.4696258>.

I’m a historian. But in the past decade the nature of my research has changed quite profoundly. Instead of heading off to the archives, taking lots of notes, and writing up a book or an article, I now make things. Generally these things are online, and open to the public. I make things for people to use, to explore, to play, and to ponder.

I started down this track before I realised there was a name for what I do – practice-led research. The things that I make even have their own acronym – they’re NTROs, or Non Traditional Research Outputs.

But practice-led research is not just about making things. New knowledge is generated through cycles of creation and reflection. My aim in making is not to follow a blueprint, or check off a list of requirements, but to end up asking ‘What is this thing?’, ‘What does it do?’, ‘How does it do it?’.

In the past, I’ve tended to talk about my research practice as playing with data. I think there’s an important argument to be made for the role of play in research, particularly when confronted with large cultural datasets. But ‘play’ doesn’t quite capture what I do, nor does it look very convincing in a research proposal. So what do I really do?

Let’s play a game.

Headline Roulette is a very simple game. Presented with the title of a digitised newspaper article drawn at random from Trove’s collection of more than 200 million you are challenged to guess the year in which the article was published. Sounds easy, but you only get ten guesses. It’s sort of like a cross between hangman and The Price is Right.

Despite its simplicity, I’ve known it to unleash the competitive instincts of a workshop full of historians. But for me, Headline Roulette is important because it provides an example of what becomes possible once we make cultural heritage collections available online. Our interactions are no longer limited to conventional modes of viewing or reading – we can play, and we can build.

I made the first version of Headline Roulette back in 2010. It was a game, but it was also an argument about access and possibilities.

Perhaps we should first take a step back. Who’s used Trove?

Trove is a fundamental part of Australia’s research infrastructure – and not just for those of us in the humanities or social sciences. Trove is a lot more than digitised newspapers, but access to more than 150 years worth of digitised newspapers has profoundly changed historical practice.

I say this not just because you have been spared the pain and suffering wrought by microfilm readers upon a generation of historians, but because the meaning of access itself has changed. Headline Roulette is just one simple and silly example of how once cultural heritage resources are in digital form we can use them differently. We can see them differently.

Imagine your search in Trove’s newspapers zone returns 10,000 or 100,000 results. How do you make sense of that? How do you get an understanding of the whole, when all you see is page after page of search results?

QueryPic extracts data from Trove to visualise your search as a single chart – showing you the number of articles per year that match your query. You can even compare the occurrence of particular words or phrases.

But that’s only the beginning, because once you think about web resources as data rather than just another type of publication you can aggregate and analyse – you can look for big, dramatic pictures as well as tiny, fragile fragments.

Trove Harvester is tool that delivers historical newspaper articles in bulk – thousands, even millions of articles saved to your computer for offline exploration.

What might you do with a million newspaper articles?

Research using digital resources like Trove is not constrained to the window of your web browser. You can ask new types of questions.

But back in 2010–11 when I created the first versions of Headline Roulette, QueryPic and the Trove Harvester there was no easy way of getting data out of Trove. The thing is, web pages are good for delivering data to human beings, but not so good for computers. Computers are actually pretty dumb, and you need to be quite explicit in packaging up data for them. Nowadays Trove has a thing called an API (an Application Programming Interface) which delivers data in a carefully structured format that even computers can understand. You can use APIs to harvest data, or to build new tools or interfaces. APIs are cool.

Without an API, the first versions of my tools had to turn human-readable web pages into computer-readable data – a process known as screen scraping. They were, therefore, not only useful or interesting applications in their own right, they were arguments about why things like APIs matter. Why web pages aren’t enough. Why researchers need access to data.

These are arguments we’re still making. Next week I’m heading to a workshop in California where we’ll be discussing how libraries and other cultural institutions can deliver their data in ways that support new forms of research.

But we don’t have to wait. By screen scraping web pages, by reverse engineering online databases, we can continue to develop the argument for access by extracting, sharing, and using data.

What could you do with 70gb of digitised surveillance files from the Australian Security Intelligence Organisation (ASIO)? I’f you’d like a copy I have them here on a USB drive.

Don’t worry – we’re not about to be raided by the security services. These are all files that have been carefully examined and released to the public through the National Archives of Australia. You can find them by searching the Archives’ online database – RecordSearch.

Who’s used RecordSearch? It’s not the most friendly system, but the collection it documents, and the metadata it provides, is rich and wondrous. I’ve spent a lot of time trying to get useful data out of RecordSearch – not just ASIO files, also records documenting the administration of the White Australia Policy, as well as higher-level data aimed at building my understanding of how the Archives, and its descriptive systems, actually work.

It is painful and frustrating work. But, I would argue, it is research. Terms like ‘data mining’ and ‘text mining’ fly around all the time, making it seem as if the the accumulation of data is a mechanical process – as if we’re just digging it up. But the practice of screen scraping, or of liberating data from any cultural heritage source, is not simply extractive – it’s iterative and interpretative. It’s a process through which you begin to understand how the data is organised, what its limits and assumptions are, what its history is. What it means. We’re not just taking things out, we’re putting them back.

Frederick Gibbs and Trevor Owens argue that historical data need not be deployed solely as statistical evidence. ‘It can also help’, they suggest, ‘with discovering and framing research questions’ – questions, not answers; interpretation not calculation. Gibbs and Owens describe an ‘iterative interaction with data as part of the hermeneutic process’.

For me, RecordSearch is like an archaeological site. Excavating data from it involves digging through layers of technology, institutional history, and descriptive practice to try and understand why we have what we have.

Those of you undertaking projects using the collections of the National Archives will almost certainly come across the process of ‘access examination’. Under the Archives Act, government records more than twenty years old are expected to be opened to the public. However, the act also defines a number of exceptions to this rule – for example, records that endanger national security or infringe an individual’s privacy can be completely, or partially, withheld from scrutiny. The process of assessing records against this set of exemptions is called ‘access examination’.

The vast majority of records are opened without problem – they are, after all, more than 20 years old. But a significant number are not. While you can’t use these records, RecordSearch does provide some information about them. So I decided to see what we couldn’t see.

In January 2016 I fired up my screen scraper and harvested details of all the files in RecordSearch that have the access status of ‘closed’ – there were 14,370 of these files that had been through the process of access examination and withheld from public view. I then created my own interface that lets you explore this data from a variety of angles – such as the reasons why files were closed, when decisions were made about them, how old they are, and which government agencies created them.

It is perhaps the most frustrating search interface ever devised, given that you’re not allowed to see any of the files you find.

Those of you currently planning research projects might be interested to know where most of these files come from. It’s not defence or the intelligence agencies, but what is now the Department of Foreign Affairs and Trade (DFAT) – in January 2016, there were 1,747 closed files from just one DFAT series. But if you dig deeper you see that most of these files aren’t withheld for one of the reasons defined by the Archives Act, they are described as ‘closed pending advice’. The National Archives is still waiting to hear back from DFAT about them. Using my interface you can see that there were 54 files in this series where the Archives has been waiting for more than five years. So if you’re embarking on a project using the National Archives, make sure you get your access examination requests in early. Just in case.

My aim in extracting and sharing this data is to better understand access examination itself as a historical process. It’s work that enables us to ask different types of questions, but it also makes a change in the process itself. My interface is public, offering a critical commentary on the ‘official’ system. As a result of my research, the Archives has made changes to the way it describes closed files. It’s both research and intervention, history and hack.

‘Hack’ has a number of definitions, both positive and negative. Mark Olsen describes the ‘hacker ethos’ as:

‘a way of feeling your way forward through trial and error, up to and perhaps beyond the limits of your expertise, in order to make something, perhaps even something new. It is provisional, sometimes ludic, and involves a willingness to transgress boundaries, to practice where you don’t belong… Whether eloquent or a kludge, a hack gets things done.’1

Olsen explores what hacking means in the context of the humanities, arguing not only that hacking has a legitimate place in humanities practice, but that the humanities itself needs to be hacked to foster the development of new skills and literacies.

At this point you’re probably thinking, ‘But I don’t do any of this wacky digital stuff, what has this got to do with me?’

Who’s heard of filter bubbles, or search personalisation? Who’s read one of the many reports recently about the way computer algorithms are shaping our online experience? Olsen argues for a humanities practice that equips us to wrestle with complex techno-social systems.

And we’re not just talking about Google.

Last year Matthew Reidsma published an analysis of algorithmic bias in library discovery systems. He hacked a common commercial library product to show some of the biases underlying its recommendations system. The interfaces we use to access information are never neutral. The databases we search are products of selection and exclusion. Hacking enables us to interact with these systems as critics, and not just consumers.

Using the Trove API you can create a chart showing the number of digitised newspaper articles available per year from 1803 onwards. If you do this, you’ll notice two significant features. First, there is a dramatic drop-off in the number of articles after 1954. This is the ‘copyright cliff of death’. Few things are certain in our overly-complex copyright system, but 1954 provides a practical cut-off point. History stops in 1954.

You’ll also notice a substantial peak in the number of articles around 1914. Why might this be? Did something significant happen in 1914?

In fact, it’s all about money. In the lead up to the centenary of WWI it was decided to focus limited digitisation resources on newspapers from the WWI period. It was a perfectly reasonable decision, but the consequences are effectively invisible to any user of the web interface. You don’t know what you’re searching.

The power of Google encourages us to put a lot of faith in search interfaces. We trust that they will just work. And if we can’t find what we’re looking for, we often assume that it doesn’t exist.

Hansard, the recorded proceedings of the Australian parliament from 1901 can be searched using the ParlInfo database on the Australian Parliament House website. Perhaps you’ve used it – it’s a wonderfully rich resource. Powering the search results are a series of well-structured XML files, one for each sitting day, that identify individual debates and speeches.

Last year I reverse-engineered ParlInfo and harvested all those XML files. I thought they’d provide a great dataset for exploring changes in political speech, and so I created a repository containing all the files for the House of Representatives and the Senate from 1901 to 1980. Feel free to download and play.

But in the process of harvesting the files I noticed that some of the XML files were empty. After a bit more analysis I realised that about 100 sitting days were missing – they didn’t show up in search results on ParlInfo.

The ‘missing’ days were concentrated in the Senate between 1910 and 1920. So anyone relying on ParlInfo to research the WWI period would have missed significant amounts of content. This ‘black hole’ was effectively invisible to any user of the web interface. It was only though hacking that its shape and extent was revealed.

Fortunately staff at the Parliamentary Library have investigated and fixed the problem. But it’s a good example of why we should, as researchers, start from the assumption that search interfaces lie. Processes of selection and description shape the ‘reality’ of online collections. We then explore them through complex technological systems that appear comprehensive, even when they are not. You can’t find what’s not there. Online collections hide as much as they reveal.

Of course this is true of all historical sources. We are trained to analyse both context and content, to make judgements about authenticity and accuracy. These same skills need to applied to digital resources, to data. Indeed, Gibbs and Owen argue that ‘historians must treat data as text, which needs to be approached from multiple points of view and as openly as possible’. But how do we find multiple points of view when interfaces construct our experiences and limit our perspectives. How do we open data to new possibilities? How do we see data differently?

No doubt you’ve been encouraged to find a way of expressing your research questions succinctly, in a way that communicates with a non-specialist audience – yes, I mean the dreaded elevator pitch. You’re not the only one.

I’ve landed back in academia after a number of years working in cultural heritage institutions, and pursuing my own research interests with the support of the international digital humanities community.

Believe me when I say, Twitter changed my life. There I was, hacking away on cultural heritage data without any real assistance or encouragement, when I discovered, via Twitter, that there were people out there like me. Many of these people are now my friends, and I’ve been lucky to travel around the world to meet and work with them.

But coming back to academia I’ve found that my collection of projects, tools, experiments, and obsessions was not quite enough – my research needs a ‘narrative’.

So, like you, I’ve had to think about why I do what I do. What motivates my research? What matters?

For me it comes back to the nature of this thing we call ‘access’. Cultural heritage organisations talk about ‘access’ all the time, particularly in relation to online collections. But what does it actually mean? I want to overturn our assumptions about access – exploring it not as a process of opening things up, but as a system of controls and limits. It’s not a state of being, it’s a struggle for meaning and power.

My methodology, and I think I can call it that, is the multiplication of contexts. Context is, of course, critical to cultural heritage collections – it enables us to locate them within history and culture, to analyse their authenticity, to mobilise their value as evidence. But the descriptive systems we use to manage and explore collections represent only a privileged subset of possible contexts.

Now I’m still figuring this out, but I think what my work does is that it removes collections from these highly-controlled systems and lets them loose in a variety of new contexts. This allows unexpected features, or new uses, to emerge – we see them differently, and in that moment, the nature of access shifts, however slightly. It’s those moments I’m trying to catch and observe.

If you’ve ever tried to use Hansard through the ParlInfo database you’ll realise that it’s just really difficult to read. You’re presented with a series of nested fragments, so it’s hard to get a sense of the context and flow of the day’s proceedings. Having downloaded all those XML files, I thought I’d have a go at presenting Hansard in a form that privileged reading over search.

So I created Historic Hansard – dedicated to lovers of political speech. It does nothing very fancy, but I think it does it pretty well.

In the end, however, Hansard is still just text. What’s lost in the documentation process is the performance – the theatre of parliament. But not completely. As well as formal speeches, many interjections have been recorded and preserved.

A few weeks ago I extracted all those interjections from 1901 to 1980, about a million of them, and saved them to a new database. As I fiddled with different presentation methods, I started to see them as something akin to tweets – quick, pithy, and pointed. What would happen, I wondered, if we reimagined interjections from a century ago in an age of social media.

Like many of my projects, this whatever it is took me a couple of days to build. No research grants were harmed in its creation, no committees were needlessly formed. This is not because I’m a whizz-bang coder – I’m certainly not. It has to do with the nature of this work – it’s rapid, experimental, and sometimes even ephemeral. I don’t design websites, I make interventions – things that are not only of the world, but in the world. They do something.

Stephen Ramsay explores the hermeneutical possibilities of screwing around with technology and texts. The ‘screwmeneutical imperative’ he suggests is based on the fact that:

’a writerly, anarchic text… is more useful than the readerly, institutional text. Useful and practical not in spite of its anarchic nature, but as a natural consequence of the speed and scale that inhere in all anarchic systems.

Digital technologies give us the opportunity to play with scale and speed. We can manipulate millions of newspaper articles, and we can build a new version of Hansard in a weekend. But this shift also applies to the way we communicate. Instead of waiting months or years for an article to appear in print, we can post it on a blog, or in a digital repository. It is fundamental to the work that I do that it is shared, it is public by default – not just the results, but the code, the data, the process, and yes the licensing. Access is not just what we take, it’s what we do.

The multiplication of contexts has some interesting precedents as a research methodology. In the literary world the Oulipo movement sought to play with the contraints of composition. Lisa Samuels and Jerome McGann suggested that the deliberate misreading of a text, what they termed ‘deformance’, could yield critical insights. More recently, Mark Sample has argued for a ‘deformed humanities’ where we learn about things by breaking them.

In history we have the counterfactual – a creative reimagining of a past that never was, aimed at revealing perspectives and possibilities too quickly closed and forgotten. As Sean Scalmer argues, ‘counterfactuals are fun’:

‘Conventions can be disregarded, or even mocked. Worlds might be remade, the tyrannical overthrown, and the humble elevated. New orders can be imagined.’2

But counterfactuals are not fiction. They work best when they sail close to an accepted version of the past; when they play with the constraints of documentary evidence rather than just ignore them. Just because an approach is playful, it doesn’t mean that there are no rules. As Ian Bogost has recently argued, the fun of play is ‘not doing what we want, but doing what we can with what is given’.3 Play is an investigation of limits.

While some of the ASIO files held by the National Archives are closed to the public, most are ‘open with exception’. This means that sensitive parts of the files have been removed. Whole pages can be withheld, or sections of text blacked out – a process known as redaction.

A redaction is, by definition, an absence of information, and yet the frequency, density, and placement of redactions across a large collection of documents could conceivably tell us something interesting. So last year I wrote a kludgy computer vision script that found and extracted redactions from digitised ASIO files. I now have a collection of 250,000 redactions which I’ve shared on Figshare – grab a copy now!

I’m continuing to explore the possibilities of these redactions as data points. But there was also something visually interesting about the redactions, particularly when they were assembled on masse.

Here you can browse all 250,000 redactions. But that’s not all, you can also use them as entry points to the documents they were intended to obscure.

Contexts here have been reversed, the files have been turned inside out – the limits remain, indeed the scale of redaction is emphasised, and yet within these limits, perhaps even because of these limits, we can experience the files quite differently. We are no longer simply the subjects of state surveillance, we can reverse the gaze, inspect the process, and ask new questions.

The manipulation of contexts is not mere invention. The limits of access offer both meaning and rules. We have skin in this game, its outcomes matter, what is at stake is our ability to see, and be seen, within the cultural record. Access changes who we can imagine ourselves to be.

In the first half of the twentieth century, if you were deemed not to be ‘white’ and wanted to travel overseas from your home here in Australia, you had to carry special documents. Without them, you’d probably be stopped from returning – from coming home. This was ‘extreme vetting’ White Australia style.

Many thousands of these documents are now held by the National Archives of Australia. In 2011, I used my screen scraper to harvest about 12,000 images like this from RecordSearch. I then ran them through a facial detection script and created The Real Face of White Australia.

There are about 7,000 faces in this seemingly endless scrolling wall. And that’s from just a small sample of the White Australia records. It’s powerful, compelling and discomfiting. But the power comes not from any technical magic, but from the faces themselves – from what we feel when meet their gaze. Once again the records have been turned inside out – instead of seeing files, metadata, or a list of search results, we see the people inside.

Play can be serious. It can make you feel things you don’t expect. It can challenge your certainties and take you to the limits of what you know.

That sounds a lot like research to me.

1. M. J. Olson, ‘Hacking the humanities: Twenty-first-Century literacies and the “becoming other”of the humanities’, in Eleonora Belfiore and Anna Upchurch (eds), Humanities in the Twenty-first Century: Beyond utility and markets, Palgrave Macmillan, 2013, pp. 237–250.
2. Sean Scalmer, ‘Introduction’, in Stuart Macintyre and Sean Scalmer (eds), What if? Australian history as it might have been, Melbourne University Press, Melbourne, 2006, pp. 1–11.
3. Ian Bogost, Play Anything: The Pleasure of Limits, the Uses of Boredom, and the Secret of Games,Basic Books, New York, 2016, p. 236.

The image above is from Geoff Hinchcliffe’s awesome visualisation of more than 12,000 #fundTrove tweets.

This year I sadly left the wonderful team at Trove and took up a full-time academic post at the University of Canberra. But it was Trove that dominated the early part of the year, as the impact of continual funding cuts on the National Library of Australia became clear. Users of Trove shared their feelings on Twitter and Facebook, organisations posted statements of support, and numerous articles appeared in the media. In the lead up to the federal election, both the Greens and ALP made commitments to support Trove and our national cultural institutions.

In the last few days, we’ve learnt that the Government will provide $16.4 million over four years to the NLA ‘for digitisation of material and upgrade of critical infrastructure for its Trove digital information resource and to upgrade other critical infrastructure’. While we wait to hear exactly what this means for the future of Trove, it’s important to remember that it comes after many cuts and job losses across the cultural sector. The lesson of #fundTrove is that we cannot take the future of our collecting organisations for granted. We need to show why they matter and fight for the resources they need.

Access is important — both its politics and its practicalities. This year I’ve tried to be a bit more rigorous in the way I share information and document my projects. I created a Digital Heritage Handbook where I publish workshops, activities, and other bits and pieces. Much of it is in draft form, but I decided it was better just to push everything out in the hope that it might be useful. Similarly, I created an Open Research Notebook to share work in progress. The Handbook also includes details of the two undergraduate units I taught in second semester — Working with collections, and Exploring digital heritage. I think they went pretty well, but I’ve got a few improvements planned for 2017.

This year I accidentally built my own version of Historic Hansard, created an interface to National Archives files we’re not allowed to see, and mined ASIO surveillance files for redactions. As well as these major projects, there were lots of little hacks and harvests aimed at exploring the idea of ‘access’. You can follow my main research obsessions in my notebook:

• Closed Access
• Historic Hansard
• Immigration, Recordkeeping, and Surveillance
• LODBooks

Talking and making details follow…

2016 — the making:

• Locating Trove newspapers
  Updated code, data, and interface to geolocate and display Trove newspaper titles. Now with maps!
• Headline Roulette
  Much needed update for my old game. Now on it’s own domain and with better handling of Trove API errors.
• DFAT Documents
  Demonstration code to harvest the Department of Foreign Affairs and Trade’s collection of historical documents and extract some metadata. The harvested documents are available in Markdown format and can be explored through a simple website.
• People of Australia
  @people_aus is a Twitter bot sharing random names drawn from late 19th and early 20th century naturalisation records held by the National Archives of Australia. Many names. Many cultures. These are the people of Australia.
• RecordSearch Series Harvests
  Code to harvest the metadata and digitised images of all items in a series from the National Archives of Australia. Data from an assortment of harvested series are available as CSV files.
• SRNSW indexes
  Code for harvesting indexes from the State Records of NSW website. Data from 59 harvested indexes is available as CSV files.
• Facial detection demo
  Code and website to demonstrate the principles of facial detection using OpenCV.
• Show Redactions userscript
  Code for inserting details of redacted files into RecordSearch results.
• ASIO Experiments
  Code used for the extraction of redactions and other experiments with digitised ASIO files.
• Redactions dataset
  Redactions extracted from ASIO surveillance records in National Archives of Australia Series A6119, <https://dx.doi.org/10.6084/m9.figshare.4101765.v1>
• Non redactions dataset
  False positives (non-redactions) extracted from ASIO surveillance records in National Archives of Australia Series A6119, <https://dx.doi.org/10.6084/m9.figshare.4104651.v1<
• Redacted
  Web interface for exploring redactions extracted from digitised ASIO files. Includes a collection of redaction art.
• Open with Exception browser
  Code and website providing an experimental browser for digitised ASIO files from the National Archives of Australia.
• Invisible Australians browser
  Updated code and website providing an experimental browser for digitised records from the National Archives of Australia relating to the administration of the White Australia Policy. Now includes a landscape view for exploring records by their orientation.
• Closed Access harvester
  Updated code for harvesting and analysing records from the National Archives of Australia with the access status of ‘closed’.
• Closed Access dataset
  Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 1 January 2016.
• Closed Access website
  Public web interface for the exploration, analysis, and visualisation of ‘closed’ records in the National Archives of Australia.
• RecordSearch Functions
  Code and documentation for analysing the performance of functions by Commonwealth government agencies over time, using data from the National Archives of Australia.
• Commonwealth Hansard XML repository
  A repository of the (almost) complete proceedings of the Commonwealth House of Representatives and Senate from 1901–1980. This comprises several gigabytes of XML-formatted files harvested from the ParlInfo database.
• Historic Hansard
  A public website that presents the proceedings of the Commonwealth House of Representatives and Senate from 1901–1980 in a form that is optimised for browsing and reading. It includes additional features such as indexes to people and legislation, and the integration of tools for text analysis and annotation. Documentation is also provided.
• Trove Harvester
  Code and documentation to support the creation of large datasets for research and analysis from Trove’s digitised newspapers.
• Gadfly front pages
  Code and documentation to demonstrate how to harvest page images from Trove’s digitised newspapers.
• Trove Proxy
  Code and active proxy service that generates links to download PDFs from Trove’s digitised newspapers, and provides a https wrapper around the Trove API.
• DIY Headline Roulette
  Code and documentation that makes it easy for anyone to create their own simple game using Trove’s digitised newspapers.
• Radio National program data
  Updated dataset of programs broadcast on Radio National from 2000–2016 harvested from Trove.
• PMs Transcripts repository
  Repository of more than 20,000 XML transcripts of speeches by Australian Prime Ministers harvested from the PMs Transcripts site.
• UMA Ellis Photos
  Repository of data and images from a collection of political photos by John Ellis held by the University of Melbourne Archives. Harvested using the Trove API.

2016 — the talking

• 8 November 2016 – Digital research seminar and workshop at Griffith University.
• 10 November 2016 – Presentation as part of the ‘Access and innovation’ panel at Digital Directions 2016, National Film and Sound Archive, ‘Caring about access’, <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>.
  Featured on dh+lib (digital humanities and libraries).
• 19 October 2016 – Keynote presentation at Forging Links, Australian Society of Archivists conference, Parramatta, ‘Turning the inside out’, <https://dx.doi.org/10.6084/m9.figshare.4055013.v2>.
• 8 September 2016 – Invited presentation to Data and libraries: harnessing the possibilities, ALIA URLs seminar, Canberra, ‘Slow data? Small data? Exploring human-sized alternatives in the big data deluge’.
• 26 August 2016 – Keynote presentation at Migrant (R)e-Collections, Lorenz Centre, Leiden, ‘A life reduced to data’.
  Featured as an ‘Editors’ choice’ by Digital Humanities Now.
• 19 August 2016 – Keynote presentation at Working History, Professional Historians’ Association conference, Melbourne, ‘Telling stories with data’.
• 29–31 July 2016 – GovHack Heritage Node, University of Canberra
• 18 July 2016 – Keynote presentation at internal staff conference, State Library of Victoria.
• 15 July 2016 – Invited presentation at DigitalGLAM Symposium, University of Melbourne, ‘Hacking heritage: power and participation in digital cultural collections’.
  Featured on dh+lib (digital humanities and libraries).
• 5 July 2016 – Interview on ABC 936 (Hobart) Evenings about Trove.
• 24 June 2016 – Interview on ABCRN afternoons, ‘The fight to save Trove’
• 22 June 2016 – Paper at DHA2016 Conference in Hobart, ‘Closed Access’
• 3 June 2016 – Invited contributor to the REMIX Sydney panel ‘Blurring the digital and the physical: How can we add new layers to history?’.
• 2 June 2016 – Invited presentation at the Digital Research Methodologies Forum, La Trobe University, ‘The revolution will not be digital’
• 31 May 2016 – Invited presentation on digital research methods to the Deakin Contemporary Histories Research Group.
• 10 May 2016 – Presented two workshops on ‘Digital Tools and Techniques for the Adventurous Historian’, organised by the History Council of South Australia for the SA History Festival 2016, Adelaide.
• 7 March 2016 – Interview on ABCRN Late Night Live, ‘Treasure Trove under threat’
• 6 March 2016 – Contributor to panel discussing the government’s Innovation Agenda, Electronic Visualisation and the Arts Australasia, University of Canberra.
• 1 March 2016 – Interview with ABC 666 (Canberra) Drive about #fundTrove.
• 1 March 2016 – Interview with RTRFM (Perth) on #fundTrove.
• 19 February 2016 – Invited presentation to symposium on Commonwealth Department of Immigration – Then and Now, La Trobe University, ‘Digital perspectives on the archives of immigration’
• 12 February 2016 – Keynote presentation to ANZREG 2016 (Ex Libris Australia and New Zealand Regional User’s Group), Melbourne, ‘Linked Open Data’.
• 10 February 2016 – Invited contributor to the Gale Cengage sponsored Digital Humanities Panel at VALA2016, Melbourne.

Presented at DHA 2016 in Hobart. Follow my research. View the slides.

A good type

In January 1954, the economic historian Ernst Boehm requested access to Commonwealth government records as part of his research into the depressions of the 1890s and 1930s. The request was forwarded from the Archives Office to the Prime Minister’s Department where it was noted ‘Boehm is I believe a good type + will agree to the usual conditions’.[1]

This sort of character assessment was important because there were few formal guidelines around access to government records. The Commonwealth Archives Committee reported in 1955 that the use of government archives was ‘seriously hampered by the lack of an access policy’.[2] The Prime Minister’s Department noted that requests for access were ‘treated on an “ad hoc” basis’.[3] Prospective researchers first needed to establish their ‘bone fides’. Archives staff would then attempt to identify records of interest and submit them to the relevant government departments for clearance. The Archives Committee noted that the ‘inevitable uncertainties and delays inherent in this system’ often left researchers ‘puzzled and disappointed’.[4]

In the years that followed, interdepartmental agreements and Cabinet decisions gradually refined the process, but it would be another thirty years before the public’s right to access was enshrined in legislation.

Right to Access

In 1955, the Cold War was near its height. The Royal Commission on Espionage was grilling public servants, scientists, journalists and others in its hunt for Communist infiltrators. Amidst the hysteria about ‘secrets’ and spies the Commonwealth Archives Committee emphasised the ‘moral obligation’ of democratic governments to make the records of their activities available for research. ‘In doing so’, the Committee argued, ‘they make a significant contribution to the continuing search for truth which is an essential part of the continuing democratic process’.[5]

The idea that the public had a ‘right’ to access government records did not go unchallenged, however. In a meeting with the visiting American archivist Theodore Schellenberg the previous year, a representative of the Department of Defence had admitted that he ‘found it difficult to accept the concept that the general public had a right of access to public records’.[6]

But the obligations of war and empire actually pushed Australia towards a more liberal policy. In the 1950s, Australia agencies tended to follow British precedent, applying a version of the 50 year rule. Given that the Commonwealth of Australia was founded in 1900, a 50 year embargo didn’t leave a lot of history to write. In 1966, however, the UK government opened access to records through to 1922 – encouraging research into World War I and its aftermath. Australia had little choice but to follow their example. As John Gorton noted in his submission to Cabinet, ‘we should not impose greater restrictions on our own scholars than are imposed in Britain’.[7] When the UK shifted to a 30 year rule a few years later, Australia again adjusted its policy.

In 1972 the Whitlam Labor government brought with it a professed commitment to ‘open government’. Access to archives started to be linked to broader questions around ‘freedom of information’. It took a few attempts, but the Archives Act was finally passed in 1983. In 2010 ammendments reduced the closed period from thirty to twenty years.

The public interest

Under the Archives Act (in its current form) the public has a right to access Commonwealth records more than twenty years old. But this right has limits. In 1954, Schellenberg described the ‘public interest’ as a balance between a liberal access regime and the protections necessary to shield both governments and individuals from the dangers of over exposure.[8] As access policy evolved, so did the articulation of these limits.

Protections for national security and the privacy of individuals have been long accepted as grounds to withhold records from public access. But the scope and meaning of such exceptions has shifted. In 1966, restrictions on ‘the release of papers relating to the personal expenditure of the Governor-General’ were provided as an example of how individuals were being protected from ‘embarrassment or distress’.[9] Similarly it was assumed that records ‘commenting on the person of the Sovereign’ would naturally be withheld.[10]

But the core of these exemptions remained and was codified within section 33 of Archives Act. Section 33(1)(a), for example, allows records to be withheld from access if they could ‘cause damage to the security, defence, or international relations of the Commonwealth’. Section 33(1)(g) exempts records that might result in ‘the unreasonable disclosure of information relating to the personal affairs of any person’.

The ad hoc approach of the 1950s has been replaced by a well-documented bureaucratic process with clearly defined outcomes. Under the Act a researcher can ask for a record to be examined for public access. The contents of the file is then assessed against the list of defined exemptions and an access status is assigned – Open, Open with exception, or Closed (sometimes closed files are described as ‘withheld’).

Statistics on the number of files examined and their access outcomes are reported annually. The vast majority are opened to the public. But some are not.

Open Access

What do we mean by ‘open access’? Back in 1964 the Interdepartmental Committee on Commonwealth Archives defined ‘open access’ as referring to documents ‘which have been cleared for consultation by any member of the public’.[11] Open could only really be understood by invoking its limits – open was not closed.

I’ve shared this potted history of Australian archival policy because it continues to shape the research that we do and what we can know.

These shifting policies and priorities have shaped the data now available online through RecordSearch, the National Archives of Australia’s database. As digital scholars we should recognise that there’s nothing natural or inevitable about these systems and their contents. They are deeply and profoundly historical.

But recognising this is both an obligation and opportunity, because we can view such systems as archaeological sites and dig back through the layers of data. We can meet the history of policy, with an analysis of practice as it has been captured in the descriptions, dates and decisions.

The access status of each item in RecordSearch is recorded and displayed – ‘Not yet examined’, ‘Open’, ‘Open with Exception’, or ‘Closed’. If a file is ‘closed’ or ‘open with exception’ the reasons for the restriction are provided along with the date when the decision was made. This is interesting and important data. But while you can filter your search by status, you can’t specify a reason or date. So I can browse a list of ‘closed’ files, but there’s no way of getting an overview of why they’re closed.

Until now.

Last year, as an experiment, I screen-scraped RecordSearch to capture the details of files that had an access status of closed. On the 1 January this year, I repeated the harvest with a bit more care.

I now have a dataset of 14,370 closed files. These are the files we’re not allowed to see.

Closed Access

You can explore them online using Closed Access, a public site I developed to play with the data.

You can view the aggregated metadata from a number of different angles – by the reason for exemption, by record series, by the age of the contents, and by the date of the decision. You can drill down from any of these views to a list of closed files. And of course you can search – want to know which files were closed in May 1989 for national security reasons? ClosedAccess can tell you!

Given that none of the files are actually publicly available, it’s possibly the most frustrating search interface yet devised. However, you can always ask for a file’s access status to be re-examined.

ClosedAccess is really more of a workbench than a discovery interface, with additional experiments documented in my open research notebook.

There for example, you can read about TF-IDF analysis of file titles, or my failed attempts to reconcile the statistics reported in the National Archives’ annual reports with the data from RecordSearch.

Reasons

So, why have these files been closed? Here’s the list of reasons.

The first thing to note is while there are 17 exemptions defined by the Archives Act, there are 28 reasons for closing files cited in RecordSearch. Some of these extra reasons relate to specific classes of records mentioned elsewhere in the legislation, while others, such as ‘MAKE YOUR SELECTION’ tell us something about the design of the data entry interface.

It’s no surprise that sections 33(1)(a) and 33(1)(g) of the Act are the most frequently cited exceptions. As I’ve noted, these relate to national security and individual privacy – longstanding grounds for withholding access.

More interestingly, these exemptions appear very frequently in combination with others. The DH standard network graph illustrates their centrality. For example of the 425 closed records citing 33(1)(b) – which refers to confidential information supplied by another government – 399 (94%) also cite 33(1)(a). This sort of pattern, combined with the low level of use for many of the exemptions, suggests that the whole legislative framework could be simplified – how many reasons to close files do we really need?

What’s more surprising are the two highly-cited reasons at the bottom of this graph, neither of which are defined in the legislation or anywhere that I’ve been able to find on the National Archives website. They’re ‘Pre access recorder’ and ‘Withheld pending advice’.

Pre-access recorder

According to the brief explanatory note in RecordSearch, ‘Pre access recorder’ denotes ‘non standard reasons for restriction’ – not very illuminating. But using ClosedAccess you can dig into the metadata to get an idea of what this actually means.

The average age of files citing ‘Pre access recorder’ is 75 years, significantly older than the overall average of 58 years. Indeed, if you look at the dates on which decisions were made about these files, you’ll see that most of these were closed before 1983 when the Archives Act was introduced. So it seems these files were examined and withheld from public access under the less formal arrangements that existed prior to the legislation.

This was (sort of) confirmed by the National Archives who told me that yes they were ‘made prior to the introduction of the Archives Act’, but they were also decisions ‘where the reason for restriction didn’t map to one of the reasons for restriction set out in the Act’. I still don’t quite understand this.

If you take all the file titles and throw them into Voyant you’ll see the word ‘staff’ figures prominently, reflecting the fact that many of them are personnel files. Surely these might have mapped to the privacy provisions of section 33(1)(g)?

From there things get even more complicated. Just as in quantum theory, by observing the phenomena of pre-access recorder, I have changed it. As a result of my inquiry, which the Archives seemed to interpret as a criticism of their metadata, they changed the status of most the ‘Pre access recorder’ files from ‘closed’ to ‘not yet examined’. They’re not any more open, but they no longer show up in a search for closed files. Fortunately, I’ve preserved their historical status in Closed Access. I’m planning to repeat my harvest on the 1 January each year, tracking newly closed files as well as those whose status has changed. So next year I’ll be able to check up on the ‘Pre access recorder’ files to see what’s really happened to them.

Withheld pending advice

Withheld pending advice is easier to interpret. The files have been referred back to the government agencies that created or controlled them to check whether they really can be unleashed on the public.

This process is known and feared by researchers because of the long delays it can create. In 2014 Philip Dorling wrote in the Canberra Times about a file from 1932 relating to Australia’s annexation of Antarctica which languished with the Department of Foreign Affairs and Trade for five years before being released in a redacted form.[12]

Delays in access examination are well-known, but the scale and impact aren’t obvious.

I think that part of the problem I was having in reconciling access examination data from annual reports was that the Archives doesn’t include the ‘Withheld pending advice’ files in its statistics. Perhaps this is fair enough – the access examination process isn’t complete. On the other hand, under the Archives Act if examination requests aren’t completed within 90 days then access is assumed to have been refused. Are they less closed because they may yet be opened? I’ve asked the National Archives what the status of these files really is, and how this affects the public’s right to access.

In the meantime we can use Closed Access to see the scale of the backlog and indulge in a bit of agency shaming. You can easily search for ‘Withheld pending advice’ files and order them by the date they were closed.

Ignoring the one file with the title ‘unknown’ that has been waiting 18 years for a decision (I suspect it’s been mislaid), there’s still a file that’s been waiting for 9 years, and 9 files that have been waiting for 7 years. Indeed, you can limit your search to files that have been waiting more than three years for a decision – there’s 1467 of them.

The Defence department gets the longest in limbo award for the 9 year delay, but DFAT is the overall champion with series A1838 containing almost 1500 files awaiting departmental determinations – 54 of which have been waiting more than five years. Too bad if you’re doing a PhD. Researchers are still being left ‘puzzled and disappointed’.

It’s important to make these processes visible because it’s all to easy to forget that the default status for any Commonwealth government file more than 20 years old is ‘open’. The reasons for closing files defined under the Archives Act are called exemptions for this very reason.

Philip Dorling suggested that agencies are becoming more conservative in their decisions. Once I’ve harvested a few year’s worth of closed files, I’m hoping I’ll be able to trace the fate of those poor souls currently ‘withheld pending advice’. But there are other measures of conservatism we can start to explore.

You’d think that the older a file was the more likely it was not to be closed. So what happens when we look at the gap between the date a decision was made to close a file and the date of it’s latest content? I’ve added an experimental view to Closed Access where you can explore the files than seem to have provoked the most conservative approach.

Strangely, at the top of this list are the proceedings of the Senate from 1903. But in fact, no decision was made to close this file. It belongs to a category of records called ‘Parliament Class A’. Regulations drawn up under the Archives Act simply declare these records as ‘exempt’. You need the permission of Parliament to access them.

It’s a reminder that at least we do have the metadata for closed files. The intelligence agencies hold unknown quantities of files where access to the recordkeeping systems themselves is restricted. We don’t know what we don’t know.

I’m hoping to explore these gaps further, but for now if you’d like a copy of the 300,000 digitised pages contained within publicly accessible ASIO files let me know.

No magic

It’s important to recognise all of this as the workings of a system. Digging through the layers of policy and metadata helps us to see that there is nothing intrinsically dangerous in a closed file. A judgement has been made against a list of historically determined criteria.

But it’s also important to resist the temptation to claim privileged access to the truth through our mastery of such systems. Next time you see words like ‘recently released documents’ in a book or article, remember that access examination is a process without magic or wonder. We need to remain critical of the process without adding layers of mystification – without succumbing to the cultural power of the secret.

Presented at the DigitalGLAM Symposium, University of Melbourne, 15 July 2016.
Play with the slides.

Who’s used Commonwealth Hansard through the ParlInfo Database — particularly the early stuff, pre 1980?

Was it fun?

It’s great that all of that content has been digitised, but the user interface is not very friendly — one of the main problems is that it’s just really hard to read anything in context, to get a sense of proceedings in parliament as they unfolded.

However, you can bypass the interface and go straight to the data. There’s an XML file — with the proceedings in a nicely-structured, machine-readable form — for each day.

To save the world much searching and clicking, I wrote a script to crawl through ParlInfo and download all of those XML files. There’s about 4gb in total, and I’ve shared them all through my GitHub account. Feel free to download and explore!

I initially harvested the files because I thought they’d provide a useful example for use in teaching about text analysis tools and techniques. But then I reflected on that ‘readability’ question and wondered how much work would be involved in creating a version of Hansard that was focused on browsing and reading. The answer was about two days — over the space of a weekend I created Historic Hansard.

I wanted to mention this today because I think it’s a good example of a ‘hack’. Contrary to mainstream media coverage, most hackers don’t want to steal your credit card details. Hackers, within the diverse community of coders, builders, programmers, and developers, are people who use code to solve problems, to get around obstacles — their solutions might not be the most elegant, but they work.

My bio reads ‘historian and hacker’. I’m a historian who uses digital tools and techniques to get around obstacles. In particular, I use code to open digital cultural collections.

In the case of Hansard I have, for minimal cost, created something that allows you to explore the proceedings of parliament in context. It’s made some historians pretty excited. Since that first weekend I’ve added an index to people and bills. I’ve also integrated tools for text analysis and annotation. Shortly I’ll add the rest of the House of Reps and the Senate through until 1980. I was very pleased that the new PMs site from the Museum of Australian Democracy was able to harvest from Historic Hansard to boost their content.

So this is an example of what happens when governments or cultural heritage organisations provide openly-licensed data for people to play with — although I should note that the contents of the parliament website, including Hansard, have a ‘no derivatives’ licence which creates a lot of uncertainty around what you can actually do.

I’ve talked a lot in recent years about building stuff with cultural heritage data — particularly in relation to Trove. And I’ve created a lot of tools, experiments and interfaces — some significant, some silly, some downright creepy.

But while open data does enable mildly-obsessed hackers like me to build new, shiny things, it also enables us to ask different questions — to analyse what’s not there, as well as what is.

In the case of Commonwealth Hansard what’s not there is about two years’ worth of Senate proceedings. I stumbled on this while harvesting the XML files and started doing some analysis. Between 1901 and 1980, 94 days of Senate proceedings, and 8 days of the House of Representatives are effectively invisible. PDFs exist, but the XML files are empty and the content is not searchable.

As you can see, most of the ‘invisible’ days come from the WWI period. So if you’ve been relying on the online version of Hansard to research WWI, you probably need to think about what you might have missed.

Needless to say, it’s unlikely that anyone would have noticed these gaps using ParlInfo’s web interface. It’s yet another example of why you always be suspicious of search engines. But it’s also an example of how hacking collections opens them to new forms of analysis and enables us to critique the very notion of access.

Spend sometime reading the strategic plans of cultural heritage organisations and you’ll see that ‘access’ figures prominently — access is ‘provided’, ‘given’, and ‘opened’. Online access is a ‘deliverable’ to be measured in hits and sessions.

The problem with this is it casts the public as consumers of access, rather than creators. Hacking heritage collections is one way demonstrating that access is not a one way flow of information — it’s a struggle and it must always be. We should always be uncomfortable with the categories used to structure our past. We should always want more. If we run out of questions and criticisms then something’s gone badly wrong.

Who’s used the National Archives of Australia?

You’ve probably come across the process of ‘access examination’. Records more than 20 years old are assessed against a list of criteria to see if they can be released to the public — the criteria are defined under the Archives Act and relate to things like national security and privacy.

Most records end up with an access status of ‘open’, but some end up ‘closed’ and are withheld from the public. Last year I started wondering what happens when we flip around the idea of ‘access’ and focus on the files we’re not allowed to see?

The National Archives’ online database, RecordSearch, provides basic metadata about ‘closed’ files, including the reasons why they’ve been withheld. But you can’t easily search or analyse this data within RecordSearch itself. So I wrote another script that harvested details of all 14000 files with the access status of closed and created my own interface to them — Closed Access.

While my Hansard hacking was aimed at opening up new forms of use, Closed Access is about understanding the processes and practices that determine whether we’re allowed to see a file. It pushes beyond legislative definitions in an attempt to reveal ‘access’ as a process that is historically contingent — there is nothing magical, mysterious, or dangerous about a ‘closed’ file.

By hacking collections we can also see them differently. Amongst its millions of records the National Archives holds the remnants of the White Australia Policy. About 5 years ago I wrote a script to harvest several thousands of these documents. I ran them through a facial detection script, cropped out the portraits, and created this seemingly endless wall of faces.

It was another weekend project, another hack. In this case it exposed the remnants of a racist bureaucratic system in a completely different way. Instead of databases, record series, and files, you could see the people inside.

Of course the danger in raving about the wonders of hacking is that we simply create new structures of power, new limits on participation based around coding ability. I don’t believe in hacker elites liberating the masses from ignorance. So part of the struggle around access must also be the struggle to share — to share technology, code, examples, tools, data, knowledge, possibilities, and passions. To give people the opportunity to participate and not just consume.

You might think sharing is easy, but it’s fucking hard work — both for institutions and individuals. And it’s work that is rarely rewarded in terms of professional acknowledgement, advancement or funding.

But I’m inspired by open access activists around the world, by radical librarians setting up Tor exit relays, by digital humanists critically examining unequal access to technology, and exploring the possibilities of minimal computing. By all the people who freely share their work and ideas to make ‘access’ just that little bit easier for others.

I share because I think it’s important, because it’s part of the obligation of being a hacker and historian, academic and activist.

So I share things like:

• articles and examples on my blog
• code and data on GitHub
• current research notes and hacks in my research notebook
• documentation, activities, and tutorials in my digital heritage handbook
• web apps and sites all over the place

I’m also trying to develop a few DIY, low code examples that enable people to build things using Trove data and some freely available tools. I’m hoping that’ll give them the confidence to explore further. A starting point perhaps…

Just last week I ran a free digital heritage workshop in the lead up to GovHack — the national, open data competition. We’re hosting a specially-themed ‘Heritage Hack’ node at the University of Canberra as part of the GovHack this year. Feel free to join us!

Not that any of this is particularly special or remarkable. It should just be what we do — as researchers, as cultural institutions, as citizens, as hackers. The main thing we need to share is, of course, our power.

So let’s just just stop talking about access to cultural heritage as if it’s a destination, and explore it as journey full of tensions, dangers and difficulties; but also full of joy, discovery, and meaning. Let’s hack heritage to build critiques as well as websites, conversations as well as code.

What are the functions of government? You’ll find one possible answer in the Australian Governments’ Interactive Functions Thesaurus (AGIFT) maintained by the National Archives of Australia. AGIFT sets out a standard series of terms to describe common business activities across the different levels of government.

Functions are important to the National Archives — they aid discovery, inform appraisal, and guide government agencies in understanding what records they should keep keep. Functions also offer researchers an alternative way of exploring the Archives’ rich collections — instead of blindly searching for keywords, you can track the activities of government agencies through time.

RecordSearch, the National Archives’ online database, provides a simple way of browsing by function, though it’s not very obvious. Click on the ‘Advanced Search’ tab in RecordSearch and you’ll see an interactive diagram representing the way that records are described and arranged using the Commonwealth Records Series (CRS) system. I made this little widget a very long time ago, and there’s an alternative version, with links to the CRS manual, over at WraggeLabs.

You’ll notice that functions are associated with agencies — they’re a way of describing what government departments do. Click on the ‘Functions’ button to open up a very basic browse interface that lists preferred and alternative names for functions. There seem to be a few differences between AGIFT and the terms used in RecordSearch, I’m not sure why. The CRS Manual includes a summary of the thesaurus used in RecordSearch, and for convenience I’ve extracted a flat list of all the functions.

Just click on any function in RecordSearch to find agencies associated with it. It’s basic, but it does at least provide an alternative to the tyranny of the search box, so well described by Mitchell Whitelaw. If you’re starting a research project and don’t know what keywords are relevant, perhaps following functions will help you map the territory.

Functions also provide a way of exploring how the organisation of government changes to meet social and political needs. How are the government’s priorities in areas such as ‘migration’, ‘education’ or ‘security’ reflected in the sorts of bureaucratic structures they create? To answer this type of question you really need to extract all the data related to a function for further analysis and manipulation, but that’s not something you can easily do through RecordSearch’s web interface. Time for some hacking…

Harvesting data from RecordSearch has been something of an obsession over the past five or six years. Over that time I’ve developed a set of tools that pull structured data out of the web interface. I’ve used these tools to expose records relating to the White Australia Policy, to grab a copy of every publicly available ASIO file, and to analyse records that remain closed to public access. But to harvest functions I needed to extend my toolkit — improving the handling of agency data, and adding a simple search module to find all agencies associated with a function. Once this was done I could build a harvester — a script that just makes a series of requests to RecordSearch until all of the desired records have been retrieved, processed and stored.

All the code for my toolkit and harvester are freely available on GitHub — please download, use and improve! I’ve tried to provide fairly detailed instructions with the harvester, but you will need to fiddle about on the command line.

I haven’t done anything very exciting yet — I just wanted to make sure it was all possible. So far I’ve harvested data about agencies related to ‘MIGRATION’ and its associated functions ‘CITIZENSHIP’, ‘DEPORTATION’, ‘MIGRANT SERVICES’, ‘MULTICULTURALISM’, ‘PASSENGER ENTRY CONTROL’, and ‘REFUGEES’. You can view CSV-formatted summaries for each function on GitHub.

The CSV file for MIGRATION, for example, lists 191 associated agencies. These range from departments of state, such as the current Department of Immigration and Border Protection, Central Office, to smaller bodies like the Committee on Overseas Professional Qualifications. For each agency I’ve listed it’s status, location, life dates, and the dates during it was associated with the migration function. This is only a summary of the data I have for each agency, which includes relationships with other agencies, as well as details of any other related functions. It would be interesting to use this to draw out possible connections between functions.

As a preliminary experiment I thought it’d be interesting to create a simple timeline displaying the different government departments associated with immigration since 1900. I was trying to avoid too much coding, so I ended up hacking Plot.ly’s basic line chart. As a visualisation it’s not very exciting, but as an initial poke at the data it’s quite encouraging.

Hover over the lines to see the agency names and date ranges. Click on one of the agency identifiers (the CA numbers) to open its page in RecordSearch.

The code to create the CSV summaries as well as simple charts like this is all included with the harvester in my RecordSearch Functions repository. I’ll add tools and examples as work proceeds or time allows. Hope you find it useful.

I’m deeply in love with the collections of the National Archives of Australia. They move me, they inspire me, they make me want to do something. How do I express my love? I’ve written stories about things like atomic bombs, progress, astronomy and weather forecasting — pursuing lives and events documented in the Archives’ rich holdings. I work on projects like Invisible Australians, hoping to bring the compelling remnants of the White Australia Policy to broader public attention. And I build things. I make tools that help other people explore, understand and use the Archives. I do this because these riches need to be used. They need to be shared. They need to be part of the fabric of our lives.

A few years ago I created a little script for Firefox that put a fresh face on the display of digitised records in the National Archives’ RecordSearch database. It’s publicly available and has been installed more than 500 times. Demonstrating this script at the ‘Doing our bit’ Build-a-thon a few weeks ago made me realise again both how useful it was and how much work it still needed.

One of the most exciting features when I first created the script was the ability to display the records on a ‘3D wall’, courtesy of a Firefox plugin called CoolIris. But CoolIris uses Flash and is no longer being supported. Time for a new approach.

Say hello to the Archives Viewer (naming things isn’t really one of my strengths). Instead of rewriting my existing script I decided to create a completely new web application. Why? Mainly because it gave me a lot more flexibility. I could also make use of a variety of existing tools and frameworks like Django, Bootstrap, Isotope and FancyBox. Standing upon the code of giants, I had the whole thing up and running in a single weekend. The code is available on GitHub.

What does it do? Simply put, just feed the Archives Viewer the barcode of a digitised file in RecordSearch and it grabs the metadata and images and displays them in a variety of useful ways. It’s really pretty simple, both in execution and design.

Yep, there’s a wall. It’s not quite as spacey and zoom-y as the CoolIris version, but perhaps that’s a good thing. It’s just a flat wall of page image thumbnails with a bit of lightbox-style magic thrown in. But when I say just, well… look for yourself. There’s something a bit magical about seeing all the pages of a file at once, taking in their shapes and colours as well as their content. This digital wall provides a strangely powerful reminder of the physical object.

Of course you can also view the file page by page if you want. Printing is a snap — just type in any combination of pages or page ranges and hit the button. The images and metadata are assembled ready to print. No more wondering ‘which file did this print out come from?’.

But perhaps the most important feature is that each page has it’s own unique, persistent url. Basic stuff, but oh, so important. With a good url you can share and cite. Find something exciting? Tell the world about it! I’ve included your typical social media share buttons to help you along.

1891 draft of the Australian Constitution — with doodles. http://t.co/aCulSUhd

— Tim Sherratt (@wragge) August 23, 2012

One disadvantage over the original userscript is that the viewer isn’t directly linked to RecordSearch. You probably don’t want to have to cut and paste the barcode every time you view a file. So I’ve also created a couple of connectors that ummm… connect things up.

The first connector is just a bookmarklet. A bookmarklet is just a little piece of javascript code disguised as a browser bookmark. Just drag this link — Archives Viewer — to your browser’s bookmark toolbar. Then when you’re on the item page of a digitised file in RecordSearch, just click the bookmarklet and you’ll be instantly transported to the wall.

The second connector is a bit smarter. It’s an enhanced version of another userscript I wrote to display the number of pages in a digitised file. It still does that, but now it also rewrites the links to the digitised files so that they automatically open in the Archives Viewer. It’s a bit harder to install. You need Chrome or Firefox and the add-ons Greasemonkey (for Firefox) or Tampermonkey (for Chrome). Then just go to the userscript page and hit the big ‘Install’ button.

You might be wondering about Zotero (at least I hope you are). My Zotero-RecordSearch translator lets you capture page images and metadata direct to your own research database, so what happens when you’re transported across to the Archives Viewer? Never fear, I’ve written a new translator that lets you save pages as you could in RecordSearch. Even better, you get a persistent, context-enriched url, and the ability to capture multiple pages at once. Yippee!

But that’s not quite all. Buried within the pages is some lovely Linked Open Data. To be truthful, it’s not really very ‘linked’ yet, but it does expose the basic metadata in a machine-readable form, borrowing from the vocabularies of projects like Locah and the Archival Ontology. It’s an experiment, as is the Archives Viewer itself. We can learn by doing.

Makin' triples from dhistory Archives Viewer RDFa: file – http://t.co/D0xCehjr & page http://t.co/gSd87cRX

— Tim Sherratt (@wragge) August 26, 2012

I’ve given quite a few talks over recent times encouraging people to take up their tools and start hacking away at the digital collections of our cultural institutions. Yes, I admit it, I’m an impatient historian (and a grumpy one at that). But it’s also because I think it’s important that we recognise that access is never just something you’re given. It’s something that we make through our stories, our projects, and our tools. It’s something that’s grounded in respect and powered by love.

I tell the full story of the newspaper’s campaign in Inigo Jones: The Weather Prophet. but I’ve always wanted to do something more with the letters. Inspiration finally arrived this year when a conference on rural media was organised to mark the The Land‘s centenary. I decided to create a little exhibition using the letters and, in doing so, provide myself with a platform for further research.

Step one: assembling the letters with Zotero

I’d already asked for the file containing the letters to be digitised, so images of all of the letters were available through RecordSearch. Zotero ships with a RecordSearch translator that I wrote some years ago, so 102 clicks later, I’d captured both the metadata and all of the images into my own database.

The Zotero translator saves individual pages with a generic title, so I then had to do some manual editing. I also had partial transcripts of the letters which I’d created during my original research. I simply cut and pasted the transcripts into a note field in Zotero and tidied them up. Pretty soon my collection was complete.

Step two: creating an exhibition with Omeka

I actually started building the exhibition during an Omeka Bootcamp session I ran at THATCamp Melbourne. Starting with nothing but a LAMP server and my Zotero library, I had the basics of site up within the hour.

It was simply a matter of installing Omeka, adding the Zotero import plugin, creating an API key in my Zotero profile, and then pointing the Zotero import plugin to my collection. (The API key is necessary if you want to download files, such as my images.)  With a press of a button the internets started chugging away and pretty soon all the letters were in Omeka. Hey presto — instant exhibition!

With the framework built I could start to play a bit. First I installed Omeka’s geolocation plugin and started mapping where the letters had come from. Google’s geocoder did a pretty good job of finding many of the small country towns, but I also made use of Geoscience Australia’s gazetteer. In the end, I managed to geolocate all but one letter. Hey presto — a map!

I thought the pre-installed ‘Summer’ theme suited my exhibition, but I did make a few minor tweaks here and there. This included adding some RDFa into the list of references (which were also imported from Zotero).

Step three: going deeper with Voyeur Tools

The transcripts of the letters were all available on the site, but I thought it would be interesting to analyse their content a bit more systematically. To do this, I exported the collection from Zotero and wrote a few lines of Python code to save each transcript in a separate text file, named after the letter’s author. I could then zip up the transcripts and feed them to Voyeur Tools.

Most of the letters are pretty short, so there’s a limited amount to be gleaned from them in isolation. Voyeur Tools did, however, make it easy for me to explore further the prevalence of the phrase ‘on the right track‘.

What will be more interesting will be to compare the letters to other bodies of text about Inigo Jones, weather forecasting and rural life. I’m hoping to start mining some of this sort of material from the Trove newspapers database.

Next steps?

Using Zotero , Omeka and Voyeur Tools it was quick and easy to build my own little research hub. It’s still a work-in-progress of course — that’s the point! Now I can continue to assemble and analyse my sources, while giving my work a public face. Magic!

Here’s the slides from the talk I gave recently in the Scholars’ Lab at the University of Virginia. Thanks to everyone who came and to the Scholars’ Lab for their hospitality.

A podcast version is now available.