Time to take stock of the year that was…

2017 certainly had its ups and downs. In the middle of the year, my university’s management decided that they could do without undergraduate teaching in cultural heritage. We managed to hold on, but it’s the sort of thing that makes you wonder why you bother.

On the flipside, the highlight of the year was working with my undergraduate class in Exploring Digital Heritage to create and promote a site for the transcription of records used in the administration of the White Australia Policy. It’s something Kate and I have been talking about for many years, so it was a rather emotional experience to see it finally happen. But it wasn’t just about the website — it was about the way my students responded to the records (many of them knew little about the White Australia Policy before we started); it was about seeing the records on the ABC news; and it was about seeing the faces of the people who lived under the weight of the White Australia Policy projected onto the walls of Old Parliament House. That’s why we bother.

Also in 2017 (and somewhat related to the above) I decided to adjust my working arrangements in attempt to find some balance. You can now find me over on Patreon.

The making

• Closed Access 2017 update! Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 9 January 2017.
• Wrestled with RecordSearch to extract summaries for all series. Along the way I found out some interesting stuff about the use of functions. Putting the two together, I tried to build a picture of what NAA holdings look like through the perspective of government functions.
• Played around with Twarc to create a collection of #invasionday tweets. Later in the year I captured and shared tweets with the tag #australianvalues.
• Have you ever wondered how the cut and thrust of parliaments past might translate to the world of social media? Wonder no longer, for here you can explore interjections in the Australian parliament from 1901 to 1980, reimagined as tweets. You might even find some emoji…
• Code and work-in-progress website exploring the language of Hansard.
• Went back to basics with LODBook — now developing a series of components using Jekyll and javascript.
• The Redaction Zoo — this collection of creatures was discovered amidst thousands of ASIO surveillance files held by the National Archives of Australia. While the practice of redaction is intended to withhold information from public view, an unknown archivist has used redactions to add an artistic flourish to the files. They are reminders that the processes that limit our access to information are human in their operation and design. There is nothing magical about the ‘secrets’ preserved in government archives. Video first exhibited as part of ‘Beauties and Beasts’, Belconnen Arts Centre, 6-28 May 2017.
• Here’s a repository of images in JPG and SVG format drawn from a collection of #redactionart discovered in ASIO surveillance files held by the National Archives of Australia. Use them to create your own #redactionart projects!
• I finally added a decent full-text search facility to Historic Hansard. Yay! Search for speeches or bills. Filter by date, house, and speaker. There’s even an option to save your complete result set as a CSV file for further analysis and exploration.
• The Tribune was a newspaper published by the Communist Party of Australia. The State Library of NSW holds more than 60,000 negatives and photos from the Tribune which document a wide range of political events and social issues from 1964 to 1991. This is a work-in-progress site documenting my exploration of the negatives as a DXLab Digital Drop-in.
• Help transcribe records that document the lives of ordinary people living under the restrictions of the White Australia Policy. The Real Face of White Australia site was built using the Scribe Framework with the help of students from my Exploring Digital Heritage class.
• Explore regular updates of data generated through the Real Face of White Australia project. Includes CSV files with transcribed fields, as well as photos and handprints.
• There’s lots of exciting new digitised content being added to Trove’s journals zone, but it’s not always easy to find and search. I made an app that lists journals that have been digitised by the NLA and have searchable records for individual articles. This means you can search inside the journal, just like you do in the newspapers zone. The code for the harvester and web app is on GitHub.
• Demonstrated how to harvest Parliamentary press releases from Trove. The repository includes a sample dataset of politicians talking about refugees.
• Kicked off my new 101 Digital Heritage Hacks site with a userscript to add a ‘copy permalink’ button to Trove.
• I made a little bot to post stories from the Real Face of White Australia data to the @invisibleaus Twitter account.
• Created two new DIY Trove bots using Glitch. The Trove Tag Bot tweets items with a specific tag, while the Trove List Bot tweets items from one or more lists. Both bots come with detailed instructions and, because they’re hosted on Glitch, it’s easy for anyone to modify the code and launch their own bots.
• Added a battle mode to Headline Roulette. It’s really just a way of generating a url to specific newspaper article in Trove. This means groups can compete on a level playing field, but it could have other uses. You could create a curated set of challenges for students, or use them in something like a scavenger hunt.

The talking

• 13 February 2017 – Digital humanities workshop at ALIAOnline.
• 24 February 2017 – Keynote presentation at HDR Summer School, Deakin University, The practice of play, <https://doi.org/10.6084/m9.figshare.4696258>
• 1-3 March 2017, attended ‘Always Already Computational: Library Collections as Data’ national forum in Santa Barbara. Read the position papers and the final statement.
• 20 April 2017 – Public lecture and workshop, University of Wollongong.
• 4 May 2017 – Presentation with Professor David Lowe, ‘The Political Language of the 1970s in Australia’, at The 1970s: Australian and Indian Perspectives on a Decade of Transition, Deakin University.
• 23 June 2017 – Workshop, Random acts of meaning: Digital skills for a post-truth world, NLS8.
• 4 August 2017, ‘The struggle for access’, presentation at DHPathways 2017, Canberra.
• 10 August 2017 — ‘Doing DH’ presentation for ACT teacher librarians.
• 13 October 2017 — ‘Trove tips and tricks’, at the Deniliquin Family History Expo.
• 8 November 2017 — ‘Making this happen’, presentation at Collaborating around Collections, ANU.
• 20 November 2017 – ‘”The badge of the outsider”: Open Access and Closed Boundaries’, invited presentation at Sharing is Caring 2017, Aarhus. Also put together a last minute ‘Hack the world’ workshop.
• 4 December 2017 — Digital methods workshop for the UTS Legal History group

Keynote presentation at the Deakin University Faculty of Arts and Education HDR Summer School, Geelong, 24 February 2017.

Cite as: Tim Sherratt, ‘The Practice of Play’, presented at Deakin University Faculty of Arts and Education HDR Summer School, Geelong, 24 February 2017, online at <https://doi.org/10.6084/m9.figshare.4696258>.

I’m a historian. But in the past decade the nature of my research has changed quite profoundly. Instead of heading off to the archives, taking lots of notes, and writing up a book or an article, I now make things. Generally these things are online, and open to the public. I make things for people to use, to explore, to play, and to ponder.

I started down this track before I realised there was a name for what I do – practice-led research. The things that I make even have their own acronym – they’re NTROs, or Non Traditional Research Outputs.

But practice-led research is not just about making things. New knowledge is generated through cycles of creation and reflection. My aim in making is not to follow a blueprint, or check off a list of requirements, but to end up asking ‘What is this thing?’, ‘What does it do?’, ‘How does it do it?’.

In the past, I’ve tended to talk about my research practice as playing with data. I think there’s an important argument to be made for the role of play in research, particularly when confronted with large cultural datasets. But ‘play’ doesn’t quite capture what I do, nor does it look very convincing in a research proposal. So what do I really do?

Let’s play a game.

Headline Roulette is a very simple game. Presented with the title of a digitised newspaper article drawn at random from Trove’s collection of more than 200 million you are challenged to guess the year in which the article was published. Sounds easy, but you only get ten guesses. It’s sort of like a cross between hangman and The Price is Right.

Despite its simplicity, I’ve known it to unleash the competitive instincts of a workshop full of historians. But for me, Headline Roulette is important because it provides an example of what becomes possible once we make cultural heritage collections available online. Our interactions are no longer limited to conventional modes of viewing or reading – we can play, and we can build.

I made the first version of Headline Roulette back in 2010. It was a game, but it was also an argument about access and possibilities.

Perhaps we should first take a step back. Who’s used Trove?

Trove is a fundamental part of Australia’s research infrastructure – and not just for those of us in the humanities or social sciences. Trove is a lot more than digitised newspapers, but access to more than 150 years worth of digitised newspapers has profoundly changed historical practice.

I say this not just because you have been spared the pain and suffering wrought by microfilm readers upon a generation of historians, but because the meaning of access itself has changed. Headline Roulette is just one simple and silly example of how once cultural heritage resources are in digital form we can use them differently. We can see them differently.

Imagine your search in Trove’s newspapers zone returns 10,000 or 100,000 results. How do you make sense of that? How do you get an understanding of the whole, when all you see is page after page of search results?

QueryPic extracts data from Trove to visualise your search as a single chart – showing you the number of articles per year that match your query. You can even compare the occurrence of particular words or phrases.

But that’s only the beginning, because once you think about web resources as data rather than just another type of publication you can aggregate and analyse – you can look for big, dramatic pictures as well as tiny, fragile fragments.

Trove Harvester is tool that delivers historical newspaper articles in bulk – thousands, even millions of articles saved to your computer for offline exploration.

What might you do with a million newspaper articles?

Research using digital resources like Trove is not constrained to the window of your web browser. You can ask new types of questions.

But back in 2010–11 when I created the first versions of Headline Roulette, QueryPic and the Trove Harvester there was no easy way of getting data out of Trove. The thing is, web pages are good for delivering data to human beings, but not so good for computers. Computers are actually pretty dumb, and you need to be quite explicit in packaging up data for them. Nowadays Trove has a thing called an API (an Application Programming Interface) which delivers data in a carefully structured format that even computers can understand. You can use APIs to harvest data, or to build new tools or interfaces. APIs are cool.

Without an API, the first versions of my tools had to turn human-readable web pages into computer-readable data – a process known as screen scraping. They were, therefore, not only useful or interesting applications in their own right, they were arguments about why things like APIs matter. Why web pages aren’t enough. Why researchers need access to data.

These are arguments we’re still making. Next week I’m heading to a workshop in California where we’ll be discussing how libraries and other cultural institutions can deliver their data in ways that support new forms of research.

But we don’t have to wait. By screen scraping web pages, by reverse engineering online databases, we can continue to develop the argument for access by extracting, sharing, and using data.

What could you do with 70gb of digitised surveillance files from the Australian Security Intelligence Organisation (ASIO)? I’f you’d like a copy I have them here on a USB drive.

Don’t worry – we’re not about to be raided by the security services. These are all files that have been carefully examined and released to the public through the National Archives of Australia. You can find them by searching the Archives’ online database – RecordSearch.

Who’s used RecordSearch? It’s not the most friendly system, but the collection it documents, and the metadata it provides, is rich and wondrous. I’ve spent a lot of time trying to get useful data out of RecordSearch – not just ASIO files, also records documenting the administration of the White Australia Policy, as well as higher-level data aimed at building my understanding of how the Archives, and its descriptive systems, actually work.

It is painful and frustrating work. But, I would argue, it is research. Terms like ‘data mining’ and ‘text mining’ fly around all the time, making it seem as if the the accumulation of data is a mechanical process – as if we’re just digging it up. But the practice of screen scraping, or of liberating data from any cultural heritage source, is not simply extractive – it’s iterative and interpretative. It’s a process through which you begin to understand how the data is organised, what its limits and assumptions are, what its history is. What it means. We’re not just taking things out, we’re putting them back.

Frederick Gibbs and Trevor Owens argue that historical data need not be deployed solely as statistical evidence. ‘It can also help’, they suggest, ‘with discovering and framing research questions’ – questions, not answers; interpretation not calculation. Gibbs and Owens describe an ‘iterative interaction with data as part of the hermeneutic process’.

For me, RecordSearch is like an archaeological site. Excavating data from it involves digging through layers of technology, institutional history, and descriptive practice to try and understand why we have what we have.

Those of you undertaking projects using the collections of the National Archives will almost certainly come across the process of ‘access examination’. Under the Archives Act, government records more than twenty years old are expected to be opened to the public. However, the act also defines a number of exceptions to this rule – for example, records that endanger national security or infringe an individual’s privacy can be completely, or partially, withheld from scrutiny. The process of assessing records against this set of exemptions is called ‘access examination’.

The vast majority of records are opened without problem – they are, after all, more than 20 years old. But a significant number are not. While you can’t use these records, RecordSearch does provide some information about them. So I decided to see what we couldn’t see.

In January 2016 I fired up my screen scraper and harvested details of all the files in RecordSearch that have the access status of ‘closed’ – there were 14,370 of these files that had been through the process of access examination and withheld from public view. I then created my own interface that lets you explore this data from a variety of angles – such as the reasons why files were closed, when decisions were made about them, how old they are, and which government agencies created them.

It is perhaps the most frustrating search interface ever devised, given that you’re not allowed to see any of the files you find.

Those of you currently planning research projects might be interested to know where most of these files come from. It’s not defence or the intelligence agencies, but what is now the Department of Foreign Affairs and Trade (DFAT) – in January 2016, there were 1,747 closed files from just one DFAT series. But if you dig deeper you see that most of these files aren’t withheld for one of the reasons defined by the Archives Act, they are described as ‘closed pending advice’. The National Archives is still waiting to hear back from DFAT about them. Using my interface you can see that there were 54 files in this series where the Archives has been waiting for more than five years. So if you’re embarking on a project using the National Archives, make sure you get your access examination requests in early. Just in case.

My aim in extracting and sharing this data is to better understand access examination itself as a historical process. It’s work that enables us to ask different types of questions, but it also makes a change in the process itself. My interface is public, offering a critical commentary on the ‘official’ system. As a result of my research, the Archives has made changes to the way it describes closed files. It’s both research and intervention, history and hack.

‘Hack’ has a number of definitions, both positive and negative. Mark Olsen describes the ‘hacker ethos’ as:

‘a way of feeling your way forward through trial and error, up to and perhaps beyond the limits of your expertise, in order to make something, perhaps even something new. It is provisional, sometimes ludic, and involves a willingness to transgress boundaries, to practice where you don’t belong… Whether eloquent or a kludge, a hack gets things done.’1

Olsen explores what hacking means in the context of the humanities, arguing not only that hacking has a legitimate place in humanities practice, but that the humanities itself needs to be hacked to foster the development of new skills and literacies.

At this point you’re probably thinking, ‘But I don’t do any of this wacky digital stuff, what has this got to do with me?’

Who’s heard of filter bubbles, or search personalisation? Who’s read one of the many reports recently about the way computer algorithms are shaping our online experience? Olsen argues for a humanities practice that equips us to wrestle with complex techno-social systems.

And we’re not just talking about Google.

Last year Matthew Reidsma published an analysis of algorithmic bias in library discovery systems. He hacked a common commercial library product to show some of the biases underlying its recommendations system. The interfaces we use to access information are never neutral. The databases we search are products of selection and exclusion. Hacking enables us to interact with these systems as critics, and not just consumers.

Using the Trove API you can create a chart showing the number of digitised newspaper articles available per year from 1803 onwards. If you do this, you’ll notice two significant features. First, there is a dramatic drop-off in the number of articles after 1954. This is the ‘copyright cliff of death’. Few things are certain in our overly-complex copyright system, but 1954 provides a practical cut-off point. History stops in 1954.

You’ll also notice a substantial peak in the number of articles around 1914. Why might this be? Did something significant happen in 1914?

In fact, it’s all about money. In the lead up to the centenary of WWI it was decided to focus limited digitisation resources on newspapers from the WWI period. It was a perfectly reasonable decision, but the consequences are effectively invisible to any user of the web interface. You don’t know what you’re searching.

The power of Google encourages us to put a lot of faith in search interfaces. We trust that they will just work. And if we can’t find what we’re looking for, we often assume that it doesn’t exist.

Hansard, the recorded proceedings of the Australian parliament from 1901 can be searched using the ParlInfo database on the Australian Parliament House website. Perhaps you’ve used it – it’s a wonderfully rich resource. Powering the search results are a series of well-structured XML files, one for each sitting day, that identify individual debates and speeches.

Last year I reverse-engineered ParlInfo and harvested all those XML files. I thought they’d provide a great dataset for exploring changes in political speech, and so I created a repository containing all the files for the House of Representatives and the Senate from 1901 to 1980. Feel free to download and play.

But in the process of harvesting the files I noticed that some of the XML files were empty. After a bit more analysis I realised that about 100 sitting days were missing – they didn’t show up in search results on ParlInfo.

The ‘missing’ days were concentrated in the Senate between 1910 and 1920. So anyone relying on ParlInfo to research the WWI period would have missed significant amounts of content. This ‘black hole’ was effectively invisible to any user of the web interface. It was only though hacking that its shape and extent was revealed.

Fortunately staff at the Parliamentary Library have investigated and fixed the problem. But it’s a good example of why we should, as researchers, start from the assumption that search interfaces lie. Processes of selection and description shape the ‘reality’ of online collections. We then explore them through complex technological systems that appear comprehensive, even when they are not. You can’t find what’s not there. Online collections hide as much as they reveal.

Of course this is true of all historical sources. We are trained to analyse both context and content, to make judgements about authenticity and accuracy. These same skills need to applied to digital resources, to data. Indeed, Gibbs and Owen argue that ‘historians must treat data as text, which needs to be approached from multiple points of view and as openly as possible’. But how do we find multiple points of view when interfaces construct our experiences and limit our perspectives. How do we open data to new possibilities? How do we see data differently?

No doubt you’ve been encouraged to find a way of expressing your research questions succinctly, in a way that communicates with a non-specialist audience – yes, I mean the dreaded elevator pitch. You’re not the only one.

I’ve landed back in academia after a number of years working in cultural heritage institutions, and pursuing my own research interests with the support of the international digital humanities community.

Believe me when I say, Twitter changed my life. There I was, hacking away on cultural heritage data without any real assistance or encouragement, when I discovered, via Twitter, that there were people out there like me. Many of these people are now my friends, and I’ve been lucky to travel around the world to meet and work with them.

But coming back to academia I’ve found that my collection of projects, tools, experiments, and obsessions was not quite enough – my research needs a ‘narrative’.

So, like you, I’ve had to think about why I do what I do. What motivates my research? What matters?

For me it comes back to the nature of this thing we call ‘access’. Cultural heritage organisations talk about ‘access’ all the time, particularly in relation to online collections. But what does it actually mean? I want to overturn our assumptions about access – exploring it not as a process of opening things up, but as a system of controls and limits. It’s not a state of being, it’s a struggle for meaning and power.

My methodology, and I think I can call it that, is the multiplication of contexts. Context is, of course, critical to cultural heritage collections – it enables us to locate them within history and culture, to analyse their authenticity, to mobilise their value as evidence. But the descriptive systems we use to manage and explore collections represent only a privileged subset of possible contexts.

Now I’m still figuring this out, but I think what my work does is that it removes collections from these highly-controlled systems and lets them loose in a variety of new contexts. This allows unexpected features, or new uses, to emerge – we see them differently, and in that moment, the nature of access shifts, however slightly. It’s those moments I’m trying to catch and observe.

If you’ve ever tried to use Hansard through the ParlInfo database you’ll realise that it’s just really difficult to read. You’re presented with a series of nested fragments, so it’s hard to get a sense of the context and flow of the day’s proceedings. Having downloaded all those XML files, I thought I’d have a go at presenting Hansard in a form that privileged reading over search.

So I created Historic Hansard – dedicated to lovers of political speech. It does nothing very fancy, but I think it does it pretty well.

In the end, however, Hansard is still just text. What’s lost in the documentation process is the performance – the theatre of parliament. But not completely. As well as formal speeches, many interjections have been recorded and preserved.

A few weeks ago I extracted all those interjections from 1901 to 1980, about a million of them, and saved them to a new database. As I fiddled with different presentation methods, I started to see them as something akin to tweets – quick, pithy, and pointed. What would happen, I wondered, if we reimagined interjections from a century ago in an age of social media.

Like many of my projects, this whatever it is took me a couple of days to build. No research grants were harmed in its creation, no committees were needlessly formed. This is not because I’m a whizz-bang coder – I’m certainly not. It has to do with the nature of this work – it’s rapid, experimental, and sometimes even ephemeral. I don’t design websites, I make interventions – things that are not only of the world, but in the world. They do something.

Stephen Ramsay explores the hermeneutical possibilities of screwing around with technology and texts. The ‘screwmeneutical imperative’ he suggests is based on the fact that:

’a writerly, anarchic text… is more useful than the readerly, institutional text. Useful and practical not in spite of its anarchic nature, but as a natural consequence of the speed and scale that inhere in all anarchic systems.

Digital technologies give us the opportunity to play with scale and speed. We can manipulate millions of newspaper articles, and we can build a new version of Hansard in a weekend. But this shift also applies to the way we communicate. Instead of waiting months or years for an article to appear in print, we can post it on a blog, or in a digital repository. It is fundamental to the work that I do that it is shared, it is public by default – not just the results, but the code, the data, the process, and yes the licensing. Access is not just what we take, it’s what we do.

The multiplication of contexts has some interesting precedents as a research methodology. In the literary world the Oulipo movement sought to play with the contraints of composition. Lisa Samuels and Jerome McGann suggested that the deliberate misreading of a text, what they termed ‘deformance’, could yield critical insights. More recently, Mark Sample has argued for a ‘deformed humanities’ where we learn about things by breaking them.

In history we have the counterfactual – a creative reimagining of a past that never was, aimed at revealing perspectives and possibilities too quickly closed and forgotten. As Sean Scalmer argues, ‘counterfactuals are fun’:

‘Conventions can be disregarded, or even mocked. Worlds might be remade, the tyrannical overthrown, and the humble elevated. New orders can be imagined.’2

But counterfactuals are not fiction. They work best when they sail close to an accepted version of the past; when they play with the constraints of documentary evidence rather than just ignore them. Just because an approach is playful, it doesn’t mean that there are no rules. As Ian Bogost has recently argued, the fun of play is ‘not doing what we want, but doing what we can with what is given’.3 Play is an investigation of limits.

While some of the ASIO files held by the National Archives are closed to the public, most are ‘open with exception’. This means that sensitive parts of the files have been removed. Whole pages can be withheld, or sections of text blacked out – a process known as redaction.

A redaction is, by definition, an absence of information, and yet the frequency, density, and placement of redactions across a large collection of documents could conceivably tell us something interesting. So last year I wrote a kludgy computer vision script that found and extracted redactions from digitised ASIO files. I now have a collection of 250,000 redactions which I’ve shared on Figshare – grab a copy now!

I’m continuing to explore the possibilities of these redactions as data points. But there was also something visually interesting about the redactions, particularly when they were assembled on masse.

Here you can browse all 250,000 redactions. But that’s not all, you can also use them as entry points to the documents they were intended to obscure.

Contexts here have been reversed, the files have been turned inside out – the limits remain, indeed the scale of redaction is emphasised, and yet within these limits, perhaps even because of these limits, we can experience the files quite differently. We are no longer simply the subjects of state surveillance, we can reverse the gaze, inspect the process, and ask new questions.

The manipulation of contexts is not mere invention. The limits of access offer both meaning and rules. We have skin in this game, its outcomes matter, what is at stake is our ability to see, and be seen, within the cultural record. Access changes who we can imagine ourselves to be.

In the first half of the twentieth century, if you were deemed not to be ‘white’ and wanted to travel overseas from your home here in Australia, you had to carry special documents. Without them, you’d probably be stopped from returning – from coming home. This was ‘extreme vetting’ White Australia style.

Many thousands of these documents are now held by the National Archives of Australia. In 2011, I used my screen scraper to harvest about 12,000 images like this from RecordSearch. I then ran them through a facial detection script and created The Real Face of White Australia.

There are about 7,000 faces in this seemingly endless scrolling wall. And that’s from just a small sample of the White Australia records. It’s powerful, compelling and discomfiting. But the power comes not from any technical magic, but from the faces themselves – from what we feel when meet their gaze. Once again the records have been turned inside out – instead of seeing files, metadata, or a list of search results, we see the people inside.

Play can be serious. It can make you feel things you don’t expect. It can challenge your certainties and take you to the limits of what you know.

That sounds a lot like research to me.

1. M. J. Olson, ‘Hacking the humanities: Twenty-first-Century literacies and the “becoming other”of the humanities’, in Eleonora Belfiore and Anna Upchurch (eds), Humanities in the Twenty-first Century: Beyond utility and markets, Palgrave Macmillan, 2013, pp. 237–250.
2. Sean Scalmer, ‘Introduction’, in Stuart Macintyre and Sean Scalmer (eds), What if? Australian history as it might have been, Melbourne University Press, Melbourne, 2006, pp. 1–11.
3. Ian Bogost, Play Anything: The Pleasure of Limits, the Uses of Boredom, and the Secret of Games,Basic Books, New York, 2016, p. 236.

Contribution to a panel discussion on ‘Access and Innovation’ at Digital Directions 2016, Canberra, 10 November 2016.
View the slides.

Please cite as: Tim Sherratt, ‘Caring about access’, presented at the Digital Directions, 2016. <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>

You might have seen that the Department of Prime Minister and Cabinet has opened discussion on an Open Government National Action Plan. Last week I was watching the Twitter stream from a briefing event in Melbourne, when I saw this tweet from Asher Wolf quoting the PM&C spokesperson:

. @pmc_gov_au: we don’t want ppl to search for flaws or try to crack gvt datasets. #OGPAU

Now I’m not sure of the context of this statement, but it is a reminder that we can’t take the meaning of words like ‘open’ or ‘access’ for granted.

They are what we make of them.

I’m a historian and hacker. I don’t steal credit card details, I use digital tools to open cultural heritage collections – to see them differently, to feel differently.

Earlier this year I wrote a script to harvest many gigabytes of parliamentary proceedings – Hansard – from the ParlInfo database maintained by the Department of Parliamentary Services. I shared the files, covering the period from 1901 to 1980, through my GitHub repository. My plan was simply to make this rich source of political history easily available to anyone who wanted to explore new forms of digital analysis.

In the end I created my own version of Historic Hansard, one that privileges the experience of reading rather than searching. I also added a few nifty features such as indexes of legislation and people, and integration with text analysis and annotation tools.

But in the process of harvesting the data I noticed that around 100 sitting days were missing from ParlInfo. The files were empty – unable to be searched. Given an average of about 50 sitting days a year, that’s about two years worth that were simply invisible.

Thanks to the Parliamentary Library these problems have now been fixed. But this experience highlighted two issues. First, search interfaces lie. We have to develop our critical capacities to be more aware of what we are not being shown and why. And second, this problem only became visible because I was hacking their database, because I went beyond what was offered by their website, because I wanted more. Unlike the PM&C spokesperson I think one of strongest reasons for opening data is to encourage people to find its flaws. This is particularly the case for cultural heritage data where the processes of selection, representation, normalisation, control, and preservation all have an impact of the types of stories we can tell about ourselves.

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

A few years ago Merete Sanderhoff, from the National Gallery of Denmark, compiled a collection of essays on opening cultural heritage collections for reuse online. It was called Sharing is Caring. I think it’s worth reflecting on the dual meanings of ‘caring’ – both looking after, and giving a shit. Sharing helps us foster communities and preserve collections. But it also matters, it has an impact, it can change the world.

Bethany Nowviskie, the Director of the Digital Library Federation, recently gave a talk on ‘speculative collections’ asking how we shift the temporal orientation of our libraries and archives away from a closed and linear past towards an exploration of what might be.

Seeing differently. Feeling differently.

This is a visualisation created by Geoff Hinchcliffe showing more than 12,000 tweets since February this year using the hashtag #fundTrove. Trove has radically changed our access to the past. It’s not perfect, it’s not everything, but it changes lives. And yet the government seems to think it can be left to wither, and nobody will really care. I care. Do you? Then what do we do about it?

I also care deeply about the collections of the National Archives – a love expressed through many painful hours spent hacking RecordSearch, their online database. Last year I harvested details of more than 12,000 publicly available ASIO files – you can grab the results from my GitHub site – and downloaded about 300,000 page images. These are mostly surveillance files, documenting the lives of writers, academics, unionists, Indigenous activists and others – identified as potential threats to the nation.

This year I wrote a script to search those images for redactions – sections of text blacked out for security reasons. If you’d like to explore the 239,000 redactions I’ve extracted, you can – marvel at their blackness, their very lack of information. But here the redactions are not dead ends, they’re starting points, ways of exploring the workings of ASIO. As the power and reach of state surveillance continues to expand, we can find creative ways to reverse the panoptic gaze.

We don’t have to accept what we’re given. We can take collections and turn them inside out.

The National Archives also preserves remnants of Australia’s racist past. There are many thousands of records documenting the workings of the White Australia Policy. They include special certificates, with portrait photos and handprints, which were needed if you were deemed non-white, travelled overseas, and simply wanted to come home.

About five years ago I harvested about 12,000 images of these certificates from RecordSearch and ran them through a facial detection script. In a little over a weekend I’d created the Real Face of White Australia – a seemingly endless scrolling wall of more than 7,000 faces. It’s compelling. It’s uncomfortable. But this is who we are.

As I saw the One Nation senators cracking open the champagne outside Parliament House yesterday, I thought again of how important its is the tell these stories, to share these collections. To give a shit.

Innovation can be measured in more than shiny apps, or cool new visualisations. By struggling with access to our past we can imagine new futures.

Keynote presented at Forging Links, the Annual Conference of the Australian Society of Archivists, Parramatta, 19 October 2016.

Please cite as: Tim Sherratt, ‘Turning the inside out’, presented at the Australian Society of Archivists Annual Conference, Parramatta, 2016, <https://dx.doi.org/10.6084/m9.figshare.4055013.v1>.

This is RecordSearch, but not as you know it.

This is my hacked version of RecordSearch, the online collection database of the National Archives of Australia. Unlike the regular version it displays the number of pages in each file. But more interestingly, if you search in Series ST84/1 you see more than metadata – you see the people inside.

As Barbara Reed noted in her article ‘Reinventing access’, ‘records are imbued with people’. Series ST84/1 goes by the fairly benign title, ‘Certificates of Domicile and Certificates of Exemption from Dictation Test, chronological series’. But of course the Dictation Test was the administrative backbone of a racist system designed to exclude people who did not fit the widely-accepted vision of ‘White Australia’. ST84/1 is full of people just trying to live their lives under the weight of the White Australia Policy.

The certificates in ST84/1 allowed people, born or resident in Australia, to return home after travelling overseas. If your ‘whiteness’ was suspect and you had no certificate, you would be subjected to the Dictation Test, and you would fail. The certificates usually include photographs and handprints – they are compelling and confronting documents. But you have to dig through layers of metadata in RecordSearch to see that. Or do you?

About five years ago, Kate Bagnall, a historian of Chinese Australia, and I were thinking about ways of drawing attention to these records. In a little over a weekend, I harvested about 12,000 page images from ST84/1 and ran them through a facial detection script. The result was ‘The Real Face of White Australia’.

You may have seen it before. It’s had a remarkable life, travelling around the world as an example of how we can use digital tools to see records differently. But of course the power is in the faces themselves – in the connections we make through time. We cannot escape their discomfiting gaze.

You may think that the certificates in ST84/1 are merely a form of identity document. But remember than in the early years of the 20th century, passports were still evolving, and the use of photographs and fingerprints for identification were generally confined to prisoners and criminals.

The sociologist Richard Jenkins talks about identity not as an essence, a noun, but as ‘something that we do, a process of identification’.1 But self-identification is constrained by broader systems of categorisation, or ‘social sorting’, that decide who belongs, who is a threat, who needs to be watched.

The records in ST84/1 were embedded within a system of surveillance that extended outwards from Australia’s ports to the offices of shipping companies around the world, and inwards to anyone who seemed out of place in White Australia. Technologies of identification and surveillance do not simply enforce boundaries, they create them. Their existence demonstrates why they are needed. These records did not document identity, they defined it according to a set of racial categories.

Modern parallels are not hard to find. Last year in a bungled operation that became known as ‘#borderfarce’, immigration officials planned to prowl the streets of Melbourne on the hunt for illegal immigrants. The focus of border surveillance once again turned inwards, to those who seemed out of place. Watching in horror as events unfolded on social media, I helpfully pointed people in Melbourne to a convenient source of identity documents.

In Melbourne this weekend? Get your Border Force identity papers here: http://t.co/H3dlXHZy6q #borderfarce #wap pic.twitter.com/j2TrkKEPkR

— Tim Sherratt (@wragge) August 28, 2015

The link I tweeted was not to RecordSearch, but to an experimental interface where Kate and I are continuing to think about ways of exposing the bureaucratic remnants of White Australia.

So far I’ve harvested metadata from more than 20,000 files and downloaded around 150,000 page images. Amongst other things, I’m working on an updated wall of faces.

Most recently I created a way of sorting and viewing pages by their orientation – by the ratio of height to width. Why? Kate wanted an easy way of finding birth certificates which, in this period, tended to be short and wide. It was a simple little hack, but it revealed the records in a very different way.

There’s a big bunch of birth certificates, but there’s also envelopes, photographs, and an assortment of slips and notes. It makes you think about how we see the written world through the frame of ‘portrait’ orientation. It’s perhaps also worth noting that RecordSearch displays thumbnails as cropped squares – shape is subsumed to the regularity of the grid.

The records in our landscape view are no more accessible than they were before, but they can be accessed differently. In his contribution to the ASA’s 30th anniversary symposium, Eric Ketelaar described the relationship between archival access and human rights – highlighting the importance of access not only to democratic accountability, but also to our rights as ‘victims’ of official surveillance: ‘As human beings, subjected to the panoptic sort of governments and private enterprise, we have the right to know’.2 Ketelaar concludes by stating that ‘access is not the actual use of archives, access enables use’. I’d like to extend this a bit. What my own work reveals is the complex relationship between access and use. Not only does access enable use, use changes what we mean by ‘access’.

My bio nowadays describes me as a ‘historian and hacker’. ‘Historian’ describes my orientation to the world – I see the past in the present. ‘Hacker’ refers to the tools I use to make connections through time. Hacking is creative and positive, despite what the mainstream media might say – it’s about finding solutions, exploring alternatives, and pushing the limits of what’s possible.

My RecordSearch hack is just a little piece of Javascript code that you can install in your browser – it changes the pages as they load.

Userscripts, as they’re called, allow anyone to alter the way webpages look and behave within their own browser. They give users greater control over their online experience, but they also open opportunities for experimentation. A lot of my own research is guided by questions like: What would happen if I…? What would it look like? What would change? What would I feel? Userscripts are one way of playing with the complexities of access.

Surveillance too can be hacked. If you’re concerned about technologies such as facial detection you’ll be pleased to know that thanks to the work of artist Adam Harvey, not only can you confuse detection algorithms, you can make a dramatic fashion statement.

Gary Marx, a major figure in the field of surveillance studies, has catalogued the ways in which individuals can resist the growing encroachments of surveillance. Amongst possible tactics he identifies ‘discovery’ – the attempt to undercover the scope of surveillance.3 Access to records can empower such acts of everyday resistance, but in other ways surveillance and access are more alike than opposed. Both start from a place of concealment. Access cannot be given unless it is first restricted. Both depend on asymmetries of power. Decisions about what we can know are ultimately made by others. Access is as much a process of control, as it is an act of release.

This is not necessarily a bad thing. I think we can all agree there should be limits to access, particularly relating to individual privacy and cultural sensitivity. But just as ‘identity’ is defined through acts of ‘identification’, so access is elaborated through instances of deployment and use. Access is not a state of being, it’s a process to be negotiated. And so the question is, what can we know about what we can know?

On 1 January this year I harvested the metadata of all the files in RecordSearch with the access status of ‘closed’.

These are records that the access examination process has determined should be withheld from public scrutiny. While the files themselves can’t be seen, RecordSearch does tell us a fair bit about them, including when the decision was made and the reasons behind it. Unfortunately you can’t search or filter on this data, so it’s difficult to look for patterns within RecordSearch itself.

I’ve taken the data and loaded into a new site where you can examine it from a number of different angles. You can explore the reasons why files were closed, the series they came from, the age of their contents, and the dates when decisions were made. It’s more of a workbench than a discovery interface, and it’s likely to change as I ask different questions of the data.

Of course, the outlines of the examination process, including the grounds for exemption, are defined by the Archives Act. So what more is there to know?

Section 33 of the Archives Act does indeed spell out 17 reasons why records can be withheld from public access. But the data from RecordSearch includes an additional 11 categories. Some, like ‘Parliament Class A’, relate to other definitions under the Act. Others, like ‘MAKE YOUR SELECTION’, tell us something about the RecordSearch interface. But two of the most heavily cited reasons – ‘Pre-access recorder’ and ‘Withheld pending advice’ are not defined under the Act or anywhere that I could find on the Archives’ website.

Being an archivally-educated audience you can probably guess what these labels refer to, but if you need a little help you can look at when the access decisions in these categories were made.

The majority of decisions on ‘pre access recorder’ files were made before the introduction of the Archives Act in 1983. I checked this with the Archives and they confirmed that these records were examined before the existence of the Act. They explained that ‘pre access recorder’ was used when the original exemptions couldn’t be mapped to those later defined under Section 33.

Conversely, most decisions on ‘Withheld pending advice’ files were recorded in the last five or six years. If you look at the series that contain the most files citing this reason you can see that almost half come from A1838 – DFAT’s main correspondence series. As I’m sure you’ve realised, these are files that have been referred back to agencies for advice. And DFAT has been particularly slow in responding. They’re listed as ‘Closed’ on RecordSearch even though their access status has not been finalised. They’re not, however, included in the count of ‘closed’ files that the Archives reports in its annual summary of access outcomes.

This is probably fair enough, but if you search the closed files you can see that 1,467 of them files have been waiting for more than three years for a final decision. They might not be officially closed, but for a PhD student wanting to see them they are effectively closed.

My point here is not to be critical of the National Archives, or even of DFAT. What I’m interested in is the inevitable gap between legislation and practice. Access examination is subject to a range of influences and constraints, resourcing amongst them, and needs to be understood not as the application of a set of rules, but as a process that is historically contingent. A human process.

Earlier this year I harvested several gigabytes of parliamentary proceedings from the Australian Parliament’s ParlInfo database and created my own version of Historic Hansard.

As you do.

In the process of harvesting the files I discovered that data was missing for about 100 sitting days – most of them from the Senate between 1912 and 1919.

There’s no conspiracy at work here, it’s just some sort of processing error. However, Parliament staff weren’t aware of the problem, and it’s unlikely that anyone would’ve noticed using the web interface. You can’t find what you can’t find. Fortunately, Parliament staff are now working on a fix, but if you’ve been relying on ParlInfo for access to debates relating to World War I, you might want to do some more checking.

These things happen. Systems go wrong. Mistakes are made. Again what interests me is not finding who’s to blame, but exploring the gap between design and outcome, between ideal and reality. This is the gap where access is made and experienced. A gap that can only be understood through the complexities and contradictions of use. Access does not exist until its limits are tested. It’s not a process of opening, it’s a constant ongoing struggle over the very meaning of ‘open’.

And that’s a good thing.

We’re here at this conference to explore the possibilities of ‘forging links’. But of course collaborations don’t have to be comfortable to be constructive. The struggle over access may sometimes be tense, frustrating, and annoying, but it is also productive. Users of archives do not just consume access, they create it.

About a year ago I fired up my RecordSearch harvester and downloaded the metadata and page images for most of the ASIO (Australian Security Intelligence Organisation) files publicly available through the National Archives. I ended up with about 70gb of images. These are mostly dossiers on individuals and organisations – odd collections of gossip, published articles, and records of surveillance.

I’ve made this data available for anyone who wants it. Some of the images were recently used in the GovHack open data competition to create the ‘Cute Commies’ site.

To be honest, I didn’t have a clear purpose in mind when I harvested the data. It was another one of those ‘What would happen if?’ moments. I was, however, thinking generally about possible points of comparison between the ASIO files and the archival remnants of the White Australia Policy. Both built systems of identification, classification, and surveillance in which recordkeeping was crucial.

Kate and other historians of Chinese Australia have noted that the administration of the White Australia Policy was not uniform or consistent. Similar cases could result in quite different outcomes depending on the location and those involved. Understanding this is important, not only for documenting the workings of the system, but for recovering the agency of those subjected to it. Non-white residents were not mere victims, they found ways of negotiating, and even manipulating, the state’s racist bureaucracy. In her work on colonial archives, Ann Laura Stoller identifies this ‘disjuncture between prescription and practice, between state mandates and the manoeuvres people made in response to them’ as part of the ‘ethnographic space’ of the archive.4

How do we explore this space? One of the things I’ve found interesting in working with the closed files is the way we can use available metadata to show us what we can’t see. It’s like creating a negative image of access. Kate and I have been thinking for a number of years now about how we might use digital tools to mine the White Australia records for traces, gaps, and shadows that together build a picture of the policy in action. Who knew who? Who was where and when? What records remain and why?

The workings of ASIO, on the other hand, are deliberately obscured. Many of the files in the Archives include a note explaining why details have been withheld. Some warn that the ‘public disclosure of information concerning the procedures and techniques used by ASIO’ would enable people of interest to formulate counter-measures ‘based on an analysis of ASIO modus operandi’. David Horner’s recent history of ASIO notes that he was required to remove ASIO file references from his footnotes ‘because of the nature of ASIO’s filing system, which itself is classified’.5 We don’t even know how many files ASIO has on people and organisations, although David McKnight suggests that it’s somewhere in the hundreds of thousands.6 My harvest includes about 12,000 files.

Just like systems of racial classification, intelligence services exist within a circle of self-justification. The fact they exist proves they need to exist. We are denied information that might enable us to imagine alternatives. And yet as limited as the provisions under the Archives Act are, we do have access.

How can we use this narrow, shuttered window to reverse the gaze of state surveillance and rebuild a context that has been deliberately erased. Just as with Closed Access and the White Australia records can we give meaning to the gaps and the absences? Can we see what’s not there?

This is one of the questions being explored by Columbia University’s History Lab. They’ve created the Declassification Engine – a huge database of previously classified government documents that they’re using to analyse the nature of official secrecy. By identifying non-redacted copies of previously redacted documents, they’ve also been able to track the words, concepts and events most likely to censored.

The History Lab’s collection of documents on foreign policy and world events is rather different to ASIO’s archive of the lives, habits and beliefs of ordinary Australians. But I’m hoping that they too can tell us something about the culture that created them.

I’d intended to have a wonderfully compelling suite of examples and arguments to demonstrate today, but time has run short. Instead I have a set of half-baked experiments which sort of look a bit interesting. But perhaps that’s better. It’s important to me to try and be open about my own processes. I share my code and data, and I’ve started documenting most of what I’m up to in a open research notebook. If access is a struggle, then we should be sharing our stories of loss and frustration, and not merely celebrating our victories.

All of these experiments are online in some sort of form. So please explore.

Experiment A is nothing more than a browse interface to all the digitised records I’ve harvested. It’s just a clone of my work with the White Australia records, but I think there’s real conceptual power in the ability to browse.

Experiment B started with a problem. From RecordSearch I could harvest data on access status and find out how many ASIO files were in each of the three categories – Open, Open with Exception, and Closed. But how much of the ‘Open with Exception’ files are actually open?

Most of the files include a summary which tells you how many pages have been completely or partially exempted. That’s great, but did I really want to open up 12,000 files and manually scan for summaries? By playing around with the Tesseract OCR engine I’ve created a simple filter that extracts text from the images and searches for words like ‘exemption’, ‘archives’, and ‘folio’. I now have a good sized collection of summaries awaiting data entry…

Experiment C began as another attempt to quantify the scale of exemption. The summaries told me how many pages had redactions – bits of information like names and ids that are blacked out, or sometimes even cut out of the page. But if I could identify individual redactions I could both test the summaries and create a new measure of openness… or redactedness…

Looking for redactions in ASIO files — that looks hopeful… pic.twitter.com/heeWMAQiwe

— Tim Sherratt (@wragge) September 11, 2016

Through trial and error I developed a computer vision script that did a pretty good job of finding redactions – despite many variations in redaction style, paper colour, and print quality. It took a couple of days to work through the 300,000 page images, but in the end I had a collection of about 300,000 redactions. Unfortunately about 20 percent of these were false positives, so I spent a number of nights manually sorting the results.

But the false positives themselves are sort of wonderful… pic.twitter.com/Jn9l04bUaY

— Tim Sherratt (@wragge) September 29, 2016

My redaction finder still needs a lot of refinement, and plenty of errors have slipped through. But, within the files that are currently digitised, the scale of exemption seems about ten times greater than Margaret Kenna estimated when giving evidence to Parliamentary Joint Committee on ASIO in 2000. She thought every file contained about 10 exemptions ‘be it a word or a folio or a paragraph’. I’m seeing an average of about 100 redactions per file.

I’ve started adding information about the size and position of the redactions to my database and aggregating this data by page. When I left Canberra, the script was still running, but you can explore the current standings in my top 50 lists of the most redacted files and pages.

Once the data processing is completed you’ll be able to filter files by the amount of area blacked out, or the total number of redactions. Many more opportunities to see what you can’t see.

Experiment D was an attempt to build a composite image of all the redactions to visualise what parts of a page were most likely to be be removed – something like a heatmap. It sort of worked, but by the time I’d added all the redactions I had nothing but a very large black blob.

My composite of 170,000 #ASIO #redactions has opened a portal to another universe… pic.twitter.com/39iQewjst4

— Tim Sherratt (@wragge) October 8, 2016

A rethink is required…

Experiment E had two aims. First to highlight the visual character of the redactions themselves – there’s a strange sort of beauty in a massed collection of blobs. Secondly, just as with the Real Face of White Australia, I wanted to turn the files inside out. Instead of being dead ends, I wanted the redactions to be discovery points, signposts, ways of exploring the files.

It’s online now so play. You can view a random sample of redactions, or browse page by page through the entire collection.

Talking about her own ASIO file in the book Dirty Secrets, the politician and academic Meredith Burgmann noted that the ‘blacking out process seems totally arbitrary and for the reader terribly frustrating, like reading a detective novel with the last page torn out’.7 But in hunting for redactions I found they could also bring moments of unexpected joy. It seems that someone got a bit bored and has left us with a glorious collection of redaction art.

So what’s to come? I need to rework my redaction finder to improve its accuracy.

It’s interesting, and perhaps ironic, that the removal of information has given me an identifiable data point that I can potentially track against other characteristics of the files. Can I identify patterns by time or topic?

Apparently ASIO assessments have become less conservative over the years – I can test this by looking at changes in redaction rates over time.

I also want to explore the context of redactions. By expanding the window around redactions and OCRing the result, I hope to identify the words that occur most commonly appear near redactions.

Those of you coming to the workshop on Friday will hear more about some of the tools and technologies I’ve used in these experiments. But I wanted to give a brief overview today because this is access.

Digital tools and technologies give us the opportunity to use databases like RecordSearch as archaeological sites to sift through layers of metadata in search of new connections and meanings. This is access.

We can turn digitised collections inside out, revealing the people, the processes, the structures, the form. This is access.

We can reveal the processes through which records are controlled, concealed, and withheld. This is access.

Access is not a deliverable or a product. It’s a struggle for understanding and power – not just to see, but to see differently.

This is RecordSearch but not as you know it.

Experiment F is a userscript that puts the redactions back into RecordSearch. Access is an honest acknowledgement of its own limits, and an invitation to push beyond.

1. Richard Jenkins, ‘Identity, surveillance and modernity: Sorting out who’s who’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 159.
2. Eric Ketelaar, ‘Access, the democratic imperative’, Archives and Manuscripts, vol. 34, no. 2, November 2006, p. 73.
3. John Gilliom and Torin Monahan, ‘Everyday resistance’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 407.
4. Ann Laura Stoller, Along the Archival Gran: Epistemic Anxieties and Colonial Common Sense, Princeton University Press, Princton, 2009, p. 32.
5. David Horner, The Spy Catchers: The Official History of ASIO, Allen & Unwin, Sydney, 2014, p. 581.
6. David McKnight, ‘How to read your ASIO file’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 38.
7. Meredith Burgmann, ‘The secret life of B/77/26 (and friends)’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 454.

Presented at DHA 2016 in Hobart. Follow my research. View the slides.

A good type

In January 1954, the economic historian Ernst Boehm requested access to Commonwealth government records as part of his research into the depressions of the 1890s and 1930s. The request was forwarded from the Archives Office to the Prime Minister’s Department where it was noted ‘Boehm is I believe a good type + will agree to the usual conditions’.[1]

This sort of character assessment was important because there were few formal guidelines around access to government records. The Commonwealth Archives Committee reported in 1955 that the use of government archives was ‘seriously hampered by the lack of an access policy’.[2] The Prime Minister’s Department noted that requests for access were ‘treated on an “ad hoc” basis’.[3] Prospective researchers first needed to establish their ‘bone fides’. Archives staff would then attempt to identify records of interest and submit them to the relevant government departments for clearance. The Archives Committee noted that the ‘inevitable uncertainties and delays inherent in this system’ often left researchers ‘puzzled and disappointed’.[4]

In the years that followed, interdepartmental agreements and Cabinet decisions gradually refined the process, but it would be another thirty years before the public’s right to access was enshrined in legislation.

Right to Access

In 1955, the Cold War was near its height. The Royal Commission on Espionage was grilling public servants, scientists, journalists and others in its hunt for Communist infiltrators. Amidst the hysteria about ‘secrets’ and spies the Commonwealth Archives Committee emphasised the ‘moral obligation’ of democratic governments to make the records of their activities available for research. ‘In doing so’, the Committee argued, ‘they make a significant contribution to the continuing search for truth which is an essential part of the continuing democratic process’.[5]

The idea that the public had a ‘right’ to access government records did not go unchallenged, however. In a meeting with the visiting American archivist Theodore Schellenberg the previous year, a representative of the Department of Defence had admitted that he ‘found it difficult to accept the concept that the general public had a right of access to public records’.[6]

But the obligations of war and empire actually pushed Australia towards a more liberal policy. In the 1950s, Australia agencies tended to follow British precedent, applying a version of the 50 year rule. Given that the Commonwealth of Australia was founded in 1900, a 50 year embargo didn’t leave a lot of history to write. In 1966, however, the UK government opened access to records through to 1922 – encouraging research into World War I and its aftermath. Australia had little choice but to follow their example. As John Gorton noted in his submission to Cabinet, ‘we should not impose greater restrictions on our own scholars than are imposed in Britain’.[7] When the UK shifted to a 30 year rule a few years later, Australia again adjusted its policy.

In 1972 the Whitlam Labor government brought with it a professed commitment to ‘open government’. Access to archives started to be linked to broader questions around ‘freedom of information’. It took a few attempts, but the Archives Act was finally passed in 1983. In 2010 ammendments reduced the closed period from thirty to twenty years.

The public interest

Under the Archives Act (in its current form) the public has a right to access Commonwealth records more than twenty years old. But this right has limits. In 1954, Schellenberg described the ‘public interest’ as a balance between a liberal access regime and the protections necessary to shield both governments and individuals from the dangers of over exposure.[8] As access policy evolved, so did the articulation of these limits.

Protections for national security and the privacy of individuals have been long accepted as grounds to withhold records from public access. But the scope and meaning of such exceptions has shifted. In 1966, restrictions on ‘the release of papers relating to the personal expenditure of the Governor-General’ were provided as an example of how individuals were being protected from ‘embarrassment or distress’.[9] Similarly it was assumed that records ‘commenting on the person of the Sovereign’ would naturally be withheld.[10]

But the core of these exemptions remained and was codified within section 33 of Archives Act. Section 33(1)(a), for example, allows records to be withheld from access if they could ‘cause damage to the security, defence, or international relations of the Commonwealth’. Section 33(1)(g) exempts records that might result in ‘the unreasonable disclosure of information relating to the personal affairs of any person’.

The ad hoc approach of the 1950s has been replaced by a well-documented bureaucratic process with clearly defined outcomes. Under the Act a researcher can ask for a record to be examined for public access. The contents of the file is then assessed against the list of defined exemptions and an access status is assigned – Open, Open with exception, or Closed (sometimes closed files are described as ‘withheld’).

Statistics on the number of files examined and their access outcomes are reported annually. The vast majority are opened to the public. But some are not.

Open Access

What do we mean by ‘open access’? Back in 1964 the Interdepartmental Committee on Commonwealth Archives defined ‘open access’ as referring to documents ‘which have been cleared for consultation by any member of the public’.[11] Open could only really be understood by invoking its limits – open was not closed.

I’ve shared this potted history of Australian archival policy because it continues to shape the research that we do and what we can know.

These shifting policies and priorities have shaped the data now available online through RecordSearch, the National Archives of Australia’s database. As digital scholars we should recognise that there’s nothing natural or inevitable about these systems and their contents. They are deeply and profoundly historical.

But recognising this is both an obligation and opportunity, because we can view such systems as archaeological sites and dig back through the layers of data. We can meet the history of policy, with an analysis of practice as it has been captured in the descriptions, dates and decisions.

The access status of each item in RecordSearch is recorded and displayed – ‘Not yet examined’, ‘Open’, ‘Open with Exception’, or ‘Closed’. If a file is ‘closed’ or ‘open with exception’ the reasons for the restriction are provided along with the date when the decision was made. This is interesting and important data. But while you can filter your search by status, you can’t specify a reason or date. So I can browse a list of ‘closed’ files, but there’s no way of getting an overview of why they’re closed.

Until now.

Last year, as an experiment, I screen-scraped RecordSearch to capture the details of files that had an access status of closed. On the 1 January this year, I repeated the harvest with a bit more care.

I now have a dataset of 14,370 closed files. These are the files we’re not allowed to see.

Closed Access

You can explore them online using Closed Access, a public site I developed to play with the data.

You can view the aggregated metadata from a number of different angles – by the reason for exemption, by record series, by the age of the contents, and by the date of the decision. You can drill down from any of these views to a list of closed files. And of course you can search – want to know which files were closed in May 1989 for national security reasons? ClosedAccess can tell you!

Given that none of the files are actually publicly available, it’s possibly the most frustrating search interface yet devised. However, you can always ask for a file’s access status to be re-examined.

ClosedAccess is really more of a workbench than a discovery interface, with additional experiments documented in my open research notebook.

There for example, you can read about TF-IDF analysis of file titles, or my failed attempts to reconcile the statistics reported in the National Archives’ annual reports with the data from RecordSearch.

Reasons

So, why have these files been closed? Here’s the list of reasons.

The first thing to note is while there are 17 exemptions defined by the Archives Act, there are 28 reasons for closing files cited in RecordSearch. Some of these extra reasons relate to specific classes of records mentioned elsewhere in the legislation, while others, such as ‘MAKE YOUR SELECTION’ tell us something about the design of the data entry interface.

It’s no surprise that sections 33(1)(a) and 33(1)(g) of the Act are the most frequently cited exceptions. As I’ve noted, these relate to national security and individual privacy – longstanding grounds for withholding access.

More interestingly, these exemptions appear very frequently in combination with others. The DH standard network graph illustrates their centrality. For example of the 425 closed records citing 33(1)(b) – which refers to confidential information supplied by another government – 399 (94%) also cite 33(1)(a). This sort of pattern, combined with the low level of use for many of the exemptions, suggests that the whole legislative framework could be simplified – how many reasons to close files do we really need?

What’s more surprising are the two highly-cited reasons at the bottom of this graph, neither of which are defined in the legislation or anywhere that I’ve been able to find on the National Archives website. They’re ‘Pre access recorder’ and ‘Withheld pending advice’.

Pre-access recorder

According to the brief explanatory note in RecordSearch, ‘Pre access recorder’ denotes ‘non standard reasons for restriction’ – not very illuminating. But using ClosedAccess you can dig into the metadata to get an idea of what this actually means.

The average age of files citing ‘Pre access recorder’ is 75 years, significantly older than the overall average of 58 years. Indeed, if you look at the dates on which decisions were made about these files, you’ll see that most of these were closed before 1983 when the Archives Act was introduced. So it seems these files were examined and withheld from public access under the less formal arrangements that existed prior to the legislation.

This was (sort of) confirmed by the National Archives who told me that yes they were ‘made prior to the introduction of the Archives Act’, but they were also decisions ‘where the reason for restriction didn’t map to one of the reasons for restriction set out in the Act’. I still don’t quite understand this.

If you take all the file titles and throw them into Voyant you’ll see the word ‘staff’ figures prominently, reflecting the fact that many of them are personnel files. Surely these might have mapped to the privacy provisions of section 33(1)(g)?

From there things get even more complicated. Just as in quantum theory, by observing the phenomena of pre-access recorder, I have changed it. As a result of my inquiry, which the Archives seemed to interpret as a criticism of their metadata, they changed the status of most the ‘Pre access recorder’ files from ‘closed’ to ‘not yet examined’. They’re not any more open, but they no longer show up in a search for closed files. Fortunately, I’ve preserved their historical status in Closed Access. I’m planning to repeat my harvest on the 1 January each year, tracking newly closed files as well as those whose status has changed. So next year I’ll be able to check up on the ‘Pre access recorder’ files to see what’s really happened to them.

Withheld pending advice

Withheld pending advice is easier to interpret. The files have been referred back to the government agencies that created or controlled them to check whether they really can be unleashed on the public.

This process is known and feared by researchers because of the long delays it can create. In 2014 Philip Dorling wrote in the Canberra Times about a file from 1932 relating to Australia’s annexation of Antarctica which languished with the Department of Foreign Affairs and Trade for five years before being released in a redacted form.[12]

Delays in access examination are well-known, but the scale and impact aren’t obvious.

I think that part of the problem I was having in reconciling access examination data from annual reports was that the Archives doesn’t include the ‘Withheld pending advice’ files in its statistics. Perhaps this is fair enough – the access examination process isn’t complete. On the other hand, under the Archives Act if examination requests aren’t completed within 90 days then access is assumed to have been refused. Are they less closed because they may yet be opened? I’ve asked the National Archives what the status of these files really is, and how this affects the public’s right to access.

In the meantime we can use Closed Access to see the scale of the backlog and indulge in a bit of agency shaming. You can easily search for ‘Withheld pending advice’ files and order them by the date they were closed.

Ignoring the one file with the title ‘unknown’ that has been waiting 18 years for a decision (I suspect it’s been mislaid), there’s still a file that’s been waiting for 9 years, and 9 files that have been waiting for 7 years. Indeed, you can limit your search to files that have been waiting more than three years for a decision – there’s 1467 of them.

The Defence department gets the longest in limbo award for the 9 year delay, but DFAT is the overall champion with series A1838 containing almost 1500 files awaiting departmental determinations – 54 of which have been waiting more than five years. Too bad if you’re doing a PhD. Researchers are still being left ‘puzzled and disappointed’.

It’s important to make these processes visible because it’s all to easy to forget that the default status for any Commonwealth government file more than 20 years old is ‘open’. The reasons for closing files defined under the Archives Act are called exemptions for this very reason.

Philip Dorling suggested that agencies are becoming more conservative in their decisions. Once I’ve harvested a few year’s worth of closed files, I’m hoping I’ll be able to trace the fate of those poor souls currently ‘withheld pending advice’. But there are other measures of conservatism we can start to explore.

You’d think that the older a file was the more likely it was not to be closed. So what happens when we look at the gap between the date a decision was made to close a file and the date of it’s latest content? I’ve added an experimental view to Closed Access where you can explore the files than seem to have provoked the most conservative approach.

Strangely, at the top of this list are the proceedings of the Senate from 1903. But in fact, no decision was made to close this file. It belongs to a category of records called ‘Parliament Class A’. Regulations drawn up under the Archives Act simply declare these records as ‘exempt’. You need the permission of Parliament to access them.

It’s a reminder that at least we do have the metadata for closed files. The intelligence agencies hold unknown quantities of files where access to the recordkeeping systems themselves is restricted. We don’t know what we don’t know.

I’m hoping to explore these gaps further, but for now if you’d like a copy of the 300,000 digitised pages contained within publicly accessible ASIO files let me know.

No magic

It’s important to recognise all of this as the workings of a system. Digging through the layers of policy and metadata helps us to see that there is nothing intrinsically dangerous in a closed file. A judgement has been made against a list of historically determined criteria.

But it’s also important to resist the temptation to claim privileged access to the truth through our mastery of such systems. Next time you see words like ‘recently released documents’ in a book or article, remember that access examination is a process without magic or wonder. We need to remain critical of the process without adding layers of mystification – without succumbing to the cultural power of the secret.

Presented at the DigitalGLAM Symposium, University of Melbourne, 15 July 2016.
Play with the slides.

Who’s used Commonwealth Hansard through the ParlInfo Database — particularly the early stuff, pre 1980?

Was it fun?

It’s great that all of that content has been digitised, but the user interface is not very friendly — one of the main problems is that it’s just really hard to read anything in context, to get a sense of proceedings in parliament as they unfolded.

However, you can bypass the interface and go straight to the data. There’s an XML file — with the proceedings in a nicely-structured, machine-readable form — for each day.

To save the world much searching and clicking, I wrote a script to crawl through ParlInfo and download all of those XML files. There’s about 4gb in total, and I’ve shared them all through my GitHub account. Feel free to download and explore!

I initially harvested the files because I thought they’d provide a useful example for use in teaching about text analysis tools and techniques. But then I reflected on that ‘readability’ question and wondered how much work would be involved in creating a version of Hansard that was focused on browsing and reading. The answer was about two days — over the space of a weekend I created Historic Hansard.

I wanted to mention this today because I think it’s a good example of a ‘hack’. Contrary to mainstream media coverage, most hackers don’t want to steal your credit card details. Hackers, within the diverse community of coders, builders, programmers, and developers, are people who use code to solve problems, to get around obstacles — their solutions might not be the most elegant, but they work.

My bio reads ‘historian and hacker’. I’m a historian who uses digital tools and techniques to get around obstacles. In particular, I use code to open digital cultural collections.

In the case of Hansard I have, for minimal cost, created something that allows you to explore the proceedings of parliament in context. It’s made some historians pretty excited. Since that first weekend I’ve added an index to people and bills. I’ve also integrated tools for text analysis and annotation. Shortly I’ll add the rest of the House of Reps and the Senate through until 1980. I was very pleased that the new PMs site from the Museum of Australian Democracy was able to harvest from Historic Hansard to boost their content.

So this is an example of what happens when governments or cultural heritage organisations provide openly-licensed data for people to play with — although I should note that the contents of the parliament website, including Hansard, have a ‘no derivatives’ licence which creates a lot of uncertainty around what you can actually do.

I’ve talked a lot in recent years about building stuff with cultural heritage data — particularly in relation to Trove. And I’ve created a lot of tools, experiments and interfaces — some significant, some silly, some downright creepy.

But while open data does enable mildly-obsessed hackers like me to build new, shiny things, it also enables us to ask different questions — to analyse what’s not there, as well as what is.

In the case of Commonwealth Hansard what’s not there is about two years’ worth of Senate proceedings. I stumbled on this while harvesting the XML files and started doing some analysis. Between 1901 and 1980, 94 days of Senate proceedings, and 8 days of the House of Representatives are effectively invisible. PDFs exist, but the XML files are empty and the content is not searchable.

As you can see, most of the ‘invisible’ days come from the WWI period. So if you’ve been relying on the online version of Hansard to research WWI, you probably need to think about what you might have missed.

Needless to say, it’s unlikely that anyone would have noticed these gaps using ParlInfo’s web interface. It’s yet another example of why you always be suspicious of search engines. But it’s also an example of how hacking collections opens them to new forms of analysis and enables us to critique the very notion of access.

Spend sometime reading the strategic plans of cultural heritage organisations and you’ll see that ‘access’ figures prominently — access is ‘provided’, ‘given’, and ‘opened’. Online access is a ‘deliverable’ to be measured in hits and sessions.

The problem with this is it casts the public as consumers of access, rather than creators. Hacking heritage collections is one way demonstrating that access is not a one way flow of information — it’s a struggle and it must always be. We should always be uncomfortable with the categories used to structure our past. We should always want more. If we run out of questions and criticisms then something’s gone badly wrong.

Who’s used the National Archives of Australia?

You’ve probably come across the process of ‘access examination’. Records more than 20 years old are assessed against a list of criteria to see if they can be released to the public — the criteria are defined under the Archives Act and relate to things like national security and privacy.

Most records end up with an access status of ‘open’, but some end up ‘closed’ and are withheld from the public. Last year I started wondering what happens when we flip around the idea of ‘access’ and focus on the files we’re not allowed to see?

The National Archives’ online database, RecordSearch, provides basic metadata about ‘closed’ files, including the reasons why they’ve been withheld. But you can’t easily search or analyse this data within RecordSearch itself. So I wrote another script that harvested details of all 14000 files with the access status of closed and created my own interface to them — Closed Access.

While my Hansard hacking was aimed at opening up new forms of use, Closed Access is about understanding the processes and practices that determine whether we’re allowed to see a file. It pushes beyond legislative definitions in an attempt to reveal ‘access’ as a process that is historically contingent — there is nothing magical, mysterious, or dangerous about a ‘closed’ file.

By hacking collections we can also see them differently. Amongst its millions of records the National Archives holds the remnants of the White Australia Policy. About 5 years ago I wrote a script to harvest several thousands of these documents. I ran them through a facial detection script, cropped out the portraits, and created this seemingly endless wall of faces.

It was another weekend project, another hack. In this case it exposed the remnants of a racist bureaucratic system in a completely different way. Instead of databases, record series, and files, you could see the people inside.

Of course the danger in raving about the wonders of hacking is that we simply create new structures of power, new limits on participation based around coding ability. I don’t believe in hacker elites liberating the masses from ignorance. So part of the struggle around access must also be the struggle to share — to share technology, code, examples, tools, data, knowledge, possibilities, and passions. To give people the opportunity to participate and not just consume.

You might think sharing is easy, but it’s fucking hard work — both for institutions and individuals. And it’s work that is rarely rewarded in terms of professional acknowledgement, advancement or funding.

But I’m inspired by open access activists around the world, by radical librarians setting up Tor exit relays, by digital humanists critically examining unequal access to technology, and exploring the possibilities of minimal computing. By all the people who freely share their work and ideas to make ‘access’ just that little bit easier for others.

I share because I think it’s important, because it’s part of the obligation of being a hacker and historian, academic and activist.

So I share things like:

• articles and examples on my blog
• code and data on GitHub
• current research notes and hacks in my research notebook
• documentation, activities, and tutorials in my digital heritage handbook
• web apps and sites all over the place

I’m also trying to develop a few DIY, low code examples that enable people to build things using Trove data and some freely available tools. I’m hoping that’ll give them the confidence to explore further. A starting point perhaps…

Just last week I ran a free digital heritage workshop in the lead up to GovHack — the national, open data competition. We’re hosting a specially-themed ‘Heritage Hack’ node at the University of Canberra as part of the GovHack this year. Feel free to join us!

Not that any of this is particularly special or remarkable. It should just be what we do — as researchers, as cultural institutions, as citizens, as hackers. The main thing we need to share is, of course, our power.

So let’s just just stop talking about access to cultural heritage as if it’s a destination, and explore it as journey full of tensions, dangers and difficulties; but also full of joy, discovery, and meaning. Let’s hack heritage to build critiques as well as websites, conversations as well as code.

I’d just arrived home on Monday evening when a tweet from the local ABC radio station appeared in my stream: ‘NOW: @nlagovau informs its staff of budget cuts’. An all-staff email describing the impact of continuing government funding cuts on the Library had become public – it was not good news.

Everyone knows there are harsh times ahead for our national cultural institutions – the effect of the latest cuts (let’s not call them ‘efficiency dividends’) was vigorously discussed at Senate Estimates. Although the Australian War Memorial was, rather significantly, spared the bayonet. These ‘dividends’ have been forcibly extracted by governments of both persuasions, but the latest round has pushed the already struggling organisations beyond their capacity to adjust. Staff and services are going to be lost.

Soon after the ABC segment, more details about the Library’s plans appeared on the Canberra Times website. As the recently departed manager of Trove I was worried for my former team mates and for the service we had worked so hard to develop. The ABC referred to major cutbacks in the Library’s digitisation program. The Canberra Times report went on to note:

The library will also cease aggregating content in Trove from museums and universities unless it is fully funded to do so.

Trove, used by more than 70,000 people a day, has previously avoided cutbacks. But no longer. Fewer collections will be added, less digitised content will appear – not quite a content freeze, but certainly a slowdown.

I started tweeting about the reports. I also invited others to share their thoughts on the value of Trove with the Minister for the Arts, Mitch Fifield, using the hashtag #fundTrove. As I write this post the total number of #fundTrove tweets and retweets is approaching 3,000. Many have shared their discoveries or the details of their research – all made possible by Trove. Others have just expressed their outrage. But what is most powerful is the overwhelming sense of gratitude and affection (even love) for what Trove provides. Trove matters.

Trove & the NLA were an absolutely indispensable resource & support for our Eureka Prize-winning climate history research project #fundTrove

— Pew pew! (@pewpewlaserfire) February 23, 2016

So many historians I publish say @TroveAustralia has transformed their work, made much more possible. And they're right. #fundTrove #ozhist

— Phillipa McGuinness (@pipmcg) February 23, 2016

The #fundTrove tweets are bringing me close to tears. Is this what we've come to? How shameful.

— Molly (@madradish) February 23, 2016

Trove is unique and globally important. It is the most significant public sector DH project in the world – Australia needs to #fundTrove

— Tim Hitchcock (@TimHitchcock) February 23, 2016

#fundTrove my mum & I swap and share family history from @TroveAustralia , all the secrets out of the bag.

— TheVelvetNap (@TheVelvetNap) February 22, 2016

Let’s be clear, Trove is not going to be suddenly turned off. Nor is it a particular target of government cuts. But these new limits on Trove are a clear sign of how deeply these ‘efficiency dividends’ are striking at one of our most important cultural institutions. And if the government thinks that’s ok, what does it mean for the future of Trove?

I’ve written previously about the way that Trove’s digitised newspapers are changing the practice of history, even changing our relationship with the past. But Trove is not just newspapers. Trove also brings together the collections of hundreds of other organisations – large and small. Together with the National Library’s own digital collections these two halves of Trove create a resource profound in depth and meaning, and brimming with the capacity to surprise.

It seems the National Library has simply been too efficient for its own good. Instead of sitting back and waiting for a big pot of money, it took the decision some years ago to push ahead with the development of Trove. Many libraries, organisations and individuals have helped fund the digitisation of newspapers, but the Library built Trove from within its existing budgets – a triumph of innovation and efficiency. And now it’s suffering for it.

The current round of cuts have made it clear – it’s time for Trove to be appropriately funded. Not as an add-on, or a ‘nice to have’, but as key component in our cultural landscape.

Although #fundTrove has focused its attention on the Minister for the Arts, I don’t think he’s the only one in the firing line. Trove is a fundamental piece of research infrastructure, as important as a telescope or a particle accelerator. But, like most cultural institutions, the Library has limited access to infrastructure funding. That’s just stupid, reflecting an outdated understanding of the nature of research. Let’s fix that as well and find support for Trove in education and research budgets.

And then there’s the ‘ideas boom’. As I’ve frequently argued, Trove is not just a website or a portal, it’s a platform. Portals are for visiting, platforms are for building on. All those hundreds of aggregated collections, all those millions of digitised newspaper articles are available in a form computers can understand via an API (Application Programming Interface). That means clever developers, innovative industries, hackers and tinkerers can take Trove’s data and BUILD NEW THINGS. BOOM! Ideas have to start somewhere, and Trove offers plenty to play with.

I don’t know if Senator Fifield will ever see any of the #fundTrove tweets. But it’s important to talk about what we value and why. I’m a historian and a hacker and I love Trove because it helps me to see things differently. Our history is not in the past – it lives as a source of inspiration and unease, understanding and critique. Trove helps me to connect to that. I want it to be safe, and I want it to grow.

So #fundTrove.

Things you can do right now

• Tweet or post about the value of Trove using the hashtag #fundTrove.
• Like the #fundTrove FaceBook page
• Write to Senator Fifield (or other government members) – the folks at the eScholarship Research Centre at Melbourne University have already drafted a sample letter.
• Sign one or all of these petitions:
  • Fund Trove! created by Edwina Byrne on Megaphone.org.au
  • Stop Cuts to National Library of Australia & Save Trove created by Paddy O’Toole at Change.org
  • Stop the Australian government from wrecking Trove created by Ian Gardener at Change.org

#fundTrove news & posts

Latest updates will be posted on the #fundTrove Facebook page!

In the media:

• Treasure Trove: why defunding Trove leaves Australia poorer by @mikejonesmelb & @bestqualitycrab in The Conversation
• Turnbull, cuts and the culture of forgetting by @AndrewThackrah on The Drum
• My radio interview with Perth’s community radio station RTRFM
• My radio interview with Phillip Adams and Peter FitzSimons on ABCRN Late Night Live
• Video except of Peter FitzSimons on ABCRN Late Night Live
• Budget cuts threaten National Library’s online Trove, by Patrick Avenell in Campus Review
• The researchers’ treasure, Trove, is under threat by Gideon Haigh in The Australian
• Australian and international arts institutions, researchers throw support behind Trove by Alexandra Back in The Canberra Times
• Trove online database’s future in doubt due to National Library funding cuts on @ABCNewsCanberra
• Cuts to National Library funding will hurt researchers from all walks of life by Erica Cervini (@thirddegreeblog) in the Sydney Morning Herald
• National Library’s Trove: a great digital democracy under threat by Paul Daley (@ppdaley) in The Guardian (Australia)
• Trove and the case for radical openness by Hugh Rundle (@hughrundle) in Overland

Statements of support:

• Statement by The Greens, 23 February 2016
• Statement in parliament by Gai Brodtmann MP (ALP), 24 February 2016
• Statement in parliament by Tim Watts MP (ALP), 29 February 2016 — see also the video on Facebook
• Statement in parliament by Gai Brodtmann MP (ALP), 3 March 2016
• Statement by peak GLAM organisations in support of Trove
• Statement by the Australian Academy of the Humanities in support of Trove
• Open letter from the Professional Historians Association (Victoria) to Senator Mitch Fifield
• Open letter from the History Council of NSW to Senator Mitch Fifield
• Open letter from the Australian Historical Association to Prime Minister Malcolm Turnbull (and others)
• Statement by ALIA, NSLA & MA, Peak bodies call on infrastructure minister to support Trove
• Statement by the Council for the Humanities, Arts and Social Sciences

Blog posts:

• #FundTROVE by @annelie_de
• On the importance of Trove by @cassmob
• #FundTrove by @jessb3
• #fundTrove by @jayechats
• An open letter to Senator Fifield by @morethangrass
• A Treasure Trove of Newspapers by @diamond_marion
• #fundTrove by @ladyredjess
• Funding smart infrastructure by @peterneish
• Trove beyond borders by @KarinESpeedy
• TROVE: the library community’s greatest contribution at risk by @57GPS
• What is it about Trove? by Chris Loudon, WAGS
• Threats to funding of the National Library of Australia’s ‘Trove’ by Katrina Grant, Melbourne Art Network
• Airships against the Mad Mullah; or, #fundTrove by @airminded
• Another reason to #FundTrove: Communist Party of Australia material now digitised by @hatfulofhistory
• Four reasons why the National Library of Australia is so valuable to Australia and Australians by David Stephens at Honest History
• Guarding the treasure trove by Bob Wilson at Friday on My Mind
• Teaching and Troving by @CowPaddock
• Fund Trove by Jackie French
• Monday musings on Australian literature: Save Trove by @whisperinggums
• I Love Trove! (Ask anyone who knows me, I say it all the time!) by @linksinachain
• One cut too many – how much more efficient can our national culture get? by Chris Bowen at Music Australia

What are the functions of government? You’ll find one possible answer in the Australian Governments’ Interactive Functions Thesaurus (AGIFT) maintained by the National Archives of Australia. AGIFT sets out a standard series of terms to describe common business activities across the different levels of government.

Functions are important to the National Archives — they aid discovery, inform appraisal, and guide government agencies in understanding what records they should keep keep. Functions also offer researchers an alternative way of exploring the Archives’ rich collections — instead of blindly searching for keywords, you can track the activities of government agencies through time.

RecordSearch, the National Archives’ online database, provides a simple way of browsing by function, though it’s not very obvious. Click on the ‘Advanced Search’ tab in RecordSearch and you’ll see an interactive diagram representing the way that records are described and arranged using the Commonwealth Records Series (CRS) system. I made this little widget a very long time ago, and there’s an alternative version, with links to the CRS manual, over at WraggeLabs.

You’ll notice that functions are associated with agencies — they’re a way of describing what government departments do. Click on the ‘Functions’ button to open up a very basic browse interface that lists preferred and alternative names for functions. There seem to be a few differences between AGIFT and the terms used in RecordSearch, I’m not sure why. The CRS Manual includes a summary of the thesaurus used in RecordSearch, and for convenience I’ve extracted a flat list of all the functions.

Just click on any function in RecordSearch to find agencies associated with it. It’s basic, but it does at least provide an alternative to the tyranny of the search box, so well described by Mitchell Whitelaw. If you’re starting a research project and don’t know what keywords are relevant, perhaps following functions will help you map the territory.

Functions also provide a way of exploring how the organisation of government changes to meet social and political needs. How are the government’s priorities in areas such as ‘migration’, ‘education’ or ‘security’ reflected in the sorts of bureaucratic structures they create? To answer this type of question you really need to extract all the data related to a function for further analysis and manipulation, but that’s not something you can easily do through RecordSearch’s web interface. Time for some hacking…

Harvesting data from RecordSearch has been something of an obsession over the past five or six years. Over that time I’ve developed a set of tools that pull structured data out of the web interface. I’ve used these tools to expose records relating to the White Australia Policy, to grab a copy of every publicly available ASIO file, and to analyse records that remain closed to public access. But to harvest functions I needed to extend my toolkit — improving the handling of agency data, and adding a simple search module to find all agencies associated with a function. Once this was done I could build a harvester — a script that just makes a series of requests to RecordSearch until all of the desired records have been retrieved, processed and stored.

All the code for my toolkit and harvester are freely available on GitHub — please download, use and improve! I’ve tried to provide fairly detailed instructions with the harvester, but you will need to fiddle about on the command line.

I haven’t done anything very exciting yet — I just wanted to make sure it was all possible. So far I’ve harvested data about agencies related to ‘MIGRATION’ and its associated functions ‘CITIZENSHIP’, ‘DEPORTATION’, ‘MIGRANT SERVICES’, ‘MULTICULTURALISM’, ‘PASSENGER ENTRY CONTROL’, and ‘REFUGEES’. You can view CSV-formatted summaries for each function on GitHub.

The CSV file for MIGRATION, for example, lists 191 associated agencies. These range from departments of state, such as the current Department of Immigration and Border Protection, Central Office, to smaller bodies like the Committee on Overseas Professional Qualifications. For each agency I’ve listed it’s status, location, life dates, and the dates during it was associated with the migration function. This is only a summary of the data I have for each agency, which includes relationships with other agencies, as well as details of any other related functions. It would be interesting to use this to draw out possible connections between functions.

As a preliminary experiment I thought it’d be interesting to create a simple timeline displaying the different government departments associated with immigration since 1900. I was trying to avoid too much coding, so I ended up hacking Plot.ly’s basic line chart. As a visualisation it’s not very exciting, but as an initial poke at the data it’s quite encouraging.

Hover over the lines to see the agency names and date ranges. Click on one of the agency identifiers (the CA numbers) to open its page in RecordSearch.

The code to create the CSV summaries as well as simple charts like this is all included with the harvester in my RecordSearch Functions repository. I’ll add tools and examples as work proceeds or time allows. Hope you find it useful.

As the end of the year approaches it’s time to take stock with my annual talking/making summary. This was a year of change. In March, I started combining my work at Trove with a part-time appointment at the University of Canberra. I think there’s a great need for joint appointments between the cultural sector and academia, but juggling two part-time jobs has been a struggle. As much as I love Trove, I’m finishing up at the National Library of Australia at Christmas time and heading into 2016 looking for new challenges!

2015 was a big year for talking, with DH2015 an obvious highlight.

• 3 February — ‘Seams & edges: Dreams of aggregation, access and discovery in a broken world’, ALIA Online, Sydney
• 13 February — ‘Myths, mega-projects, and making’, joint keynote at EuropeanaTech 2015, Paris (via video)
• 30 March, ‘The many meanings of Trove’ (slides), 14th Australasian Congress on Genealogy and Heraldry, Canberra
• 10 April — ‘Stories for machines, data for humans’, New Factual Storytelling Symposium, University of Canberra, Canberra
• 15 April — Presentation at ‘Your Library in 2020’, Deakin University, Melbourne
• 15 May — ‘Significance at scale: Understanding the value of 150 million newspaper articles’ (slides), (In)significance, University of Canberra
• 4 June — Part of a panel discussing copyright, State Library of NSW, Sydney
• 29 June — Speedo on my LODBook project (video), LODLAM Summit, Sydney
• 3 July — ‘Unremembering the forgotten’, keynote at DH2015, Parramatta
• 13 August — ‘Trove and beyond’ (slides), Griffith University, Brisbane
• 7 September — ‘#Borderfarce, building, and a hackers’ reward’, National Scholarly Communications Forum, Canberra
• 13 October — ‘The perfect face’, NDF2015, Wellington
• 21 October — ‘Towards a manifesto for tactical DH infrastructure’, University of Canterbury, Christchurch
• 28 October — ‘Digital disruptions’ (slides), workshop at University of Otago, Dunedin
• 5 November — Introduced the closing panel on ‘Perspective and scale’, Recasting the Question: Digital Approaches to Art History, University of Sydney
• 5 December — ‘The digital revolution and the practice of history’ (slides), keynote at History Teachers’ Day, Macquarie University, Sydney

It’s been a bit of a frustrating year on the making front — just didn’t seem to have the time or energy to pursue the projects I was interested in. But I did have a bit of fun with faces and ASIO files…

• Radio National data — harvesting current affairs programs from Trove
• Closed Access — some experiments in the meaning of ‘access’ through the National Archives of Australia’s RecordSearch database
• The Vintage Face Depot (background on Storify)
• Face API — a simple API to a collection of faces extracted from Trove newspapers
• Faces dataset — faces from Trove in an easily downloadable form (via Figshare)
• The People Inside (Trove edition) — userscript to add faces to Trove newspaper results (I also updated the RecordSearch edition)
• Have you got what it takes to protect Australia’s secrets?
• ASIO files — harvesting images and data from ASIO files held by the National Archives of Australia
• Trove Lists Exhibition — turning Trove lists into simple exhibitions

There’s various other updates and unfinished stuff on GitHub. I hope to expand on a few of these projects in the new year.

I’ll still be at the University of Canberra, working on developing some new teaching possibilities around digital cultural heritage. Get in touch if you’re interested in pursuing postgraduate research!

Here’s the video of the talk I gave at the University of Canterbury in Christchurch on 21 October.