The image above is from Geoff Hinchcliffe’s awesome visualisation of more than 12,000 #fundTrove tweets.

This year I sadly left the wonderful team at Trove and took up a full-time academic post at the University of Canberra. But it was Trove that dominated the early part of the year, as the impact of continual funding cuts on the National Library of Australia became clear. Users of Trove shared their feelings on Twitter and Facebook, organisations posted statements of support, and numerous articles appeared in the media. In the lead up to the federal election, both the Greens and ALP made commitments to support Trove and our national cultural institutions.

In the last few days, we’ve learnt that the Government will provide $16.4 million over four years to the NLA ‘for digitisation of material and upgrade of critical infrastructure for its Trove digital information resource and to upgrade other critical infrastructure’. While we wait to hear exactly what this means for the future of Trove, it’s important to remember that it comes after many cuts and job losses across the cultural sector. The lesson of #fundTrove is that we cannot take the future of our collecting organisations for granted. We need to show why they matter and fight for the resources they need.

Access is important — both its politics and its practicalities. This year I’ve tried to be a bit more rigorous in the way I share information and document my projects. I created a Digital Heritage Handbook where I publish workshops, activities, and other bits and pieces. Much of it is in draft form, but I decided it was better just to push everything out in the hope that it might be useful. Similarly, I created an Open Research Notebook to share work in progress. The Handbook also includes details of the two undergraduate units I taught in second semester — Working with collections, and Exploring digital heritage. I think they went pretty well, but I’ve got a few improvements planned for 2017.

This year I accidentally built my own version of Historic Hansard, created an interface to National Archives files we’re not allowed to see, and mined ASIO surveillance files for redactions. As well as these major projects, there were lots of little hacks and harvests aimed at exploring the idea of ‘access’. You can follow my main research obsessions in my notebook:

• Closed Access
• Historic Hansard
• Immigration, Recordkeeping, and Surveillance
• LODBooks

Talking and making details follow…

2016 — the making:

• Locating Trove newspapers
  Updated code, data, and interface to geolocate and display Trove newspaper titles. Now with maps!
• Headline Roulette
  Much needed update for my old game. Now on it’s own domain and with better handling of Trove API errors.
• DFAT Documents
  Demonstration code to harvest the Department of Foreign Affairs and Trade’s collection of historical documents and extract some metadata. The harvested documents are available in Markdown format and can be explored through a simple website.
• People of Australia
  @people_aus is a Twitter bot sharing random names drawn from late 19th and early 20th century naturalisation records held by the National Archives of Australia. Many names. Many cultures. These are the people of Australia.
• RecordSearch Series Harvests
  Code to harvest the metadata and digitised images of all items in a series from the National Archives of Australia. Data from an assortment of harvested series are available as CSV files.
• SRNSW indexes
  Code for harvesting indexes from the State Records of NSW website. Data from 59 harvested indexes is available as CSV files.
• Facial detection demo
  Code and website to demonstrate the principles of facial detection using OpenCV.
• Show Redactions userscript
  Code for inserting details of redacted files into RecordSearch results.
• ASIO Experiments
  Code used for the extraction of redactions and other experiments with digitised ASIO files.
• Redactions dataset
  Redactions extracted from ASIO surveillance records in National Archives of Australia Series A6119, <https://dx.doi.org/10.6084/m9.figshare.4101765.v1>
• Non redactions dataset
  False positives (non-redactions) extracted from ASIO surveillance records in National Archives of Australia Series A6119, <https://dx.doi.org/10.6084/m9.figshare.4104651.v1<
• Redacted
  Web interface for exploring redactions extracted from digitised ASIO files. Includes a collection of redaction art.
• Open with Exception browser
  Code and website providing an experimental browser for digitised ASIO files from the National Archives of Australia.
• Invisible Australians browser
  Updated code and website providing an experimental browser for digitised records from the National Archives of Australia relating to the administration of the White Australia Policy. Now includes a landscape view for exploring records by their orientation.
• Closed Access harvester
  Updated code for harvesting and analysing records from the National Archives of Australia with the access status of ‘closed’.
• Closed Access dataset
  Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 1 January 2016.
• Closed Access website
  Public web interface for the exploration, analysis, and visualisation of ‘closed’ records in the National Archives of Australia.
• RecordSearch Functions
  Code and documentation for analysing the performance of functions by Commonwealth government agencies over time, using data from the National Archives of Australia.
• Commonwealth Hansard XML repository
  A repository of the (almost) complete proceedings of the Commonwealth House of Representatives and Senate from 1901–1980. This comprises several gigabytes of XML-formatted files harvested from the ParlInfo database.
• Historic Hansard
  A public website that presents the proceedings of the Commonwealth House of Representatives and Senate from 1901–1980 in a form that is optimised for browsing and reading. It includes additional features such as indexes to people and legislation, and the integration of tools for text analysis and annotation. Documentation is also provided.
• Trove Harvester
  Code and documentation to support the creation of large datasets for research and analysis from Trove’s digitised newspapers.
• Gadfly front pages
  Code and documentation to demonstrate how to harvest page images from Trove’s digitised newspapers.
• Trove Proxy
  Code and active proxy service that generates links to download PDFs from Trove’s digitised newspapers, and provides a https wrapper around the Trove API.
• DIY Headline Roulette
  Code and documentation that makes it easy for anyone to create their own simple game using Trove’s digitised newspapers.
• Radio National program data
  Updated dataset of programs broadcast on Radio National from 2000–2016 harvested from Trove.
• PMs Transcripts repository
  Repository of more than 20,000 XML transcripts of speeches by Australian Prime Ministers harvested from the PMs Transcripts site.
• UMA Ellis Photos
  Repository of data and images from a collection of political photos by John Ellis held by the University of Melbourne Archives. Harvested using the Trove API.

2016 — the talking

• 8 November 2016 – Digital research seminar and workshop at Griffith University.
• 10 November 2016 – Presentation as part of the ‘Access and innovation’ panel at Digital Directions 2016, National Film and Sound Archive, ‘Caring about access’, <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>.
  Featured on dh+lib (digital humanities and libraries).
• 19 October 2016 – Keynote presentation at Forging Links, Australian Society of Archivists conference, Parramatta, ‘Turning the inside out’, <https://dx.doi.org/10.6084/m9.figshare.4055013.v2>.
• 8 September 2016 – Invited presentation to Data and libraries: harnessing the possibilities, ALIA URLs seminar, Canberra, ‘Slow data? Small data? Exploring human-sized alternatives in the big data deluge’.
• 26 August 2016 – Keynote presentation at Migrant (R)e-Collections, Lorenz Centre, Leiden, ‘A life reduced to data’.
  Featured as an ‘Editors’ choice’ by Digital Humanities Now.
• 19 August 2016 – Keynote presentation at Working History, Professional Historians’ Association conference, Melbourne, ‘Telling stories with data’.
• 29–31 July 2016 – GovHack Heritage Node, University of Canberra
• 18 July 2016 – Keynote presentation at internal staff conference, State Library of Victoria.
• 15 July 2016 – Invited presentation at DigitalGLAM Symposium, University of Melbourne, ‘Hacking heritage: power and participation in digital cultural collections’.
  Featured on dh+lib (digital humanities and libraries).
• 5 July 2016 – Interview on ABC 936 (Hobart) Evenings about Trove.
• 24 June 2016 – Interview on ABCRN afternoons, ‘The fight to save Trove’
• 22 June 2016 – Paper at DHA2016 Conference in Hobart, ‘Closed Access’
• 3 June 2016 – Invited contributor to the REMIX Sydney panel ‘Blurring the digital and the physical: How can we add new layers to history?’.
• 2 June 2016 – Invited presentation at the Digital Research Methodologies Forum, La Trobe University, ‘The revolution will not be digital’
• 31 May 2016 – Invited presentation on digital research methods to the Deakin Contemporary Histories Research Group.
• 10 May 2016 – Presented two workshops on ‘Digital Tools and Techniques for the Adventurous Historian’, organised by the History Council of South Australia for the SA History Festival 2016, Adelaide.
• 7 March 2016 – Interview on ABCRN Late Night Live, ‘Treasure Trove under threat’
• 6 March 2016 – Contributor to panel discussing the government’s Innovation Agenda, Electronic Visualisation and the Arts Australasia, University of Canberra.
• 1 March 2016 – Interview with ABC 666 (Canberra) Drive about #fundTrove.
• 1 March 2016 – Interview with RTRFM (Perth) on #fundTrove.
• 19 February 2016 – Invited presentation to symposium on Commonwealth Department of Immigration – Then and Now, La Trobe University, ‘Digital perspectives on the archives of immigration’
• 12 February 2016 – Keynote presentation to ANZREG 2016 (Ex Libris Australia and New Zealand Regional User’s Group), Melbourne, ‘Linked Open Data’.
• 10 February 2016 – Invited contributor to the Gale Cengage sponsored Digital Humanities Panel at VALA2016, Melbourne.

Contribution to a panel discussion on ‘Access and Innovation’ at Digital Directions 2016, Canberra, 10 November 2016.
View the slides.

Please cite as: Tim Sherratt, ‘Caring about access’, presented at the Digital Directions, 2016. <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>

You might have seen that the Department of Prime Minister and Cabinet has opened discussion on an Open Government National Action Plan. Last week I was watching the Twitter stream from a briefing event in Melbourne, when I saw this tweet from Asher Wolf quoting the PM&C spokesperson:

. @pmc_gov_au: we don’t want ppl to search for flaws or try to crack gvt datasets. #OGPAU

Now I’m not sure of the context of this statement, but it is a reminder that we can’t take the meaning of words like ‘open’ or ‘access’ for granted.

They are what we make of them.

I’m a historian and hacker. I don’t steal credit card details, I use digital tools to open cultural heritage collections – to see them differently, to feel differently.

Earlier this year I wrote a script to harvest many gigabytes of parliamentary proceedings – Hansard – from the ParlInfo database maintained by the Department of Parliamentary Services. I shared the files, covering the period from 1901 to 1980, through my GitHub repository. My plan was simply to make this rich source of political history easily available to anyone who wanted to explore new forms of digital analysis.

In the end I created my own version of Historic Hansard, one that privileges the experience of reading rather than searching. I also added a few nifty features such as indexes of legislation and people, and integration with text analysis and annotation tools.

But in the process of harvesting the data I noticed that around 100 sitting days were missing from ParlInfo. The files were empty – unable to be searched. Given an average of about 50 sitting days a year, that’s about two years worth that were simply invisible.

Thanks to the Parliamentary Library these problems have now been fixed. But this experience highlighted two issues. First, search interfaces lie. We have to develop our critical capacities to be more aware of what we are not being shown and why. And second, this problem only became visible because I was hacking their database, because I went beyond what was offered by their website, because I wanted more. Unlike the PM&C spokesperson I think one of strongest reasons for opening data is to encourage people to find its flaws. This is particularly the case for cultural heritage data where the processes of selection, representation, normalisation, control, and preservation all have an impact of the types of stories we can tell about ourselves.

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

A few years ago Merete Sanderhoff, from the National Gallery of Denmark, compiled a collection of essays on opening cultural heritage collections for reuse online. It was called Sharing is Caring. I think it’s worth reflecting on the dual meanings of ‘caring’ – both looking after, and giving a shit. Sharing helps us foster communities and preserve collections. But it also matters, it has an impact, it can change the world.

Bethany Nowviskie, the Director of the Digital Library Federation, recently gave a talk on ‘speculative collections’ asking how we shift the temporal orientation of our libraries and archives away from a closed and linear past towards an exploration of what might be.

Seeing differently. Feeling differently.

This is a visualisation created by Geoff Hinchcliffe showing more than 12,000 tweets since February this year using the hashtag #fundTrove. Trove has radically changed our access to the past. It’s not perfect, it’s not everything, but it changes lives. And yet the government seems to think it can be left to wither, and nobody will really care. I care. Do you? Then what do we do about it?

I also care deeply about the collections of the National Archives – a love expressed through many painful hours spent hacking RecordSearch, their online database. Last year I harvested details of more than 12,000 publicly available ASIO files – you can grab the results from my GitHub site – and downloaded about 300,000 page images. These are mostly surveillance files, documenting the lives of writers, academics, unionists, Indigenous activists and others – identified as potential threats to the nation.

This year I wrote a script to search those images for redactions – sections of text blacked out for security reasons. If you’d like to explore the 239,000 redactions I’ve extracted, you can – marvel at their blackness, their very lack of information. But here the redactions are not dead ends, they’re starting points, ways of exploring the workings of ASIO. As the power and reach of state surveillance continues to expand, we can find creative ways to reverse the panoptic gaze.

We don’t have to accept what we’re given. We can take collections and turn them inside out.

The National Archives also preserves remnants of Australia’s racist past. There are many thousands of records documenting the workings of the White Australia Policy. They include special certificates, with portrait photos and handprints, which were needed if you were deemed non-white, travelled overseas, and simply wanted to come home.

About five years ago I harvested about 12,000 images of these certificates from RecordSearch and ran them through a facial detection script. In a little over a weekend I’d created the Real Face of White Australia – a seemingly endless scrolling wall of more than 7,000 faces. It’s compelling. It’s uncomfortable. But this is who we are.

As I saw the One Nation senators cracking open the champagne outside Parliament House yesterday, I thought again of how important its is the tell these stories, to share these collections. To give a shit.

Innovation can be measured in more than shiny apps, or cool new visualisations. By struggling with access to our past we can imagine new futures.

Keynote presented at Forging Links, the Annual Conference of the Australian Society of Archivists, Parramatta, 19 October 2016.

Please cite as: Tim Sherratt, ‘Turning the inside out’, presented at the Australian Society of Archivists Annual Conference, Parramatta, 2016, <https://dx.doi.org/10.6084/m9.figshare.4055013.v1>.

This is RecordSearch, but not as you know it.

This is my hacked version of RecordSearch, the online collection database of the National Archives of Australia. Unlike the regular version it displays the number of pages in each file. But more interestingly, if you search in Series ST84/1 you see more than metadata – you see the people inside.

As Barbara Reed noted in her article ‘Reinventing access’, ‘records are imbued with people’. Series ST84/1 goes by the fairly benign title, ‘Certificates of Domicile and Certificates of Exemption from Dictation Test, chronological series’. But of course the Dictation Test was the administrative backbone of a racist system designed to exclude people who did not fit the widely-accepted vision of ‘White Australia’. ST84/1 is full of people just trying to live their lives under the weight of the White Australia Policy.

The certificates in ST84/1 allowed people, born or resident in Australia, to return home after travelling overseas. If your ‘whiteness’ was suspect and you had no certificate, you would be subjected to the Dictation Test, and you would fail. The certificates usually include photographs and handprints – they are compelling and confronting documents. But you have to dig through layers of metadata in RecordSearch to see that. Or do you?

About five years ago, Kate Bagnall, a historian of Chinese Australia, and I were thinking about ways of drawing attention to these records. In a little over a weekend, I harvested about 12,000 page images from ST84/1 and ran them through a facial detection script. The result was ‘The Real Face of White Australia’.

You may have seen it before. It’s had a remarkable life, travelling around the world as an example of how we can use digital tools to see records differently. But of course the power is in the faces themselves – in the connections we make through time. We cannot escape their discomfiting gaze.

You may think that the certificates in ST84/1 are merely a form of identity document. But remember than in the early years of the 20th century, passports were still evolving, and the use of photographs and fingerprints for identification were generally confined to prisoners and criminals.

The sociologist Richard Jenkins talks about identity not as an essence, a noun, but as ‘something that we do, a process of identification’.1 But self-identification is constrained by broader systems of categorisation, or ‘social sorting’, that decide who belongs, who is a threat, who needs to be watched.

The records in ST84/1 were embedded within a system of surveillance that extended outwards from Australia’s ports to the offices of shipping companies around the world, and inwards to anyone who seemed out of place in White Australia. Technologies of identification and surveillance do not simply enforce boundaries, they create them. Their existence demonstrates why they are needed. These records did not document identity, they defined it according to a set of racial categories.

Modern parallels are not hard to find. Last year in a bungled operation that became known as ‘#borderfarce’, immigration officials planned to prowl the streets of Melbourne on the hunt for illegal immigrants. The focus of border surveillance once again turned inwards, to those who seemed out of place. Watching in horror as events unfolded on social media, I helpfully pointed people in Melbourne to a convenient source of identity documents.

In Melbourne this weekend? Get your Border Force identity papers here: http://t.co/H3dlXHZy6q #borderfarce #wap pic.twitter.com/j2TrkKEPkR

— Tim Sherratt (@wragge) August 28, 2015

The link I tweeted was not to RecordSearch, but to an experimental interface where Kate and I are continuing to think about ways of exposing the bureaucratic remnants of White Australia.

So far I’ve harvested metadata from more than 20,000 files and downloaded around 150,000 page images. Amongst other things, I’m working on an updated wall of faces.

Most recently I created a way of sorting and viewing pages by their orientation – by the ratio of height to width. Why? Kate wanted an easy way of finding birth certificates which, in this period, tended to be short and wide. It was a simple little hack, but it revealed the records in a very different way.

There’s a big bunch of birth certificates, but there’s also envelopes, photographs, and an assortment of slips and notes. It makes you think about how we see the written world through the frame of ‘portrait’ orientation. It’s perhaps also worth noting that RecordSearch displays thumbnails as cropped squares – shape is subsumed to the regularity of the grid.

The records in our landscape view are no more accessible than they were before, but they can be accessed differently. In his contribution to the ASA’s 30th anniversary symposium, Eric Ketelaar described the relationship between archival access and human rights – highlighting the importance of access not only to democratic accountability, but also to our rights as ‘victims’ of official surveillance: ‘As human beings, subjected to the panoptic sort of governments and private enterprise, we have the right to know’.2 Ketelaar concludes by stating that ‘access is not the actual use of archives, access enables use’. I’d like to extend this a bit. What my own work reveals is the complex relationship between access and use. Not only does access enable use, use changes what we mean by ‘access’.

My bio nowadays describes me as a ‘historian and hacker’. ‘Historian’ describes my orientation to the world – I see the past in the present. ‘Hacker’ refers to the tools I use to make connections through time. Hacking is creative and positive, despite what the mainstream media might say – it’s about finding solutions, exploring alternatives, and pushing the limits of what’s possible.

My RecordSearch hack is just a little piece of Javascript code that you can install in your browser – it changes the pages as they load.

Userscripts, as they’re called, allow anyone to alter the way webpages look and behave within their own browser. They give users greater control over their online experience, but they also open opportunities for experimentation. A lot of my own research is guided by questions like: What would happen if I…? What would it look like? What would change? What would I feel? Userscripts are one way of playing with the complexities of access.

Surveillance too can be hacked. If you’re concerned about technologies such as facial detection you’ll be pleased to know that thanks to the work of artist Adam Harvey, not only can you confuse detection algorithms, you can make a dramatic fashion statement.

Gary Marx, a major figure in the field of surveillance studies, has catalogued the ways in which individuals can resist the growing encroachments of surveillance. Amongst possible tactics he identifies ‘discovery’ – the attempt to undercover the scope of surveillance.3 Access to records can empower such acts of everyday resistance, but in other ways surveillance and access are more alike than opposed. Both start from a place of concealment. Access cannot be given unless it is first restricted. Both depend on asymmetries of power. Decisions about what we can know are ultimately made by others. Access is as much a process of control, as it is an act of release.

This is not necessarily a bad thing. I think we can all agree there should be limits to access, particularly relating to individual privacy and cultural sensitivity. But just as ‘identity’ is defined through acts of ‘identification’, so access is elaborated through instances of deployment and use. Access is not a state of being, it’s a process to be negotiated. And so the question is, what can we know about what we can know?

On 1 January this year I harvested the metadata of all the files in RecordSearch with the access status of ‘closed’.

These are records that the access examination process has determined should be withheld from public scrutiny. While the files themselves can’t be seen, RecordSearch does tell us a fair bit about them, including when the decision was made and the reasons behind it. Unfortunately you can’t search or filter on this data, so it’s difficult to look for patterns within RecordSearch itself.

I’ve taken the data and loaded into a new site where you can examine it from a number of different angles. You can explore the reasons why files were closed, the series they came from, the age of their contents, and the dates when decisions were made. It’s more of a workbench than a discovery interface, and it’s likely to change as I ask different questions of the data.

Of course, the outlines of the examination process, including the grounds for exemption, are defined by the Archives Act. So what more is there to know?

Section 33 of the Archives Act does indeed spell out 17 reasons why records can be withheld from public access. But the data from RecordSearch includes an additional 11 categories. Some, like ‘Parliament Class A’, relate to other definitions under the Act. Others, like ‘MAKE YOUR SELECTION’, tell us something about the RecordSearch interface. But two of the most heavily cited reasons – ‘Pre-access recorder’ and ‘Withheld pending advice’ are not defined under the Act or anywhere that I could find on the Archives’ website.

Being an archivally-educated audience you can probably guess what these labels refer to, but if you need a little help you can look at when the access decisions in these categories were made.

The majority of decisions on ‘pre access recorder’ files were made before the introduction of the Archives Act in 1983. I checked this with the Archives and they confirmed that these records were examined before the existence of the Act. They explained that ‘pre access recorder’ was used when the original exemptions couldn’t be mapped to those later defined under Section 33.

Conversely, most decisions on ‘Withheld pending advice’ files were recorded in the last five or six years. If you look at the series that contain the most files citing this reason you can see that almost half come from A1838 – DFAT’s main correspondence series. As I’m sure you’ve realised, these are files that have been referred back to agencies for advice. And DFAT has been particularly slow in responding. They’re listed as ‘Closed’ on RecordSearch even though their access status has not been finalised. They’re not, however, included in the count of ‘closed’ files that the Archives reports in its annual summary of access outcomes.

This is probably fair enough, but if you search the closed files you can see that 1,467 of them files have been waiting for more than three years for a final decision. They might not be officially closed, but for a PhD student wanting to see them they are effectively closed.

My point here is not to be critical of the National Archives, or even of DFAT. What I’m interested in is the inevitable gap between legislation and practice. Access examination is subject to a range of influences and constraints, resourcing amongst them, and needs to be understood not as the application of a set of rules, but as a process that is historically contingent. A human process.

Earlier this year I harvested several gigabytes of parliamentary proceedings from the Australian Parliament’s ParlInfo database and created my own version of Historic Hansard.

As you do.

In the process of harvesting the files I discovered that data was missing for about 100 sitting days – most of them from the Senate between 1912 and 1919.

There’s no conspiracy at work here, it’s just some sort of processing error. However, Parliament staff weren’t aware of the problem, and it’s unlikely that anyone would’ve noticed using the web interface. You can’t find what you can’t find. Fortunately, Parliament staff are now working on a fix, but if you’ve been relying on ParlInfo for access to debates relating to World War I, you might want to do some more checking.

These things happen. Systems go wrong. Mistakes are made. Again what interests me is not finding who’s to blame, but exploring the gap between design and outcome, between ideal and reality. This is the gap where access is made and experienced. A gap that can only be understood through the complexities and contradictions of use. Access does not exist until its limits are tested. It’s not a process of opening, it’s a constant ongoing struggle over the very meaning of ‘open’.

And that’s a good thing.

We’re here at this conference to explore the possibilities of ‘forging links’. But of course collaborations don’t have to be comfortable to be constructive. The struggle over access may sometimes be tense, frustrating, and annoying, but it is also productive. Users of archives do not just consume access, they create it.

About a year ago I fired up my RecordSearch harvester and downloaded the metadata and page images for most of the ASIO (Australian Security Intelligence Organisation) files publicly available through the National Archives. I ended up with about 70gb of images. These are mostly dossiers on individuals and organisations – odd collections of gossip, published articles, and records of surveillance.

I’ve made this data available for anyone who wants it. Some of the images were recently used in the GovHack open data competition to create the ‘Cute Commies’ site.

To be honest, I didn’t have a clear purpose in mind when I harvested the data. It was another one of those ‘What would happen if?’ moments. I was, however, thinking generally about possible points of comparison between the ASIO files and the archival remnants of the White Australia Policy. Both built systems of identification, classification, and surveillance in which recordkeeping was crucial.

Kate and other historians of Chinese Australia have noted that the administration of the White Australia Policy was not uniform or consistent. Similar cases could result in quite different outcomes depending on the location and those involved. Understanding this is important, not only for documenting the workings of the system, but for recovering the agency of those subjected to it. Non-white residents were not mere victims, they found ways of negotiating, and even manipulating, the state’s racist bureaucracy. In her work on colonial archives, Ann Laura Stoller identifies this ‘disjuncture between prescription and practice, between state mandates and the manoeuvres people made in response to them’ as part of the ‘ethnographic space’ of the archive.4

How do we explore this space? One of the things I’ve found interesting in working with the closed files is the way we can use available metadata to show us what we can’t see. It’s like creating a negative image of access. Kate and I have been thinking for a number of years now about how we might use digital tools to mine the White Australia records for traces, gaps, and shadows that together build a picture of the policy in action. Who knew who? Who was where and when? What records remain and why?

The workings of ASIO, on the other hand, are deliberately obscured. Many of the files in the Archives include a note explaining why details have been withheld. Some warn that the ‘public disclosure of information concerning the procedures and techniques used by ASIO’ would enable people of interest to formulate counter-measures ‘based on an analysis of ASIO modus operandi’. David Horner’s recent history of ASIO notes that he was required to remove ASIO file references from his footnotes ‘because of the nature of ASIO’s filing system, which itself is classified’.5 We don’t even know how many files ASIO has on people and organisations, although David McKnight suggests that it’s somewhere in the hundreds of thousands.6 My harvest includes about 12,000 files.

Just like systems of racial classification, intelligence services exist within a circle of self-justification. The fact they exist proves they need to exist. We are denied information that might enable us to imagine alternatives. And yet as limited as the provisions under the Archives Act are, we do have access.

How can we use this narrow, shuttered window to reverse the gaze of state surveillance and rebuild a context that has been deliberately erased. Just as with Closed Access and the White Australia records can we give meaning to the gaps and the absences? Can we see what’s not there?

This is one of the questions being explored by Columbia University’s History Lab. They’ve created the Declassification Engine – a huge database of previously classified government documents that they’re using to analyse the nature of official secrecy. By identifying non-redacted copies of previously redacted documents, they’ve also been able to track the words, concepts and events most likely to censored.

The History Lab’s collection of documents on foreign policy and world events is rather different to ASIO’s archive of the lives, habits and beliefs of ordinary Australians. But I’m hoping that they too can tell us something about the culture that created them.

I’d intended to have a wonderfully compelling suite of examples and arguments to demonstrate today, but time has run short. Instead I have a set of half-baked experiments which sort of look a bit interesting. But perhaps that’s better. It’s important to me to try and be open about my own processes. I share my code and data, and I’ve started documenting most of what I’m up to in a open research notebook. If access is a struggle, then we should be sharing our stories of loss and frustration, and not merely celebrating our victories.

All of these experiments are online in some sort of form. So please explore.

Experiment A is nothing more than a browse interface to all the digitised records I’ve harvested. It’s just a clone of my work with the White Australia records, but I think there’s real conceptual power in the ability to browse.

Experiment B started with a problem. From RecordSearch I could harvest data on access status and find out how many ASIO files were in each of the three categories – Open, Open with Exception, and Closed. But how much of the ‘Open with Exception’ files are actually open?

Most of the files include a summary which tells you how many pages have been completely or partially exempted. That’s great, but did I really want to open up 12,000 files and manually scan for summaries? By playing around with the Tesseract OCR engine I’ve created a simple filter that extracts text from the images and searches for words like ‘exemption’, ‘archives’, and ‘folio’. I now have a good sized collection of summaries awaiting data entry…

Experiment C began as another attempt to quantify the scale of exemption. The summaries told me how many pages had redactions – bits of information like names and ids that are blacked out, or sometimes even cut out of the page. But if I could identify individual redactions I could both test the summaries and create a new measure of openness… or redactedness…

Looking for redactions in ASIO files — that looks hopeful… pic.twitter.com/heeWMAQiwe

— Tim Sherratt (@wragge) September 11, 2016

Through trial and error I developed a computer vision script that did a pretty good job of finding redactions – despite many variations in redaction style, paper colour, and print quality. It took a couple of days to work through the 300,000 page images, but in the end I had a collection of about 300,000 redactions. Unfortunately about 20 percent of these were false positives, so I spent a number of nights manually sorting the results.

But the false positives themselves are sort of wonderful… pic.twitter.com/Jn9l04bUaY

— Tim Sherratt (@wragge) September 29, 2016

My redaction finder still needs a lot of refinement, and plenty of errors have slipped through. But, within the files that are currently digitised, the scale of exemption seems about ten times greater than Margaret Kenna estimated when giving evidence to Parliamentary Joint Committee on ASIO in 2000. She thought every file contained about 10 exemptions ‘be it a word or a folio or a paragraph’. I’m seeing an average of about 100 redactions per file.

I’ve started adding information about the size and position of the redactions to my database and aggregating this data by page. When I left Canberra, the script was still running, but you can explore the current standings in my top 50 lists of the most redacted files and pages.

Once the data processing is completed you’ll be able to filter files by the amount of area blacked out, or the total number of redactions. Many more opportunities to see what you can’t see.

Experiment D was an attempt to build a composite image of all the redactions to visualise what parts of a page were most likely to be be removed – something like a heatmap. It sort of worked, but by the time I’d added all the redactions I had nothing but a very large black blob.

My composite of 170,000 #ASIO #redactions has opened a portal to another universe… pic.twitter.com/39iQewjst4

— Tim Sherratt (@wragge) October 8, 2016

A rethink is required…

Experiment E had two aims. First to highlight the visual character of the redactions themselves – there’s a strange sort of beauty in a massed collection of blobs. Secondly, just as with the Real Face of White Australia, I wanted to turn the files inside out. Instead of being dead ends, I wanted the redactions to be discovery points, signposts, ways of exploring the files.

It’s online now so play. You can view a random sample of redactions, or browse page by page through the entire collection.

Talking about her own ASIO file in the book Dirty Secrets, the politician and academic Meredith Burgmann noted that the ‘blacking out process seems totally arbitrary and for the reader terribly frustrating, like reading a detective novel with the last page torn out’.7 But in hunting for redactions I found they could also bring moments of unexpected joy. It seems that someone got a bit bored and has left us with a glorious collection of redaction art.

So what’s to come? I need to rework my redaction finder to improve its accuracy.

It’s interesting, and perhaps ironic, that the removal of information has given me an identifiable data point that I can potentially track against other characteristics of the files. Can I identify patterns by time or topic?

Apparently ASIO assessments have become less conservative over the years – I can test this by looking at changes in redaction rates over time.

I also want to explore the context of redactions. By expanding the window around redactions and OCRing the result, I hope to identify the words that occur most commonly appear near redactions.

Those of you coming to the workshop on Friday will hear more about some of the tools and technologies I’ve used in these experiments. But I wanted to give a brief overview today because this is access.

Digital tools and technologies give us the opportunity to use databases like RecordSearch as archaeological sites to sift through layers of metadata in search of new connections and meanings. This is access.

We can turn digitised collections inside out, revealing the people, the processes, the structures, the form. This is access.

We can reveal the processes through which records are controlled, concealed, and withheld. This is access.

Access is not a deliverable or a product. It’s a struggle for understanding and power – not just to see, but to see differently.

This is RecordSearch but not as you know it.

Experiment F is a userscript that puts the redactions back into RecordSearch. Access is an honest acknowledgement of its own limits, and an invitation to push beyond.

1. Richard Jenkins, ‘Identity, surveillance and modernity: Sorting out who’s who’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 159.
2. Eric Ketelaar, ‘Access, the democratic imperative’, Archives and Manuscripts, vol. 34, no. 2, November 2006, p. 73.
3. John Gilliom and Torin Monahan, ‘Everyday resistance’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 407.
4. Ann Laura Stoller, Along the Archival Gran: Epistemic Anxieties and Colonial Common Sense, Princeton University Press, Princton, 2009, p. 32.
5. David Horner, The Spy Catchers: The Official History of ASIO, Allen & Unwin, Sydney, 2014, p. 581.
6. David McKnight, ‘How to read your ASIO file’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 38.
7. Meredith Burgmann, ‘The secret life of B/77/26 (and friends)’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 454.

Presented at DHA 2016 in Hobart. Follow my research. View the slides.

A good type

In January 1954, the economic historian Ernst Boehm requested access to Commonwealth government records as part of his research into the depressions of the 1890s and 1930s. The request was forwarded from the Archives Office to the Prime Minister’s Department where it was noted ‘Boehm is I believe a good type + will agree to the usual conditions’.[1]

This sort of character assessment was important because there were few formal guidelines around access to government records. The Commonwealth Archives Committee reported in 1955 that the use of government archives was ‘seriously hampered by the lack of an access policy’.[2] The Prime Minister’s Department noted that requests for access were ‘treated on an “ad hoc” basis’.[3] Prospective researchers first needed to establish their ‘bone fides’. Archives staff would then attempt to identify records of interest and submit them to the relevant government departments for clearance. The Archives Committee noted that the ‘inevitable uncertainties and delays inherent in this system’ often left researchers ‘puzzled and disappointed’.[4]

In the years that followed, interdepartmental agreements and Cabinet decisions gradually refined the process, but it would be another thirty years before the public’s right to access was enshrined in legislation.

Right to Access

In 1955, the Cold War was near its height. The Royal Commission on Espionage was grilling public servants, scientists, journalists and others in its hunt for Communist infiltrators. Amidst the hysteria about ‘secrets’ and spies the Commonwealth Archives Committee emphasised the ‘moral obligation’ of democratic governments to make the records of their activities available for research. ‘In doing so’, the Committee argued, ‘they make a significant contribution to the continuing search for truth which is an essential part of the continuing democratic process’.[5]

The idea that the public had a ‘right’ to access government records did not go unchallenged, however. In a meeting with the visiting American archivist Theodore Schellenberg the previous year, a representative of the Department of Defence had admitted that he ‘found it difficult to accept the concept that the general public had a right of access to public records’.[6]

But the obligations of war and empire actually pushed Australia towards a more liberal policy. In the 1950s, Australia agencies tended to follow British precedent, applying a version of the 50 year rule. Given that the Commonwealth of Australia was founded in 1900, a 50 year embargo didn’t leave a lot of history to write. In 1966, however, the UK government opened access to records through to 1922 – encouraging research into World War I and its aftermath. Australia had little choice but to follow their example. As John Gorton noted in his submission to Cabinet, ‘we should not impose greater restrictions on our own scholars than are imposed in Britain’.[7] When the UK shifted to a 30 year rule a few years later, Australia again adjusted its policy.

In 1972 the Whitlam Labor government brought with it a professed commitment to ‘open government’. Access to archives started to be linked to broader questions around ‘freedom of information’. It took a few attempts, but the Archives Act was finally passed in 1983. In 2010 ammendments reduced the closed period from thirty to twenty years.

The public interest

Under the Archives Act (in its current form) the public has a right to access Commonwealth records more than twenty years old. But this right has limits. In 1954, Schellenberg described the ‘public interest’ as a balance between a liberal access regime and the protections necessary to shield both governments and individuals from the dangers of over exposure.[8] As access policy evolved, so did the articulation of these limits.

Protections for national security and the privacy of individuals have been long accepted as grounds to withhold records from public access. But the scope and meaning of such exceptions has shifted. In 1966, restrictions on ‘the release of papers relating to the personal expenditure of the Governor-General’ were provided as an example of how individuals were being protected from ‘embarrassment or distress’.[9] Similarly it was assumed that records ‘commenting on the person of the Sovereign’ would naturally be withheld.[10]

But the core of these exemptions remained and was codified within section 33 of Archives Act. Section 33(1)(a), for example, allows records to be withheld from access if they could ‘cause damage to the security, defence, or international relations of the Commonwealth’. Section 33(1)(g) exempts records that might result in ‘the unreasonable disclosure of information relating to the personal affairs of any person’.

The ad hoc approach of the 1950s has been replaced by a well-documented bureaucratic process with clearly defined outcomes. Under the Act a researcher can ask for a record to be examined for public access. The contents of the file is then assessed against the list of defined exemptions and an access status is assigned – Open, Open with exception, or Closed (sometimes closed files are described as ‘withheld’).

Statistics on the number of files examined and their access outcomes are reported annually. The vast majority are opened to the public. But some are not.

Open Access

What do we mean by ‘open access’? Back in 1964 the Interdepartmental Committee on Commonwealth Archives defined ‘open access’ as referring to documents ‘which have been cleared for consultation by any member of the public’.[11] Open could only really be understood by invoking its limits – open was not closed.

I’ve shared this potted history of Australian archival policy because it continues to shape the research that we do and what we can know.

These shifting policies and priorities have shaped the data now available online through RecordSearch, the National Archives of Australia’s database. As digital scholars we should recognise that there’s nothing natural or inevitable about these systems and their contents. They are deeply and profoundly historical.

But recognising this is both an obligation and opportunity, because we can view such systems as archaeological sites and dig back through the layers of data. We can meet the history of policy, with an analysis of practice as it has been captured in the descriptions, dates and decisions.

The access status of each item in RecordSearch is recorded and displayed – ‘Not yet examined’, ‘Open’, ‘Open with Exception’, or ‘Closed’. If a file is ‘closed’ or ‘open with exception’ the reasons for the restriction are provided along with the date when the decision was made. This is interesting and important data. But while you can filter your search by status, you can’t specify a reason or date. So I can browse a list of ‘closed’ files, but there’s no way of getting an overview of why they’re closed.

Until now.

Last year, as an experiment, I screen-scraped RecordSearch to capture the details of files that had an access status of closed. On the 1 January this year, I repeated the harvest with a bit more care.

I now have a dataset of 14,370 closed files. These are the files we’re not allowed to see.

Closed Access

You can explore them online using Closed Access, a public site I developed to play with the data.

You can view the aggregated metadata from a number of different angles – by the reason for exemption, by record series, by the age of the contents, and by the date of the decision. You can drill down from any of these views to a list of closed files. And of course you can search – want to know which files were closed in May 1989 for national security reasons? ClosedAccess can tell you!

Given that none of the files are actually publicly available, it’s possibly the most frustrating search interface yet devised. However, you can always ask for a file’s access status to be re-examined.

ClosedAccess is really more of a workbench than a discovery interface, with additional experiments documented in my open research notebook.

There for example, you can read about TF-IDF analysis of file titles, or my failed attempts to reconcile the statistics reported in the National Archives’ annual reports with the data from RecordSearch.

Reasons

So, why have these files been closed? Here’s the list of reasons.

The first thing to note is while there are 17 exemptions defined by the Archives Act, there are 28 reasons for closing files cited in RecordSearch. Some of these extra reasons relate to specific classes of records mentioned elsewhere in the legislation, while others, such as ‘MAKE YOUR SELECTION’ tell us something about the design of the data entry interface.

It’s no surprise that sections 33(1)(a) and 33(1)(g) of the Act are the most frequently cited exceptions. As I’ve noted, these relate to national security and individual privacy – longstanding grounds for withholding access.

More interestingly, these exemptions appear very frequently in combination with others. The DH standard network graph illustrates their centrality. For example of the 425 closed records citing 33(1)(b) – which refers to confidential information supplied by another government – 399 (94%) also cite 33(1)(a). This sort of pattern, combined with the low level of use for many of the exemptions, suggests that the whole legislative framework could be simplified – how many reasons to close files do we really need?

What’s more surprising are the two highly-cited reasons at the bottom of this graph, neither of which are defined in the legislation or anywhere that I’ve been able to find on the National Archives website. They’re ‘Pre access recorder’ and ‘Withheld pending advice’.

Pre-access recorder

According to the brief explanatory note in RecordSearch, ‘Pre access recorder’ denotes ‘non standard reasons for restriction’ – not very illuminating. But using ClosedAccess you can dig into the metadata to get an idea of what this actually means.

The average age of files citing ‘Pre access recorder’ is 75 years, significantly older than the overall average of 58 years. Indeed, if you look at the dates on which decisions were made about these files, you’ll see that most of these were closed before 1983 when the Archives Act was introduced. So it seems these files were examined and withheld from public access under the less formal arrangements that existed prior to the legislation.

This was (sort of) confirmed by the National Archives who told me that yes they were ‘made prior to the introduction of the Archives Act’, but they were also decisions ‘where the reason for restriction didn’t map to one of the reasons for restriction set out in the Act’. I still don’t quite understand this.

If you take all the file titles and throw them into Voyant you’ll see the word ‘staff’ figures prominently, reflecting the fact that many of them are personnel files. Surely these might have mapped to the privacy provisions of section 33(1)(g)?

From there things get even more complicated. Just as in quantum theory, by observing the phenomena of pre-access recorder, I have changed it. As a result of my inquiry, which the Archives seemed to interpret as a criticism of their metadata, they changed the status of most the ‘Pre access recorder’ files from ‘closed’ to ‘not yet examined’. They’re not any more open, but they no longer show up in a search for closed files. Fortunately, I’ve preserved their historical status in Closed Access. I’m planning to repeat my harvest on the 1 January each year, tracking newly closed files as well as those whose status has changed. So next year I’ll be able to check up on the ‘Pre access recorder’ files to see what’s really happened to them.

Withheld pending advice

Withheld pending advice is easier to interpret. The files have been referred back to the government agencies that created or controlled them to check whether they really can be unleashed on the public.

This process is known and feared by researchers because of the long delays it can create. In 2014 Philip Dorling wrote in the Canberra Times about a file from 1932 relating to Australia’s annexation of Antarctica which languished with the Department of Foreign Affairs and Trade for five years before being released in a redacted form.[12]

Delays in access examination are well-known, but the scale and impact aren’t obvious.

I think that part of the problem I was having in reconciling access examination data from annual reports was that the Archives doesn’t include the ‘Withheld pending advice’ files in its statistics. Perhaps this is fair enough – the access examination process isn’t complete. On the other hand, under the Archives Act if examination requests aren’t completed within 90 days then access is assumed to have been refused. Are they less closed because they may yet be opened? I’ve asked the National Archives what the status of these files really is, and how this affects the public’s right to access.

In the meantime we can use Closed Access to see the scale of the backlog and indulge in a bit of agency shaming. You can easily search for ‘Withheld pending advice’ files and order them by the date they were closed.

Ignoring the one file with the title ‘unknown’ that has been waiting 18 years for a decision (I suspect it’s been mislaid), there’s still a file that’s been waiting for 9 years, and 9 files that have been waiting for 7 years. Indeed, you can limit your search to files that have been waiting more than three years for a decision – there’s 1467 of them.

The Defence department gets the longest in limbo award for the 9 year delay, but DFAT is the overall champion with series A1838 containing almost 1500 files awaiting departmental determinations – 54 of which have been waiting more than five years. Too bad if you’re doing a PhD. Researchers are still being left ‘puzzled and disappointed’.

It’s important to make these processes visible because it’s all to easy to forget that the default status for any Commonwealth government file more than 20 years old is ‘open’. The reasons for closing files defined under the Archives Act are called exemptions for this very reason.

Philip Dorling suggested that agencies are becoming more conservative in their decisions. Once I’ve harvested a few year’s worth of closed files, I’m hoping I’ll be able to trace the fate of those poor souls currently ‘withheld pending advice’. But there are other measures of conservatism we can start to explore.

You’d think that the older a file was the more likely it was not to be closed. So what happens when we look at the gap between the date a decision was made to close a file and the date of it’s latest content? I’ve added an experimental view to Closed Access where you can explore the files than seem to have provoked the most conservative approach.

Strangely, at the top of this list are the proceedings of the Senate from 1903. But in fact, no decision was made to close this file. It belongs to a category of records called ‘Parliament Class A’. Regulations drawn up under the Archives Act simply declare these records as ‘exempt’. You need the permission of Parliament to access them.

It’s a reminder that at least we do have the metadata for closed files. The intelligence agencies hold unknown quantities of files where access to the recordkeeping systems themselves is restricted. We don’t know what we don’t know.

I’m hoping to explore these gaps further, but for now if you’d like a copy of the 300,000 digitised pages contained within publicly accessible ASIO files let me know.

No magic

It’s important to recognise all of this as the workings of a system. Digging through the layers of policy and metadata helps us to see that there is nothing intrinsically dangerous in a closed file. A judgement has been made against a list of historically determined criteria.

But it’s also important to resist the temptation to claim privileged access to the truth through our mastery of such systems. Next time you see words like ‘recently released documents’ in a book or article, remember that access examination is a process without magic or wonder. We need to remain critical of the process without adding layers of mystification – without succumbing to the cultural power of the secret.

Keynote presented at DH2015, 3 July 2015. Full slides available on SlideShare.

This, you might be surprised to learn, is not the first time that Australia has welcomed some of the world’s leading thinkers to its shores. Just over a hundred years ago, the British Association for the Advancement of Science held its annual meeting in Australia. In earlier years the Association had journeyed to Canada and South Africa, but this was it’s first tour of Australia. One senior Australian scientist heralded the Association’s arrival as ‘a great event in the history of Imperial unity’.

More than 300 scientists made the trip, including such notables as Ernest Rutherford and William Bateson. I’m a little embarrassed to admit that their travel was heavily subsidised by the Federal government. But then, it did take them more than a month to get here. Think about that on your flight home.

The eminent Australian geologist Edgeworth David described the Association’s visit as ‘an epoch making event’. He expected Australian researchers to be ‘strengthened and confirmed’ in their work, reaffirmed through the ‘inspiration which comes alone from personal contact with master minds’.

It was also an occasion to celebrate the ideals of science. War had been declared while the scientists were at sea, but events proceeded nonetheless with delegates barnstorming across the country from Adelaide to Melbourne, Sydney and Brisbane. The spirit of proceedings was summed up in Melbourne where the presentation of an honorary degree to the German geologist Johannes Walter was greeted with a ‘perfect storm of applause’. ‘Truly science knows not distinction between belligerent and belligerent’, noted one newspaper. Australia’s Governor General, Sir Ronald Munro Ferguson, welcomed the scientists with the observation that the looming dangers of war had at least ‘enabled them to realise that all men of science were brothers’.

And of course, they were mostly men.

If you’d like a bit of data around that, you can grab a digitised copy of the report of the meeting from the Internet Archive and run a script over the list of members, grouping them by title – Miss, Mrs and Lady. Here’s what you get.

You can do the same for the people who joined the Association at one of the Australian venues.

Ok, so this 10 minute analysis might not show anything unexpected, but I love the fact that with a digitised text and a few lines of Python I can ask a question and get an almost instant answer.

What the official report doesn’t say is that despite these proclamations of scientific brotherhood, not all German scientists were welcome in wartime Australia. Those who extended their stay beyond the meeting dates fell under suspicion.

Two of them, Fritz Graebner and Peter Pringsheim, were interned as suspected spies and imprisoned for the remainder of the war. The press which had fawned over the travelling savants now railed against these ‘scientists in disguise’ whose ‘supreme act of treachery’ was undoubtedly part of a German plot to capture Australia. The Minister of Defence noted that the case emphasised the ‘real and pressing nature’ of the wartime emergency. Honorary degrees awarded to two German scientists by the University of Adelaide were expunged from the record.

At this point I feel I should warn all our international visitors that legislation introduced in recent years to combat the so-called ‘war on terror’ has added new limits to our freedom of speech and movement. We are all under suspicion.

The German scientists were interned alongside many thousands of others. Most had had no charges brought against them. Many were naturalised British subjects, or Australian-born of German descent. Australia was their home. That didn’t stop the government repatriating many of them to Germany at war’s end.

To orient them on their antipodean adventure, visitors for the British Association meeting were supplied with specially-prepared handbooks that described conditions in Australia. At a time when violence against Indigenous people was still common along the frontiers of settlement, the Commonwealth Handbook informed visitors that Australian Aboriginals ‘represent the most backward race extant’.

Australia was big, but its population was small. The Commonwealth Handbook noted the challenges of maintaining ‘control of so large a territory by a mere handful of people’, pointing to the significance of the ‘White Australia’ policy in avoiding the ‘difficulties’ of ‘heterogenous’ populations. Chris Watson, who served as Australia’s first Labor prime minister a decade earlier, expanded on this theme in the NSW Handbook. Concerns about the financial impact of ‘coloured’ labour, he explained, had been fused with an ‘abhorrence of racial admixture’ to create ‘practically a unanimous demand for a “White Australia”’. ‘White Australia’ was both an ideal and an obligation, an opportunity and a threat. Watson observed:

The aboriginal natives are numerically a negligible quantity, so there is every opportunity for the building up of a great white democracy if the community can maintain possession against the natural desire of the brown and yellow races to participate in the good things to be found in the Commonwealth. That the Asiatic will for ever tamely submit to be excluded from a country which, while presenting golden opportunities, is yet comparatively unpeopled, can hardly be expected. Therefore Australians are realising that to maintain their ideals they must fill their waste spaces and prepare for effective defence.

Welcome to Australia a hundred years later where we remember 1914 not for its institutionalised racism, but because it marked the beginning of a war that has come to be strongly associated with ideas of Australian nationhood.

You have arrived here amidst the ‘Anzac Centenary’ which, the official website notes, ‘is a milestone of special significance to all Australians’. It must be true because, according to the Honest History site, we’re spending more than half a billion dollars on commemorative activities. That’s a lot of remembering.

Amidst the travelling roadshows, the memorials, the exhibitions, and the rolling anniversaries, are of course many worthy digital projects. Some of these will provide new access to war-related collections, or gather community content and memories. They will result in important new historical resources. But who are we remembering and why? As a historian and hacker, as a maker of tools and a scraper of sites, I want today to poke around for a while amidst the complexities of memory.

It’s not all about the war. Recent decades have brought attempts to remember more difficult histories. Peter Read coined the phrase ‘stolen generations’ to draw attention to the devastating effects of official policies that resulted in the forced removal of Indigenous children from their families, through until the 1970s. The damaging experiences of children in institutional care, the ‘forgotten Australians’, have also been opened to scrutiny. Both of these have brought official apologies from the Commonwealth government. Even now, almost every day brings more horrifying testimony as the Royal Commission into Institutional Responses to Child Sexual Abuse continues its hearings.

In each case we have learnt to our shame of continuing failures to protect the most vulnerable in Australian society – children.

Often these investigations are cast as attempts to bring to the surface forgotten aspects of our history. But to those who suffered through these events, who have continued to live with the consequences, they have never been far from memory.

Nor have they been entirely lost to the historical record. One of the responses to these inquiries has been to discover, marshal and deploy existing archival resources. The National Archives of Australia created an exhibition based on the experiences of some of the Stolen Generation. They also developed a new name index to their collections to help Indigenous people reconnect with their families through official records.

The eScholarship Research Centre at the University of Melbourne drew on its experience in documenting a wide variety of archival collections to create Find & Connect – a web resource that assembles information about institutional care in Australia and assists care leavers in recovering their own stories. Official records have been supplemented by oral history programs and other collecting initiatives to ensure that these memories are secure.

Such histories are ‘forgotten’ not because they are unremembered or undocumented, but because they sit uncomfortably alongside more widely promulgated visions of Australia’s past. As researchers on the Find & Connect project noted, the stories of care leavers ‘did not “fit in” with the narratives in the public domain. Their memories were “outside discourse”’.1 Remembering the forgotten is not just a matter of recall or rediscovery, but a battle over the boundaries of what matters.

Libraries, archives and museums are often referred to as memory institutions. Rhetorically it can be a useful way of positioning cultural institutions in respect to structures of governance and assessments of public value. The idea of losing our memory, whether as a society or an individual, is frightening.

But there are contradictions here. We frequently talk about memory in terms of storage – the ability of our technologies to tuck away useful pieces of information for retrieval later. There’s the ‘M’ in RAM and ROM, the fields in our database, our backups in the cloud. Memory is an accumulation of key/value pairs. Each time we query a particular key, we expect to get the same value back.

Memory, as we experience it, is something quite different. It’s fragmentary, uncertain, and shaped by context. The process of recall is unpredictable and sometimes disturbing – memories are often triggered involuntarily. Within a society memories are contested and contradictory. Who controls the keys?

Cultural institutions are trying to respond to this complexity. On the one hand they offer the security of authority – sources to be trusted in world overflowing with information. But they are also looking for ways of capturing and representing alternative voices.

I think we can help with that.

Both in my work at Trove and my own noodling about I use the word ‘access’ a lot. But the more I use it the more I suspect it really doesn’t mean very much. What does it say that we now distinguish between ‘open’ and ‘closed’ access?

We tend to think of ‘access’ as the way we get to stuff. It’s the pathway along which we can explore our cultural collections. But as Mitchell Whitelaw argues, one of our primary means of access, the common or garden variety search box, constrains our view of the resources beyond.  Search provides not an open door, but a grumpy ‘Yes, what?’

I’d suggest that these sort of constraints don’t stand in the way of access, they construct it. Through legislation, technology, and professional practice, through the metadata we create and the interfaces we build, limits are created around what we can see and what we can do. Access is a process of control rather than liberation.

In 1952, in another notable act of ‘imperial unity’, Britain exploded an atomic bomb off the coast of Western Australia. A further 11 atomic tests were carried out here, most at a mainland testing site called Maralinga in South Australia. As a young research student in 1984, the British atomic tests introduced me both to the gloriously rich collections of the National Archives of Australia, and to the contradictions of access.

Under the Archives Act, most government records are opened to the public after 20 years (this was reduced from 30 years in 2010). However, before they are released they undergo examination to see whether they contain material that is exempted from public access – for example any secret squirrel business that could endanger our national security. The access process can therefore result in records that are ‘closed’ or ‘open with exception’.

What does ‘closed’ access look like? A few weeks ago I harvested details of all the files in the National Archives’ online database that have the access status ‘closed’. The records include the reasons why the files remain restricted. If you group them by reason, you can see that the most common grounds for restriction is Section 33(1)(g) of the Archives Act which seeks to prevent the ‘unreasonable disclosure of information relating to the personal affairs of any person’. Fair enough. Coming second is the rather less obvious category of ‘withheld pending advice’. These are files that have gone back to the government agencies that created or controlled them to check that they really can be released. So they’re actually part way through the process.

Using the contents dates of the files we can see how old they are. Section 33(1)(a) of the Archives Act exempts records from public scrutiny if they might ‘cause damage to the security, defence or international relations of the Commonwealth’. Most of the records closed on these grounds are over 50 years old, with a peak in 1956.

And here’s a word cloud of the closed file titles from 1956. I’m sure that we all feel a lot safer knowing all those Cold War secrets are still being protected.

Back in 1984 I asked for some of those secret files to be opened so I could write my Honours thesis on the role of Australian scientists in the British Atomic tests. A number of the files I was interested in went off to agencies for advice, and some even made their way to the British High Commission. Being young, optimistic, and on a deadline, I wrote to the British High Commissioner asking if anything could be done to speed the process up.

I received a very polite reply explaining that they were obligated under the Nuclear Non-Profileration Treaty to make sure that they didn’t unleash any atomic bomb secrets upon the world. This was hilariously and tragically ironic, as the argument of my thesis was that the British government withheld information from their Australian hosts to curry favour with the USA. There was no way that atomic bomb plans would be in Australian government files. Yeah – hilarious.

Access is political. Cassie Findlay has contrasted the Australian government’s processes for the release of records with the creation and use of the WikiLeaks Cablegate archive.2 Cassie argues that the ‘hyper-dissemination’ model of WikiLeaks, through which large volumes of material are shared across multiple platforms, creates a ‘pluralised archive’ that ‘exists beyond spatial and temporal boundaries, transcends state and economic controls and encourages and incorporates people’s participation and comment’. Instead of gatekeepers and reading rooms there are hackers and torrents.

Traditional forms of access are often celebrated as if they are a gift to a grateful nation. As Cassie notes, the release of Cabinet documents by the National Archives is a yearly ritual where stories of 30 year old political manoeuvring are mixed with the comforts of nostalgia. But with each release more files are closed, withheld from public access. The workings of a bureaucratic process developed to control the release of information is recast as an opportunity to party. Invested with the cultural power of the secret and the political weight of national security, access itself becomes mysterious and magical.

We are left to ponder such wonders as: ‘Named country [imposed title, original title wholly exempt]’.

At the same time, governments are pumping out ‘open data’, bringing the promise of greater transparency, and new fuel for the engines of innovation. But for all its benefits, open data isn’t. It only exists because decisions have been made about what is valuable to record and to keep – structures have been defined, categories have been closed. As Geoffrey Bowker and Susan Leigh Starr remind us, the definition, elaboration and enforcement of categories lies at the heart of bureaucracy and the infrastructure of the state.3 Data is not just a product of government, it is implicated in the workings of power.

Chris Watson’s vision of a White Australia was, by 1914, well established as a system of bureaucratic surveillance and control. The Commonwealth Handbook benignly noted that ‘an immigrant may be required to pass a dictation test before being admitted into the Commonwealth’. It added that ‘in general practice this test is not imposed upon persons of European race’. The dictation test was a mechanism of exclusion. Any intending immigrant deemed not to be ‘white’ would be subjected to the dictation test and they would fail. But there were already many people born or resident in Australia of Asian descent. If they wanted to travel overseas they were forced to carry official documents to protect them from the application of the dictation test – otherwise they might not be allowed to return home. Many thousands of these documents are now preserved in the National Archives of Australia. With portrait photographs and inky-black handprints, they are visually compelling, and disturbing, documents. They need to be seen.

A few years ago, my partner Kate Bagnall and I harvested thousands of these documents from the National Archives website, ran them through a facial detection script and created ‘The Real Face of White Australia’.

You might have seen it before. It’s been widely cited, and it’s probably one of the main reasons I’m standing here today. For Kate and me this was part of our ongoing attempts to use the bureaucratic remnants of the White Australia policy to reconstruct the lives of those who lived within its grasp. But it’s also an example of the complications of access.

In the past I’ve tended to gloss over the hardest part of this project – just harvesting those 12,000 images. It was only possible because I’d spent a lot of time, over a number of years, wrestling with RecordSearch, the National Archives’ online database. I think it was back in 2008 that I wrote my first Zotero translator to extract structured data from RecordSearch. It was one of those Eureka moments. Although I’d been developing web applications for a long time, I hadn’t really thought of the web as a source to be mined, manipulated and transformed. I could take what was delivered in my browser and change it.

Thanks to the Bill Turkel and the Programming Historian, I taught myself enough Python to be dangerous and was soon creating screen scrapers for a variety of sites – taking their HTML and turning it into data. I was no longer bound to a particular interface. The meaning of access had changed.

But screen scrapers are a pain. Sites change and scrapers break. I don’t know how many hours I’ve spent inspecting RecordSearch response headers, trying to figure out where my requests were going. I’ve given up several times, but always gone back, because there’s always more to do.

Amongst the enthusiasm for open data there’s perhaps a tendency to overlook the opening of data – the way that hackers, tinkerers, journalists, activists and others have been stretching the limits of access.

The various projects of the Open Australia Foundation are a great example of this – they’ve even established their own public scraping framework called Morph, to share both the code and the data that’s been liberated from websites and pdfs.

The Australian Parliament recently passed changes to the Copyright Act that will enable copyright holders to apply to the Federal Court to block piracy-related websites. Of all the changes needed to copyright, this is the one that went to Parliament.

But what I love is that even before the legislation had passed, even before the first application has been made or site blocked, there was a website and Twitter account ready to document and publicise any site-blocking orders – created not by government, but by an ABC journalist.

Archivists Wendy Duff and Verne Harris have talked about records ‘as always in the process of being made’, not locked in the past but ‘opening out of the future’.4 Cassie Findlay similarly notes that the Cablegate archive ‘is still forming’. She argues for models of participation and access around archives that open ‘more directly from the affairs that they document’.

The act of opening – records, archives, sources – is contingent and contextual. It creates a connection between inside and outside, past and present, us and them. What we do with that connection is up to us.

What would have happened if instead of hearing about ‘prohibited immigrants’, instead of seeing ‘wanted’ posters of escaped Chinese seaman, Australians in 1914 had seen something like our wall of faces?

What would happen now if instead of hearing about ‘illegal maritime arrivals’ (IMAs) we were exposed to the stories of those who arrive in Australia in search of asylum?

Access will never be open. Every CSV is an expression of power, every API is an argument. While I would gladly take back the time I’ve spent wresting data from HTML I recognise the value of the struggle. The bureaucratic structures of the White Australia policy live on in the descriptive hierarchies of the National Archives. To build our wall of faces we had to dismantle these structures – to drill down through series, items, documents and images until we found the people inside. I feel differently about the records because of that. Access can never simply be given, at some level it has to be taken.

In 1987 I ended up outside the gates of Pine Gap, a US intelligence facility near Alice Springs, dressed as a kangaroo. Having finished my honours thesis on the British atomic tests, I couldn’t ignore the parallels between the bombs and the bases. I even organised a conference entitled, ‘From Maralinga to Pine Gap: The historical fallout’. I remember pulling over on the road to Alice Springs because there was one point where you could just glimpse the top of one of the white domes that protected Pine Gap’s receivers. It was a pretty thrilling moment.

Now you can just type ‘Pine Gap’ into Google Maps and there it is.

It’s still secret, it’s still gathering unknown quantities of electronic intelligence, but last time I checked it also had 21 reviews and an average rating of 3.6 stars. Keep it in mind for your next Aussie holiday!

Digital tools enable us to see things differently – to demystify the secret, to expose patterns and trends locked up in tables, statistics, or cultural collections.

Mapping Police Violence, for example, displays your chances of being killed by police in the US based on your location. It also presents the photos and details of more than 100 unarmed black people killed by police in 2014.

@CongressEdits is a Twitter bot, created by Ed Summers, that tweets anonymous edits to Wikipedia made within the US Congress. A similar bot exists for Australian state and federal governments.

I love the way that Twitter bots, in particular, can play around with our ideas of context and significance. I’ve created a few myself that automatically tweet content from Trove, and I’m interested in what happens when we mobilise cultural collections and let them loose in the places where people already congregate. Steve Lubar argues that ‘the randomness of the museumbot calls attention to the choices that we take for granted’. Twitter bots can challenge the sense of control and authority that adhere to our collection databases.

But bots can be more. Mark Sample’s important essay on ‘bots of conviction’ explores the possibilities for protest and intervention. He describes protest bots as ‘tactical media’ creating ‘messy moments that destabilise narratives, perspectives, and events’. Wendy Duff and Verne Harris warn archivists of the dangers of the story in disguising the exercise of power, in stealing from individuals what they need to construct their own narratives – ‘space, confusion, [and] a sense of meaninglessness’. Against the brutal logic of the state, a bot’s algorithmic nonsense can help us to see differently, to feel differently.

Caleb McDaniel’s bot @Every3Minutes is an example of how powerful these interventions can be. Working from estimates of the volume of the slave trade in the American South, it tweets a reminder every three minutes – a person was just traded, a child was just bought – often with links to historical sources. Mark Sample notes that ‘it is in the aggregate that a protest bot’s tweets attain power’ and it is through simple, unyielding repetition that @Every3Minutes reaches us. As Alex Madrigal noted: ‘To follow this bot is to agree to reweave the horrors of slavery into the fabric of your life’.

My own protest bot is trivial by comparison to Mark or Caleb’s work. @OperationBot merely assembles random words to create new names for national security operations. It’s a bot born of frustration and fury as the Australian government responded to the plight of asylum seekers by launching ‘Operation Sovereign Borders’. As @OperationBot proudly proclaims, its aim is to ‘protect Australia from meaning’.

Perhaps more significant than @OperationBot’s supposed subversions is the fact that I could create it in a couple of hours sitting in front of the TV. Digital skills and tools allow us to try things, to create and experiment, without any expectation of significance or impact.

One of the more controversial sessions at the British Association meeting in Australia in 1914 was devoted to the structure of the atom. Ernest Rutherford reported on experiments that pointed to the now familiar model where the atom’s mass is concentrated in a tiny, central nucleus. Firing charged particles at a thin sheet of gold foil, Rutherford, Geiger and Marsden had expected the particles to pass through largely undeflected. But some bounced back. As Rutherford later noted: ‘It was almost as incredible as if you fired a 15-inch shell at a piece of tissue paper and it came back and hit you’.

I wonder if that’s what we’re doing – firing off experiments into the net, waiting for one to hit something solid and bounce back. PING!

@Every3Minutes – PING!

The Real Face of White Australia – PING!

In 2012 Kate and I received an email from Mayu Kanamori, an artist researching the life of an early Japanese Australian photographer. She described her reaction to the Real Face of White Australia:

When I scrolled down the Faces section of your website, browsing through the faces, tears welled up, and I couldn’t stop crying as if some sort of flood gates had been removed.

We knew that that the documents and the images were powerful, but displaying the faces on that seemingly endless scrolling wall did something more than we were expecting.

Jenny Edkins has been exploring the politics of faces, and she suggests that alongside our attempts to ‘read’ portrait photographs we also respond in a more visceral fashion, provoking responses such as ‘guilt, obligation, and reciprocity’.5 Like the ‘messy moments’ of protest bots, she argues that the connections we make through photos of faces can disrupt the ‘linear narrative temporality’ on which sovereign power depends. We are connected through time, not with history, not with the past, but with people. And that has implications.

Last year I tried extracting faces, and eyes within those faces, from photos I’d harvested via Trove’s digitised newspapers. The result was Eyes on the Past. It presents a random selection of eyes, slowly blinking on and off. Clicking on an eye reveals the full face and the source of the image. Where the Real Face of White Australia overwhelms with scale and meaning, Eyes on the Past is minimal and mysterious. Eyes on the Past emphasises absence, and the fragility of our connection with the past, even while it provides a new way of exploring the digitised newspapers. Perhaps the best thing about it is the range of responses it has provoked – from those who found it beautiful, to those who thought it was just creepy.

More recently I’ve been playing around with the possibility of connection, and creepiness, through The Vintage Face Depot. Tweet a photo of yourself to @FaceDepot and a bot will select a face at random from my collection of newspaper images and superimpose that face over yours – tweeting you back the result. It sounds stupid, and it probably is. I’m still waiting for it to go viral like Microsoft’s age detection thing. But sometimes… PING!

One night I started fiddling with the transparency of the superimposed images. All of a sudden I could see the colour of my face showing through. I could see my glasses on this face from the past.

Experimenting on Kate, I saw the blue of her eyes peering through eyes of another person. Again, the potential is there to mess around with the barriers that put some people on the other side of this wall we call the past – to explore what Devon Elliot suggested on Twitter was an ‘uncanny temporal valley’.

The Australian historian Greg Dening has argued:

Nothing can be returned to the past. Not life to its dead. Not justice to its victimised. But we take something from the past with our hindsighted clarity. That which we take we can return. We disempower the people of the past when we rob them of their present moments.

There is no open access to the past. There is no key we can enter to recall a life. I do this sort of stuff not because I want to contribute to some form of national memory, but because I want to unsettle what it means to remember – to go beyond the listing of names and the cataloguing of files to develop modes of access that are confusing, challenging, inspiring, uncomfortable and sometimes creepy.

Perhaps my favourite experiments are a couple of simple userscripts. They sit in your browser and change the behaviour of Trove and RecordSearch.

Instead of pulling faces out of documents, they put them back in. Instead of seeing lists of search results, you see the people inside. Like the faces on our wall the people bubble up though the interfaces. They are present.

Despite the apparent enthusiasm for the visit of the British Association in 1914, there was in Australia a lingering suspicion of scientists as ‘impractical dreamers’, as mere theorists unwilling to address the nation’s most urgent needs. In debates over the application of knowledge to Australian development, the scientist commonly battled it out against the supposed virtues of the ‘practical man’.

I imagine my grandfather, Henry Sherratt, was a practical man. He was a brass moulder with a workshop in Brunswick, a suburb of Melbourne. His father and brother, both brass workers, lived and laboured nearby. I have a small brass ashtray that Henry made.

Henry’s name isn’t amongst those who joined the British Association in Melbourne, though perhaps he attended one of the ‘Public or Citizens Lectures’ which, until the 1911 meeting, had been known as ‘Lectures for the Operative Classes’. Neither is Henry’s name amongst those who journeyed to the battlefields of Europe and the Middle East. He is not one of those honoured by the Anzac Centenary for having ‘served our country and worn our nation’s uniform’. And yet he went to war.

Henry Sherratt was amongst a select group of tradesmen who travelled to Britain in 1916 to help meet the desperate need for skilled workers in munitions factories. He worked as a foreman brass moulder in Scotland, before an accident in which he ‘strained his heart’ carrying a ladle of molten iron. He never really recovered and as his income suffered, so did his family at home. Henry finally returned in 1919 and was offered £50 compensation with no admission of liability. He died in 1955. I never knew him.

Who do we remember and why?

The Commonwealth Bureau of Census and Statistics reported that 159 people died as a result of industrial accidents in 1914. But these were only the accidents that had been reported under the provisions of state legislation. There must have been more. Where is their memorial? What about mothers who died in childbirth, or victims of domestic violence? How do we remember them?

At a recent workshop organised by Europeana, Lucy Delap described how her project to historicise child sex abuse in 20th century Britain was making use of digitised newspapers. As well as documenting individual cases, the researchers hope to create ‘a map of change over time in the reporting of child sexual abuse’ that would enable them to test theories about how different organisations respond to abuse.

In the week that the British Association met in Melbourne, newspapers tell us that David Phillips, an engine driver, was fatally injured at Flinders Street Station. I’m thinking about how we might use Trove’s digitised newspapers to collect the stories of those who went off to work, but never returned. What might we learn about economic history, unionism, industrial legislation – about the value we place on an individual life?

As I’ve often said in regard to our work on the White Australia records – it just seems too important not to try.

As I was writing this talk I was also keeping an eye on my harvesting scripts which were chugging away, pulling down more images from the National Archives. For the original wall of faces I downloaded about 12,000 images from one series, I’ve now got more than 150,000 from about 10 series. You’ll see more of that soon.

As I was writing this talk I stopped at various times to play around with code – to look at the gender balance at the British Association, to investigate ‘closed’ files in the National Archives, to create a public Face API for anyone to use. The code and apps are all out there now for you to play with or improve.

Writing, making, thinking, playing, sharing. It all happens together. I’m a maker like my grandfather. While he poured metal I cut code. I do it because I want to find ways to connect with people like him, ordinary people living their lives. Those connections will always be fleeting and fragile, lacking the certainty of commemoration, but hopefully bearing some of the meaning and complexity of memory.

It’s a task that needs to be both playful and political. It’s not about making things, but trying to make a difference.

Permalink

http://discontents.com.au/unremembering-the-forgotten/

DOI

http://dx.doi.org/10.6084/m9.figshare.1536150

Internet Archive

https://web.archive.org/web/20150905062506/http://discontents.com.au/unremembering-the-forgotten

GitHub

https://github.com/wragge/discontents-blog/blob/master/_posts/2015-07-03-unremembering-the-forgotten.md

1. Shurlee Swain, Leonie Sheedy, and Cate O’Neill, ‘Responding to “Forgotten Australians”: historians and the legacy of out-of-home “care”’, Journal of Australian Studies, vol. 36, no. 1, 1 March 2012, pp. 17–28. <doi:10.1080/14443058.2011.646283>
2. Cassie Findlay, ‘People, records and power: what archives can learn from WikiLeaks’, Archives and Manuscripts, vol. 41, no. 1, 2013, pp. 7–22. <http://www.tandfonline.com/doi/abs/10.1080/01576895.2013.779926>
3. Geoffrey C. Bowker and Susan Leigh Star, Sorting Things Out: Classification and Its Consequences, MIT Press, 2000.
4. Wendy M Duff and Verne Harris, ‘Stories and names: Archival description as narrating records and constructing meanings’, Archival Science, vol. 2, 2002, pp. 263–85.
5. Jenny Edkins, Face Politics, Routledge, 2015.