Presented at Sharing is Caring 2017, 20 November 2017, in Aarhus, Denmark.
You can also watch the video.

In 1946, Britain decided that Australia would be the perfect place to test missiles. The Australian government, keen to play its part in the defence of the Empire, readily agreed. Ignoring, yet again, the presence of Australia’s Indigenous peoples, defence planners thought Australia was attractive to because it was ‘empty’, flat, and far from ‘prying eyes’.

The town of Woomera was built in the South Australian desert to house scientists, workers, and military personnel. It was a town where no housewife could go to the shop without her security pass; where curiosity was ’the badge of the outsider’.

But while Australia’s land seemed ideal for secret military operations, its people remained suspect. Britain’s plans were threatened by concerns about Communist infiltration of the Australian government. Under pressure from the UK and USA, Australia sought to lift its spy game through the establishment of a new agency to monitor such threats — the Australian Security Intelligence Organisation, or ASIO.

Legislation defined ASIO’s functions in very broad terms, ‘to obtain, correlate and evaluate intelligence relevant to security’. From the 1950s to the 1970s, this was used to justify surveillance of a wide range of potential ‘subversives’ — not just known Communists, but writers, artists, academics, scientists, Indigenous activists, and more. Many thousands of files were created to document their beliefs, activities, connections, and personal lives. Recordkeeping was critical to the practice of state surveillance.

We don’t know how many files were kept on ordinary Australians because ASIO is exempt from many of the key provisions of the government’s archives legislation. Unlike other agencies, ASIO does not routinely transfer records or indexes to the National Archives of Australia. Researchers have to go on a fishing expedition, asking the National Archives to ask ASIO whether they might have a file relating to a particular person or organisation. If ASIO admits it has a relevant file in the open period (more than twenty years old), the file goes through an ‘access examination’ process to determine whether it contains information that should be withheld for reasons of national security, or individual privacy. If anything is left, it is finally opened to public access.

Despite these hurdles, more than 12,000 ASIO surveillance files have been made public, though most include redactions — black boxes obscure words too sensitive to be read.

The files have been used in biographies, family histories, and studies of Australia’s literary community. One recent book invited the subjects of ASIO surveillance to reflect on the contents of their own files — to see their lives through a different set of eyes; to explore the intrusions and innuendo that passed for ‘intelligence’.

I’m currently working with a set of 60,000 photographs held by the State Library of New South Wales. These photos were taken for The Tribune, a Communist Party newspaper published in Sydney, and document protest and political activity in Australia from the 1960s to the 1990s. One of the things I’m interested in is finding overlaps between the Tribune photographs and ASIO surveillance files. For example, in February 1972 there was a demonstration on Indigenous rights held outside Parliament House. Because both sets of records have been digitised and made public, we can compare perspectives — spies versus ‘subversives’.

This is a reminder that the impact of digitisation is not simply easier, more immediate, access. We can also see the same things differently. We can interrogate the meaning of access itself.

RecordSearch, the National Archives of Australia’s online database, provides access to about 64,000 series descriptions, 11 million item descriptions, and 1.8 million digitised pages. There’s currently no API, or downloadable datasets at item level, so I make my own.

For six or seven years now I’ve relied on my own little library of screen scrapers to get data out of RecordSearch. They’re slow and they break easily, but they do the job.

Late last year I embarked upon what was probably my most ambitious data harvest. I gathered information about every series listed on RecordSearch and calculated, for each, the quantity of records (in linear metres), the number of individual items described, and the number of items digitised. I then aggregated the series by the top-level functions of agencies associated with them. Basically I grouped them by subject — defence, security, education etc.

Why?

Because digitisation shapes our perceptions of reality. The more we have in digital form, the easier cultural heritage collections are to find and use, the more likely we are to assume that everything (or at least everything important) is online. Ease of access bears an ontological weight — if we can’t find it online, does it exist?

Now that might not be a problem if what was digitised somehow provided a representative sample of the whole. But we all know how such decisions are shaped by political priorities, funding opportunities, user demand, public events, and happy accidents. There’s nothing necessarily wrong with that, it’s just the environment within which we work. There are never enough resources. We have to do what we can, when we can.

The problem is, we rarely expose the impact of these decisions to the users of our digital collections. We rarely give them the chance to reflect on how our decisions shape their assumptions.

The National Archives of Australia documents the workings of our democracy. If offers one important perspective on who we are as a nation. If we look at the quantity of records associated with each top-level function, we see a fairly even distribution. Nothing stands out.

But what happens when we view the activities of government through the number of files digitised in each subject area?

The prominence of defence is really no surprise. Service records are heavily used by family historians, and in 2007 the Australian government funded the digitisation of all 375,000 World War I service records in what was branded as ‘A Gift to the Nation’.

The National Archives is not alone. I often show people this graph of the number of digitised newspaper articles in Trove, pointing out the fairly dramatic peak around 1914. Did something happen in 1914? Were there more articles published, more newspapers? No, there’s just more money. In the lead up to the centenary of WWI, funding was directed towards the digitisation of newspapers from the wartime period.

Again, there’s nothing wrong with this. It’s just that these biases are not obvious to someone typing queries into a search box. In the context of Australian history, these decisions around digitisation help to reinforce the long-held belief that Australian national identity was somehow forged on the battlefields of WWI. It helps to put war at the centre of our history, at the centre of who we are.

But of course while digitisation can shape our assumptions, it also gives us new opportunities to critique them. I could only analyse the holdings of the National Archives because their collection data is online. We don’t have to take just what the search box delivers — we can ask our own questions. But this is only possible if people have the skills, the tools, and the confidence to poke around in the data. This too is access. Institutions should invite the public not to swoon at their digital delights, but to hack away at difficult questions — not to see collections, but to see them in unexpected and challenging ways.

I mentioned that ASIO files go through an process known as ‘access examination’ before they’re released to the public. This is the case for all records more than twenty years old, not just the super secret ones. The vast majority of files are simply opened without restriction. Some, including most of the ASIO files, are opened ‘with exceptions’ — pages can be withheld, and text redacted. A few are withheld from the public completely. They have entries in RecordSearch, but you can’t see them — their access status is officially ‘closed’.

But because the metadata about access decisions is available online, we can start to build a picture of what we’re not allowed to see.

At the start of 2016, I harvested the details of all files in the National Archives of Australia with the access status of ‘closed’. I’ve aggregated and sliced the data in a number of different ways, so you can explore the age of the files, what series they came from, and when decisions were made about their access status. At any point you can drill down to a list of the files you cannot see — making it perhaps the most frustrating search interface ever devised.

You can also examine the reasons why files have been withheld. Many of these exceptions are defined by the legislation that established the National Archives. Clause 33(1)(a), for example, relates to national security, 33(1)(g) is concerned with individual privacy. But the metadata reveals that files are withheld for a number of other reasons, such as ‘Pre Access Recorder’ and ‘Withheld Pending Advice’. There’s also, you might note, a category entitled ‘MAKE YOUR SELECTION’ — which reveals something about the limits of the data entry interface.

By poking around you in the data you can make some guesses as to how these additional categories are used. ‘Pre Access Recorder’ is used as a catch-all for records that were withheld from public access before the archives legislation was passed. ‘Withheld Pending Advice’ is used to label files that have been sent off to other government agencies for their assessment — they’re not yet finally closed, but as this process can take years, they’re sort of closed. Indeed, my interface shows that 1,467 files have been waiting more than three years for advice.

The point of this is not to embarrass the National Archives, nor the Department of Foreign Affairs and Trade which holds the most files in limbo. The point is to examine the ways in which access itself is constructed. Legislation defines an ideal, but the reality is more messy and human. By tracking patterns in the way access decisions are made we can explore the historical processes at work. Access is not allowed, it is made.

Remember those 12,000 ASIO files publicly available through the National Archives? You might not be surprised to know that I’ve harvested them all — both the metadata and the 300,000 digitised pages. There’s about 70gb of images.

Using these files we can dig a little deeper into the nature of access. I wrote a computer vision script to find redactions. It took a lot of trial and error, and I’m about to start work on a smarter version that incorporates machine learning, but it did the job. From one series of ASIO files, about 230,000 pages, I extracted 239,000 redactions — lots and lots of little black boxes. You’ll be pleased to know that not only can you download the complete set of redactions from the research repository Figshare, you can browse them. All of them! Hours of fun for all the family!

The interesting thing about this interface is that if you click on a redaction you can view the page that it was extracted from. So it’s sort of an inside out discovery interface. Instead of the redactions being a brick wall or a dead end, they’re a starting point. A practice intended to remove information, to limit access, becomes a gateway for exploration. Indeed, the redactions themselves provide an identifiable data point — something that can be analysed to turn the gaze of government surveillance upon itself.

But something else was hiding in those ASIO files. As I was reviewing the collection of redactions for false positives I discovered that someone tasked with the removal of information, decided to add a little creative flair.

I discovered #redactionart.

I assure you that these creations really are sitting inside ASIO files held by the National Archives. But since I’ve discovered them, they’ve developed a life of their own. Not only can you browse through them online, you can wear them.

I gave away about 80 of these badges at an exhibition earlier this year. To create the badges, I simply traced around the original images and saved the results as SVG files. These files themselves are shared through GitHub for anyone wanting to create their own #redactionart.

This amazing #redactionart dress was made by Bonnie Wildie, a librarian in NSW. My SVG files have also been turned into a set of 3D printable cookie cutters, as well as a range of t-shirts and stickers on RedBubble.

This escape from the archives is not only creative and fun, it’s important.

It’s important because it emphasises that the practices through which government information is controlled and withheld are profoundly human. People make decisions and they leave their marks. There is nothing mysterious or otherworldly in the secret — it is an exercise of power.

Archives are not just made of documents — there are people inside.

‘Surveillance’ is not included in the National Archives’ official thesaurus of government functions, yet the movements and activities of individuals are recorded in many thousands of files across an assortment of agencies.

A simple query of my harvested data reveals that the phrase ‘alien registration’ appears in the titles of only 29 series. But these series contain more than half a million files. 4.7% of digitised files in the National Archives document the movements of so-called ‘aliens’. While these registration systems were created during wartime, they lingered beyond. And they were not the only means of keeping track of potential threats. Just as at Woomera, boundaries were drawn, and outsiders marked for attention.

When I was last in this part of the world, I talked a bit about some work that Kate Bagnall and I had done with records of the White Australia Policy held by the National Archives.

A quick recap — when the Australian colonies federated in 1901, it was generally assumed that the new nation’s future could only be assured through strict racial homogeneity. A ’white’ Australia was a strong Australia. Legislation was quickly passed to restrict immigration and set the foundations for what became known as the White Australia Policy.

However, in 1901 there were around 40,000 people living in Australia whose background was neither European nor Indigenous — they were Chinese, Japanese, Syrian, Indian, and Malay. Some had been born in Australia, or had lived there for many years — raising families, building businesses; just living their lives.

If any of these people wanted to travel overseas they had to carry special documents, or they might not be allowed to return home. Customs officials at Australian ports would ask anyone who seemed not to be ‘white’ for identification. The badge of the outsider was the colour of their skin.

Many thousands of these documents, the remnants of a racist bureaucratic system, are preserved in the National Archives.

Back in 2011, I downloaded about 12,000 of these documents from RecordSearch and ran them through a facial detection script to create a seemingly endless scrolling wall of faces. We called it ‘The Real Face of White Australia’. It’s another inside-out interface — instead of showing the files, you see the people inside.

That was then, this is now! In the last few months, I’ve been working with a group of my digital heritage students to develop a website for the collaborative transcription of these same records. We want to put names to the faces. We want to chart their journeys. We want to document their lives.

Our project has no funding, and was only possible because Zooniverse and the New York Public Library created and shared Scribe, a framework for the transcription of structured documents — an easy way to get usable data out of forms, ledgers, and certificates.

The site was launched at a ‘transcribe-a-thon’ held at the Museum of Australian Democracy in Canberra, which just happens to be located in Australia’s first parliament house. The building didn’t exist when the Immigration Restriction Act was passed in 1901, but it was where the White Australia Policy was elaborated and maintained.

Transcription continues. There’s still much work to do on the documents, but data is already flowing. I’m making regular dumps available for download through a GitHub repository.

But it was never just about the data. Many more people now know that these records, this history, exists. Through the process of transcription you are confronted by the disturbing reality of the records — you’re surprised, puzzled, shocked, and often moved. Creating a space for these sorts of experiences is important in itself.

The Museum of Australian Democracy not only gave us their building for a weekend, they let us play with their data projectors. In some ways, I would have been happy if all we had achieved was this — to put these faces in this space.

Once again the gaze of surveillance is reversed. In the home of Australian democracy, people who lives were monitored under a racist system of exclusion and control were looking at us, asking questions of us.

Amongst those Tribune photos at the State Library of NSW, I recently found this. Believe it or not, I’m the spy on the right. This compelling piece of street theatre was performed at the gates of Pine Gap, a US electronic surveillance facility right in the centre of Australia. Pine Gap’s lease was due for renewal in 1987, so hundreds of protestors converged on the site, hoping that the Australian government might withdraw it’s support. Needless to say, it didn’t. Pine Gap remains, and in recent times has been implicated in US drone strikes

I found another photo of myself amongst the Tribune archives. A group of us climbed over the outer perimeter fence in the middle of the night and took up positions on a rocky outcrop that overlooked the main gate. At a predetermined time, we leapt out of our hiding places and lit smoke flares. I was arrested soon after, charged with trespass, and fined $100.

Another group of Pine Gap protestors are currently on trial in Australia. They made it through the protective fences and dared to play music and pray. For this they have been charged under the Defence (Special Undertakings) Act which carries a maximum sentence of seven years in prison. This Act was passed in 1952 when Britain decided to expand its weapons testing program in Australia to include atomic bombs. It expanded upon earlier legislation that had been intended to protect Woomera from Communist interference. This is one of the very few times anyone has been charged under the Act, despite there being hundreds of arrests like mine in the past.

As security services gain new powers, and electronic surveillance expands, it’s hard not to see the Pine Gap proceedings as an attempt to discourage criticism of the government’s tough on terrorism stance.

At a recent symposium on collaboration between researchers and collecting institutions, Seb Chan described some of the advances that had taken place in opening up collections, but then asked ‘So what?’.

I suppose that’s the question we’re hear to discuss. Why do we put all this effort into digitising collections, building interfaces, and sharing data? Easier access is great, beautiful interfaces are cool, but… so what? For me, as a historian, hacker, and sometime heritage professional, the answer is straightforward — it’s all about bringing the past into conversation with the present. It’s about mobilising our collections as critical resources in debates about who we are, what matters, and why we should care.

Those inky, black handprints on the White Australia records moved one of my students to reflect on her experience as a recent immigrant from Canada, required by the Australian government to supply a set of her fingerprints. She wrote a beautiful talk and presented it during the transcribe-a-thon in the original House of Representatives chamber at Old Parliament House. Another student noted in her final essay that the documents made non-white residents seem like criminals, pointing to parallels with the current treatment of refugees. On the flip-side, our efforts attracted the attention of a few racist trolls, one of whom referred to the White Australia Policy as ‘the good old days’.

Once again ’outsiders’ are being targeted as threats to our security. Boundaries are being reinforced, and efforts being made to define who belongs. We know this. We’ve seen this before. Europeana’s new project on the history of migration is an important initiative — we need to tell our stories, share our resources, grapple with our difficult and painful pasts. I don’t think this is a time to reassert the authority of our cultural institutions as reservoirs of truth. We are implicated in all of this. Our collections are built upon systems of surveillance, on attempts to put humans into categories. They are products of power and privilege. We are not the guardians of enlightenment, we are the keepers of horrors.

Just like #redactionart, the value of our collections lies in their complexity and contradictions — in their very humanity and all the confusion that entails. Digital collections lend themselves to an exploration of complexity. We can shift scales and perspectives, we can manipulate contexts, we can set collections loose in public spaces, we can turn them inside out. We can see differently, but perhaps more importantly, we can feel differently.

When you think about it, ‘impact’ is a pretty violent sort of word. There are perhaps a few people around the world we’d like to ‘wallop’ with our digital collections. But I suspect most of the time we’re after something more subtle — to expand possibilities, to undermine assumed certainties, maybe even to expose a glitch in the Matrix.

Perhaps we can offer a glimpse of an alternative reality, where we recognise the outsider as us.

Maybe in the end we will be able to see the outsider as us. Hear hear! #ShareCare17 pic.twitter.com/TWWYqyvCDg

— Johanna Berg (@johannaberg) November 20, 2017

Contribution to a panel discussion on ‘Access and Innovation’ at Digital Directions 2016, Canberra, 10 November 2016.
View the slides.

Please cite as: Tim Sherratt, ‘Caring about access’, presented at the Digital Directions, 2016. <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>

You might have seen that the Department of Prime Minister and Cabinet has opened discussion on an Open Government National Action Plan. Last week I was watching the Twitter stream from a briefing event in Melbourne, when I saw this tweet from Asher Wolf quoting the PM&C spokesperson:

. @pmc_gov_au: we don’t want ppl to search for flaws or try to crack gvt datasets. #OGPAU

Now I’m not sure of the context of this statement, but it is a reminder that we can’t take the meaning of words like ‘open’ or ‘access’ for granted.

They are what we make of them.

I’m a historian and hacker. I don’t steal credit card details, I use digital tools to open cultural heritage collections – to see them differently, to feel differently.

Earlier this year I wrote a script to harvest many gigabytes of parliamentary proceedings – Hansard – from the ParlInfo database maintained by the Department of Parliamentary Services. I shared the files, covering the period from 1901 to 1980, through my GitHub repository. My plan was simply to make this rich source of political history easily available to anyone who wanted to explore new forms of digital analysis.

In the end I created my own version of Historic Hansard, one that privileges the experience of reading rather than searching. I also added a few nifty features such as indexes of legislation and people, and integration with text analysis and annotation tools.

But in the process of harvesting the data I noticed that around 100 sitting days were missing from ParlInfo. The files were empty – unable to be searched. Given an average of about 50 sitting days a year, that’s about two years worth that were simply invisible.

Thanks to the Parliamentary Library these problems have now been fixed. But this experience highlighted two issues. First, search interfaces lie. We have to develop our critical capacities to be more aware of what we are not being shown and why. And second, this problem only became visible because I was hacking their database, because I went beyond what was offered by their website, because I wanted more. Unlike the PM&C spokesperson I think one of strongest reasons for opening data is to encourage people to find its flaws. This is particularly the case for cultural heritage data where the processes of selection, representation, normalisation, control, and preservation all have an impact of the types of stories we can tell about ourselves.

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

A few years ago Merete Sanderhoff, from the National Gallery of Denmark, compiled a collection of essays on opening cultural heritage collections for reuse online. It was called Sharing is Caring. I think it’s worth reflecting on the dual meanings of ‘caring’ – both looking after, and giving a shit. Sharing helps us foster communities and preserve collections. But it also matters, it has an impact, it can change the world.

Bethany Nowviskie, the Director of the Digital Library Federation, recently gave a talk on ‘speculative collections’ asking how we shift the temporal orientation of our libraries and archives away from a closed and linear past towards an exploration of what might be.

Seeing differently. Feeling differently.

This is a visualisation created by Geoff Hinchcliffe showing more than 12,000 tweets since February this year using the hashtag #fundTrove. Trove has radically changed our access to the past. It’s not perfect, it’s not everything, but it changes lives. And yet the government seems to think it can be left to wither, and nobody will really care. I care. Do you? Then what do we do about it?

I also care deeply about the collections of the National Archives – a love expressed through many painful hours spent hacking RecordSearch, their online database. Last year I harvested details of more than 12,000 publicly available ASIO files – you can grab the results from my GitHub site – and downloaded about 300,000 page images. These are mostly surveillance files, documenting the lives of writers, academics, unionists, Indigenous activists and others – identified as potential threats to the nation.

This year I wrote a script to search those images for redactions – sections of text blacked out for security reasons. If you’d like to explore the 239,000 redactions I’ve extracted, you can – marvel at their blackness, their very lack of information. But here the redactions are not dead ends, they’re starting points, ways of exploring the workings of ASIO. As the power and reach of state surveillance continues to expand, we can find creative ways to reverse the panoptic gaze.

We don’t have to accept what we’re given. We can take collections and turn them inside out.

The National Archives also preserves remnants of Australia’s racist past. There are many thousands of records documenting the workings of the White Australia Policy. They include special certificates, with portrait photos and handprints, which were needed if you were deemed non-white, travelled overseas, and simply wanted to come home.

About five years ago I harvested about 12,000 images of these certificates from RecordSearch and ran them through a facial detection script. In a little over a weekend I’d created the Real Face of White Australia – a seemingly endless scrolling wall of more than 7,000 faces. It’s compelling. It’s uncomfortable. But this is who we are.

As I saw the One Nation senators cracking open the champagne outside Parliament House yesterday, I thought again of how important its is the tell these stories, to share these collections. To give a shit.

Innovation can be measured in more than shiny apps, or cool new visualisations. By struggling with access to our past we can imagine new futures.

Keynote presented at Forging Links, the Annual Conference of the Australian Society of Archivists, Parramatta, 19 October 2016.

Please cite as: Tim Sherratt, ‘Turning the inside out’, presented at the Australian Society of Archivists Annual Conference, Parramatta, 2016, <https://dx.doi.org/10.6084/m9.figshare.4055013.v1>.

This is RecordSearch, but not as you know it.

This is my hacked version of RecordSearch, the online collection database of the National Archives of Australia. Unlike the regular version it displays the number of pages in each file. But more interestingly, if you search in Series ST84/1 you see more than metadata – you see the people inside.

As Barbara Reed noted in her article ‘Reinventing access’, ‘records are imbued with people’. Series ST84/1 goes by the fairly benign title, ‘Certificates of Domicile and Certificates of Exemption from Dictation Test, chronological series’. But of course the Dictation Test was the administrative backbone of a racist system designed to exclude people who did not fit the widely-accepted vision of ‘White Australia’. ST84/1 is full of people just trying to live their lives under the weight of the White Australia Policy.

The certificates in ST84/1 allowed people, born or resident in Australia, to return home after travelling overseas. If your ‘whiteness’ was suspect and you had no certificate, you would be subjected to the Dictation Test, and you would fail. The certificates usually include photographs and handprints – they are compelling and confronting documents. But you have to dig through layers of metadata in RecordSearch to see that. Or do you?

About five years ago, Kate Bagnall, a historian of Chinese Australia, and I were thinking about ways of drawing attention to these records. In a little over a weekend, I harvested about 12,000 page images from ST84/1 and ran them through a facial detection script. The result was ‘The Real Face of White Australia’.

You may have seen it before. It’s had a remarkable life, travelling around the world as an example of how we can use digital tools to see records differently. But of course the power is in the faces themselves – in the connections we make through time. We cannot escape their discomfiting gaze.

You may think that the certificates in ST84/1 are merely a form of identity document. But remember than in the early years of the 20th century, passports were still evolving, and the use of photographs and fingerprints for identification were generally confined to prisoners and criminals.

The sociologist Richard Jenkins talks about identity not as an essence, a noun, but as ‘something that we do, a process of identification’.1 But self-identification is constrained by broader systems of categorisation, or ‘social sorting’, that decide who belongs, who is a threat, who needs to be watched.

The records in ST84/1 were embedded within a system of surveillance that extended outwards from Australia’s ports to the offices of shipping companies around the world, and inwards to anyone who seemed out of place in White Australia. Technologies of identification and surveillance do not simply enforce boundaries, they create them. Their existence demonstrates why they are needed. These records did not document identity, they defined it according to a set of racial categories.

Modern parallels are not hard to find. Last year in a bungled operation that became known as ‘#borderfarce’, immigration officials planned to prowl the streets of Melbourne on the hunt for illegal immigrants. The focus of border surveillance once again turned inwards, to those who seemed out of place. Watching in horror as events unfolded on social media, I helpfully pointed people in Melbourne to a convenient source of identity documents.

In Melbourne this weekend? Get your Border Force identity papers here: http://t.co/H3dlXHZy6q #borderfarce #wap pic.twitter.com/j2TrkKEPkR

— Tim Sherratt (@wragge) August 28, 2015

The link I tweeted was not to RecordSearch, but to an experimental interface where Kate and I are continuing to think about ways of exposing the bureaucratic remnants of White Australia.

So far I’ve harvested metadata from more than 20,000 files and downloaded around 150,000 page images. Amongst other things, I’m working on an updated wall of faces.

Most recently I created a way of sorting and viewing pages by their orientation – by the ratio of height to width. Why? Kate wanted an easy way of finding birth certificates which, in this period, tended to be short and wide. It was a simple little hack, but it revealed the records in a very different way.

There’s a big bunch of birth certificates, but there’s also envelopes, photographs, and an assortment of slips and notes. It makes you think about how we see the written world through the frame of ‘portrait’ orientation. It’s perhaps also worth noting that RecordSearch displays thumbnails as cropped squares – shape is subsumed to the regularity of the grid.

The records in our landscape view are no more accessible than they were before, but they can be accessed differently. In his contribution to the ASA’s 30th anniversary symposium, Eric Ketelaar described the relationship between archival access and human rights – highlighting the importance of access not only to democratic accountability, but also to our rights as ‘victims’ of official surveillance: ‘As human beings, subjected to the panoptic sort of governments and private enterprise, we have the right to know’.2 Ketelaar concludes by stating that ‘access is not the actual use of archives, access enables use’. I’d like to extend this a bit. What my own work reveals is the complex relationship between access and use. Not only does access enable use, use changes what we mean by ‘access’.

My bio nowadays describes me as a ‘historian and hacker’. ‘Historian’ describes my orientation to the world – I see the past in the present. ‘Hacker’ refers to the tools I use to make connections through time. Hacking is creative and positive, despite what the mainstream media might say – it’s about finding solutions, exploring alternatives, and pushing the limits of what’s possible.

My RecordSearch hack is just a little piece of Javascript code that you can install in your browser – it changes the pages as they load.

Userscripts, as they’re called, allow anyone to alter the way webpages look and behave within their own browser. They give users greater control over their online experience, but they also open opportunities for experimentation. A lot of my own research is guided by questions like: What would happen if I…? What would it look like? What would change? What would I feel? Userscripts are one way of playing with the complexities of access.

Surveillance too can be hacked. If you’re concerned about technologies such as facial detection you’ll be pleased to know that thanks to the work of artist Adam Harvey, not only can you confuse detection algorithms, you can make a dramatic fashion statement.

Gary Marx, a major figure in the field of surveillance studies, has catalogued the ways in which individuals can resist the growing encroachments of surveillance. Amongst possible tactics he identifies ‘discovery’ – the attempt to undercover the scope of surveillance.3 Access to records can empower such acts of everyday resistance, but in other ways surveillance and access are more alike than opposed. Both start from a place of concealment. Access cannot be given unless it is first restricted. Both depend on asymmetries of power. Decisions about what we can know are ultimately made by others. Access is as much a process of control, as it is an act of release.

This is not necessarily a bad thing. I think we can all agree there should be limits to access, particularly relating to individual privacy and cultural sensitivity. But just as ‘identity’ is defined through acts of ‘identification’, so access is elaborated through instances of deployment and use. Access is not a state of being, it’s a process to be negotiated. And so the question is, what can we know about what we can know?

On 1 January this year I harvested the metadata of all the files in RecordSearch with the access status of ‘closed’.

These are records that the access examination process has determined should be withheld from public scrutiny. While the files themselves can’t be seen, RecordSearch does tell us a fair bit about them, including when the decision was made and the reasons behind it. Unfortunately you can’t search or filter on this data, so it’s difficult to look for patterns within RecordSearch itself.

I’ve taken the data and loaded into a new site where you can examine it from a number of different angles. You can explore the reasons why files were closed, the series they came from, the age of their contents, and the dates when decisions were made. It’s more of a workbench than a discovery interface, and it’s likely to change as I ask different questions of the data.

Of course, the outlines of the examination process, including the grounds for exemption, are defined by the Archives Act. So what more is there to know?

Section 33 of the Archives Act does indeed spell out 17 reasons why records can be withheld from public access. But the data from RecordSearch includes an additional 11 categories. Some, like ‘Parliament Class A’, relate to other definitions under the Act. Others, like ‘MAKE YOUR SELECTION’, tell us something about the RecordSearch interface. But two of the most heavily cited reasons – ‘Pre-access recorder’ and ‘Withheld pending advice’ are not defined under the Act or anywhere that I could find on the Archives’ website.

Being an archivally-educated audience you can probably guess what these labels refer to, but if you need a little help you can look at when the access decisions in these categories were made.

The majority of decisions on ‘pre access recorder’ files were made before the introduction of the Archives Act in 1983. I checked this with the Archives and they confirmed that these records were examined before the existence of the Act. They explained that ‘pre access recorder’ was used when the original exemptions couldn’t be mapped to those later defined under Section 33.

Conversely, most decisions on ‘Withheld pending advice’ files were recorded in the last five or six years. If you look at the series that contain the most files citing this reason you can see that almost half come from A1838 – DFAT’s main correspondence series. As I’m sure you’ve realised, these are files that have been referred back to agencies for advice. And DFAT has been particularly slow in responding. They’re listed as ‘Closed’ on RecordSearch even though their access status has not been finalised. They’re not, however, included in the count of ‘closed’ files that the Archives reports in its annual summary of access outcomes.

This is probably fair enough, but if you search the closed files you can see that 1,467 of them files have been waiting for more than three years for a final decision. They might not be officially closed, but for a PhD student wanting to see them they are effectively closed.

My point here is not to be critical of the National Archives, or even of DFAT. What I’m interested in is the inevitable gap between legislation and practice. Access examination is subject to a range of influences and constraints, resourcing amongst them, and needs to be understood not as the application of a set of rules, but as a process that is historically contingent. A human process.

Earlier this year I harvested several gigabytes of parliamentary proceedings from the Australian Parliament’s ParlInfo database and created my own version of Historic Hansard.

As you do.

In the process of harvesting the files I discovered that data was missing for about 100 sitting days – most of them from the Senate between 1912 and 1919.

There’s no conspiracy at work here, it’s just some sort of processing error. However, Parliament staff weren’t aware of the problem, and it’s unlikely that anyone would’ve noticed using the web interface. You can’t find what you can’t find. Fortunately, Parliament staff are now working on a fix, but if you’ve been relying on ParlInfo for access to debates relating to World War I, you might want to do some more checking.

These things happen. Systems go wrong. Mistakes are made. Again what interests me is not finding who’s to blame, but exploring the gap between design and outcome, between ideal and reality. This is the gap where access is made and experienced. A gap that can only be understood through the complexities and contradictions of use. Access does not exist until its limits are tested. It’s not a process of opening, it’s a constant ongoing struggle over the very meaning of ‘open’.

And that’s a good thing.

We’re here at this conference to explore the possibilities of ‘forging links’. But of course collaborations don’t have to be comfortable to be constructive. The struggle over access may sometimes be tense, frustrating, and annoying, but it is also productive. Users of archives do not just consume access, they create it.

About a year ago I fired up my RecordSearch harvester and downloaded the metadata and page images for most of the ASIO (Australian Security Intelligence Organisation) files publicly available through the National Archives. I ended up with about 70gb of images. These are mostly dossiers on individuals and organisations – odd collections of gossip, published articles, and records of surveillance.

I’ve made this data available for anyone who wants it. Some of the images were recently used in the GovHack open data competition to create the ‘Cute Commies’ site.

To be honest, I didn’t have a clear purpose in mind when I harvested the data. It was another one of those ‘What would happen if?’ moments. I was, however, thinking generally about possible points of comparison between the ASIO files and the archival remnants of the White Australia Policy. Both built systems of identification, classification, and surveillance in which recordkeeping was crucial.

Kate and other historians of Chinese Australia have noted that the administration of the White Australia Policy was not uniform or consistent. Similar cases could result in quite different outcomes depending on the location and those involved. Understanding this is important, not only for documenting the workings of the system, but for recovering the agency of those subjected to it. Non-white residents were not mere victims, they found ways of negotiating, and even manipulating, the state’s racist bureaucracy. In her work on colonial archives, Ann Laura Stoller identifies this ‘disjuncture between prescription and practice, between state mandates and the manoeuvres people made in response to them’ as part of the ‘ethnographic space’ of the archive.4

How do we explore this space? One of the things I’ve found interesting in working with the closed files is the way we can use available metadata to show us what we can’t see. It’s like creating a negative image of access. Kate and I have been thinking for a number of years now about how we might use digital tools to mine the White Australia records for traces, gaps, and shadows that together build a picture of the policy in action. Who knew who? Who was where and when? What records remain and why?

The workings of ASIO, on the other hand, are deliberately obscured. Many of the files in the Archives include a note explaining why details have been withheld. Some warn that the ‘public disclosure of information concerning the procedures and techniques used by ASIO’ would enable people of interest to formulate counter-measures ‘based on an analysis of ASIO modus operandi’. David Horner’s recent history of ASIO notes that he was required to remove ASIO file references from his footnotes ‘because of the nature of ASIO’s filing system, which itself is classified’.5 We don’t even know how many files ASIO has on people and organisations, although David McKnight suggests that it’s somewhere in the hundreds of thousands.6 My harvest includes about 12,000 files.

Just like systems of racial classification, intelligence services exist within a circle of self-justification. The fact they exist proves they need to exist. We are denied information that might enable us to imagine alternatives. And yet as limited as the provisions under the Archives Act are, we do have access.

How can we use this narrow, shuttered window to reverse the gaze of state surveillance and rebuild a context that has been deliberately erased. Just as with Closed Access and the White Australia records can we give meaning to the gaps and the absences? Can we see what’s not there?

This is one of the questions being explored by Columbia University’s History Lab. They’ve created the Declassification Engine – a huge database of previously classified government documents that they’re using to analyse the nature of official secrecy. By identifying non-redacted copies of previously redacted documents, they’ve also been able to track the words, concepts and events most likely to censored.

The History Lab’s collection of documents on foreign policy and world events is rather different to ASIO’s archive of the lives, habits and beliefs of ordinary Australians. But I’m hoping that they too can tell us something about the culture that created them.

I’d intended to have a wonderfully compelling suite of examples and arguments to demonstrate today, but time has run short. Instead I have a set of half-baked experiments which sort of look a bit interesting. But perhaps that’s better. It’s important to me to try and be open about my own processes. I share my code and data, and I’ve started documenting most of what I’m up to in a open research notebook. If access is a struggle, then we should be sharing our stories of loss and frustration, and not merely celebrating our victories.

All of these experiments are online in some sort of form. So please explore.

Experiment A is nothing more than a browse interface to all the digitised records I’ve harvested. It’s just a clone of my work with the White Australia records, but I think there’s real conceptual power in the ability to browse.

Experiment B started with a problem. From RecordSearch I could harvest data on access status and find out how many ASIO files were in each of the three categories – Open, Open with Exception, and Closed. But how much of the ‘Open with Exception’ files are actually open?

Most of the files include a summary which tells you how many pages have been completely or partially exempted. That’s great, but did I really want to open up 12,000 files and manually scan for summaries? By playing around with the Tesseract OCR engine I’ve created a simple filter that extracts text from the images and searches for words like ‘exemption’, ‘archives’, and ‘folio’. I now have a good sized collection of summaries awaiting data entry…

Experiment C began as another attempt to quantify the scale of exemption. The summaries told me how many pages had redactions – bits of information like names and ids that are blacked out, or sometimes even cut out of the page. But if I could identify individual redactions I could both test the summaries and create a new measure of openness… or redactedness…

Looking for redactions in ASIO files — that looks hopeful… pic.twitter.com/heeWMAQiwe

— Tim Sherratt (@wragge) September 11, 2016

Through trial and error I developed a computer vision script that did a pretty good job of finding redactions – despite many variations in redaction style, paper colour, and print quality. It took a couple of days to work through the 300,000 page images, but in the end I had a collection of about 300,000 redactions. Unfortunately about 20 percent of these were false positives, so I spent a number of nights manually sorting the results.

But the false positives themselves are sort of wonderful… pic.twitter.com/Jn9l04bUaY

— Tim Sherratt (@wragge) September 29, 2016

My redaction finder still needs a lot of refinement, and plenty of errors have slipped through. But, within the files that are currently digitised, the scale of exemption seems about ten times greater than Margaret Kenna estimated when giving evidence to Parliamentary Joint Committee on ASIO in 2000. She thought every file contained about 10 exemptions ‘be it a word or a folio or a paragraph’. I’m seeing an average of about 100 redactions per file.

I’ve started adding information about the size and position of the redactions to my database and aggregating this data by page. When I left Canberra, the script was still running, but you can explore the current standings in my top 50 lists of the most redacted files and pages.

Once the data processing is completed you’ll be able to filter files by the amount of area blacked out, or the total number of redactions. Many more opportunities to see what you can’t see.

Experiment D was an attempt to build a composite image of all the redactions to visualise what parts of a page were most likely to be be removed – something like a heatmap. It sort of worked, but by the time I’d added all the redactions I had nothing but a very large black blob.

My composite of 170,000 #ASIO #redactions has opened a portal to another universe… pic.twitter.com/39iQewjst4

— Tim Sherratt (@wragge) October 8, 2016

A rethink is required…

Experiment E had two aims. First to highlight the visual character of the redactions themselves – there’s a strange sort of beauty in a massed collection of blobs. Secondly, just as with the Real Face of White Australia, I wanted to turn the files inside out. Instead of being dead ends, I wanted the redactions to be discovery points, signposts, ways of exploring the files.

It’s online now so play. You can view a random sample of redactions, or browse page by page through the entire collection.

Talking about her own ASIO file in the book Dirty Secrets, the politician and academic Meredith Burgmann noted that the ‘blacking out process seems totally arbitrary and for the reader terribly frustrating, like reading a detective novel with the last page torn out’.7 But in hunting for redactions I found they could also bring moments of unexpected joy. It seems that someone got a bit bored and has left us with a glorious collection of redaction art.

So what’s to come? I need to rework my redaction finder to improve its accuracy.

It’s interesting, and perhaps ironic, that the removal of information has given me an identifiable data point that I can potentially track against other characteristics of the files. Can I identify patterns by time or topic?

Apparently ASIO assessments have become less conservative over the years – I can test this by looking at changes in redaction rates over time.

I also want to explore the context of redactions. By expanding the window around redactions and OCRing the result, I hope to identify the words that occur most commonly appear near redactions.

Those of you coming to the workshop on Friday will hear more about some of the tools and technologies I’ve used in these experiments. But I wanted to give a brief overview today because this is access.

Digital tools and technologies give us the opportunity to use databases like RecordSearch as archaeological sites to sift through layers of metadata in search of new connections and meanings. This is access.

We can turn digitised collections inside out, revealing the people, the processes, the structures, the form. This is access.

We can reveal the processes through which records are controlled, concealed, and withheld. This is access.

Access is not a deliverable or a product. It’s a struggle for understanding and power – not just to see, but to see differently.

This is RecordSearch but not as you know it.

Experiment F is a userscript that puts the redactions back into RecordSearch. Access is an honest acknowledgement of its own limits, and an invitation to push beyond.

1. Richard Jenkins, ‘Identity, surveillance and modernity: Sorting out who’s who’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 159.
2. Eric Ketelaar, ‘Access, the democratic imperative’, Archives and Manuscripts, vol. 34, no. 2, November 2006, p. 73.
3. John Gilliom and Torin Monahan, ‘Everyday resistance’, in Kristie Bell, Kevin D. Haggerty and David Lyon (eds), Routledge Companion of Surveillance Studies, Routledge, 2014, p. 407.
4. Ann Laura Stoller, Along the Archival Gran: Epistemic Anxieties and Colonial Common Sense, Princeton University Press, Princton, 2009, p. 32.
5. David Horner, The Spy Catchers: The Official History of ASIO, Allen & Unwin, Sydney, 2014, p. 581.
6. David McKnight, ‘How to read your ASIO file’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 38.
7. Meredith Burgmann, ‘The secret life of B/77/26 (and friends)’, in Meredith Burgmann (ed.), Dirty Secrets: Our ASIO Files, NewSouth, Sydney, 2014, p. 454.

Presented at DHA 2016 in Hobart. Follow my research. View the slides.

A good type

In January 1954, the economic historian Ernst Boehm requested access to Commonwealth government records as part of his research into the depressions of the 1890s and 1930s. The request was forwarded from the Archives Office to the Prime Minister’s Department where it was noted ‘Boehm is I believe a good type + will agree to the usual conditions’.[1]

This sort of character assessment was important because there were few formal guidelines around access to government records. The Commonwealth Archives Committee reported in 1955 that the use of government archives was ‘seriously hampered by the lack of an access policy’.[2] The Prime Minister’s Department noted that requests for access were ‘treated on an “ad hoc” basis’.[3] Prospective researchers first needed to establish their ‘bone fides’. Archives staff would then attempt to identify records of interest and submit them to the relevant government departments for clearance. The Archives Committee noted that the ‘inevitable uncertainties and delays inherent in this system’ often left researchers ‘puzzled and disappointed’.[4]

In the years that followed, interdepartmental agreements and Cabinet decisions gradually refined the process, but it would be another thirty years before the public’s right to access was enshrined in legislation.

Right to Access

In 1955, the Cold War was near its height. The Royal Commission on Espionage was grilling public servants, scientists, journalists and others in its hunt for Communist infiltrators. Amidst the hysteria about ‘secrets’ and spies the Commonwealth Archives Committee emphasised the ‘moral obligation’ of democratic governments to make the records of their activities available for research. ‘In doing so’, the Committee argued, ‘they make a significant contribution to the continuing search for truth which is an essential part of the continuing democratic process’.[5]

The idea that the public had a ‘right’ to access government records did not go unchallenged, however. In a meeting with the visiting American archivist Theodore Schellenberg the previous year, a representative of the Department of Defence had admitted that he ‘found it difficult to accept the concept that the general public had a right of access to public records’.[6]

But the obligations of war and empire actually pushed Australia towards a more liberal policy. In the 1950s, Australia agencies tended to follow British precedent, applying a version of the 50 year rule. Given that the Commonwealth of Australia was founded in 1900, a 50 year embargo didn’t leave a lot of history to write. In 1966, however, the UK government opened access to records through to 1922 – encouraging research into World War I and its aftermath. Australia had little choice but to follow their example. As John Gorton noted in his submission to Cabinet, ‘we should not impose greater restrictions on our own scholars than are imposed in Britain’.[7] When the UK shifted to a 30 year rule a few years later, Australia again adjusted its policy.

In 1972 the Whitlam Labor government brought with it a professed commitment to ‘open government’. Access to archives started to be linked to broader questions around ‘freedom of information’. It took a few attempts, but the Archives Act was finally passed in 1983. In 2010 ammendments reduced the closed period from thirty to twenty years.

The public interest

Under the Archives Act (in its current form) the public has a right to access Commonwealth records more than twenty years old. But this right has limits. In 1954, Schellenberg described the ‘public interest’ as a balance between a liberal access regime and the protections necessary to shield both governments and individuals from the dangers of over exposure.[8] As access policy evolved, so did the articulation of these limits.

Protections for national security and the privacy of individuals have been long accepted as grounds to withhold records from public access. But the scope and meaning of such exceptions has shifted. In 1966, restrictions on ‘the release of papers relating to the personal expenditure of the Governor-General’ were provided as an example of how individuals were being protected from ‘embarrassment or distress’.[9] Similarly it was assumed that records ‘commenting on the person of the Sovereign’ would naturally be withheld.[10]

But the core of these exemptions remained and was codified within section 33 of Archives Act. Section 33(1)(a), for example, allows records to be withheld from access if they could ‘cause damage to the security, defence, or international relations of the Commonwealth’. Section 33(1)(g) exempts records that might result in ‘the unreasonable disclosure of information relating to the personal affairs of any person’.

The ad hoc approach of the 1950s has been replaced by a well-documented bureaucratic process with clearly defined outcomes. Under the Act a researcher can ask for a record to be examined for public access. The contents of the file is then assessed against the list of defined exemptions and an access status is assigned – Open, Open with exception, or Closed (sometimes closed files are described as ‘withheld’).

Statistics on the number of files examined and their access outcomes are reported annually. The vast majority are opened to the public. But some are not.

Open Access

What do we mean by ‘open access’? Back in 1964 the Interdepartmental Committee on Commonwealth Archives defined ‘open access’ as referring to documents ‘which have been cleared for consultation by any member of the public’.[11] Open could only really be understood by invoking its limits – open was not closed.

I’ve shared this potted history of Australian archival policy because it continues to shape the research that we do and what we can know.

These shifting policies and priorities have shaped the data now available online through RecordSearch, the National Archives of Australia’s database. As digital scholars we should recognise that there’s nothing natural or inevitable about these systems and their contents. They are deeply and profoundly historical.

But recognising this is both an obligation and opportunity, because we can view such systems as archaeological sites and dig back through the layers of data. We can meet the history of policy, with an analysis of practice as it has been captured in the descriptions, dates and decisions.

The access status of each item in RecordSearch is recorded and displayed – ‘Not yet examined’, ‘Open’, ‘Open with Exception’, or ‘Closed’. If a file is ‘closed’ or ‘open with exception’ the reasons for the restriction are provided along with the date when the decision was made. This is interesting and important data. But while you can filter your search by status, you can’t specify a reason or date. So I can browse a list of ‘closed’ files, but there’s no way of getting an overview of why they’re closed.

Until now.

Last year, as an experiment, I screen-scraped RecordSearch to capture the details of files that had an access status of closed. On the 1 January this year, I repeated the harvest with a bit more care.

I now have a dataset of 14,370 closed files. These are the files we’re not allowed to see.

Closed Access

You can explore them online using Closed Access, a public site I developed to play with the data.

You can view the aggregated metadata from a number of different angles – by the reason for exemption, by record series, by the age of the contents, and by the date of the decision. You can drill down from any of these views to a list of closed files. And of course you can search – want to know which files were closed in May 1989 for national security reasons? ClosedAccess can tell you!

Given that none of the files are actually publicly available, it’s possibly the most frustrating search interface yet devised. However, you can always ask for a file’s access status to be re-examined.

ClosedAccess is really more of a workbench than a discovery interface, with additional experiments documented in my open research notebook.

There for example, you can read about TF-IDF analysis of file titles, or my failed attempts to reconcile the statistics reported in the National Archives’ annual reports with the data from RecordSearch.

Reasons

So, why have these files been closed? Here’s the list of reasons.

The first thing to note is while there are 17 exemptions defined by the Archives Act, there are 28 reasons for closing files cited in RecordSearch. Some of these extra reasons relate to specific classes of records mentioned elsewhere in the legislation, while others, such as ‘MAKE YOUR SELECTION’ tell us something about the design of the data entry interface.

It’s no surprise that sections 33(1)(a) and 33(1)(g) of the Act are the most frequently cited exceptions. As I’ve noted, these relate to national security and individual privacy – longstanding grounds for withholding access.

More interestingly, these exemptions appear very frequently in combination with others. The DH standard network graph illustrates their centrality. For example of the 425 closed records citing 33(1)(b) – which refers to confidential information supplied by another government – 399 (94%) also cite 33(1)(a). This sort of pattern, combined with the low level of use for many of the exemptions, suggests that the whole legislative framework could be simplified – how many reasons to close files do we really need?

What’s more surprising are the two highly-cited reasons at the bottom of this graph, neither of which are defined in the legislation or anywhere that I’ve been able to find on the National Archives website. They’re ‘Pre access recorder’ and ‘Withheld pending advice’.

Pre-access recorder

According to the brief explanatory note in RecordSearch, ‘Pre access recorder’ denotes ‘non standard reasons for restriction’ – not very illuminating. But using ClosedAccess you can dig into the metadata to get an idea of what this actually means.

The average age of files citing ‘Pre access recorder’ is 75 years, significantly older than the overall average of 58 years. Indeed, if you look at the dates on which decisions were made about these files, you’ll see that most of these were closed before 1983 when the Archives Act was introduced. So it seems these files were examined and withheld from public access under the less formal arrangements that existed prior to the legislation.

This was (sort of) confirmed by the National Archives who told me that yes they were ‘made prior to the introduction of the Archives Act’, but they were also decisions ‘where the reason for restriction didn’t map to one of the reasons for restriction set out in the Act’. I still don’t quite understand this.

If you take all the file titles and throw them into Voyant you’ll see the word ‘staff’ figures prominently, reflecting the fact that many of them are personnel files. Surely these might have mapped to the privacy provisions of section 33(1)(g)?

From there things get even more complicated. Just as in quantum theory, by observing the phenomena of pre-access recorder, I have changed it. As a result of my inquiry, which the Archives seemed to interpret as a criticism of their metadata, they changed the status of most the ‘Pre access recorder’ files from ‘closed’ to ‘not yet examined’. They’re not any more open, but they no longer show up in a search for closed files. Fortunately, I’ve preserved their historical status in Closed Access. I’m planning to repeat my harvest on the 1 January each year, tracking newly closed files as well as those whose status has changed. So next year I’ll be able to check up on the ‘Pre access recorder’ files to see what’s really happened to them.

Withheld pending advice

Withheld pending advice is easier to interpret. The files have been referred back to the government agencies that created or controlled them to check whether they really can be unleashed on the public.

This process is known and feared by researchers because of the long delays it can create. In 2014 Philip Dorling wrote in the Canberra Times about a file from 1932 relating to Australia’s annexation of Antarctica which languished with the Department of Foreign Affairs and Trade for five years before being released in a redacted form.[12]

Delays in access examination are well-known, but the scale and impact aren’t obvious.

I think that part of the problem I was having in reconciling access examination data from annual reports was that the Archives doesn’t include the ‘Withheld pending advice’ files in its statistics. Perhaps this is fair enough – the access examination process isn’t complete. On the other hand, under the Archives Act if examination requests aren’t completed within 90 days then access is assumed to have been refused. Are they less closed because they may yet be opened? I’ve asked the National Archives what the status of these files really is, and how this affects the public’s right to access.

In the meantime we can use Closed Access to see the scale of the backlog and indulge in a bit of agency shaming. You can easily search for ‘Withheld pending advice’ files and order them by the date they were closed.

Ignoring the one file with the title ‘unknown’ that has been waiting 18 years for a decision (I suspect it’s been mislaid), there’s still a file that’s been waiting for 9 years, and 9 files that have been waiting for 7 years. Indeed, you can limit your search to files that have been waiting more than three years for a decision – there’s 1467 of them.

The Defence department gets the longest in limbo award for the 9 year delay, but DFAT is the overall champion with series A1838 containing almost 1500 files awaiting departmental determinations – 54 of which have been waiting more than five years. Too bad if you’re doing a PhD. Researchers are still being left ‘puzzled and disappointed’.

It’s important to make these processes visible because it’s all to easy to forget that the default status for any Commonwealth government file more than 20 years old is ‘open’. The reasons for closing files defined under the Archives Act are called exemptions for this very reason.

Philip Dorling suggested that agencies are becoming more conservative in their decisions. Once I’ve harvested a few year’s worth of closed files, I’m hoping I’ll be able to trace the fate of those poor souls currently ‘withheld pending advice’. But there are other measures of conservatism we can start to explore.

You’d think that the older a file was the more likely it was not to be closed. So what happens when we look at the gap between the date a decision was made to close a file and the date of it’s latest content? I’ve added an experimental view to Closed Access where you can explore the files than seem to have provoked the most conservative approach.

Strangely, at the top of this list are the proceedings of the Senate from 1903. But in fact, no decision was made to close this file. It belongs to a category of records called ‘Parliament Class A’. Regulations drawn up under the Archives Act simply declare these records as ‘exempt’. You need the permission of Parliament to access them.

It’s a reminder that at least we do have the metadata for closed files. The intelligence agencies hold unknown quantities of files where access to the recordkeeping systems themselves is restricted. We don’t know what we don’t know.

I’m hoping to explore these gaps further, but for now if you’d like a copy of the 300,000 digitised pages contained within publicly accessible ASIO files let me know.

No magic

It’s important to recognise all of this as the workings of a system. Digging through the layers of policy and metadata helps us to see that there is nothing intrinsically dangerous in a closed file. A judgement has been made against a list of historically determined criteria.

But it’s also important to resist the temptation to claim privileged access to the truth through our mastery of such systems. Next time you see words like ‘recently released documents’ in a book or article, remember that access examination is a process without magic or wonder. We need to remain critical of the process without adding layers of mystification – without succumbing to the cultural power of the secret.

I obviously did a lot of talking in 2012, but I also made a few things…

The evolution of QueryPic

At the start of 2012 QueryPic was a fairly messy Python script that scraped data from the Trove newspaper database and generated a local html file. It worked well enough and was generously reviewed in the Journal of Digital Humanities. But QueryPic’s ability to generate a quick visualisation of a newspaper search was undermined by the work necessary to get the script running in the first place. I wanted it to be easy and accessible for everyone.

Fortunately the folks at the National Library of Australia had already started work on an API. Once it became available for beta testing, I started rebuilding QueryPic — replacing the Python and screen-scraping with Javascript and JSON.

In the meantime, I headed over the New Zealand for a digital history workshop and began to wonder about building a NZ version of QueryPic based on the content of Papers Past, available through the DigitialNZ API. The work I’d already done with the Trove API made this remarkable easy and QueryPic NZ was born.

Once the Trove API was publicly released I finished off the new version of QueryPic. Instead of a Python script that had to be downloaded and run from the command line, QueryPic was now a simple web form that generated visualisations on demand.

The new version also included a ‘shareable’ link, but all this really did was regenerate the query. There was no way of citing a visualisation as it existed at a certain point in time. If QueryPic was going to be of scholarly use, it needed to be properly citable. I also wanted to make it possible to visualise more complex queries.

And so the next step in QueryPic’s evolution was to hook the web form to a backend database that would store queries and make them available through persistent urls. With the addition of various other bells and whistles, QueryPic became a fully-fledged web application — a place for people to play, to share and to explore.

Headlines and history

Back in 2011 I started examining ways of finding and extracting editorials from digitised newspapers.  Because the location of editorials is often tied up with the main news stories, this started me thinking about when the news moved to the front page. And of course this meant that I ended up downloading the metadata for four million newspaper articles and building a public web application — The Front Page — to explore the results.

The Front Page was also the first resource published on my new dhistory site (since joined by the Archives Viewer and QueryPic). dhistory — ‘your digital history workbench’ — is where I hope to collect tools and resources that have graduated from WraggeLabs.

Viewing archives

In 2012 I also revisited some older projects. After much hair-pulling and head-scratching, I finally managed to get the Zotero translator for the National Archives of Australia’s RecordSearch database working nicely again. I also updated it to work with the latest versions of Zotero, including the new bookmarklet.

My various userscripts for RecordSearch also needed some maintenance. This prompted me to reconsider my hacked together alternative interface for viewing digitised files in RecordSearch. While the userscript worked pretty well, there were limits to what I could do. The alternative was to build a separate web interface… and so the Archives Viewer was born.

Stories and data

In the ‘work-in-progress’ category is the demo I put together for my NDF2012 talk, Small stories in a big data world. Expect to see more of this…

My favourite things

Two things I made in 2012 are rather special (to me at least). Instead of responding to particular needs or frustrations, these projects emerged from late night flashes of inspiration — ‘what if…?’ moments. They’re not particularly useful, but both have encouraged me to think about what I do in different ways.

The Future of the Past is a way of exploring a set of newspaper articles from Trove. I’ve told the story of its creation elsewhere — I simply fell in love with the evocative combinations of words that were being generated by text analysis and wanted to share them. It’s playful, surprising and frustrating. And you can make your own tweetable fridge poetry!

One night I was thinking about The Real Face of White Australia and the work I’d done extracting photos of people from the records of the National Archives of Australia’s database. I wondered what would happen if we went the other way — if we put the people back into RecordSearch. The result was The People Inside — an experiment in rethinking archival interfaces.

Earlier this week it was announced that the Mosman Library had been awarded a Library Development Grant for an innovative project that aims to document stories and artefacts relating to the First World War. I’m very excited to be part of it. As well as working with the local community in the creation of a new resource, the project offers an interesting opportunity to explore how we can link in with the ever-increasing volume of WWI material being published as linked data around the world.

But thinking about this new project has also made me reflect again on the creation of Mapping Our Anzacs — a project that still fills me with great pride and immense frustration. I thought I might as well finally post a couple of things I wrote about the project back in 2009. They’re a bit out of date, but I think there’s still a few useful lessons to be gleaned.

The first is a case-study that focuses on the crowdsourcing aspects of Mapping Our Anzacs. The second looks at the project as an example of a mashup. Thanks to Kate Theimer for initiating and publishing both pieces.

Bringing life to records

2009 preprint version of case-study originally published in Kate Theimer (ed), A Different Kind of Web: New Connections between Archives and Our Users, Society of American Archivists, Chicago, 2011. [Order here]

Overview of repository

The National Archives of Australia is responsible for preserving and making accessible the records of the Commonwealth of Australia. It employs more than 400 staff, with offices in Canberra and every state capital. Its holdings include more than 360 shelf km of records – around 69 million items. Through its digitisation program more than 1.6 million items have been fully digitised, making nearly 20 million digital images available online. The National Archives’ website now provides the main point of access for researchers, with more than 2 million images viewed through the online database RecordSearch in the year 2007–8.

Business drivers

Most people now experience the collections of the National Archives of Australia online. With an obligation to provide ‘an accessible, and interpreted, national archival collection’ the Archives is looking to new technologies to enhance access and improve efficiency.

The idea for Mapping our Anzacs arose during planning for a travelling exhibition on the impact of World War I, timed to coincide with the 90^th anniversary of the war’s end. Public interest in commemorating Australia’s war effort was as strong as ever, so a website that encouraged local participation seemed a useful way of extending the exhibition and its accompanying education program.

The major focus of both the exhibition and the website was to be the 376,000 service records documenting the experiences of Australian men and women during World War I held by the National Archives. These records had been fully digitised and described as part of a major project entitled ‘A Gift to the Nation’, but were still somewhat buried within our collection database.

Mapping our Anzacs was intended to highlight these records and open them up to local communities. First a map interface would allow service records to be discovered by place of birth or enlistment. Secondly, users would be able to add tributes – online versions of the war memorials that remain a feature of just about every town, large or small.

While the exhibition and the records themselves provided the main drivers for the project, there was also a growing desire within the institution to explore some of the possibilities of Web 2.0 technologies. This desire was tempered somewhat by a range of familiar concerns centred on issues of authority and control. Would user contributions detract from the reliability of the records? Who would take responsibility for any errors in user-created content? Would the potential for abuse demand vigilant moderation? Mapping our Anzacs gave us a chance to start working through such issues.

Setting the stage

We had an idea, a budget and a launch date, what we needed was a plan. While in theory we had around six months to play with, the project had to be fitted in around the ongoing work of our small web team. On the content side we had one person cleaning up the data. At the technical end we had someone connecting up the various components and making it all work within the Archives’ web environment. In the middle there were two of us trying to marry content and technology and create a usable resource. While we had a range of useful skills, none of us had tackled a project quite like this. We all had to learn on the job.

With few models or examples to work from, we began to experiment – researching available technologies, throwing around possibilities. Our first efforts were largely focused on the map interface and before long we had a working prototype using javascript and Google Maps. But what we also needed was a better understanding of how users might interact with the site.

We started from the idea of the online memorial – a list of names compiled by users that would be linked through to service records. Our example was a local historical society creating a site to commemorate their community’s war effort. But what if they had more information – photographs or family histories – how could this sort of material be incorporated? Further inspiration came from a visit to the local historical museum in the small Victorian town of Chiltern. On one wall was a typical roll of honour, listing the names of those who had served in the war. But underneath were framed portraits of many of those listed. They were people, not just records. Could we create something like this online?

There were some exciting possibilities emerging, but concerns remained. Would anybody actually want to contribute? Strong interest in family history and a growing community desire to commemorate the experience of World War I offered anecdotal support. We just had to ensure that this interest could be translated into engagement – that the barriers of participation were low enough to encourage visitors to become collaborators.

But what of concerns that such material might detract from the authority of the records, or open our institution up to liability? We needed to make it clear where public contributions began and archival data ended.

Welcoming, but separate; open, but managed – a tricky balancing act was required. The answer, we decided was to create a separate ‘scrapbook’ using the blogging service Tumblr. The ‘scrapbook’ label was intended to be encouraging – this was not a database, or formal register, it was a place to leave your thoughts, comments, information or memorabilia. This was reinforced by our terms of service which simply required contributions to be relevant and respectful.

A ‘scrapbook’ was also something quite different to a finding aid. The informality helped to make the boundary clear between record and response. The separation was physical as well as intellectual. While the scrapbook shared many of the design elements of the main site, it was hosted by Tumblr not the National Archives. By using the Tumblr API, however, it was easy to pass information between the two sites. We could also use the API to provide a basic moderation facility.

But this meant that an important part of the site’s functionality would be dependent on an outside service. To make sure we considered fully all the implications of this, we developed a risk analysis and contacted Tumblr staff to inform them of our plans. Our major concern was simply the continuity of the service. While there could be no guarantees, we judged that this risk was manageable. Tumblr staff were interested in the project and offered their assistance if necessary.

Results

On 25 April each year, Anzac Day, Australians remember the sacrifices made in war. Over the Anzac Day weekend in 2009, we were astonished to receive more than 200 scrapbook posts. Of course we had expected an increase in use, particularly after the site was featured on the Australian version of the Today Show, but this remarkable response certainly confirmed the site’s success. In the six months since its launch there had been almost 94,000 visitors to Mapping our Anzacs. More than 1,000 scrapbook posts had been contributed and 280 tributes created.

But the greatest success was in the type of posts being contributed rather than their sheer volume. Our ‘scrapbook’ had proved to be just that – as well as photographs of service people and their families, there were pictures of medals, headstones, letters, newspaper clippings, pay books, identity disks, diaries, postcards and certificates. Some people simply commented ‘my grandfather’, while others wrote detailed accounts of family history. Perhaps most moving were those who took the opportunity to leave a message for their loved one: ‘You were the best dad’.

Some have taken a systematic approach. Our most frequent contributor is gradually attaching photographs of headstones and memorial plaques that she has gathered from local cemeteries. Others are posting their own contact details in the hope of linking up with family. Perhaps most interesting are the notes that provide links to other people or documents – to family members, for example, or to a later service record. These are helping build a rich web of contextual data. Equally valuable are the corrections and additions that are being offered by eagle-eyed users, pointing out transcription errors or helping us track down elusive locations.

The success of the scrapbook has somewhat overshadowed the tributes, or online memorials, which really provided our starting point. Many tributes have been created and, as we had hoped, schools and other groups are using them to document the impact of war on their local communities. However, some compromises at the implementation stage have meant that it is not as easy to build them as we had hoped. There has also been some confusion by users between the tributes and the scrapbook. This is one area of the site we certainly hope to improve.

Even though the digitised service records had been available online for sometime though our collection database, it’s clear that many people are discovering them for the first time through Mapping our Anzacs. It was ‘a stunning find for me and my siblings’ wrote one grateful user. The scrapbook has aided discovery, providing another way into the records. Indeed, with the addition of a MediaRSS feed for CoolIris, the scrapbook provides two new entry points – one of them a 3D wall of faces and families. By embedding the records in these new contexts and making them easier to find, Mapping our Anzacs has successfully garnered extra value from an existing asset.

The site has also been recognised by others for its successful use of Web 2.0 technologies. We were pleased to be joint winners of the Best Archives on the Web Award, and surprised to be cited by the Federal Minister for Finance in a speech launching a Gov 2.0 taskforce. Recognition such as this has helped strengthen the case for future innovation in the Web 2.0 sphere.

Challenges

Success brings its own problems. One of the main challenges has been simply managing the sheer volume of posts and feedback. This was particularly acute of course after the Anzac Day deluge. As a result we have had to consider ways of streamlining our processes.

The Tumblr API allows us to set the status of a new post as ‘private’. We can then examine the post using the Tumblr dashboard before making it public. This works well enough as a basic form of moderation, however, the dashboard is not really designed for this purpose and it takes several clicks to release each post to the world.

But while moderation takes considerable time, it requires little intellectual effort. Despite concerns about abuse, our contributors have caused us few dilemmas. The only significant questions that have arisen concern the re-use of materials from other sources. This has made consider whether pre-emptive moderation is necessary or appropriate.

While the site includes detailed help information, it’s clear from the feedback that there are certain aspects that continue to cause difficulty. This provides useful data on how the site might be improved, but it has also made us think about how we communicate with our users. At the moment the content we provide is fairly static – there is no way of informing visitors of recent updates, or developing quick guides to common problems. If we took a more active approach to communication we might be able to decrease the number of help requests, while building a greater sense of community.

Similarly, while we have been excited by the number of corrections submitted by users, we can now see ways in which we might have structured the feedback process to capture their corrections more easily and efficiently. For example, a ‘submit a correction’ link on each individual’s page could automatically capture the person’s details, saving both us and our contributors from potential confusion.

We have suffered through the expected number of software glitches, and have a growing list of things we’d like to improve or develop, but overall the experience has been much more rewarding than painful.

Lessons Learned

Perhaps the most valuable lessons revolve around trust. Having entered into the project uncertain of what to expect from public participation, we have found ourselves in an evolving, creative partnership. Our users have defined what the scrapbook is and have taken an active role in improving and developing the resource. Our trust has been repaid many times over, helping us build something that in many ways has exceeded our expectations.

Trust is also necessary in the support of new ideas. Mapping our Anzacs was a very different type of project for the National Archives, challenging ideas both of access and user engagement. By taking the risk we have not only gained valuable publicity and user support, we have opened up the realm of possibilities for future development.

In terms of technology, the project demonstrated the power of the mashup and the efficiencies that can be gained by using existing web services. Tumblr, Google Maps and their associated APIs gave us a kickstart that enabled us to do a lot with a little.

Next Steps

There are so many exciting possibilities! Obviously our first priority is to improve those areas of the site that continue to cause our users grief. There are a number of navigation and usability tweaks that should improve the overall experience. Similarly, we can now see ways in which we might streamline moderation and management processes.

We hope to build on the success of the scrapbook and tributes by enhancing and extending their functionality. Improved editing and creation tools could assist contributors while also enriching the web of connections they build. We might, for example, provide widgets that make it easier to link the records of family members or friends. Over time this could develop into a complex network of relationships, providing new means of finding and visualising the records. Similarly, there are ways in which we might reuse the existing content of the scrapbook posts to develop new modes of discovery.

We could also do more to feature the labours and passions of our contributors. We could give them the option of exposing a public profile that lists all of their scrapbook posts. This would help foster a sense of community while providing yet another means of exploring connections between records.

Recent developments in geospatial technology and mobile devices perhaps offer the most exciting possibilities. Our original aim was to give the World War I service records back to local communities, to imbue the records with a greater sense of context, locality and belonging. Perhaps we will have succeeded when a tourist exploring a small country town can press a button on their mobile phone to retrieve a list of service people born near their current location.

Perhaps they will take a photo of a name on the local war memorial and use it to automatically retrieve that person’s service record or create an online tribute.

Perhaps they will come across a headstone in the local cemetery and immediately upload a geocoded photograph to the Mapping our Anzacs scrapbook.

Instead of merely being markers on a map, the records will start to overlay and inform the very spaces in which we move. The stories they contain will become part of our journeys, the people they document will have found their way home.

Creating a mashup

2009 preprint version of interview originally published in Kate Theimer, Web 2.0 Tools and Strategies for Archives and Local History Collections, Neal-Schuman, New York, 2010. [Order here]

What made you interested in creating a mashup?

It really started with the records. We hold the records of more than 375,000 World War I service people, identified by their places of birth and enlistment. With war memorials in just about every town across Australia, the connection between local communities and the memory of war remains strong. So we wondered how we could we give the service people in our records back to their communities. Having played around with the Google Maps API the answer seemed obvious — find the places, put them on a map and let people explore the connections for themselves.

What information, tools and processes did you need to begin?

The main thing we needed was the confidence to experiment. The process seemed straightforward in principle: first we had to extract the data we needed from the file titles in our collection database, then we had to find the latitude and longitude of each of the place names we extracted, and finally we had to plot these coordinates on a map with links back to details of the service records themselves. Web services, such as those provided by Google Maps, had the potential to do much of the work for us, and we scoured online documentation, user forums and blog posts for hints. But there were many things we could not know until we actually started. How consistent was our data? How many of the places would we be able to find? How would we be able to display thousands of places at once?

Moving from file titles through to coordinates obviously required a lot of data manipulation and we used Perl for much of the grunt work. Because our data set was large and variations in spelling and formatting were often unpredictable (including 13 different spellings of ‘lieutenant’!), we often had to work by trial and error — seeing what results we obtained and then adjusting our processes accordingly.

Once we had a list of place names in a consistent format we could begin to find latitudes and longitudes through a process known as geocoding. Google’s geocoding service was an easy option: it was well documented, reasonably comprehensive and it worked! We fed it our place names through a Perl script and soon we had a list of coordinates. Of course, many places were not found or returned multiple results, but the basic principle was sound. Our places were no longer just names, but points in space — we could begin making maps.

How did you determine what to include?

What we were creating was an archival finding aid, but one which placed the people, their homes and their communities up front rather than the systems that control their records. By browsing from a map a user would be able to find the details of a loved one, read a digitised copy of their service record and then follow a link through to our collection database. These links provide crucial context about the records, but we realised that this project also gave us an opportunity to capture other contexts and meanings. Who were these people? What did they look like? What happened to them after the war? By adding an online ‘scrapbook’ we gave users the chance to enrich the resource by adding notes or photographs about individuals.

This meant we had to deal with three sets of interlinked data: geocoded places, details from our records, and scrapbook posts provided by the public. To bring these all together with limited resources we had to make clever use of what was already out there. Why create our own maps when all you needed to do was write a bit of Javascript to embed a Google Map? Why build our own scrapbook application when the blogging service Tumblr provides free accounts and a simple API to manipulate posts? While a substantial amount of custom scripting was required to glue everything together, much of the core functionality was provided by free web services, available to anyone.

What challenges did you face?

Perhaps the first challenge to overcome was that of imagination. It was difficult for people to understand what the project was until we had a prototype to show them.

The process of handling and cleaning the data at times threatened to overwhelm us. While the geocoding service got us to the point where we could make maps, it also left us with many place names that needed to be manually checked. Often this was the result of misspellings in the original data, or because places either no longer existed or had changed their names. This data cleanup consumed much effort and continues still, though now with the help of our users who regularly point out errors and inconsistencies.

Once we had our coordinates we had to display them on a map without killing anyone’s computer. Showing thousands of markers on a Google Map is a challenge to slower web browsers and can end up hindering navigation. By dividing up our maps, clustering markers and changing the way they were rendered, we managed to greatly improve performance while maintaining the browsing experience. Once again it was trial and error coupled with the advice of the online community that guided us through the roadblocks.

What kinds of positive results have you had? (And any negative ones?)

From the messages we receive it’s clear that Mapping our Anzacs allows people to find records they didn’t know existed. Some have met a great-great-uncle for the first time. Others have learned about the war experience of a much-loved grandparent. Local communities have embraced the project and the scrapbook has developed into a rich and often moving resource. We wanted to give users a new way to explore and interact with our collection, and it seems we have succeeded.

Our users have also become our collaborators, providing corrections and comments that help us improve our data. They have extended the idea of the scrapbook, using it, for example, as a noticeboard for family history research, or as a way of creating crosslinks between related resources.

Success brings problems of its own and the work of moderating the scrapbook and responding to feedback has proved considerable. Issues with performance remain for people on slow connections, and while many are familiar with the Google Maps interface, some find it difficult to navigate. We are planning a number of enhancements based on this feedback, and hope to take advantage of the technology as it evolves to improve and extend the interface.

About how much time did it take?

While the project as a whole stretched over about eight months, much of this time was taken up cleaning and processing the data. The development of the interface was completed in under two months.

What advice would you give an organization wanting to use something similar?

Start experimenting. The technology is developing so rapidly that if you spend 12 months planning a project it’s likely to be out-of-date even before you start. New web services and data sources are becoming available every day. Perhaps you could use Open Calais to extract people’s names from a collection description, or MetaCarta to find the places. You might use the Google Books API to harvest the details of publications that cite your records. Even if you’re not a coder you can use tools like Yahoo Pipes to see what happens when you start to link data and services. Experimentation brings new ideas and possibilities. It’s all about making connections.

This is the full version of a paper I presented at the National Digital Forum, 30 November 2011.

In 1901, one of the first acts of the Commonwealth of Australia was to create a system of exclusion and control designed to keep the newly-formed nation ‘white’. But White Australia was always a myth. As well as the Indigenous population, there were already many thousands of people classified as ‘non-white‘ living in Australia — most were Chinese, but there were also Japanese, Indians, Syrians and Indonesians.

Here are some of them…

The administration of what became known as the White Australia Policy created a huge volume of records, much of which is still preserved within the National Archives of Australia. These photographs are attached to certificates that non-white residents needed to get back into the country if they decided to travel overseas. There are thousands upon thousands of these certificates in the Archives. Thousands of certificates representing thousands of lives — all monitored and controlled.

But is is too easy to see these people as the powerless victims of a repressive system. There were many acts of resistance. Some argued against the need to be identified ‘just like a criminal’. Others exercised control over their representation, submitting formal studio portraits instead of mug shots.

Most commonly and most powerfully, people resisted the policy simply by going ahead and living rich and productive lives.

My partner, Kate Bagnall, is helping to rewrite Australian-Chinese history by overthrowing the stereotype of the culturally isolated Chinese man living a lonely, meagre existence surrounded by gambling and opium dens. By mining the available records, by reading against the grain of contemporary reports and by working with family historians, Kate is documenting their intimate lives — their wives, their lovers, their families and descendants — the sorts of relationships that sent a shudder through the edifice of White Australia. Power can be reclaimed in many subtle and subversive ways.

‘The real face of White Australia’ is an experiment. It uses facial detection to technology to find and extract the photographs from digital copies of the original certificates made available through the National Archives of Australia’s collection database. The photographs you see here come from just one series, ST84/1. There’s no API to the collection so I reverse-engineered the web interface to create a script that would harvest the item metadata and download copies of all the digitised images. There are 2,756 files in this series. On the day I harvested the metadata, 347 of those files had been digitised, comprising 12,502 images. It took a few hours, but I just ran my script and soon I had a copy of all of this in my local database.

Then came the exciting part. Using a facial detection script I found through Google and an open source computer vision library, I started experimenting with ways of extracting the photos. After a few tweaks I had something that worked pretty well, so I pointed my aging laptop at the 12,502 images and watched anxiously as the CPU temperature rose and rose. It took a few emergency cooling measures, but the laptop survived and I had a folder containing 11,170 cropped images. About a third of these weren’t actually faces, but it was easy to manually remove the false positives, leaving 7,247 photos.

These photos. These people.

With my database fully primed and loaded it was just a matter of creating a simple web interface using Django for the backend and Isotope (a jQuery plugin) at the front. Both are open source projects. All together, from idea to interface, it took a bit more than a weekend to create, and most of that was waiting for the harvesting and facial detection scripts to complete. It would be silly to say it was easy, but I would say that it wasn’t hard.

What we ended up with was a new way of seeing and understanding the records — not as the remnants of bureaucratic processes, but as windows onto the lives of people. All the faces are linked to copies of the original certificates and back to the collection database of the National Archives. So this is also a finding aid. A finding aid that brings the people to the front.

According to Margaret Hedstrom the archival interface ‘is a site where power is negotiated and exercised’.1 Whether in a reading room or online, finding aids or collection databases are ‘neither neutral nor transparent’, but the product of ‘conscious design decisions’. We would like to think that this interface gives some power back to the people within the records. Their photographs challenge us to do something, to think something, to feel something. We cannot escape their discomfiting gaze.

But this interface represents another subtle shift in power. We could create it without any explicit assistance or involvement by the National Archives itself. Simply by putting part of the collection online, they provided us with the opportunity to develop a resource that both extends and critiques the existing collection database. Interfaces to cultural heritage collections are no longer controlled solely by cultural heritage institutions.

It’s these two aspects of the power of interfaces that I want to focus on today.

There are a growing number of examples where the records created by repressive or discriminatory regimes have, in Eric Ketelaar’s words, ‘become instruments of empowerment and liberation, salvation and freedom’.2 Nazi records of assets confiscated during the Holocaust have been used to inform processes of restitution and reparation. Government records have helped members of Australia’s Stolen Generations trace family members. Descendants of inmates incarcerated by American colonial authorities in what was the world’s largest leprosy colony in the Philippines, have embraced the administrative record as an affirmation of their own heritage and survival.3 Records can find new meanings. Power can be reclaimed.

Technology can help. Tim Hitchcock has described how something as simple as keyword searching can turn archives on their heads. Recordkeeping systems tend to reflect the structures and power relations of the organisations that create them. The ‘hierarchical and institutional nature of most archives’, Hitchcock argues, ‘contains an ideological component which is sucked in with every dust-filled breath’.4 But digitisation and keyword searching free us from having to follow the well-worn paths of institutional power. We can find people and follow their lives against the flow of bureaucratic convenience. We can gain a wholly new perspective on the workings of society. ‘What changes’, Hitchcock asks, ‘when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?’5

Projects such as Unknown no longer may help us answer that question.

It’s aiming to extract the names and biographical details of slaves from the 8 million manuscript documents held by the Virginia Historical Society. The documents include court records, receipts, wills and inventories. Here is a page from the ‘Inventory of Negroes at Berry Plain Plantation, King George County, Virginia’ for 1855, listing names, occupations and valuations.

Tim Hitchcock is one of the directors of London Lives a project that similarly seeks to find the people in 240,000 manuscript pages documenting the lives of plebeian Londoners in the 17th century.

More than three million names have already been extracted from the records of courts, workhouses, hospitals and other institutions. Work is continuing to link these names together, to merge these various shards of identity and piece together the experiences of London’s poorest inhabitants.

Remember me from the US Holocaust Memorial Museum is working with photographs taken by relief agencies in the aftermath of World War Two. The photographs are of displaced children who survived the Holocaust but were separated from families. What happened to them? The project is seeking public help to identify and trace the children.

These are all projects about finding people. Finding the oppressed, the vulnerable, the displaced, the marginalized and the poor and giving them their place in history. This is what Kate and I hope to do with Invisible Australians, the broader project of which our faces experiment is part.

‘Invisible Australians’ aims to extract more than just photographs. We want to record and aggregate the biographical data contained within the records of the White Australia Policy — to extract the data and rebuild identities.

But we want to do more, we want to link these identities up with with other records, with the research of family and local historians, with cemetery registers and family trees, with newspaper articles and databases we don’t even know about yet. We want to find people, families and communities.

It’s ridiculously ambitious and totally unfunded. But it is possible.

The most exciting part of online technology is the power it gives to people to pursue their passions. As with the faces, we don’t need the help of the National Archives. We need the records to be digitized, but that’s happening anyway and we can afford to be patient. Most of the tools we need already exist, and are free. In the past 12 months, for example, there have been a number of open source tools released for crowd-sourced transcription of manuscript records.

People with passions, people with dreams, people who are just annoyed and impatient, don’t have to wait for cultural institutions to create exactly what they need. They can take what’s on offer and change it.

Interfaces can be modified. It is amazingly easy to write a script that will change the way a web page looks and behaves in your browser. I was frustrated by the standard interface to digitized files in the National Archives of Australia’s Recordsearch database — so I changed it.

Not only did make it look a bit nicer, I added new functions. My script lets you print a whole file or a range of pages and display the entire contents of the file on a pretty cool 3d wall.

I’ve shared this script, and a few other Recordsearch enhancements. Anyone can install them with a click and use them.

Interfaces are sites of power and we can claim some of that power for ourselves. Online technologies not only free us from the having to brave the physical intimidation of the reading room, they free us up to engage with the records in new ways. The archivist-on-duty would probably not be pleased if I pulled out some scissors and started snipping photos out of certificates. Or if I pulled a file apart and pasted it’s contents on the wall. But online we are free to experiment.

The power of cultural heritage organisations is perhaps expressed most forcefully in their ability to control the arrangement and description of their collections. ‘Every representation, every model of description, is biased’, note Verne Harris and Wendy Duff, ‘because it reflects a particular world-view and is constructed to meet specific purposes’.6 Archives, libraries and museums are already starting to share this power, by allowing tagging, or seeking public assistance with description through crowd sourcing projects. But most of the these activities still happen within spaces created and curated by the institutions themselves. Our cathedrals of culture might be opening their doors and inviting the public to participate in their ceremonies, but that doesn’t make them bazaars. The architecture stills speaks of authority.

In any case, people already have a space where they can explore and enrich collections — it’s called the internet.

It would be great to see cultural institutions doing more to watch, understand and support what people are doing with collections in their own spaces — following them as they pursue their passions, rather than thinking of ways to motivate them.

A quick example… You might have heard of Zotero, it’s an open source project that lets you capture, annotate and organize your research materials.

One cool thing about Zotero is that you can build and contribute little screen scrapers, called translators, that let Zotero extract structured data from any old collection database. You might not be surprised to learn that I’ve created a translator for Recordsearch. Another cool thing about Zotero is that you can share the stuff that you collect in public groups.

Put those two cool things together and what do you have? Well to me they spell out user generated finding aids — parallel collection databases created by researchers simply pursuing their own passions.

Linked Open Data greatly increases opportunities for collection description to leak into the wider web. If objects and documents are identified with a unique URL, then anyone can can make and publish statements about them in machine-readable form. These statements can then be aggregated and explored. Initiatives such as the Open Annotation Collaboration will hasten the development of these shared descriptive and interpretative layers around our cultural collections.

And of course all this descriptive and interpretative work can be harvested back to enhance existing collection databases. We could start doing it now — though I will spare you today my rant about the possibilities of mining footnotes.

As well as exploring the possibilities of user-generated content, cultural institutions are starting to open up their collection data for re-use. APIs are great (though Linked Open Data is better), and New Zealand is lucky to have an organisation like DigitalNZ which just gets it. People can and will make cool things with your stuff.

But again, we don’t have to wait for everything to be delivered in a convenient, machine-readable form. If it’s on the web anybody can scrape, harvest and experiment.

You probably all know about the National Library of Australia’s newspaper digitisation project — it’s building a magnificent resource. But I wanted to do more than just find articles. I wanted to explore and analyze their content on a large scale. So I built a screen scraper to extract structured data from search results, and then used the scraper to  power a series of tools. I have a harvester that lets you download an entire results set — hundreds or thousands of articles — with metadata neatly packaged for further analysis.

Or what about a script that graphs the occurrence of search terms over time, and allows you to ask questions like When did the Great War become the First World War?.

In the end I got a bit carried away and built my own public API to the Trove newspaper database.

I think it’s important to note that the tools I developed were guided by the types of questions I wanted to ask. While we should welcome APIs and celebrate their possibilities, we should also remain critical. APIs are interfaces, they too embed power relations. Every API has an argument. What questions do they let us ask? What questions do they prevent us from asking?

Even as we move from the age of lumbering, slow-witted data silos into the rapidly-evolving realms of Linked Open Data, we have to constantly question the models we make of the world. Ontologies and vocabularies are culturally determined and historically specific. Yes, they too are interfaces, complete with their own distributions of power and authority. But we can revisit and change them. And we can relate our new models to our old models, capturing complex, long-term shifts in the way we think about the world. That’s incredibly exciting.

All of this hacking, harvesting, questioning, enriching and meaning-making makes me think about the possibilities of grassroots leadership. Online technologies enable people to take cultural institutions into unexpected realms. They can build their own interfaces, ask their own questions, determine their own needs — they can point the way instead of simply waiting to be served.

You might wonder what the National Library of Australia thinks of my various scrapers and harvesters. I can’t speak for them, but I can say that they’ve awarded me a fellowship to explore further the possibilities of text-mining in their newspaper database.

The idea of grassroots leadership brings me back to the title of this talk — ‘It’s all about the stuff’. It seems to me that we tend to model the interactions between cultural institutions and the public as transactions. The public are ‘clients’, ‘patrons’, ‘users’ or ‘visitors’. But the sorts of things I’ve been talking about today give us a chance to put the collections themselves squarely at the centre of our thoughts and actions. Instead of concentrating on the relationship between the institution and the public, we can can focus on the relationship we both have with the collections.

It’s all about the stuff.

It’s all about the respect and responsibility we both have for our collections.

It’s all about the respect and responsibility we both have for people like this.

1. Margaret Hedstrom, ‘Archives, Memory, and Interfaces with the Past’, Archival Science, vol. 2, 2002, p. 22.
2. Eric Ketelaar, ‘Archival Temples, Archival Prisons: Modes of Power and Protection’, Archival Science, vol. 2, 2002, p. 229.
3. Ricardo L Punzalan, ‘Archives and the Segregation of People with Leprosy in the Philippines’, presented at ICHORA-4, University of Western Australia, 2008, pp. 93-102.
4. Tim Hitchcock, ‘Digital Searching and the Re-formulation of Historical Knowledge’, in Mark Greengrass and Lorna Hughes (eds), The Virtual Representation of the Past, Farnham, UK: Ashgate, 2008, p.87.
5. Tim Hitchcock, p.90.
6. Wendy M. Duff and Verne Harris, ‘Stories and Names: Archival Description as Narrating Records and Constructing Meanings’, Archival Science, vol. 2, 2002, p. 275.

In many of the presentations I’ve given in recent times I’ve managed to include a question raised by Tim Hitchcock in his chapter in The Virtual Representation of the Past. Tim asks:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

The idea of turning archival systems on their head to expose the people rather than the bureaucracy is what motivates Kate Bagnall and I in our attempts to make the Invisible Australians project into a reality.

Invisible Australians aims to liberate the lives of those who suffered under the restrictions of the White Australia Policy from the rich archival holdings of the National Archives of Australia and elsewhere.

We always knew that the portrait photographs, included on a range of government documents, would provide a compelling perspective on these lives, but we weren’t quite sure how we were going to extract them. Up until last weekend, I’d assumed that we’d develop a crowdsourcing tool that contributors would use to mark-up the photos.

Now I’m not so sure.

In the space of a couple of days I’ve extracted over 7,000 photographs and built an application to browse them — here is the real face of White Australia…

How did I do it? Paul Hagon, at the National Library of Australia, gave a presentation last year in which he explored the possibilities of facial detection in developing access to photographic collections. The idea lodged in my brain somewhere and a few days ago I started to poke around looking to see how practical it might be for Invisible Australians.

It didn’t take long to find a python script that used the OpenCV library to detect faces in photographs. I tried the script on a few of the NAA documents and was impressed — there were a few false positives, but the faces were being found!

So then the excitement kicked in. I modified the script so that instead of just finding the coordinates of faces it would enlarge the selected area by 50px on each side and then crop the image. This did a great job of extracting the portraits. I tweaked a few of the settings as well to try and reduce the number of false positives. Eventually, I developed a two-pass system that repeated the detection process after the image had been cropped and it’s contrast adjusted. This seemed to weed out a few more errors. You can find the code on GitHub.

Once the script was working I had to assemble the documents. I already had a basic harvester that would retrieve both the file metadata and digitised images for any series in the NAA database. Acting on Kate’s advice, I pointed it at series ST84/1 and downloaded 12,502 page images.

All I then had to do was loop the facial detection script over the images. Simple! The only problem was that my 3-year-old laptop wasn’t quite up to the task. As it’s CPU temperature rose and rose, I was forced to employ a special high-tech cooling system.

But after running for several hours, my faithful old laptop finally worked it’s way through all the documents. The result was a directory full of 11,170 cropped images.

There were still quite a lot of false positives and so I simply worked my way through the files, manually deleting the errors. I ended up with 7,247 photos of people. That’s a strike rate of nearly 65% which seems pretty good. The classifier, which does the actual facial detection, was probably trained on conventional photographs rather than on the mixed-format documents I was feeding it.

Then it was just a matter of building a web app to display the portraits. I used Django for the backend work of managing the metadata and delivering the content, while the interface was built using a combination or Isotope, Infinite Scroll and FancyBox.

It’s important to note that the portraits provide a way of exploring the records themselves. If you click on a face you see a copy of the document from which the photo was extracted. A link is provided to examine the full context of the image in RecordSearch. This is not just an exhibition, it’s a finding aid.

What next? There are many more of these documents to be harvested and processed (and many more still yet to be digitised). I will be adding more series as I can (though I might have to wait until I can afford a new computer!). I’d also like to explore the possibilities of facial or object detection a bit more. Could I train my own classifier? Could I detect handprints, or even classify the type of form?

In the meantime, I think our experimental browser helps us to understand why the Invisible Australians project is so important — you look at their faces and you simply want to know more. Who are they? What were their lives like?

UPDATE: For more on the photos and the issues they raise, see Kate Bagnall’s posts over at the Tiger’s Mouth.

Presented at the Life of Information Symposium, 24 September 2010.
Slides are available on Slideshare.

This palm print belongs to a 12-year-old boy called Charlie Allen.

Charlie was born in Sydney in 1896.

His mother was Frances Allen (sometime sweet shop owner and brothel keeper), his father Charlie Gum (a buyer for Wing On company).

Charlie was raised by his mother, but in 1909, at the age of 13, he was taken to China by his father.

His father returned to Sydney, leaving Charlie in China. He lived with relatives in the town of Shekki (inland from Hong Kong) for 6 years.

Charlie was homesick, but had no means of getting back to Australia. His mother attempted to enlist government help but to no avail. Charlie finally returned in 1915.

The following year he enlisted in First AIF (well actually he enlisted three times, and was discharged as medically unfit each time).

Charlie married in Sydney in 1917 and had two daughters soon after. He returned to China in 1922 for 7 months.

Charlie Allen died in 1938 as the result of an industrial accident. He was 41.

How do we know all this about Charlie Allen?

We know this because there are fragments of Charlie’s life scattered throughout the holdings of the National Archives of Australia.

The CEDT from 1909 when he left Australia with his father:

Charlie’s father was Chinese, he was therefore categorised as a ‘half-caste’, as someone who was not white, and fell under the restrictions imposed by the White Australia Policy.

The certificate from 1909 granted Charlie an exemption to the Dictation Test. Without it, he may not have been allowed back into the country.

Every time one of many thousands of non-Europeans resident in Australia sought to travel overseas and return home again they needed one of these certificates.

We’re all of course familiar with the general outlines of the White Australia Policy, and the way it underpinned conceptions of Australia as a nation in the first half of the 20th century.

But what we sometimes forget is that it was also a massive bureaucratic exercise.

Forms and certificates were printed, issued, used and filed. Regulations were modified, guidelines were distributed and administering officers were managed and advised. Individual cases were reviewed, policy was changed and new forms and certificates were printed, issued, used and filed…

For example, between 1901 and 1911, 400 circulars were issued to port officers about immigration restriction. The confidential manual on immigration restriction grew from one page in 1902 to more than 200 in 1912.

Much of this system is now preserved in the National Archives.

For the years between 1902 and 1948 there remain:

• More than 50,000 CEDTs
• 90 shelf metres of records
• 15,000 case files

And within those many thousands of files are the scattered fragments of lives such as Charlie’s — lives that were controlled, monitored and documented in a vain attempt to make Australia ‘white’.

We’ve already seen today some wonderful examples of how these fragments, these slivers of existence, can be found, extracted, aggregated and displayed. But I think it’s worth considering for a moment what happens when we do this.

The historian Tim Hitchcock, behind projects such as the Old Bailey Online and London Lives, has reflected on the impact of digitisation on our access to archives. Archives, he notes, tend to reflect the assumptions and practices of the institutions that created them.

But by providing new ways into these records systems, technology can undermine the power relations that persist within their structures.

‘What changes’, asks Tim Hitchcock, ‘when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?’1

I don’t know, but I think we should find out, don’t you?

**********

I hope you’ve all collected a mini card. These themselves provide a little glimpse at the real face of White Australia and I’d invite you all to head over to the National Archives website, do battle with the monster that is RecordSearch, and look up the file references that are on each card.

The cards are part of a project that Kate Bagnall and I are trying to develop — Invisible Australians.

I should note too that the cards, and most of the examples I’m showing you here today are the product of Kate’s long and detailed research into Chinese-Australian families. In modern project management parlance, Kate is the domain expert, while I am merely the technical resource.

If we look again at one of the CEDTs, we can see that there’s a lot of useful structured data:

• name
• place of birth
• age
• height
• destination
• date of departure
• name of ship

Invisible Australians has the modest aim of extracting this data from the 50,000+ forms in the National Archives. But of course that’s just the start, because each person might have used a number of certificates — so then it’s a matter of matching these identities.

And then there are a range of other related forms, not to mention case files, alien registration documents, naturalisation applications…

Obviously we can’t do it alone. We’ll be creating a crowdsourcing tool to extract and link the data.

It’s ridiculously ambitious, totally unfunded and is likely to take over our lives.

Is it worth it?

Imagine being able to navigate the network of lives, families and relationships. To follow their journeys, to share their tragedies, to celebrate their small victories against a repressive system.

Imagine being able to watch them age.

Is it worth it? We think so.

**********

For Tim Hitchcock technology opens up the possibility of writing a new history from below, exploring how the poor, the marginalised and the powerless navigated the institutions of the modern state. But it’s not just about search engines and databases. He talks about making ‘best use of the technology of emotions and representation — how you use words and pictures and a story to impact, not just on what people think, but what they see in their mind’s eye’.2

In this project, the photos matter. I hope the irony in our project title is obvious.

This is the real face of White Australia.

The photos remind us that the project is not just about shifting data around — these are lives, these are people.

But this brings its own challenge, for if we are seeking to liberate these lives from the fragmentation and obscurity of bureaucratic systems then we should be asking what are we liberating them into?

A database?

This is not just an exercise in data creation and management. We also have to think carefully and creatively about issues of representation, access and discovery.

We have to give these lives back their freedom to associate, to have relationships, to make connections.

We need to embed these lives in a variety of contexts and combinations. To make room for serendipity, celebration, sadness, and yes, even play.

We need to bring these lives into a rich and ongoing conversation with the world.

But how?

**********

I’ve been working on a little experiment for the National Museum of Australia called The History Wall. What the History Wall does is quite simple, it pulls together data on the fly from a variety of sources including People Australia, the Australian Dictionary of Biography, the National Library’s newspapers project, historical population data from the Bureau of Statistics, photos from the Flickr accounts of the PowerHouse Museum and the National Archives, and the collection database of the National Museum itself. It chooses randomly from all this stuff, throws the results up into the air and then displays them however they happen to fall. No two views are ever quite the same.

It’s something more than a timeline. To me it’s more like a celebration of context and serendipity. There’s a richness to it, a sense of discovery and fun, but there’s also fragility — next time you look it might be gone.

It’s a bit like history itself.

It’s a bit like the world.

How do we create spaces for our data to merge and mingle? How do we encourage the development of new contexts and connections?

I think the first thing we have to do is stop thinking about databases and dictionaries, registers and encyclopaedias. Don’t get me wrong, I’m not being critical of the wonderful projects we’ve seen today. I just think we can use all this work better if we stop thinking about individual resources and start developing on a web scale, on a global scale.

Yes, we have the technology. Time today has spared you from a detailed discourse on the Semantic Web, but I do want to focus on one aspect.

You may have heard of Linked Data, it’s a set of guidelines to help you publish your data to the Semantic Web. There are only four basic principles and I’m only going to talk about one of them. It’s one of those deceptively simple things. You look at it and think, ‘yeah, ok’, but before too long it’s starting to turn your brain inside out.

Use URLs to identify things in the real world.

Yeah, ok…

You know what URLs are, web addresses, the things you type in your browser’s location field.

And hopefully you know what things in the real world are: people, places, objects, events, ideas…

Now you may have detected a problem here, because no matter how many times you click the refresh button, your web browser is not going to be able to use such a URL to magically deliver you the real world thing.

Well, unless you’re on eBay.

Fortunately, the Linked Data guidelines provide for a bit of technical trickery that allow your browser to retrieve not the real world thing, but some information about that thing — perhaps in the form of a web page.

Why bother?

Names are powerful.

We share and use names to talk about things. Computers are the same. If we use URLs to identify things in the real world, then computers can start talking about them.

We can define and explore real-world relationships in an online environment. We can create rich, meaningful linkages across databases, across disciplines, across the world.

We can start building and thinking on a web scale.

**********

Thanks to the People Australia project, I can confidently claim that this is me:

http://nla.gov.au/nla.party-479364#foaf:Person

I keep meaning to get it on a t-shirt.

The most exciting thing about People Australia is not the EAC records or the aggregation of resources — it’s the identifiers, because they enable us to say things about people anywhere on the web that computers can understand and relate back to a specific real world entity — a person.

You can start doing it now with Wragge’s Identity Browser.

This is a little tool I built using the People Australia API. It makes it easy to find identifiers for people and organisations, and it supplies you with some code that you can drop into a blog post or web page that will tell a computer that a name relates to a thing called a ‘person’ , that this person’s name has a certain standard form, and that this person can be uniquely identified by People Australia.

Even if you don’t publish a website or a blog, you can use People Australia identifiers to build semantic linkages. Wragge’s Identity Browser also creates machine tags for you. Machine tags are like normal tags but with built in semantics. When coupled with identifiers they enable you to do some pretty powerful things.

You could for example use machine tags in Flickr to tell computers that a certain photo depicts a person uniquely identified by People Australia. In fact, people have been doing just that.

The Flickr Machine Tag Challenge is a sort of scoreboard that I built to encourage people to start adding People Australia enriched machine tags to photos. More than 1200 tags have been added to over 1000 photos. Feel free to join in!

The point is that the technologies already exist to enable us to build web scale biographical resources. Not dictionaries or databases as we know them, but networks capable of constant expansion, elaboration, and cooperation.

What we need are more tools to make it simple, recipes to make it obvious, examples and applications to make it popular, and leadership to make it all seem possible.

**********

Of course most of the lives we hope to liberate through Invisible Australians will not be represented in People Australia.

Not yet.

But Invisible Australians will offer a point of aggregation and disambiguation that will enable our people to find their way from the bureaucratic recesses of the White Australia Policy to a place on the national stage.

And we will encourage others to do likewise. Basil can’t do all the work. The centralised system has to be fed through centres of aggregation and collaboration.

Similarly, there are many great resources already out there relating to Chinese-Australians. There are hordes of family and local historians compiling and publishing biographical data. We want to identify people in these resources and link to them.

We want to publish up to People Australia and link down to a single headstone in a lonely country cemetery.

But to do this we need to help people make their resources linkable. To help them create persistent, re-usable URLs, and expose their data in standard formats. To create Linked Data, even if they have no particular interest in the Semantic Web.

Invisible Australians is not just about extracting data from archives. It’s also about working with others to build capacities and demonstrate possibilities.

It’s ridiculously ambitious, totally unfunded and is likely to take over our lives.

Is it worth it?

We think so.

1. Tim Hitchcock, ‘Digital searching and the re-formulation of historical knowledge’, in Mark Greengrass and Lorna Hughes (eds),The Virtual Representation of the Past, Ashgate, Farnham, UK, 2008, pp. 81-90.
2. http://historyonics.blogspot.com/2010/04/new-history-from-below.html

About a month ago I posted a copy of my report Emerging technologies for the provision of access to archives on Scribd. It’s already edging up towards a thousand reads, so I thought it was time I put a link in from here.

The basic message is we need to experiment and find the spaces both within and between our institutions to foster such experimentation. Is that asking too much? Anyway… read, enjoy, use!