Time to take stock of the year that was…

2017 certainly had its ups and downs. In the middle of the year, my university’s management decided that they could do without undergraduate teaching in cultural heritage. We managed to hold on, but it’s the sort of thing that makes you wonder why you bother.

On the flipside, the highlight of the year was working with my undergraduate class in Exploring Digital Heritage to create and promote a site for the transcription of records used in the administration of the White Australia Policy. It’s something Kate and I have been talking about for many years, so it was a rather emotional experience to see it finally happen. But it wasn’t just about the website — it was about the way my students responded to the records (many of them knew little about the White Australia Policy before we started); it was about seeing the records on the ABC news; and it was about seeing the faces of the people who lived under the weight of the White Australia Policy projected onto the walls of Old Parliament House. That’s why we bother.

Also in 2017 (and somewhat related to the above) I decided to adjust my working arrangements in attempt to find some balance. You can now find me over on Patreon.

The making

• Closed Access 2017 update! Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 9 January 2017.
• Wrestled with RecordSearch to extract summaries for all series. Along the way I found out some interesting stuff about the use of functions. Putting the two together, I tried to build a picture of what NAA holdings look like through the perspective of government functions.
• Played around with Twarc to create a collection of #invasionday tweets. Later in the year I captured and shared tweets with the tag #australianvalues.
• Have you ever wondered how the cut and thrust of parliaments past might translate to the world of social media? Wonder no longer, for here you can explore interjections in the Australian parliament from 1901 to 1980, reimagined as tweets. You might even find some emoji…
• Code and work-in-progress website exploring the language of Hansard.
• Went back to basics with LODBook — now developing a series of components using Jekyll and javascript.
• The Redaction Zoo — this collection of creatures was discovered amidst thousands of ASIO surveillance files held by the National Archives of Australia. While the practice of redaction is intended to withhold information from public view, an unknown archivist has used redactions to add an artistic flourish to the files. They are reminders that the processes that limit our access to information are human in their operation and design. There is nothing magical about the ‘secrets’ preserved in government archives. Video first exhibited as part of ‘Beauties and Beasts’, Belconnen Arts Centre, 6-28 May 2017.
• Here’s a repository of images in JPG and SVG format drawn from a collection of #redactionart discovered in ASIO surveillance files held by the National Archives of Australia. Use them to create your own #redactionart projects!
• I finally added a decent full-text search facility to Historic Hansard. Yay! Search for speeches or bills. Filter by date, house, and speaker. There’s even an option to save your complete result set as a CSV file for further analysis and exploration.
• The Tribune was a newspaper published by the Communist Party of Australia. The State Library of NSW holds more than 60,000 negatives and photos from the Tribune which document a wide range of political events and social issues from 1964 to 1991. This is a work-in-progress site documenting my exploration of the negatives as a DXLab Digital Drop-in.
• Help transcribe records that document the lives of ordinary people living under the restrictions of the White Australia Policy. The Real Face of White Australia site was built using the Scribe Framework with the help of students from my Exploring Digital Heritage class.
• Explore regular updates of data generated through the Real Face of White Australia project. Includes CSV files with transcribed fields, as well as photos and handprints.
• There’s lots of exciting new digitised content being added to Trove’s journals zone, but it’s not always easy to find and search. I made an app that lists journals that have been digitised by the NLA and have searchable records for individual articles. This means you can search inside the journal, just like you do in the newspapers zone. The code for the harvester and web app is on GitHub.
• Demonstrated how to harvest Parliamentary press releases from Trove. The repository includes a sample dataset of politicians talking about refugees.
• Kicked off my new 101 Digital Heritage Hacks site with a userscript to add a ‘copy permalink’ button to Trove.
• I made a little bot to post stories from the Real Face of White Australia data to the @invisibleaus Twitter account.
• Created two new DIY Trove bots using Glitch. The Trove Tag Bot tweets items with a specific tag, while the Trove List Bot tweets items from one or more lists. Both bots come with detailed instructions and, because they’re hosted on Glitch, it’s easy for anyone to modify the code and launch their own bots.
• Added a battle mode to Headline Roulette. It’s really just a way of generating a url to specific newspaper article in Trove. This means groups can compete on a level playing field, but it could have other uses. You could create a curated set of challenges for students, or use them in something like a scavenger hunt.

The talking

• 13 February 2017 – Digital humanities workshop at ALIAOnline.
• 24 February 2017 – Keynote presentation at HDR Summer School, Deakin University, The practice of play, <https://doi.org/10.6084/m9.figshare.4696258>
• 1-3 March 2017, attended ‘Always Already Computational: Library Collections as Data’ national forum in Santa Barbara. Read the position papers and the final statement.
• 20 April 2017 – Public lecture and workshop, University of Wollongong.
• 4 May 2017 – Presentation with Professor David Lowe, ‘The Political Language of the 1970s in Australia’, at The 1970s: Australian and Indian Perspectives on a Decade of Transition, Deakin University.
• 23 June 2017 – Workshop, Random acts of meaning: Digital skills for a post-truth world, NLS8.
• 4 August 2017, ‘The struggle for access’, presentation at DHPathways 2017, Canberra.
• 10 August 2017 — ‘Doing DH’ presentation for ACT teacher librarians.
• 13 October 2017 — ‘Trove tips and tricks’, at the Deniliquin Family History Expo.
• 8 November 2017 — ‘Making this happen’, presentation at Collaborating around Collections, ANU.
• 20 November 2017 – ‘”The badge of the outsider”: Open Access and Closed Boundaries’, invited presentation at Sharing is Caring 2017, Aarhus. Also put together a last minute ‘Hack the world’ workshop.
• 4 December 2017 — Digital methods workshop for the UTS Legal History group

Contribution to a panel discussion on ‘Access and Innovation’ at Digital Directions 2016, Canberra, 10 November 2016.
View the slides.

Please cite as: Tim Sherratt, ‘Caring about access’, presented at the Digital Directions, 2016. <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>

You might have seen that the Department of Prime Minister and Cabinet has opened discussion on an Open Government National Action Plan. Last week I was watching the Twitter stream from a briefing event in Melbourne, when I saw this tweet from Asher Wolf quoting the PM&C spokesperson:

. @pmc_gov_au: we don’t want ppl to search for flaws or try to crack gvt datasets. #OGPAU

Now I’m not sure of the context of this statement, but it is a reminder that we can’t take the meaning of words like ‘open’ or ‘access’ for granted.

They are what we make of them.

I’m a historian and hacker. I don’t steal credit card details, I use digital tools to open cultural heritage collections – to see them differently, to feel differently.

Earlier this year I wrote a script to harvest many gigabytes of parliamentary proceedings – Hansard – from the ParlInfo database maintained by the Department of Parliamentary Services. I shared the files, covering the period from 1901 to 1980, through my GitHub repository. My plan was simply to make this rich source of political history easily available to anyone who wanted to explore new forms of digital analysis.

In the end I created my own version of Historic Hansard, one that privileges the experience of reading rather than searching. I also added a few nifty features such as indexes of legislation and people, and integration with text analysis and annotation tools.

But in the process of harvesting the data I noticed that around 100 sitting days were missing from ParlInfo. The files were empty – unable to be searched. Given an average of about 50 sitting days a year, that’s about two years worth that were simply invisible.

Thanks to the Parliamentary Library these problems have now been fixed. But this experience highlighted two issues. First, search interfaces lie. We have to develop our critical capacities to be more aware of what we are not being shown and why. And second, this problem only became visible because I was hacking their database, because I went beyond what was offered by their website, because I wanted more. Unlike the PM&C spokesperson I think one of strongest reasons for opening data is to encourage people to find its flaws. This is particularly the case for cultural heritage data where the processes of selection, representation, normalisation, control, and preservation all have an impact of the types of stories we can tell about ourselves.

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

A few years ago Merete Sanderhoff, from the National Gallery of Denmark, compiled a collection of essays on opening cultural heritage collections for reuse online. It was called Sharing is Caring. I think it’s worth reflecting on the dual meanings of ‘caring’ – both looking after, and giving a shit. Sharing helps us foster communities and preserve collections. But it also matters, it has an impact, it can change the world.

Bethany Nowviskie, the Director of the Digital Library Federation, recently gave a talk on ‘speculative collections’ asking how we shift the temporal orientation of our libraries and archives away from a closed and linear past towards an exploration of what might be.

Seeing differently. Feeling differently.

This is a visualisation created by Geoff Hinchcliffe showing more than 12,000 tweets since February this year using the hashtag #fundTrove. Trove has radically changed our access to the past. It’s not perfect, it’s not everything, but it changes lives. And yet the government seems to think it can be left to wither, and nobody will really care. I care. Do you? Then what do we do about it?

I also care deeply about the collections of the National Archives – a love expressed through many painful hours spent hacking RecordSearch, their online database. Last year I harvested details of more than 12,000 publicly available ASIO files – you can grab the results from my GitHub site – and downloaded about 300,000 page images. These are mostly surveillance files, documenting the lives of writers, academics, unionists, Indigenous activists and others – identified as potential threats to the nation.

This year I wrote a script to search those images for redactions – sections of text blacked out for security reasons. If you’d like to explore the 239,000 redactions I’ve extracted, you can – marvel at their blackness, their very lack of information. But here the redactions are not dead ends, they’re starting points, ways of exploring the workings of ASIO. As the power and reach of state surveillance continues to expand, we can find creative ways to reverse the panoptic gaze.

We don’t have to accept what we’re given. We can take collections and turn them inside out.

The National Archives also preserves remnants of Australia’s racist past. There are many thousands of records documenting the workings of the White Australia Policy. They include special certificates, with portrait photos and handprints, which were needed if you were deemed non-white, travelled overseas, and simply wanted to come home.

About five years ago I harvested about 12,000 images of these certificates from RecordSearch and ran them through a facial detection script. In a little over a weekend I’d created the Real Face of White Australia – a seemingly endless scrolling wall of more than 7,000 faces. It’s compelling. It’s uncomfortable. But this is who we are.

As I saw the One Nation senators cracking open the champagne outside Parliament House yesterday, I thought again of how important its is the tell these stories, to share these collections. To give a shit.

Innovation can be measured in more than shiny apps, or cool new visualisations. By struggling with access to our past we can imagine new futures.

Presented at the DigitalGLAM Symposium, University of Melbourne, 15 July 2016.
Play with the slides.

Who’s used Commonwealth Hansard through the ParlInfo Database — particularly the early stuff, pre 1980?

Was it fun?

It’s great that all of that content has been digitised, but the user interface is not very friendly — one of the main problems is that it’s just really hard to read anything in context, to get a sense of proceedings in parliament as they unfolded.

However, you can bypass the interface and go straight to the data. There’s an XML file — with the proceedings in a nicely-structured, machine-readable form — for each day.

To save the world much searching and clicking, I wrote a script to crawl through ParlInfo and download all of those XML files. There’s about 4gb in total, and I’ve shared them all through my GitHub account. Feel free to download and explore!

I initially harvested the files because I thought they’d provide a useful example for use in teaching about text analysis tools and techniques. But then I reflected on that ‘readability’ question and wondered how much work would be involved in creating a version of Hansard that was focused on browsing and reading. The answer was about two days — over the space of a weekend I created Historic Hansard.

I wanted to mention this today because I think it’s a good example of a ‘hack’. Contrary to mainstream media coverage, most hackers don’t want to steal your credit card details. Hackers, within the diverse community of coders, builders, programmers, and developers, are people who use code to solve problems, to get around obstacles — their solutions might not be the most elegant, but they work.

My bio reads ‘historian and hacker’. I’m a historian who uses digital tools and techniques to get around obstacles. In particular, I use code to open digital cultural collections.

In the case of Hansard I have, for minimal cost, created something that allows you to explore the proceedings of parliament in context. It’s made some historians pretty excited. Since that first weekend I’ve added an index to people and bills. I’ve also integrated tools for text analysis and annotation. Shortly I’ll add the rest of the House of Reps and the Senate through until 1980. I was very pleased that the new PMs site from the Museum of Australian Democracy was able to harvest from Historic Hansard to boost their content.

So this is an example of what happens when governments or cultural heritage organisations provide openly-licensed data for people to play with — although I should note that the contents of the parliament website, including Hansard, have a ‘no derivatives’ licence which creates a lot of uncertainty around what you can actually do.

I’ve talked a lot in recent years about building stuff with cultural heritage data — particularly in relation to Trove. And I’ve created a lot of tools, experiments and interfaces — some significant, some silly, some downright creepy.

But while open data does enable mildly-obsessed hackers like me to build new, shiny things, it also enables us to ask different questions — to analyse what’s not there, as well as what is.

In the case of Commonwealth Hansard what’s not there is about two years’ worth of Senate proceedings. I stumbled on this while harvesting the XML files and started doing some analysis. Between 1901 and 1980, 94 days of Senate proceedings, and 8 days of the House of Representatives are effectively invisible. PDFs exist, but the XML files are empty and the content is not searchable.

As you can see, most of the ‘invisible’ days come from the WWI period. So if you’ve been relying on the online version of Hansard to research WWI, you probably need to think about what you might have missed.

Needless to say, it’s unlikely that anyone would have noticed these gaps using ParlInfo’s web interface. It’s yet another example of why you always be suspicious of search engines. But it’s also an example of how hacking collections opens them to new forms of analysis and enables us to critique the very notion of access.

Spend sometime reading the strategic plans of cultural heritage organisations and you’ll see that ‘access’ figures prominently — access is ‘provided’, ‘given’, and ‘opened’. Online access is a ‘deliverable’ to be measured in hits and sessions.

The problem with this is it casts the public as consumers of access, rather than creators. Hacking heritage collections is one way demonstrating that access is not a one way flow of information — it’s a struggle and it must always be. We should always be uncomfortable with the categories used to structure our past. We should always want more. If we run out of questions and criticisms then something’s gone badly wrong.

Who’s used the National Archives of Australia?

You’ve probably come across the process of ‘access examination’. Records more than 20 years old are assessed against a list of criteria to see if they can be released to the public — the criteria are defined under the Archives Act and relate to things like national security and privacy.

Most records end up with an access status of ‘open’, but some end up ‘closed’ and are withheld from the public. Last year I started wondering what happens when we flip around the idea of ‘access’ and focus on the files we’re not allowed to see?

The National Archives’ online database, RecordSearch, provides basic metadata about ‘closed’ files, including the reasons why they’ve been withheld. But you can’t easily search or analyse this data within RecordSearch itself. So I wrote another script that harvested details of all 14000 files with the access status of closed and created my own interface to them — Closed Access.

While my Hansard hacking was aimed at opening up new forms of use, Closed Access is about understanding the processes and practices that determine whether we’re allowed to see a file. It pushes beyond legislative definitions in an attempt to reveal ‘access’ as a process that is historically contingent — there is nothing magical, mysterious, or dangerous about a ‘closed’ file.

By hacking collections we can also see them differently. Amongst its millions of records the National Archives holds the remnants of the White Australia Policy. About 5 years ago I wrote a script to harvest several thousands of these documents. I ran them through a facial detection script, cropped out the portraits, and created this seemingly endless wall of faces.

It was another weekend project, another hack. In this case it exposed the remnants of a racist bureaucratic system in a completely different way. Instead of databases, record series, and files, you could see the people inside.

Of course the danger in raving about the wonders of hacking is that we simply create new structures of power, new limits on participation based around coding ability. I don’t believe in hacker elites liberating the masses from ignorance. So part of the struggle around access must also be the struggle to share — to share technology, code, examples, tools, data, knowledge, possibilities, and passions. To give people the opportunity to participate and not just consume.

You might think sharing is easy, but it’s fucking hard work — both for institutions and individuals. And it’s work that is rarely rewarded in terms of professional acknowledgement, advancement or funding.

But I’m inspired by open access activists around the world, by radical librarians setting up Tor exit relays, by digital humanists critically examining unequal access to technology, and exploring the possibilities of minimal computing. By all the people who freely share their work and ideas to make ‘access’ just that little bit easier for others.

I share because I think it’s important, because it’s part of the obligation of being a hacker and historian, academic and activist.

So I share things like:

• articles and examples on my blog
• code and data on GitHub
• current research notes and hacks in my research notebook
• documentation, activities, and tutorials in my digital heritage handbook
• web apps and sites all over the place

I’m also trying to develop a few DIY, low code examples that enable people to build things using Trove data and some freely available tools. I’m hoping that’ll give them the confidence to explore further. A starting point perhaps…

Just last week I ran a free digital heritage workshop in the lead up to GovHack — the national, open data competition. We’re hosting a specially-themed ‘Heritage Hack’ node at the University of Canberra as part of the GovHack this year. Feel free to join us!

Not that any of this is particularly special or remarkable. It should just be what we do — as researchers, as cultural institutions, as citizens, as hackers. The main thing we need to share is, of course, our power.

So let’s just just stop talking about access to cultural heritage as if it’s a destination, and explore it as journey full of tensions, dangers and difficulties; but also full of joy, discovery, and meaning. Let’s hack heritage to build critiques as well as websites, conversations as well as code.