Time to take stock of the year that was…

2017 certainly had its ups and downs. In the middle of the year, my university’s management decided that they could do without undergraduate teaching in cultural heritage. We managed to hold on, but it’s the sort of thing that makes you wonder why you bother.

On the flipside, the highlight of the year was working with my undergraduate class in Exploring Digital Heritage to create and promote a site for the transcription of records used in the administration of the White Australia Policy. It’s something Kate and I have been talking about for many years, so it was a rather emotional experience to see it finally happen. But it wasn’t just about the website — it was about the way my students responded to the records (many of them knew little about the White Australia Policy before we started); it was about seeing the records on the ABC news; and it was about seeing the faces of the people who lived under the weight of the White Australia Policy projected onto the walls of Old Parliament House. That’s why we bother.

Also in 2017 (and somewhat related to the above) I decided to adjust my working arrangements in attempt to find some balance. You can now find me over on Patreon.

The making

• Closed Access 2017 update! Complete dataset of records held by the National Archives of Australia that had the access status of ‘closed’ (withheld from public access) on 9 January 2017.
• Wrestled with RecordSearch to extract summaries for all series. Along the way I found out some interesting stuff about the use of functions. Putting the two together, I tried to build a picture of what NAA holdings look like through the perspective of government functions.
• Played around with Twarc to create a collection of #invasionday tweets. Later in the year I captured and shared tweets with the tag #australianvalues.
• Have you ever wondered how the cut and thrust of parliaments past might translate to the world of social media? Wonder no longer, for here you can explore interjections in the Australian parliament from 1901 to 1980, reimagined as tweets. You might even find some emoji…
• Code and work-in-progress website exploring the language of Hansard.
• Went back to basics with LODBook — now developing a series of components using Jekyll and javascript.
• The Redaction Zoo — this collection of creatures was discovered amidst thousands of ASIO surveillance files held by the National Archives of Australia. While the practice of redaction is intended to withhold information from public view, an unknown archivist has used redactions to add an artistic flourish to the files. They are reminders that the processes that limit our access to information are human in their operation and design. There is nothing magical about the ‘secrets’ preserved in government archives. Video first exhibited as part of ‘Beauties and Beasts’, Belconnen Arts Centre, 6-28 May 2017.
• Here’s a repository of images in JPG and SVG format drawn from a collection of #redactionart discovered in ASIO surveillance files held by the National Archives of Australia. Use them to create your own #redactionart projects!
• I finally added a decent full-text search facility to Historic Hansard. Yay! Search for speeches or bills. Filter by date, house, and speaker. There’s even an option to save your complete result set as a CSV file for further analysis and exploration.
• The Tribune was a newspaper published by the Communist Party of Australia. The State Library of NSW holds more than 60,000 negatives and photos from the Tribune which document a wide range of political events and social issues from 1964 to 1991. This is a work-in-progress site documenting my exploration of the negatives as a DXLab Digital Drop-in.
• Help transcribe records that document the lives of ordinary people living under the restrictions of the White Australia Policy. The Real Face of White Australia site was built using the Scribe Framework with the help of students from my Exploring Digital Heritage class.
• Explore regular updates of data generated through the Real Face of White Australia project. Includes CSV files with transcribed fields, as well as photos and handprints.
• There’s lots of exciting new digitised content being added to Trove’s journals zone, but it’s not always easy to find and search. I made an app that lists journals that have been digitised by the NLA and have searchable records for individual articles. This means you can search inside the journal, just like you do in the newspapers zone. The code for the harvester and web app is on GitHub.
• Demonstrated how to harvest Parliamentary press releases from Trove. The repository includes a sample dataset of politicians talking about refugees.
• Kicked off my new 101 Digital Heritage Hacks site with a userscript to add a ‘copy permalink’ button to Trove.
• I made a little bot to post stories from the Real Face of White Australia data to the @invisibleaus Twitter account.
• Created two new DIY Trove bots using Glitch. The Trove Tag Bot tweets items with a specific tag, while the Trove List Bot tweets items from one or more lists. Both bots come with detailed instructions and, because they’re hosted on Glitch, it’s easy for anyone to modify the code and launch their own bots.
• Added a battle mode to Headline Roulette. It’s really just a way of generating a url to specific newspaper article in Trove. This means groups can compete on a level playing field, but it could have other uses. You could create a curated set of challenges for students, or use them in something like a scavenger hunt.

The talking

• 13 February 2017 – Digital humanities workshop at ALIAOnline.
• 24 February 2017 – Keynote presentation at HDR Summer School, Deakin University, The practice of play, <https://doi.org/10.6084/m9.figshare.4696258>
• 1-3 March 2017, attended ‘Always Already Computational: Library Collections as Data’ national forum in Santa Barbara. Read the position papers and the final statement.
• 20 April 2017 – Public lecture and workshop, University of Wollongong.
• 4 May 2017 – Presentation with Professor David Lowe, ‘The Political Language of the 1970s in Australia’, at The 1970s: Australian and Indian Perspectives on a Decade of Transition, Deakin University.
• 23 June 2017 – Workshop, Random acts of meaning: Digital skills for a post-truth world, NLS8.
• 4 August 2017, ‘The struggle for access’, presentation at DHPathways 2017, Canberra.
• 10 August 2017 — ‘Doing DH’ presentation for ACT teacher librarians.
• 13 October 2017 — ‘Trove tips and tricks’, at the Deniliquin Family History Expo.
• 8 November 2017 — ‘Making this happen’, presentation at Collaborating around Collections, ANU.
• 20 November 2017 – ‘”The badge of the outsider”: Open Access and Closed Boundaries’, invited presentation at Sharing is Caring 2017, Aarhus. Also put together a last minute ‘Hack the world’ workshop.
• 4 December 2017 — Digital methods workshop for the UTS Legal History group

Contribution to a panel discussion on ‘Access and Innovation’ at Digital Directions 2016, Canberra, 10 November 2016.
View the slides.

Please cite as: Tim Sherratt, ‘Caring about access’, presented at the Digital Directions, 2016. <https://dx.doi.org/10.6084/m9.figshare.4229402.v1>

You might have seen that the Department of Prime Minister and Cabinet has opened discussion on an Open Government National Action Plan. Last week I was watching the Twitter stream from a briefing event in Melbourne, when I saw this tweet from Asher Wolf quoting the PM&C spokesperson:

. @pmc_gov_au: we don’t want ppl to search for flaws or try to crack gvt datasets. #OGPAU

Now I’m not sure of the context of this statement, but it is a reminder that we can’t take the meaning of words like ‘open’ or ‘access’ for granted.

They are what we make of them.

I’m a historian and hacker. I don’t steal credit card details, I use digital tools to open cultural heritage collections – to see them differently, to feel differently.

Earlier this year I wrote a script to harvest many gigabytes of parliamentary proceedings – Hansard – from the ParlInfo database maintained by the Department of Parliamentary Services. I shared the files, covering the period from 1901 to 1980, through my GitHub repository. My plan was simply to make this rich source of political history easily available to anyone who wanted to explore new forms of digital analysis.

In the end I created my own version of Historic Hansard, one that privileges the experience of reading rather than searching. I also added a few nifty features such as indexes of legislation and people, and integration with text analysis and annotation tools.

But in the process of harvesting the data I noticed that around 100 sitting days were missing from ParlInfo. The files were empty – unable to be searched. Given an average of about 50 sitting days a year, that’s about two years worth that were simply invisible.

Thanks to the Parliamentary Library these problems have now been fixed. But this experience highlighted two issues. First, search interfaces lie. We have to develop our critical capacities to be more aware of what we are not being shown and why. And second, this problem only became visible because I was hacking their database, because I went beyond what was offered by their website, because I wanted more. Unlike the PM&C spokesperson I think one of strongest reasons for opening data is to encourage people to find its flaws. This is particularly the case for cultural heritage data where the processes of selection, representation, normalisation, control, and preservation all have an impact of the types of stories we can tell about ourselves.

Access is not something that cultural institutions bestow on a grateful public. It’s a struggle for understanding and meaning. Expect to be criticised, expect problems to be found, expect your prejudices to be exposed. That’s the point.

If cultural institutions want to celebrate their website hits, celebrity visits, or their latest glossy magazines – well that’s just fabulous. But I’d like them to celebrate every flaw that’s found in their data, every gap identified in their collection – that’s engagement, that’s access. We need to get beyond defensive posturing and embrace the risky, exciting possibilities that come from critical engagement with collection data – recognising hacking as a way of knowing.

In this new post-truth world it’s going to be more important than ever to challenge what is given, what is ‘natural’, what is ‘inevitable’. Our cultural heritage will be a crucially important resource to be mobilised in defence of complexity, nuance, and doubt – the rich and glorious reality of simply being human.

A few years ago Merete Sanderhoff, from the National Gallery of Denmark, compiled a collection of essays on opening cultural heritage collections for reuse online. It was called Sharing is Caring. I think it’s worth reflecting on the dual meanings of ‘caring’ – both looking after, and giving a shit. Sharing helps us foster communities and preserve collections. But it also matters, it has an impact, it can change the world.

Bethany Nowviskie, the Director of the Digital Library Federation, recently gave a talk on ‘speculative collections’ asking how we shift the temporal orientation of our libraries and archives away from a closed and linear past towards an exploration of what might be.

Seeing differently. Feeling differently.

This is a visualisation created by Geoff Hinchcliffe showing more than 12,000 tweets since February this year using the hashtag #fundTrove. Trove has radically changed our access to the past. It’s not perfect, it’s not everything, but it changes lives. And yet the government seems to think it can be left to wither, and nobody will really care. I care. Do you? Then what do we do about it?

I also care deeply about the collections of the National Archives – a love expressed through many painful hours spent hacking RecordSearch, their online database. Last year I harvested details of more than 12,000 publicly available ASIO files – you can grab the results from my GitHub site – and downloaded about 300,000 page images. These are mostly surveillance files, documenting the lives of writers, academics, unionists, Indigenous activists and others – identified as potential threats to the nation.

This year I wrote a script to search those images for redactions – sections of text blacked out for security reasons. If you’d like to explore the 239,000 redactions I’ve extracted, you can – marvel at their blackness, their very lack of information. But here the redactions are not dead ends, they’re starting points, ways of exploring the workings of ASIO. As the power and reach of state surveillance continues to expand, we can find creative ways to reverse the panoptic gaze.

We don’t have to accept what we’re given. We can take collections and turn them inside out.

The National Archives also preserves remnants of Australia’s racist past. There are many thousands of records documenting the workings of the White Australia Policy. They include special certificates, with portrait photos and handprints, which were needed if you were deemed non-white, travelled overseas, and simply wanted to come home.

About five years ago I harvested about 12,000 images of these certificates from RecordSearch and ran them through a facial detection script. In a little over a weekend I’d created the Real Face of White Australia – a seemingly endless scrolling wall of more than 7,000 faces. It’s compelling. It’s uncomfortable. But this is who we are.

As I saw the One Nation senators cracking open the champagne outside Parliament House yesterday, I thought again of how important its is the tell these stories, to share these collections. To give a shit.

Innovation can be measured in more than shiny apps, or cool new visualisations. By struggling with access to our past we can imagine new futures.

Presented at the DigitalGLAM Symposium, University of Melbourne, 15 July 2016.
Play with the slides.

Who’s used Commonwealth Hansard through the ParlInfo Database — particularly the early stuff, pre 1980?

Was it fun?

It’s great that all of that content has been digitised, but the user interface is not very friendly — one of the main problems is that it’s just really hard to read anything in context, to get a sense of proceedings in parliament as they unfolded.

However, you can bypass the interface and go straight to the data. There’s an XML file — with the proceedings in a nicely-structured, machine-readable form — for each day.

To save the world much searching and clicking, I wrote a script to crawl through ParlInfo and download all of those XML files. There’s about 4gb in total, and I’ve shared them all through my GitHub account. Feel free to download and explore!

I initially harvested the files because I thought they’d provide a useful example for use in teaching about text analysis tools and techniques. But then I reflected on that ‘readability’ question and wondered how much work would be involved in creating a version of Hansard that was focused on browsing and reading. The answer was about two days — over the space of a weekend I created Historic Hansard.

I wanted to mention this today because I think it’s a good example of a ‘hack’. Contrary to mainstream media coverage, most hackers don’t want to steal your credit card details. Hackers, within the diverse community of coders, builders, programmers, and developers, are people who use code to solve problems, to get around obstacles — their solutions might not be the most elegant, but they work.

My bio reads ‘historian and hacker’. I’m a historian who uses digital tools and techniques to get around obstacles. In particular, I use code to open digital cultural collections.

In the case of Hansard I have, for minimal cost, created something that allows you to explore the proceedings of parliament in context. It’s made some historians pretty excited. Since that first weekend I’ve added an index to people and bills. I’ve also integrated tools for text analysis and annotation. Shortly I’ll add the rest of the House of Reps and the Senate through until 1980. I was very pleased that the new PMs site from the Museum of Australian Democracy was able to harvest from Historic Hansard to boost their content.

So this is an example of what happens when governments or cultural heritage organisations provide openly-licensed data for people to play with — although I should note that the contents of the parliament website, including Hansard, have a ‘no derivatives’ licence which creates a lot of uncertainty around what you can actually do.

I’ve talked a lot in recent years about building stuff with cultural heritage data — particularly in relation to Trove. And I’ve created a lot of tools, experiments and interfaces — some significant, some silly, some downright creepy.

But while open data does enable mildly-obsessed hackers like me to build new, shiny things, it also enables us to ask different questions — to analyse what’s not there, as well as what is.

In the case of Commonwealth Hansard what’s not there is about two years’ worth of Senate proceedings. I stumbled on this while harvesting the XML files and started doing some analysis. Between 1901 and 1980, 94 days of Senate proceedings, and 8 days of the House of Representatives are effectively invisible. PDFs exist, but the XML files are empty and the content is not searchable.

As you can see, most of the ‘invisible’ days come from the WWI period. So if you’ve been relying on the online version of Hansard to research WWI, you probably need to think about what you might have missed.

Needless to say, it’s unlikely that anyone would have noticed these gaps using ParlInfo’s web interface. It’s yet another example of why you always be suspicious of search engines. But it’s also an example of how hacking collections opens them to new forms of analysis and enables us to critique the very notion of access.

Spend sometime reading the strategic plans of cultural heritage organisations and you’ll see that ‘access’ figures prominently — access is ‘provided’, ‘given’, and ‘opened’. Online access is a ‘deliverable’ to be measured in hits and sessions.

The problem with this is it casts the public as consumers of access, rather than creators. Hacking heritage collections is one way demonstrating that access is not a one way flow of information — it’s a struggle and it must always be. We should always be uncomfortable with the categories used to structure our past. We should always want more. If we run out of questions and criticisms then something’s gone badly wrong.

Who’s used the National Archives of Australia?

You’ve probably come across the process of ‘access examination’. Records more than 20 years old are assessed against a list of criteria to see if they can be released to the public — the criteria are defined under the Archives Act and relate to things like national security and privacy.

Most records end up with an access status of ‘open’, but some end up ‘closed’ and are withheld from the public. Last year I started wondering what happens when we flip around the idea of ‘access’ and focus on the files we’re not allowed to see?

The National Archives’ online database, RecordSearch, provides basic metadata about ‘closed’ files, including the reasons why they’ve been withheld. But you can’t easily search or analyse this data within RecordSearch itself. So I wrote another script that harvested details of all 14000 files with the access status of closed and created my own interface to them — Closed Access.

While my Hansard hacking was aimed at opening up new forms of use, Closed Access is about understanding the processes and practices that determine whether we’re allowed to see a file. It pushes beyond legislative definitions in an attempt to reveal ‘access’ as a process that is historically contingent — there is nothing magical, mysterious, or dangerous about a ‘closed’ file.

By hacking collections we can also see them differently. Amongst its millions of records the National Archives holds the remnants of the White Australia Policy. About 5 years ago I wrote a script to harvest several thousands of these documents. I ran them through a facial detection script, cropped out the portraits, and created this seemingly endless wall of faces.

It was another weekend project, another hack. In this case it exposed the remnants of a racist bureaucratic system in a completely different way. Instead of databases, record series, and files, you could see the people inside.

Of course the danger in raving about the wonders of hacking is that we simply create new structures of power, new limits on participation based around coding ability. I don’t believe in hacker elites liberating the masses from ignorance. So part of the struggle around access must also be the struggle to share — to share technology, code, examples, tools, data, knowledge, possibilities, and passions. To give people the opportunity to participate and not just consume.

You might think sharing is easy, but it’s fucking hard work — both for institutions and individuals. And it’s work that is rarely rewarded in terms of professional acknowledgement, advancement or funding.

But I’m inspired by open access activists around the world, by radical librarians setting up Tor exit relays, by digital humanists critically examining unequal access to technology, and exploring the possibilities of minimal computing. By all the people who freely share their work and ideas to make ‘access’ just that little bit easier for others.

I share because I think it’s important, because it’s part of the obligation of being a hacker and historian, academic and activist.

So I share things like:

• articles and examples on my blog
• code and data on GitHub
• current research notes and hacks in my research notebook
• documentation, activities, and tutorials in my digital heritage handbook
• web apps and sites all over the place

I’m also trying to develop a few DIY, low code examples that enable people to build things using Trove data and some freely available tools. I’m hoping that’ll give them the confidence to explore further. A starting point perhaps…

Just last week I ran a free digital heritage workshop in the lead up to GovHack — the national, open data competition. We’re hosting a specially-themed ‘Heritage Hack’ node at the University of Canberra as part of the GovHack this year. Feel free to join us!

Not that any of this is particularly special or remarkable. It should just be what we do — as researchers, as cultural institutions, as citizens, as hackers. The main thing we need to share is, of course, our power.

So let’s just just stop talking about access to cultural heritage as if it’s a destination, and explore it as journey full of tensions, dangers and difficulties; but also full of joy, discovery, and meaning. Let’s hack heritage to build critiques as well as websites, conversations as well as code.

What are the functions of government? You’ll find one possible answer in the Australian Governments’ Interactive Functions Thesaurus (AGIFT) maintained by the National Archives of Australia. AGIFT sets out a standard series of terms to describe common business activities across the different levels of government.

Functions are important to the National Archives — they aid discovery, inform appraisal, and guide government agencies in understanding what records they should keep keep. Functions also offer researchers an alternative way of exploring the Archives’ rich collections — instead of blindly searching for keywords, you can track the activities of government agencies through time.

RecordSearch, the National Archives’ online database, provides a simple way of browsing by function, though it’s not very obvious. Click on the ‘Advanced Search’ tab in RecordSearch and you’ll see an interactive diagram representing the way that records are described and arranged using the Commonwealth Records Series (CRS) system. I made this little widget a very long time ago, and there’s an alternative version, with links to the CRS manual, over at WraggeLabs.

You’ll notice that functions are associated with agencies — they’re a way of describing what government departments do. Click on the ‘Functions’ button to open up a very basic browse interface that lists preferred and alternative names for functions. There seem to be a few differences between AGIFT and the terms used in RecordSearch, I’m not sure why. The CRS Manual includes a summary of the thesaurus used in RecordSearch, and for convenience I’ve extracted a flat list of all the functions.

Just click on any function in RecordSearch to find agencies associated with it. It’s basic, but it does at least provide an alternative to the tyranny of the search box, so well described by Mitchell Whitelaw. If you’re starting a research project and don’t know what keywords are relevant, perhaps following functions will help you map the territory.

Functions also provide a way of exploring how the organisation of government changes to meet social and political needs. How are the government’s priorities in areas such as ‘migration’, ‘education’ or ‘security’ reflected in the sorts of bureaucratic structures they create? To answer this type of question you really need to extract all the data related to a function for further analysis and manipulation, but that’s not something you can easily do through RecordSearch’s web interface. Time for some hacking…

Harvesting data from RecordSearch has been something of an obsession over the past five or six years. Over that time I’ve developed a set of tools that pull structured data out of the web interface. I’ve used these tools to expose records relating to the White Australia Policy, to grab a copy of every publicly available ASIO file, and to analyse records that remain closed to public access. But to harvest functions I needed to extend my toolkit — improving the handling of agency data, and adding a simple search module to find all agencies associated with a function. Once this was done I could build a harvester — a script that just makes a series of requests to RecordSearch until all of the desired records have been retrieved, processed and stored.

All the code for my toolkit and harvester are freely available on GitHub — please download, use and improve! I’ve tried to provide fairly detailed instructions with the harvester, but you will need to fiddle about on the command line.

I haven’t done anything very exciting yet — I just wanted to make sure it was all possible. So far I’ve harvested data about agencies related to ‘MIGRATION’ and its associated functions ‘CITIZENSHIP’, ‘DEPORTATION’, ‘MIGRANT SERVICES’, ‘MULTICULTURALISM’, ‘PASSENGER ENTRY CONTROL’, and ‘REFUGEES’. You can view CSV-formatted summaries for each function on GitHub.

The CSV file for MIGRATION, for example, lists 191 associated agencies. These range from departments of state, such as the current Department of Immigration and Border Protection, Central Office, to smaller bodies like the Committee on Overseas Professional Qualifications. For each agency I’ve listed it’s status, location, life dates, and the dates during it was associated with the migration function. This is only a summary of the data I have for each agency, which includes relationships with other agencies, as well as details of any other related functions. It would be interesting to use this to draw out possible connections between functions.

As a preliminary experiment I thought it’d be interesting to create a simple timeline displaying the different government departments associated with immigration since 1900. I was trying to avoid too much coding, so I ended up hacking Plot.ly’s basic line chart. As a visualisation it’s not very exciting, but as an initial poke at the data it’s quite encouraging.

Hover over the lines to see the agency names and date ranges. Click on one of the agency identifiers (the CA numbers) to open its page in RecordSearch.

The code to create the CSV summaries as well as simple charts like this is all included with the harvester in my RecordSearch Functions repository. I’ll add tools and examples as work proceeds or time allows. Hope you find it useful.

I spoke recently at the LODLAM Summit and the New Factual Storytelling Symposium about my latest experiments in developing online historical narratives that embed rich structured data about people, places, events and resources. I also mentioned that there were plans afoot to publish a couple of substantial works using the framework. One of these will be by my partner in crime, Kate Bagnall, and I thought you might like a preview. Coming soon(ish) to a website near you…

James Minahan’s Homecoming

By Kate Bagnall

In 1908 James Minahan arrived in Australia after 26 years in China. Born in Melbourne, as a small boy he was taken by his Chinese father to live in his ancestral village in rural Guangdong to be educated in Chinese. Growing up in China James remembered little of the country of his birth or of his Irish-Australian mother, but Australia was always part of his plans for the future. His father had kept shares in the store he had run on the Indigo goldfields in the 1870s and, as an adult, James planned to return there.

What he found on his arrival in the newly federated Australia, however, was that being born in Australia did not guarantee a right to live there if you were deemed to be Chinese. No longer able to speak the English of his Victorian childhood, James was made to sit the infamous Dictation Test, which he failed. After he was arrested as a prohibited immigrant, the Chinese community rallied around him and began legal proceedings to have him released. James Minahan’s case made it all the way to the High Court of Australia, who ruled in his favour. He was allowed to remain in Australia.

A century later, the High Court’s decision in Potter v. Minahan continues to inform the legal interpretation of what it means to be an Australian. Who is an immigrant? Who is a member of the Australian community? Who belongs? Who doesn’t? In James Minahan’s Homecoming historian Kate Bagnall tells the story behind the Potter v. Minahan High Court case for the first time.

Presented at the New Factual Storytelling symposium, 10 April 2015, University of Canberra

I feel like the nerdy kid at the cool kids’ party.

There are lots of interesting and creative projects on show today and I… well I want to talk to you about metadata.

Data. According to some pundits it’s the new oil, or the new electricity. Fuel for economic development — a raw material ready to be ‘mined’ for insights, innovation and our purchasing preferences.

In the cultural heritage sector the data metaphors are more likely to be framed around liberation than exploitation. Our data wants to be ‘open’. But there’s still a tendency to think on an industrial scale — it’s about pumping out large datasets for potential re-use.

What can be lost in metaphors of extraction and scale is an appreciation of the human origins of data. We are not buoys bobbing in the ocean reporting on the heights of passing waves. Big data is made up of many small acts of living.

So today I want to talk about small-scale, free-range, artisanal data. I want to talk about data, alongside storytelling, as the product of creativity, imagination, frustration and fury.

Let’s think for a moment about the work of a historian — identifying actors, defining relationships, documenting the complex networks that bring together people, places and events over time. It’s painstaking, exhilirating and potentially soul-destroying work. It’s also an exercise in data modelling. Whether the results are preserved in a triplestore, a spreadsheet, or on a drawer full of index cards — it’s nodes and edges, it’s entities and relationships, it’s data.

And that’s ok. Making data doesn’t condemn you to a rigidly empirical, deterministic framework. There’s always room for nuance, interpretation and doubt. there’s always room for stories.

But what happens when historians undertake the oddly-named process of ‘writing up’. The complex data models are flattened down to a series of sentences neatly arranged in linear sequence — our things become strings. The data is squeezed out and discarded, glimpsed only as fragile echoes hiding in footnotes.

This is of course part of the skill of historical writing — the ability to represent complex relationships through narrative. But why can’t we have our stories and data too?

This is a question I’ve returned to a number of times over the last few years.

It’s come up because I get excited about Linked Open Data’s potential to deliver structured, machine-readable information via the web. But then I wonder, whose stories will we be telling to the machines. How can we explore the expressive possibilities of Linked Open Data and not be constrained by instrumentalist assumptions about the models we make.

It’s come up because I get excited about embedding cultural heritage collections within the passion and practice of everyday life. Why squeeze out the data from historical publications when every article could be an online exhibition, every book could be a digital portal, every footnote could be a link for exploration and aggregation?

I don’t really have answers, but I do make stuff. I’ve a few goes at trying to create narratives that embed Linked Open Data.

They’re not very exciting from a design point of view, but I keep coming back to them because there still seems to be a lack of alternatives. There’s lots of talk about publishing Linked Open Data, but much less about how the use and consumption of Linked Open Data can be built into creative practice.

So here I am again.

This exercise has a number of constraints built in. The main one is NO PLATFORMS — a historian using a series of simple tools should be able to create and publish a data-driven web page without any dependencies. It should be as simple as uploading an html page to a server.

In my idealised workflow, the historian would manage their data about people, places, events and resources in a simple database capable of exporting a flavour of Linked Open Data known as JSON-LD.

Then, having created their narrative, they’d mark it up in the tool of their choice to relate specific names or phrases in the text to the entities in their database.

Then they’d just drop the text and the data into a html page and whack it on the web. With a bit of javascript magic to activate the data, you’d have something like this…

The demo is live (though still under construction), so have a play.

• Scroll the text to see those carefully inserted identifiers create pop ups in the sidebar.
• The text has itself become data, each paragraph is an object — try filtering the text or linking to an individual paragraph.
• Browse all the people, or resources. Explore all the relationships for Inigo Jones.
• Mapping to existing identifiers from sources like Trove and Wikipedia help put the ‘linked’ into Linked Open Data.
• There’s a rather boring map, a timeline and a wall view. New views could be easily added by dropping in some extra javascript.
• It’s all data, so other visualisations and analyses might be created on the fly.

That’s what humans see, but what about machines? All the carefully curated data is exposed in a machine-readable form. Lots of triples…

The code for the viewer and maker is all available if anyone wants to play with it, and I’m intending this year to develop two substantial monographs using these tools. Both have many links into cultural collections.

My aim here is not to develop a fully-operational publishing system. I just want to get a better idea of what’s useful, what’s interesting, what’s possible. To think beyond the current limits of scholarly publishing into a world where data and narrative can live together, where interpretative work is represented in all data-inflected glory.

This my now traditional ‘what I done this year’ post, which, if nothing else, makes me check that my various experiments are still alive. It’s been a challenging year trying to balance my work as Trove Manager with my broader passions and responsibilities as a member of the digital humanities community. So yeah. My personal highlights included heading to Japan to give a keynote at the annual conference of the Japanese Association of Digital Humanities, building Eyes on the past, and resurrecting THATCamp Canberra.

2015 is shaping up as both exciting and scary. On the scary front there’s the whole giving a keynote to hundreds of the world’s leading digital humanities scholars at DH2015 thing (cue imposter syndrome). There’ll also be the launch of Copyfight from NewSouth Publishing, which includes contributions from me and some really-real writers. I’m looking forward to squeezing in some more work on Invisible Australians and a few other research projects. Stay tuned.

The making

Inserting usual disclaimer here that this is not what I get paid to do as Trove Manager. These are projects and experiments I undertake in my own time, for my own reasons, at the cost of my own sanity. So all the problems and mistakes are also mine.

• Some Python code for migrating Trove tags and lists to Zotero collections. More details…
• @TroveBot — a sibling for @TroveNewsBot, tweeting goodness from all the non-newspapery bits of Trove. Read the birth notice…
• Using the Trove API to enrich some WWI data from the State Library of South Australia.
• A new version of the Tung Wah Newspaper Index — an English-language index of two Chinese-language newspapers – the Tung Wah News 東華新報 (1898–1902) and the later Tung Wah Times 東華報 (1902–1936) – published in Sydney, Australia.  More details…
• Some experimenting with facial detection and WWI newspaper articles.
• A prototype for finding Trove newspapers near a given location. More details…
• Trove is… — a dashboard-y thing to give a current snapshot of Trove’s contents. More details…
• A simple search interface to help you find the Radio National content in Trove. More details…
• In a word…  — explore the currents of Australian affairs, 2003–2013. More details…
• Eyes on the past — an experimental interface to explore Trove’s newspapers through faces. Yep, faces (and eyes). Creepy or beautiful — what do you think? The story…
• Trove Traces — exploring Trove newspaper links on the wild, wild web.
• @OperationBot — protecting Australia from meaning. Also available in web app flavour…
• Making lots of graphs in Plotly. More details…
• Faces from Series J2483 — a new demo on an old theme, prepared for my JADH keynote.
• A webapp for Troveia — create your own Trove trivia competitions with this code
• I also solved the problems I was having with RecordSearch to get my WWI Records Finder working properly again. Yay!

The talking

• The future of Trove, State Library of WA, 18 March 2014.
• Trove — it’s bigger on the inside…, GovHack Data Jam, 3 June 2014. Bonus video edition at no extra cost!
• Life on the outside: Collections, contexts and the wild, wild web, keynote at the Japanese Association of Digital Humanities, 20 September 2014. Now available on Medium for your viewing pleasure!
• Introduction to Trove, Cultural Heritage and New Media Workshop, Curtin University, 25 September 2014.
• Digitised newspapers and the varieties of value, Europeana Newspapers workshop, British Library, 29 September 2014.
• Presentations on the future of Trove for AustLII, the ACT branch of the Australian Law Librarians Association, and the ANU Library.
• Guest lectures on digital humanities and Trove for undergraduate and graduate courses at ANU.

I’m currently trying to make some progress with my ‘seams and edges’ paper for ALIAOnline 2015 and naturally ended up writing some code (what me procrastinate?). I was wondering about ways of exploring the ‘representativeness’ of an aggregation like Trove — what’s there and what’s not — so started noodling around with the Trove API.

The first result was a graph representing the numbers of Trove contributors and resources by state, compared to the population of that state. All values are displayed as percentages of the total.

The ACT is over-represented, of course, because of the holdings of the National Library itself. The under-representation of Queensland looks interesting — something to explore in the future.

My next graph used data on languages spoken at home in Australia from the 2011 census. It compared the population speaking those languages with the number of books in that language included in Trove, again as percentages of the total. It doesn’t embed very well, so view the full-size version on Plotly.

As I was playing around I noticed a tweet from Bridget Griffen-Foley:

#CMHistory Associate Member @WillaMcDonald: by 1886 at least 48 daily newspapers circulating in Australia. @JERAAus

— Bridget Griffen-Foley (@BGriffenFoley) November 25, 2014

Being in a quick-coding sort of mood I had to see how long it would take me to create a graph showing the numbers of daily newspapers in Trove (where daily is defined as more than 300 issues in a year). The answer was about fifteen minutes.

All of the graphs are created using the web service Plotly. Plotly has an easy-to-use Python API which means all you need to do to create a graph is to add a few lines of code. There are other Python visualisation libraries, but I like Plotly because it creates something instantly shareable — perfectly suited to this sort of quick and dirty experimentation.

I don’t think any of these graphs are particularly revealing, and I’ve made some assumptions about the data that probably wouldn’t hold up under scrutiny. But what this fiddling around emphasised was how an API and some simple tools make it possible to ask quick questions of the data.

All the code is in my Trove-Sketches repository on GitHub.

Faces offer an instant connection to history, reminding us that the past is full of people. People like us, but different. People with their own lives and stories. People we might only know through a picture, a few documentary fragments, or a newspaper article.

Eyes on the Past is an experimental interface, built in a weekend. I’m exploring whether faces can provide a way to explore more than 120 million newspaper articles available on Trove.

This collection of tweets tells the story of its development.

There’s some details about the software used in the site’s about page. You can view the harvest/detection and the website code on GitHub.

Meanwhile over on the Trove blog I posted about ‘In a word’ — my quick experiment with all the lovely Radio National data now available in Trove.