What are the functions of government? You’ll find one possible answer in the Australian Governments’ Interactive Functions Thesaurus (AGIFT) maintained by the National Archives of Australia. AGIFT sets out a standard series of terms to describe common business activities across the different levels of government.

Functions are important to the National Archives — they aid discovery, inform appraisal, and guide government agencies in understanding what records they should keep keep. Functions also offer researchers an alternative way of exploring the Archives’ rich collections — instead of blindly searching for keywords, you can track the activities of government agencies through time.

RecordSearch, the National Archives’ online database, provides a simple way of browsing by function, though it’s not very obvious. Click on the ‘Advanced Search’ tab in RecordSearch and you’ll see an interactive diagram representing the way that records are described and arranged using the Commonwealth Records Series (CRS) system. I made this little widget a very long time ago, and there’s an alternative version, with links to the CRS manual, over at WraggeLabs.

You’ll notice that functions are associated with agencies — they’re a way of describing what government departments do. Click on the ‘Functions’ button to open up a very basic browse interface that lists preferred and alternative names for functions. There seem to be a few differences between AGIFT and the terms used in RecordSearch, I’m not sure why. The CRS Manual includes a summary of the thesaurus used in RecordSearch, and for convenience I’ve extracted a flat list of all the functions.

Just click on any function in RecordSearch to find agencies associated with it. It’s basic, but it does at least provide an alternative to the tyranny of the search box, so well described by Mitchell Whitelaw. If you’re starting a research project and don’t know what keywords are relevant, perhaps following functions will help you map the territory.

Functions also provide a way of exploring how the organisation of government changes to meet social and political needs. How are the government’s priorities in areas such as ‘migration’, ‘education’ or ‘security’ reflected in the sorts of bureaucratic structures they create? To answer this type of question you really need to extract all the data related to a function for further analysis and manipulation, but that’s not something you can easily do through RecordSearch’s web interface. Time for some hacking…

Harvesting data from RecordSearch has been something of an obsession over the past five or six years. Over that time I’ve developed a set of tools that pull structured data out of the web interface. I’ve used these tools to expose records relating to the White Australia Policy, to grab a copy of every publicly available ASIO file, and to analyse records that remain closed to public access. But to harvest functions I needed to extend my toolkit — improving the handling of agency data, and adding a simple search module to find all agencies associated with a function. Once this was done I could build a harvester — a script that just makes a series of requests to RecordSearch until all of the desired records have been retrieved, processed and stored.

All the code for my toolkit and harvester are freely available on GitHub — please download, use and improve! I’ve tried to provide fairly detailed instructions with the harvester, but you will need to fiddle about on the command line.

I haven’t done anything very exciting yet — I just wanted to make sure it was all possible. So far I’ve harvested data about agencies related to ‘MIGRATION’ and its associated functions ‘CITIZENSHIP’, ‘DEPORTATION’, ‘MIGRANT SERVICES’, ‘MULTICULTURALISM’, ‘PASSENGER ENTRY CONTROL’, and ‘REFUGEES’. You can view CSV-formatted summaries for each function on GitHub.

The CSV file for MIGRATION, for example, lists 191 associated agencies. These range from departments of state, such as the current Department of Immigration and Border Protection, Central Office, to smaller bodies like the Committee on Overseas Professional Qualifications. For each agency I’ve listed it’s status, location, life dates, and the dates during it was associated with the migration function. This is only a summary of the data I have for each agency, which includes relationships with other agencies, as well as details of any other related functions. It would be interesting to use this to draw out possible connections between functions.

As a preliminary experiment I thought it’d be interesting to create a simple timeline displaying the different government departments associated with immigration since 1900. I was trying to avoid too much coding, so I ended up hacking Plot.ly’s basic line chart. As a visualisation it’s not very exciting, but as an initial poke at the data it’s quite encouraging.

Hover over the lines to see the agency names and date ranges. Click on one of the agency identifiers (the CA numbers) to open its page in RecordSearch.

The code to create the CSV summaries as well as simple charts like this is all included with the harvester in my RecordSearch Functions repository. I’ll add tools and examples as work proceeds or time allows. Hope you find it useful.

I spoke recently at the LODLAM Summit and the New Factual Storytelling Symposium about my latest experiments in developing online historical narratives that embed rich structured data about people, places, events and resources. I also mentioned that there were plans afoot to publish a couple of substantial works using the framework. One of these will be by my partner in crime, Kate Bagnall, and I thought you might like a preview. Coming soon(ish) to a website near you…

James Minahan’s Homecoming

By Kate Bagnall

In 1908 James Minahan arrived in Australia after 26 years in China. Born in Melbourne, as a small boy he was taken by his Chinese father to live in his ancestral village in rural Guangdong to be educated in Chinese. Growing up in China James remembered little of the country of his birth or of his Irish-Australian mother, but Australia was always part of his plans for the future. His father had kept shares in the store he had run on the Indigo goldfields in the 1870s and, as an adult, James planned to return there.

What he found on his arrival in the newly federated Australia, however, was that being born in Australia did not guarantee a right to live there if you were deemed to be Chinese. No longer able to speak the English of his Victorian childhood, James was made to sit the infamous Dictation Test, which he failed. After he was arrested as a prohibited immigrant, the Chinese community rallied around him and began legal proceedings to have him released. James Minahan’s case made it all the way to the High Court of Australia, who ruled in his favour. He was allowed to remain in Australia.

A century later, the High Court’s decision in Potter v. Minahan continues to inform the legal interpretation of what it means to be an Australian. Who is an immigrant? Who is a member of the Australian community? Who belongs? Who doesn’t? In James Minahan’s Homecoming historian Kate Bagnall tells the story behind the Potter v. Minahan High Court case for the first time.

Presented at the New Factual Storytelling symposium, 10 April 2015, University of Canberra

I feel like the nerdy kid at the cool kids’ party.

There are lots of interesting and creative projects on show today and I… well I want to talk to you about metadata.

Data. According to some pundits it’s the new oil, or the new electricity. Fuel for economic development — a raw material ready to be ‘mined’ for insights, innovation and our purchasing preferences.

In the cultural heritage sector the data metaphors are more likely to be framed around liberation than exploitation. Our data wants to be ‘open’. But there’s still a tendency to think on an industrial scale — it’s about pumping out large datasets for potential re-use.

What can be lost in metaphors of extraction and scale is an appreciation of the human origins of data. We are not buoys bobbing in the ocean reporting on the heights of passing waves. Big data is made up of many small acts of living.

So today I want to talk about small-scale, free-range, artisanal data. I want to talk about data, alongside storytelling, as the product of creativity, imagination, frustration and fury.

Let’s think for a moment about the work of a historian — identifying actors, defining relationships, documenting the complex networks that bring together people, places and events over time. It’s painstaking, exhilirating and potentially soul-destroying work. It’s also an exercise in data modelling. Whether the results are preserved in a triplestore, a spreadsheet, or on a drawer full of index cards — it’s nodes and edges, it’s entities and relationships, it’s data.

And that’s ok. Making data doesn’t condemn you to a rigidly empirical, deterministic framework. There’s always room for nuance, interpretation and doubt. there’s always room for stories.

But what happens when historians undertake the oddly-named process of ‘writing up’. The complex data models are flattened down to a series of sentences neatly arranged in linear sequence — our things become strings. The data is squeezed out and discarded, glimpsed only as fragile echoes hiding in footnotes.

This is of course part of the skill of historical writing — the ability to represent complex relationships through narrative. But why can’t we have our stories and data too?

This is a question I’ve returned to a number of times over the last few years.

It’s come up because I get excited about Linked Open Data’s potential to deliver structured, machine-readable information via the web. But then I wonder, whose stories will we be telling to the machines. How can we explore the expressive possibilities of Linked Open Data and not be constrained by instrumentalist assumptions about the models we make.

It’s come up because I get excited about embedding cultural heritage collections within the passion and practice of everyday life. Why squeeze out the data from historical publications when every article could be an online exhibition, every book could be a digital portal, every footnote could be a link for exploration and aggregation?

I don’t really have answers, but I do make stuff. I’ve a few goes at trying to create narratives that embed Linked Open Data.

They’re not very exciting from a design point of view, but I keep coming back to them because there still seems to be a lack of alternatives. There’s lots of talk about publishing Linked Open Data, but much less about how the use and consumption of Linked Open Data can be built into creative practice.

So here I am again.

This exercise has a number of constraints built in. The main one is NO PLATFORMS — a historian using a series of simple tools should be able to create and publish a data-driven web page without any dependencies. It should be as simple as uploading an html page to a server.

In my idealised workflow, the historian would manage their data about people, places, events and resources in a simple database capable of exporting a flavour of Linked Open Data known as JSON-LD.

Then, having created their narrative, they’d mark it up in the tool of their choice to relate specific names or phrases in the text to the entities in their database.

Then they’d just drop the text and the data into a html page and whack it on the web. With a bit of javascript magic to activate the data, you’d have something like this…

The demo is live (though still under construction), so have a play.

• Scroll the text to see those carefully inserted identifiers create pop ups in the sidebar.
• The text has itself become data, each paragraph is an object — try filtering the text or linking to an individual paragraph.
• Browse all the people, or resources. Explore all the relationships for Inigo Jones.
• Mapping to existing identifiers from sources like Trove and Wikipedia help put the ‘linked’ into Linked Open Data.
• There’s a rather boring map, a timeline and a wall view. New views could be easily added by dropping in some extra javascript.
• It’s all data, so other visualisations and analyses might be created on the fly.

That’s what humans see, but what about machines? All the carefully curated data is exposed in a machine-readable form. Lots of triples…

The code for the viewer and maker is all available if anyone wants to play with it, and I’m intending this year to develop two substantial monographs using these tools. Both have many links into cultural collections.

My aim here is not to develop a fully-operational publishing system. I just want to get a better idea of what’s useful, what’s interesting, what’s possible. To think beyond the current limits of scholarly publishing into a world where data and narrative can live together, where interpretative work is represented in all data-inflected glory.

This my now traditional ‘what I done this year’ post, which, if nothing else, makes me check that my various experiments are still alive. It’s been a challenging year trying to balance my work as Trove Manager with my broader passions and responsibilities as a member of the digital humanities community. So yeah. My personal highlights included heading to Japan to give a keynote at the annual conference of the Japanese Association of Digital Humanities, building Eyes on the past, and resurrecting THATCamp Canberra.

2015 is shaping up as both exciting and scary. On the scary front there’s the whole giving a keynote to hundreds of the world’s leading digital humanities scholars at DH2015 thing (cue imposter syndrome). There’ll also be the launch of Copyfight from NewSouth Publishing, which includes contributions from me and some really-real writers. I’m looking forward to squeezing in some more work on Invisible Australians and a few other research projects. Stay tuned.

The making

Inserting usual disclaimer here that this is not what I get paid to do as Trove Manager. These are projects and experiments I undertake in my own time, for my own reasons, at the cost of my own sanity. So all the problems and mistakes are also mine.

• Some Python code for migrating Trove tags and lists to Zotero collections. More details…
• @TroveBot — a sibling for @TroveNewsBot, tweeting goodness from all the non-newspapery bits of Trove. Read the birth notice…
• Using the Trove API to enrich some WWI data from the State Library of South Australia.
• A new version of the Tung Wah Newspaper Index — an English-language index of two Chinese-language newspapers – the Tung Wah News 東華新報 (1898–1902) and the later Tung Wah Times 東華報 (1902–1936) – published in Sydney, Australia.  More details…
• Some experimenting with facial detection and WWI newspaper articles.
• A prototype for finding Trove newspapers near a given location. More details…
• Trove is… — a dashboard-y thing to give a current snapshot of Trove’s contents. More details…
• A simple search interface to help you find the Radio National content in Trove. More details…
• In a word…  — explore the currents of Australian affairs, 2003–2013. More details…
• Eyes on the past — an experimental interface to explore Trove’s newspapers through faces. Yep, faces (and eyes). Creepy or beautiful — what do you think? The story…
• Trove Traces — exploring Trove newspaper links on the wild, wild web.
• @OperationBot — protecting Australia from meaning. Also available in web app flavour…
• Making lots of graphs in Plotly. More details…
• Faces from Series J2483 — a new demo on an old theme, prepared for my JADH keynote.
• A webapp for Troveia — create your own Trove trivia competitions with this code
• I also solved the problems I was having with RecordSearch to get my WWI Records Finder working properly again. Yay!

The talking

• The future of Trove, State Library of WA, 18 March 2014.
• Trove — it’s bigger on the inside…, GovHack Data Jam, 3 June 2014. Bonus video edition at no extra cost!
• Life on the outside: Collections, contexts and the wild, wild web, keynote at the Japanese Association of Digital Humanities, 20 September 2014. Now available on Medium for your viewing pleasure!
• Introduction to Trove, Cultural Heritage and New Media Workshop, Curtin University, 25 September 2014.
• Digitised newspapers and the varieties of value, Europeana Newspapers workshop, British Library, 29 September 2014.
• Presentations on the future of Trove for AustLII, the ACT branch of the Australian Law Librarians Association, and the ANU Library.
• Guest lectures on digital humanities and Trove for undergraduate and graduate courses at ANU.

I’m currently trying to make some progress with my ‘seams and edges’ paper for ALIAOnline 2015 and naturally ended up writing some code (what me procrastinate?). I was wondering about ways of exploring the ‘representativeness’ of an aggregation like Trove — what’s there and what’s not — so started noodling around with the Trove API.

The first result was a graph representing the numbers of Trove contributors and resources by state, compared to the population of that state. All values are displayed as percentages of the total.

The ACT is over-represented, of course, because of the holdings of the National Library itself. The under-representation of Queensland looks interesting — something to explore in the future.

My next graph used data on languages spoken at home in Australia from the 2011 census. It compared the population speaking those languages with the number of books in that language included in Trove, again as percentages of the total. It doesn’t embed very well, so view the full-size version on Plotly.

As I was playing around I noticed a tweet from Bridget Griffen-Foley:

#CMHistory Associate Member @WillaMcDonald: by 1886 at least 48 daily newspapers circulating in Australia. @JERAAus

— Bridget Griffen-Foley (@BGriffenFoley) November 25, 2014

Being in a quick-coding sort of mood I had to see how long it would take me to create a graph showing the numbers of daily newspapers in Trove (where daily is defined as more than 300 issues in a year). The answer was about fifteen minutes.

All of the graphs are created using the web service Plotly. Plotly has an easy-to-use Python API which means all you need to do to create a graph is to add a few lines of code. There are other Python visualisation libraries, but I like Plotly because it creates something instantly shareable — perfectly suited to this sort of quick and dirty experimentation.

I don’t think any of these graphs are particularly revealing, and I’ve made some assumptions about the data that probably wouldn’t hold up under scrutiny. But what this fiddling around emphasised was how an API and some simple tools make it possible to ask quick questions of the data.

All the code is in my Trove-Sketches repository on GitHub.

Faces offer an instant connection to history, reminding us that the past is full of people. People like us, but different. People with their own lives and stories. People we might only know through a picture, a few documentary fragments, or a newspaper article.

Eyes on the Past is an experimental interface, built in a weekend. I’m exploring whether faces can provide a way to explore more than 120 million newspaper articles available on Trove.

This collection of tweets tells the story of its development.

There’s some details about the software used in the site’s about page. You can view the harvest/detection and the website code on GitHub.

Meanwhile over on the Trove blog I posted about ‘In a word’ — my quick experiment with all the lovely Radio National data now available in Trove.

No, nothing to do with Easter or eggs, but it’s Easter Sunday and who can resist a good opportunity for a bad pun?

This is another catch-up post, pulling together some recent experiments. If nothing else, it’ll help me keep track of things I’m otherwise likely to forget.

WWI Faces

In our last instalment I was playing around with some WWI data from the State Library of South Australia. I’m really pleased to report that SLSA staff have used my experiments to help them add Trove links to more than 6,000 of their Heroes of the Great War records. Here’s an example — note the ‘article’ link which goes straight to a digitised newspaper article in Trove. With some good data and bit of API wrangling we’ve now established rich linkages between an important WWI resource and Trove. Win!

I’ve also continued my fiddling with articles from the Adelaide Chronicle as I start to think about how Trove’s newspapers might be used in a WWI exhibition being developed by the National Library. At the end of my last post I’d created a list of articles from the Chronicle that were likely to include biographical details of WWI personnel. I knew that many of these included portrait photos, so I filtered them on Trove’s built-in ‘illustrated’ facet and saved the page images for the remaining articles. You can browse the resulting collection of pages on Dropbox. As you can see there are indeed many portraits of service people.

So the next step was to try and extract the portraits from the pages. This was rather familiar territory, as I’d already used a facial detection script to create The Real Face of White Australia. But I wasn’t sure how the pattern recognition software would cope with the lower quality newspaper images. After getting all the necessary libraries installed (the hardest bit of the whole process), I pointed the script at the page images and… it worked!

From 141 pages I extracted 1,738 images, and most of them were faces. You can browse all 1,738, but be warned, I’ve just dumped them onto a single page and added a bit of Isotope magic — so they’ll take a fair while to load and your browser might object. You’ll also notice that I haven’t tried to filter out photos of non-service people, I just wanted to see if it worked. And it does. Even in this rough form you can sense some of the emotive power. What’s really amazing is the way that even small images of faces in group photographs were identified. All I was aiming for at this stage was a proof of concept — yes, I can extract photos of WWI service people from newspapers. Hmmm…

Trove in space

All the faces above were from one South Australian newspaper. Several years ago I worked on a project to map places of birth and enlistment of WWI service people, and while I have no interest in the national mythologies surrounding WWI, I do still wonder about the local impact of war — all those small communities sending off their sons and daughters…

So I’m wondering whether we might be able to use the digitised newspapers in Trove to navigate from place to face. To choose a town anywhere in Australia, and present photographs of service personnel published in nearby newspapers.

I now know I can extract the photos, but how can we navigate Trove newspapers by location? Time for a new experiment…

The Trove API provides a complete list of digitised newspaper titles. You’ll notice that some of the titles include a place name as part of the summary information in brackets, while many others will include place names in their titles, for example:

• Illawarra Daily Mercury (Wollongong, NSW : 1950 – 1954)
• Hawkesbury Herald (Windsor, NSW : 1902 – 1945)
• Kiama Examiner (NSW : 1858 – 1859)
• Narromine News and Trangie Advocate (NSW : 1898 – 1955)

I haven’t had much luck getting automated named entity extraction tools to work on short text strings like this, so I decided to roll my own using Geoscience Australia’s Gazetteer of Australia 2012. I opened up the GML file containing all Australian places and saved the populated locations to my own Mongo database. This gave me a handy place name database, complete with geo-locations.

Next I went to work on the newspaper titles. Extracting the places from the summary information was easy because they followed a regular pattern, but finding them in the body of the title was trickier. First I had to exclude those words that were obviously not place names. Aside from the usual stopwords (‘and’ and ‘the’), there are many words that commonly occur in newspaper titles — ‘Herald’, ‘Star’, ‘Chronicle’ etc. To find these words I pulled apart all the titles and calculated the frequency of every word. You can explore the raw results — ‘Advertiser’ (116) wins by a large margin, with ‘Times’ (67) in second place. From these results I could create a list of words that I knew were not places and could safely be ignored.

Then it was just a matter of tokenising the titles (breaking them up into individual words), removing all the stopwords (the standard list and my special list), and then looking up the words in my place name database. I did this in two passes, first as bigrams (pairs of words), and then as single words — this allowed me to find compound place names like ‘North Melbourne’. The Trove API gives you s ‘state’ value for each title, so I could use this in the query to increase accuracy.

If I found a place name, I added the place details, including the latitude and longitude, to the title record from the API and included it in my own newspaper title database.

So I ended up to two databases — one with geolocated places, and another with geolocated newspapers. That meant I could build a simple interface to find newspaper titles by place. It’s nothing fancy — just another proof of concept — but it works pretty well. Just type in a place name and select a state and a query is run against the place name database. If the place is found then the latitude and longitude is fed to the titles database to find the closest newspapers. After removing some duplicates, the 10 nearest newspapers are displayed.

Building some sort of map interface on top of this is pretty trivial. What’s more important is to do some analysis of my place matching to see what I might have missed. But so far so good!

Trove is…

Trove is more than newspapers. This is a message the Trove team tries to emphasise at every opportunity. The digitised newspapers are an incredible resource of course, but there’s so much other interesting stuff to explore.

To try and give a quick and easy introduction to this richness, I created a simple dashboard-type view of Trove, imaginatively titled Trove is…

Trove is… gives a basic status report on each of the 1o Trove zones, with statistics updated daily (except for the archived websites as there’s no API access at the moment). The BIG NUMBERS are counter-balanced by a single randomly-selected example from each zone. It’s a summary, an overview, a portal and a snapshot. Reload the page and the zones will be reordered and the examples will change.

It’s pretty simple, but I think it works quite well, and thanks to Twitter Bootstrap it looks really nice on my phone! But while the idea was simple, the implementation was pretty tricky — particularly the balance between randomness and performance. If all the examples were truly random, drawn from the complete holdings to Trove on every page reload, you’d spend a lot of time watching spinning arrows waiting for content to appear. I tried a number of different approaches and finally settled on a system where random selections of 100 resources per zone are made every hour by background processes and cached. When you load the page, this cache is queried and an item selected. So if you keep hitting reload you’ll probably notice that some examples reappear. It’s random, but at any moment the pool of possibilities is quite limited. Come back later in the day and everything will be different.

Anyway, if anyone asks you what Trove is, you now know where to point them…

Who listens to the radio?

After a lot of hard work, the Trove team was excited to announce recently that more than 200,000 records from 54 ABC Radio National programs were available through Trove.

To make it a bit easier to explore this wonderful new content, I created a simple search interface. All it really does is help you build a query using the RN program titles, and then sends the query off to Trove. Not fancy, but useful (my family motto).

Of course, I couldn’t leave my Twitter bot family out of the action. @TroveBot has been Radio National enabled. Just tweet the tag #abcrn at him to receive a randomly-selected Radio National story. To search for something amidst the RN records, just tweet a keyword or two and add the #abcrn tag to limit the results. Consult the TroveBot manual for complete operating instructions.

In a word…

But the Radio National content is not just findable through the Trove web interface — all that lovely data is freely accessible through the Trove API. That includes just about every segment of every edition of the ABC’s flagship current affairs programs, AM, PM, and The World Today from 1999 onwards. What sort of questions could you ask of this data?

I’ll be writing something soon on the Trove blog about accessing these riches, but I couldn’t resist having a play. So I harvested all the RN data via the API and built a new thing…

It’s called In a word: Currents in Australian affairs, 2003–2013, and for once it’s quite well documented, so I won’t go into details here. I’ll just say that it’s one of my favourite creations, and I hope you find it interesting.

Addendum (21 April) — The Tung Wah Newspaper Index

See, I told you I forget things…

I recently finished resurrecting the Tung Wah Newspaper Index. Kate has described the original project on her blog, and there’s a fair bit of contextual information on the site, so I won’t go into details here. Suffice it to say it’s an important resource for Chinese Australian history that had succumbed to technological decay.

The original FileMaker database has been MySqld, Solrised, and Bootstrapped to get it all working nicely. I also took the opportunity to introduce a bit of LOD love, with plenty of machine-readable data built-in.

The whole site follows an interface as API type pattern. So if you want a resource as JSON-LD, you just change the file extension to .json. To help you out, there are links at the bottom of each page to the various serialisations, and of course you can also use content negotiation to get what you’re after. There’s some examples of all this in the GitHub repository, as well as a CSV dump of the whole database.

I can’t resist a challenge, particularly when it involves lots of new historical data and an excuse to muck around with the Trove API. So when Katie Hannan from the State Library of South Australia asked me about putting the API to work to enrich one of their World War I datasets, I had to dive in and have a play.

The dataset consists of references to South Australian WWI service personnel published in the Adelaide Chronicle between 1914 and 1919. In a massive effort starting back in 2000, SLSA staff manually extracted more than 13,000 references and grouped them under 9709 headings, mostly names. You can explore the data in the SLSA catalogue as part of the Heroes of the Great War collection.

It’s great data, but it would be even better if there was a direct link to each article in Trove — hence Katie’s interest in the possibilities of the API!

Katie sent me a spreadsheet containing the data. Each row corresponds to an individual entry and includes an identifier, a name, a year, and a list of references separated by semicolons. My plan was simple, for each row I’d construct a search based on the name, then loop through the search results to try find an article that matched the date and page number of each reference. This might seem a bit cumbersome, but currently there’s no way of searching Trove for newspaper articles published on a particular day.

You’ll find all of the code on GitHub. I’ve tried to include plenty of comments to make it easy to follow along.

Let’s look at the entry for Lieutenant Frank Rosevear. It includes the following references: ‘Chronicle, 7 September 1918, p. 27, col. c;Chronicle, 14 September 1918, p. 24, col. a p.38, col. d’. If you look closely, you’ll see that there’s two page numbers for 14 September 1918, so there’s actually three references included in this string. The first thing I had to do was to pull out all the references and format them in a standard way.

Assuming that the last name was the surname, I then constructed a query that searched for an exact match of the surname together with at least one of the other names. In Lieutenant Rosevear’s case the query would’ve been ‘fulltext:”Rosevear” AND (“Lieutenant” OR “Frank”)’. Note the use of the ‘fulltext’ modifier to indicate an exact match. To this query I added a date filter to limit the search to the specified year and an ‘l-title’ value to search only the Adelaide Chronicle.

You can see the results for this query in my Trove API console. Try modifying the query string to see what difference it makes.

Once the results came back from the API I compared them to the references, looking for matches on both the date and page number. You might notice that the second result from the API query, dated 7 September 1918, is a match for one of our references. Yay! This gets saved to a list of strong matches. But what about the other references?

Just in case there’s been a transcription error, or the page numbering differed across editions,  I relax the rules a bit in a second pass and accept matches on the date, but not the page. These are saved to a list of close matches.

This second pass doesn’t help much with  Lieutenant Rosevear’s missing references, so we have to broaden our search query a bit. This time we search on the surname only. Bingo! The first result is a match and points us to one of the ‘Heroes of the Great War’ series.

It took a while, but my script eventually worked it’s way through all 9709 entries like this, writing the results out to csv files containing the strong and close matches. It also created a summary for each entry, listing the original number of references alongside the number of strong and close matches.

Ever since I read Trevor Munoz’s post on using Pandas with data from the NYPL’s What’s on the Menu? project, I’ve wanted to have a play with it. So I decided to use Pandas to produce some quick stats from my results file.

>>> import pandas as pd
>>> df = pd.read_csv('data/slsa_results.csv')
# How many entries?
>>> len(df)
9709
# How many references?
>>> df['references'].sum()
13504
# How many entries had strong matches?
>>>len(df[df.strong > 0])
6440
# As a percentage thank you...
>>> 100 * len(df[df.strong > 0]) / len(df)
66
# In how many entries did the number of refs = the number of strong matches
>>> len(df[df.references == df.strong])
4399
# As a percentage thank you...
>>> 100 * len(df[df.references == df.strong]) / len(df)
45
# How many entries had at least one strong or close match?
>>> len(df[df.total > 0])
8989
# As a percentage thank you...
>>> 100 * len(df[df.strong > 0]) / len(df)
92

Not bad. The number of strong matches equalled the number of references in 45% of cases, and overall 66% of entries had a least one strong match. I might be able to get those numbers up by tweaking the search query a bit, but of course the main limiting factor is the quality of the OCR. If the article text isn’t good enough we’re never going to find the names we’re after.

Katie tells me that the State Library intends to point volunteer text correctors towards identified articles. As the correctors start to clean things up, we should be able to find more matches simply by re-running this script at regular intervals.

But what articles should they point the volunteers to? Many of them included the title ‘Heroes of the Great War’, so they’re easy to find, but there were others as well. By analysing the matches we’ve already found we can pull out the most frequent titles and build a list of likely candidates. Something like this:

title_phrases = [
'heroes of the great war they gave their lives for king and country',
'australian soldiers died for their country',
'casualty list south australia killed in action',
'on active service',
'honoring soldiers',
'military honors australians honored',
'casualty lists south australian losses list killed in action',
'australian soldiers died for his country',
'died for their country',
'australian soldiers died for the country',
'australian soldiers died for their country photographs of soldiers',
'quality lists south australian losses list killed in action',
'list killed in action',
'answered the call enlistments',
'gallant south australians how they won their honors',
'casualty list south australia died of wounds',
]

Now we can feed these phrases into a series of API queries and automatically generate a list of articles that are likely to contain details of WWI service people. This list should provide a useful starting point for keen text correctors.

I might not have completely solved Katie’s problem, but I think I’ve shown that the Trove API can be usefully called into action for these sorts of projects. Taking this approach should certainly save a lot of manual searching, clicking, cutting and pasting. And while I’ve focused on the South Australian data, there’s no reason why similar approaches couldn’t be applied to other WWI projects.

What’s the collective noun for a group of Twitter bots? Inspiration is failing me at the moment, so let’s just say that the Trove bot family recently welcomed a new member — @TroveBot.

@TroveBot is a sibling of @TroveNewsBot, who’s been tweeting away since June last year. But while @TroveNewsBot draws his inspiration from 120+ million historical newspaper articles, @TroveBot digs away in the millions of books, theses, pictures, articles, maps and archives that make up the rest of Trove. Trove, as we always like to remind people, is not just newspapers.

Like @TroveNewsBot, the newcomer tweets random finds at regular intervals during the day. But both bots also respond to queries. Just tweet a few keywords at them and they’ll have a poke around Trove and reply with something that seems relevant. There’s various ways of modifying this basic search, as explained on the GitHub pages of TroveNewsBot and TroveBot.

@TroveBot’s behaviour is a little more complex because of Trove’s zone structure. The zones bring together resources with a similar format. If you want to get a more detailed idea of what’s in them, you can play around with my Zone Explorer (an experiment for every occasion!). If you just tweet some keywords at @TroveBot he’ll search for them across all the zones, then choose one of the matching zones at random. If you want to limit your search to a particular zone or format, just add one of the format tags listed on the GitHub site.

Let’s say you want a book about robots, just tweet: ‘robots #book’. Or a photos of pelicans, try ‘pelican #photo’.  It’s really that easy.

@wragge 'Pelican' http://t.co/4dp3urRID1

— TroveBot (@TroveBot) January 19, 2014

But you don’t just have to use keywords, you can also feed @TroveBot a url. Perhaps you want to find a thesis in Trove that is related to a Wikipedia page — just tweet the url together with the tag ‘#thesis’. Yes, really.

@jeanburgess 'Enzymatic mechanisms in the respiration of rhubarb leaves / by John F. Morrison' http://t.co/zde0mrfUqt

— TroveBot (@TroveBot) January 19, 2014

Behind the scenes @TroveBot makes use of AlchemyAPI to extract keywords from the url you supply. These keywords are then bundled up and shipped off to Trove for a response.

You probably know that @TroveNewsBot is similarly url-enabled. This allows him to do things like respond to the tweets of his friends @DigitalNZBot and @DPLABot, and offer regular commentary on the ABC News headlines.

8 Feb 1926: 'GUSTY WINDS' http://t.co/8ZaFAfSEfw // re ABC: http://t.co/hRQE6GV6cS

— TroveNewsBot (@TroveNewsBot) January 22, 2014

So what would happen if @TroveBot and @TroveNewsBot started exchanging links? Would they ever stop? Would they break the internet?

With the Australian Tennis Open coming to a climax over the last few days I thought it was a suitable time to set up a game of bot tennis. To avoid the possibility of internet implosion, I decided to act as intermediary, forwarding the urls from one to the other. The rules were simple — the point was lost if a bot failed to respond or repeated a link from earlier in the match.

And now @trovebot has missed one completely. He’s looking rattled. Amazing fightback from @TroveNewsBot. Advantage TroveNewsBot.

— Tim Sherratt (@wragge) January 26, 2014

As a sporting spectacle the inaugural game wasn’t exactly scintillating. In fact, the first game is still locked at 40-40. But it’s been interesting to see the connections that were made. It’s also been a useful opportunity for me to find bugs and tweak their behaviours. You can catch up with the full, gripping coverage via Storify.

After a few more experiments I’m thinking I might try and set up a permanent, endless conversation between them. It would be fascinating to see where they’d end up over time — the links they’d find, the leaps they’d make.

Hmm, collective nouns… What about a serendipity of bots?