There seems to be a lot of topic modelling going on at the moment. Any why not? Projects like Mining the Dispatch are demonstrating the possibilities. Tools like Mallet are making it easy. And generous DHers like Ted Underwood and Scott Weingart are doing a great job explaining what it is and how it works.

I’ve talked briefly about using topic modelling to explore digitised newspapers, something that the Mapping Texts project has also been investigating. But I’ve also been following with interest Chad Black’s use of algorithmic techniques, including topic modelling, to look for local variations amidst the legal system of the early modern Spanish empire.

As part of the Invisible Australians project, Kate and I are exploring the bureaucracy of the White Australia Policy. In particular, we’re interested in the interaction between policy and practice, between the highly-centralised bureaucracy and the activities of individual port officials. Like Chad, we’re interested in mapping local variations — to try and understand the bureaucracy from the point of view of an individual forced to live within its restrictions.

I recently gave a presentation about the project at Digital Humanities Australasia (post coming soon!), and in preparation I decided to try a few topic modelling experiments. They were very simple, but I was impressed by the possibilities for exploring archival systems.

The problem I started with was this. The workings of the White Australia Policy are well documented by records held by the National Archives of Australia. Some series within the archives are specifically related to the operations of the policy — such as those containing many thousands of CEDTs. But there are also general correspondence series created by the customs offices in each state, as well as the Commonwealth Department of External Affairs which administered the Immigration Restriction Act (responsibility was later taken by the Department of Home and Territories and it’s successors). These general correspondence series are important, because they often include details of difficult or controversial cases — those that required a policy judgment, or prompted a change in existing practices. But how do you find relevant files within series that can contain large numbers of items?

Series A1, for example, is a correspondence series created by the Department of External Affairs. It contains more than 60,000 items. Past research tells us that amongst these 60,000 files are records of important policy discussions relating to White Australia. But these files tend to be labelled with the names of the people involved, so unless you know the names in advance they can be difficult to find.

Mitchell Whitelaw’s A1 Explorer, part of the Visible Archive project, lets you to explore the contents of Series A1 in a easy and engaging way. But while the A1 Explorer provides new opportunities for discovery, it doesn’t offer the fine-grained analysis we need to sift out the files we’re after. And so… topic modelling.

The process was pretty simple. While I can dip into my bag of screen-scrapers to harvest series directly from the NAA’s RecordSearch database, there was already an XML dump of A1 available from data.gov.au. So I extracted the basic file metadata from the XML and wrote the identifiers and titles out to a text file, one item per line. Following the instructions on the website I then loaded this file into Mallet:

/Applications/Mallet/bin/mallet import-file --input ./A1.txt --output A1.mallet --keep-sequence --remove-stopwords

Then it was just a matter of firing up the topic modeller:

/Applications/Mallet/bin/mallet train-topics --input ./A1.mallet --output-state ./A1.gz --output-doc-topics ./A1-topics.txt --output-topic-keys ./A1-keys.txt --num-topics 40

Again, I just followed the examples on the Mallet site.

Once it was finished I opened up A1-keys.txt to browse the ‘topics’ Mallet had found. The results were intriguing. There are a large number of applications for naturalisation in A1, so it’s no surprise that ‘naturalisation’ figures prominently in a number of the topics. What was more interesting was the way Mallet had grouped the naturalisation files. For example:

naturalization christian hans hansen jensen petersen andersen nielsen larsen christensen johannes jens niels pedersen andreas johansen martin jorgensen

and

naturalisation certificate giuseppe salvatore frank la leo samios spina sorbello leonardo fisher natale patane torrisi barbagallo luka rossi ross

Based on the co-occurrence of names within the file titles, Mallet had created groupings that roughly reflected the ethnic origins of applicants. It makes sense when you think about what Mallet is doing, but I still found it pretty amazing.

Mallet also found clusters around the major activities of the department, such as the administration of the territories. But of most interest to us was:

1 0.55539 passport ah student exemption students lee wong chinese young deserter education sing wing chong readmission son hing chin wife

The Chinese names alongside words such as ‘readmission’ and ‘wife’ suggested that this topic revolved around the administration of the White Australia Policy. This was easy to test. In A1-topics.txt was a list of every file in the series and their weightings in relation to each of the topics. I wasn’t sure what was a reasonable cut-off value to use in assessing the weightings, but after a bit of trial and error I fixed on a value of 0.7. I then just extracted the identifiers of every file that had a weighting greater than 0.7 for this topic. I used the identifiers to build a simple web page that Kate and I could browse. I also included links back to RecordSearch so we could explore further.

Browse the full list

It’s a pretty impressive result. Instead of fumbling with the uncertainties of keyword searches, we now have a list of more than 1,300 files that are clearly of relevance to Invisible Australians. There’s a few false positives and there are likely to be other files that we’ll have missed altogether, but now we have a much clearer picture of the types of files that are included and how they are described.

And that was at my first attempt, simply using the default settings. I’m now starting to play around with some of Mallet’s configuration options to see what sort of difference they make. I’m also keen to try out GenSim, a topic modelling package for Python.

I’m really excited about the possibilities of these sort of tools for analysing the contents of archival descriptive systems, something I mentioned in my Digital Humanities Australasia paper. Much more to come on this I suspect…

Earlier this week it was announced that the Mosman Library had been awarded a Library Development Grant for an innovative project that aims to document stories and artefacts relating to the First World War. I’m very excited to be part of it. As well as working with the local community in the creation of a new resource, the project offers an interesting opportunity to explore how we can link in with the ever-increasing volume of WWI material being published as linked data around the world.

But thinking about this new project has also made me reflect again on the creation of Mapping Our Anzacs — a project that still fills me with great pride and immense frustration. I thought I might as well finally post a couple of things I wrote about the project back in 2009. They’re a bit out of date, but I think there’s still a few useful lessons to be gleaned.

The first is a case-study that focuses on the crowdsourcing aspects of Mapping Our Anzacs. The second looks at the project as an example of a mashup. Thanks to Kate Theimer for initiating and publishing both pieces.

Bringing life to records

2009 preprint version of case-study originally published in Kate Theimer (ed), A Different Kind of Web: New Connections between Archives and Our Users, Society of American Archivists, Chicago, 2011. [Order here]

Overview of repository

The National Archives of Australia is responsible for preserving and making accessible the records of the Commonwealth of Australia. It employs more than 400 staff, with offices in Canberra and every state capital. Its holdings include more than 360 shelf km of records – around 69 million items. Through its digitisation program more than 1.6 million items have been fully digitised, making nearly 20 million digital images available online. The National Archives’ website now provides the main point of access for researchers, with more than 2 million images viewed through the online database RecordSearch in the year 2007–8.

Business drivers

Most people now experience the collections of the National Archives of Australia online. With an obligation to provide ‘an accessible, and interpreted, national archival collection’ the Archives is looking to new technologies to enhance access and improve efficiency.

The idea for Mapping our Anzacs arose during planning for a travelling exhibition on the impact of World War I, timed to coincide with the 90^th anniversary of the war’s end. Public interest in commemorating Australia’s war effort was as strong as ever, so a website that encouraged local participation seemed a useful way of extending the exhibition and its accompanying education program.

The major focus of both the exhibition and the website was to be the 376,000 service records documenting the experiences of Australian men and women during World War I held by the National Archives. These records had been fully digitised and described as part of a major project entitled ‘A Gift to the Nation’, but were still somewhat buried within our collection database.

Mapping our Anzacs was intended to highlight these records and open them up to local communities. First a map interface would allow service records to be discovered by place of birth or enlistment. Secondly, users would be able to add tributes – online versions of the war memorials that remain a feature of just about every town, large or small.

While the exhibition and the records themselves provided the main drivers for the project, there was also a growing desire within the institution to explore some of the possibilities of Web 2.0 technologies. This desire was tempered somewhat by a range of familiar concerns centred on issues of authority and control. Would user contributions detract from the reliability of the records? Who would take responsibility for any errors in user-created content? Would the potential for abuse demand vigilant moderation? Mapping our Anzacs gave us a chance to start working through such issues.

Setting the stage

We had an idea, a budget and a launch date, what we needed was a plan. While in theory we had around six months to play with, the project had to be fitted in around the ongoing work of our small web team. On the content side we had one person cleaning up the data. At the technical end we had someone connecting up the various components and making it all work within the Archives’ web environment. In the middle there were two of us trying to marry content and technology and create a usable resource. While we had a range of useful skills, none of us had tackled a project quite like this. We all had to learn on the job.

With few models or examples to work from, we began to experiment – researching available technologies, throwing around possibilities. Our first efforts were largely focused on the map interface and before long we had a working prototype using javascript and Google Maps. But what we also needed was a better understanding of how users might interact with the site.

World War I Honour Roll at the Chiltern Atheneum Museum

We started from the idea of the online memorial – a list of names compiled by users that would be linked through to service records. Our example was a local historical society creating a site to commemorate their community’s war effort. But what if they had more information – photographs or family histories – how could this sort of material be incorporated? Further inspiration came from a visit to the local historical museum in the small Victorian town of Chiltern. On one wall was a typical roll of honour, listing the names of those who had served in the war. But underneath were framed portraits of many of those listed. They were people, not just records. Could we create something like this online?

There were some exciting possibilities emerging, but concerns remained. Would anybody actually want to contribute? Strong interest in family history and a growing community desire to commemorate the experience of World War I offered anecdotal support. We just had to ensure that this interest could be translated into engagement – that the barriers of participation were low enough to encourage visitors to become collaborators.

But what of concerns that such material might detract from the authority of the records, or open our institution up to liability? We needed to make it clear where public contributions began and archival data ended.

Welcoming, but separate; open, but managed – a tricky balancing act was required. The answer, we decided was to create a separate ‘scrapbook’ using the blogging service Tumblr. The ‘scrapbook’ label was intended to be encouraging – this was not a database, or formal register, it was a place to leave your thoughts, comments, information or memorabilia. This was reinforced by our terms of service which simply required contributions to be relevant and respectful.

Scrapbook post

A ‘scrapbook’ was also something quite different to a finding aid. The informality helped to make the boundary clear between record and response. The separation was physical as well as intellectual. While the scrapbook shared many of the design elements of the main site, it was hosted by Tumblr not the National Archives. By using the Tumblr API, however, it was easy to pass information between the two sites. We could also use the API to provide a basic moderation facility.

But this meant that an important part of the site’s functionality would be dependent on an outside service. To make sure we considered fully all the implications of this, we developed a risk analysis and contacted Tumblr staff to inform them of our plans. Our major concern was simply the continuity of the service. While there could be no guarantees, we judged that this risk was manageable. Tumblr staff were interested in the project and offered their assistance if necessary.

Results

Images from scrapbook posts viewed via a media RSS feed in CoolIris.

On 25 April each year, Anzac Day, Australians remember the sacrifices made in war. Over the Anzac Day weekend in 2009, we were astonished to receive more than 200 scrapbook posts. Of course we had expected an increase in use, particularly after the site was featured on the Australian version of the Today Show, but this remarkable response certainly confirmed the site’s success. In the six months since its launch there had been almost 94,000 visitors to Mapping our Anzacs. More than 1,000 scrapbook posts had been contributed and 280 tributes created.

But the greatest success was in the type of posts being contributed rather than their sheer volume. Our ‘scrapbook’ had proved to be just that – as well as photographs of service people and their families, there were pictures of medals, headstones, letters, newspaper clippings, pay books, identity disks, diaries, postcards and certificates. Some people simply commented ‘my grandfather’, while others wrote detailed accounts of family history. Perhaps most moving were those who took the opportunity to leave a message for their loved one: ‘You were the best dad’.

Scrapbook post

Some have taken a systematic approach. Our most frequent contributor is gradually attaching photographs of headstones and memorial plaques that she has gathered from local cemeteries. Others are posting their own contact details in the hope of linking up with family. Perhaps most interesting are the notes that provide links to other people or documents – to family members, for example, or to a later service record. These are helping build a rich web of contextual data. Equally valuable are the corrections and additions that are being offered by eagle-eyed users, pointing out transcription errors or helping us track down elusive locations.

The success of the scrapbook has somewhat overshadowed the tributes, or online memorials, which really provided our starting point. Many tributes have been created and, as we had hoped, schools and other groups are using them to document the impact of war on their local communities. However, some compromises at the implementation stage have meant that it is not as easy to build them as we had hoped. There has also been some confusion by users between the tributes and the scrapbook. This is one area of the site we certainly hope to improve.

Even though the digitised service records had been available online for sometime though our collection database, it’s clear that many people are discovering them for the first time through Mapping our Anzacs. It was ‘a stunning find for me and my siblings’ wrote one grateful user. The scrapbook has aided discovery, providing another way into the records. Indeed, with the addition of a MediaRSS feed for CoolIris, the scrapbook provides two new entry points – one of them a 3D wall of faces and families. By embedding the records in these new contexts and making them easier to find, Mapping our Anzacs has successfully garnered extra value from an existing asset.

The site has also been recognised by others for its successful use of Web 2.0 technologies. We were pleased to be joint winners of the Best Archives on the Web Award, and surprised to be cited by the Federal Minister for Finance in a speech launching a Gov 2.0 taskforce. Recognition such as this has helped strengthen the case for future innovation in the Web 2.0 sphere.

Challenges

Success brings its own problems. One of the main challenges has been simply managing the sheer volume of posts and feedback. This was particularly acute of course after the Anzac Day deluge. As a result we have had to consider ways of streamlining our processes.

The Tumblr API allows us to set the status of a new post as ‘private’. We can then examine the post using the Tumblr dashboard before making it public. This works well enough as a basic form of moderation, however, the dashboard is not really designed for this purpose and it takes several clicks to release each post to the world.

But while moderation takes considerable time, it requires little intellectual effort. Despite concerns about abuse, our contributors have caused us few dilemmas. The only significant questions that have arisen concern the re-use of materials from other sources. This has made consider whether pre-emptive moderation is necessary or appropriate.

While the site includes detailed help information, it’s clear from the feedback that there are certain aspects that continue to cause difficulty. This provides useful data on how the site might be improved, but it has also made us think about how we communicate with our users. At the moment the content we provide is fairly static – there is no way of informing visitors of recent updates, or developing quick guides to common problems. If we took a more active approach to communication we might be able to decrease the number of help requests, while building a greater sense of community.

Similarly, while we have been excited by the number of corrections submitted by users, we can now see ways in which we might have structured the feedback process to capture their corrections more easily and efficiently. For example, a ‘submit a correction’ link on each individual’s page could automatically capture the person’s details, saving both us and our contributors from potential confusion.

We have suffered through the expected number of software glitches, and have a growing list of things we’d like to improve or develop, but overall the experience has been much more rewarding than painful.

Lessons Learned

Perhaps the most valuable lessons revolve around trust. Having entered into the project uncertain of what to expect from public participation, we have found ourselves in an evolving, creative partnership. Our users have defined what the scrapbook is and have taken an active role in improving and developing the resource. Our trust has been repaid many times over, helping us build something that in many ways has exceeded our expectations.

Trust is also necessary in the support of new ideas. Mapping our Anzacs was a very different type of project for the National Archives, challenging ideas both of access and user engagement. By taking the risk we have not only gained valuable publicity and user support, we have opened up the realm of possibilities for future development.

In terms of technology, the project demonstrated the power of the mashup and the efficiencies that can be gained by using existing web services. Tumblr, Google Maps and their associated APIs gave us a kickstart that enabled us to do a lot with a little.

Next Steps

There are so many exciting possibilities! Obviously our first priority is to improve those areas of the site that continue to cause our users grief. There are a number of navigation and usability tweaks that should improve the overall experience. Similarly, we can now see ways in which we might streamline moderation and management processes.

We hope to build on the success of the scrapbook and tributes by enhancing and extending their functionality. Improved editing and creation tools could assist contributors while also enriching the web of connections they build. We might, for example, provide widgets that make it easier to link the records of family members or friends. Over time this could develop into a complex network of relationships, providing new means of finding and visualising the records. Similarly, there are ways in which we might reuse the existing content of the scrapbook posts to develop new modes of discovery.

We could also do more to feature the labours and passions of our contributors. We could give them the option of exposing a public profile that lists all of their scrapbook posts. This would help foster a sense of community while providing yet another means of exploring connections between records.

Recent developments in geospatial technology and mobile devices perhaps offer the most exciting possibilities. Our original aim was to give the World War I service records back to local communities, to imbue the records with a greater sense of context, locality and belonging. Perhaps we will have succeeded when a tourist exploring a small country town can press a button on their mobile phone to retrieve a list of service people born near their current location.

Perhaps they will take a photo of a name on the local war memorial and use it to automatically retrieve that person’s service record or create an online tribute.

Perhaps they will come across a headstone in the local cemetery and immediately upload a geocoded photograph to the Mapping our Anzacs scrapbook.

Instead of merely being markers on a map, the records will start to overlay and inform the very spaces in which we move. The stories they contain will become part of our journeys, the people they document will have found their way home.

Creating a mashup

2009 preprint version of interview originally published in Kate Theimer, Web 2.0 Tools and Strategies for Archives and Local History Collections, Neal-Schuman, New York, 2010. [Order here]

What made you interested in creating a mashup?

It really started with the records. We hold the records of more than 375,000 World War I service people, identified by their places of birth and enlistment. With war memorials in just about every town across Australia, the connection between local communities and the memory of war remains strong. So we wondered how we could we give the service people in our records back to their communities. Having played around with the Google Maps API the answer seemed obvious — find the places, put them on a map and let people explore the connections for themselves.

What information, tools and processes did you need to begin?

The main thing we needed was the confidence to experiment. The process seemed straightforward in principle: first we had to extract the data we needed from the file titles in our collection database, then we had to find the latitude and longitude of each of the place names we extracted, and finally we had to plot these coordinates on a map with links back to details of the service records themselves. Web services, such as those provided by Google Maps, had the potential to do much of the work for us, and we scoured online documentation, user forums and blog posts for hints. But there were many things we could not know until we actually started. How consistent was our data? How many of the places would we be able to find? How would we be able to display thousands of places at once?

Moving from file titles through to coordinates obviously required a lot of data manipulation and we used Perl for much of the grunt work. Because our data set was large and variations in spelling and formatting were often unpredictable (including 13 different spellings of ‘lieutenant’!), we often had to work by trial and error — seeing what results we obtained and then adjusting our processes accordingly.

Once we had a list of place names in a consistent format we could begin to find latitudes and longitudes through a process known as geocoding. Google’s geocoding service was an easy option: it was well documented, reasonably comprehensive and it worked! We fed it our place names through a Perl script and soon we had a list of coordinates. Of course, many places were not found or returned multiple results, but the basic principle was sound. Our places were no longer just names, but points in space — we could begin making maps.

How did you determine what to include?

What we were creating was an archival finding aid, but one which placed the people, their homes and their communities up front rather than the systems that control their records. By browsing from a map a user would be able to find the details of a loved one, read a digitised copy of their service record and then follow a link through to our collection database. These links provide crucial context about the records, but we realised that this project also gave us an opportunity to capture other contexts and meanings. Who were these people? What did they look like? What happened to them after the war? By adding an online ‘scrapbook’ we gave users the chance to enrich the resource by adding notes or photographs about individuals.

This meant we had to deal with three sets of interlinked data: geocoded places, details from our records, and scrapbook posts provided by the public. To bring these all together with limited resources we had to make clever use of what was already out there. Why create our own maps when all you needed to do was write a bit of Javascript to embed a Google Map? Why build our own scrapbook application when the blogging service Tumblr provides free accounts and a simple API to manipulate posts? While a substantial amount of custom scripting was required to glue everything together, much of the core functionality was provided by free web services, available to anyone.

What challenges did you face?

Perhaps the first challenge to overcome was that of imagination. It was difficult for people to understand what the project was until we had a prototype to show them.

The process of handling and cleaning the data at times threatened to overwhelm us. While the geocoding service got us to the point where we could make maps, it also left us with many place names that needed to be manually checked. Often this was the result of misspellings in the original data, or because places either no longer existed or had changed their names. This data cleanup consumed much effort and continues still, though now with the help of our users who regularly point out errors and inconsistencies.

Once we had our coordinates we had to display them on a map without killing anyone’s computer. Showing thousands of markers on a Google Map is a challenge to slower web browsers and can end up hindering navigation. By dividing up our maps, clustering markers and changing the way they were rendered, we managed to greatly improve performance while maintaining the browsing experience. Once again it was trial and error coupled with the advice of the online community that guided us through the roadblocks.

What kinds of positive results have you had? (And any negative ones?)

From the messages we receive it’s clear that Mapping our Anzacs allows people to find records they didn’t know existed. Some have met a great-great-uncle for the first time. Others have learned about the war experience of a much-loved grandparent. Local communities have embraced the project and the scrapbook has developed into a rich and often moving resource. We wanted to give users a new way to explore and interact with our collection, and it seems we have succeeded.

Our users have also become our collaborators, providing corrections and comments that help us improve our data. They have extended the idea of the scrapbook, using it, for example, as a noticeboard for family history research, or as a way of creating crosslinks between related resources.

Success brings problems of its own and the work of moderating the scrapbook and responding to feedback has proved considerable. Issues with performance remain for people on slow connections, and while many are familiar with the Google Maps interface, some find it difficult to navigate. We are planning a number of enhancements based on this feedback, and hope to take advantage of the technology as it evolves to improve and extend the interface.

About how much time did it take?

While the project as a whole stretched over about eight months, much of this time was taken up cleaning and processing the data. The development of the interface was completed in under two months.

What advice would you give an organization wanting to use something similar?

Start experimenting. The technology is developing so rapidly that if you spend 12 months planning a project it’s likely to be out-of-date even before you start. New web services and data sources are becoming available every day. Perhaps you could use Open Calais to extract people’s names from a collection description, or MetaCarta to find the places. You might use the Google Books API to harvest the details of publications that cite your records. Even if you’re not a coder you can use tools like Yahoo Pipes to see what happens when you start to link data and services. Experimentation brings new ideas and possibilities. It’s all about making connections.

Yes, I have a suit. On 8 May at the National Library of Australia I gave my suit an outing as I delivered my Harold White Fellowship presentation. Thanks to everyone who came along.

If you missed it or want to relive the fun, the NLA has made a podcast available. My slides are also online, so you can follow along for the full audio-visual-not-quite-3D experience.

Use your arrow keys to navigate through the slides, and yes the first page is intentionally left blank. If you linger for a bit on slide two or three, you’ll see the Trove API in action. The presentation itself was constructed using deck.js.

The slides also include links to lots of different examples and demos, and introduce my new favourite plaything. I don’t really know what to call it yet, or what it’s actually for, but it makes me happy, and it makes me think. TF-IDF FTW. I’ll write up some more details shortly.

It seems a bit late to be introducing the newest version of QueryPic. Folks are already using it to explore the contents of digitised newspapers made available through Trove and Papers Past. Some, like the National Library of New Zealand, Andrew S. Bowman and the Carnamah Historical Society are already blogging about it. But I suppose I’d better document a few things…

As I noted in my post about QueryPicNZ (yes I now have a rather confusing proliferation of QueryPics), I was waiting for the Trove API to become public. Last week I noticed a little ‘API’ link pop up in the Trove footer and so I set to work…

"The past" versus "the future" in the new QueryPic

My original version of QueryPic (recently reviewed in the Journal of the Digital Humanities) used a series of Python scripts to harvest and scrape content from the Trove web pages. This meant that you had to download the scripts and be code-confident enough to run them in a terminal. It’s still a useful tool and I’ll be updating it as well, but I wanted to create something quicker and simpler that encouraged people to explore and play.

The latest version of QueryPic (QueryPic+, QueryPic Web, QueryPic 2.0?) simply runs in your browser. It uses JQuery to grab data on the fly from the Trove and DigitalNZ APIs. Like previous versions, it uses the HighCharts library to turn the data into pretty graphs.

What does it do? It’s really pretty basic. QueryPic just displays the number of articles matching your search query over time. By default, these are displayed as a proportion of the total articles available for that year, but a dropdown field lets you switch to view the raw numbers. It’s simple, but it’s also remarkably evocative, suggestive and fun. Just try it!

Why stop at just one query? To compare frequency patterns you can add as many as you like. Just keep entering new words or phrases.

If you notice an interesting peak or trough you can just click on it and another API request will be fired off to retrieve the first 20 matching articles. So it’s also a new way of exploring the newspaper databases themselves.

There are plenty of limitations — not all newspapers are digitised, for example, and the quality of the OCR is patchy. The National Library of New Zealand’s post does a great job summing up a number of issues relating to Papers Past. It’s not magic, it’s not perfect, but is it useful? I think so.

Tasks for the future:

• Create some sort of backend that makes it easy to save , share and cite your query data. The ‘share’ link just regenerates the graph which, of course, might change as new articles are added to the databases.
• Make it possible to add more complex queries — I want to keep the interface simple, so I’ll probably create a bookmarklet to take any Trove or Papers Past query and display it using QueryPic.
• As I mentioned over at the WraggeLabs Emporium, I intend to rewrite my various Trove tools to work with the new API. This will include the classic Python version of QueryPic. I still think it’s useful for harvesting your own data.

You may have noticed I have a bit on an interest in exploring ways of using digitised historical newspapers. In the last year or so I’ve spent a lot of time scraping, mining, processing and visualising content from the Trove collection of digitised Australian newspapers. But what about other countries?

Recently I was invited to a digital history workshop organised by Sydney Shep (@nzsydney) at the Victoria University of Wellington. In between sessions I started to play with the DigitalNZ API guided by Chris McDowall (@fogonwater). In anticipation of the forthcoming Trove API I’d already done a bit of work converting QueryPic to run in the browser. It didn’t take long to adapt this to work with New Zealand newspapers available through Papers Past.

So presenting for your enjoyment and education… QueryPicNZ.

Wind, rain and snow in QueryPicNZ

Like QueryPic, the New Zealand version graphs newspaper search results over time. But thanks to the DigitalNZ API it has a number of advantages:

• it runs in your browser — no need to download or run any scripts
• results appear almost instantly
• easy to combine queries — just search on a new word or phrase
• easy to remove queries — just use the ‘Clear last’ button
• easy to share — just copy the provided link or use the Tweet button

It’s limited to simple word or phrase searches at the moment, but eventually I’ll add the ability to process more sophisticated queries. I also want to add a way of saving, sharing and citing graphs. For now the ‘share’ link simply regenerates the graph, so if the content has changed the result could well be different.

The code is available on GitHub.

Ultimately, I want to combine Trove and Papers Past so that you can query and combine content from either Australia or New Zealand… perhaps even other countries?

In February I made a quick dash to Melbourne to talk at VALA2012.

The paper I originally submitted, ‘Mining the treasures of Trove: New approaches and new tools’, provided a general introduction to the use of digitised historical newspapers and the possibilities of digital history. You can download the pdf from the VALA2012 proceedings, or view online at Scribd.

I ended up presenting something a little different, focusing on my recent work around 1913 and extracting editorials from the Trove newspaper database. You can view the slides on Slideshare or watch a video of the whole presentation on the VALA2012 site.

By my own criteria I’ve already failed… I started this series of posts with the intention of documenting the process of finding and extracting editorials as I was actually doing the work. But here I am about to describe some work I finished a few weeks back. Oh well…

In my previous instalments (here and here), I focused on the Sydney Morning Herald. Having continued the hunt for missing editorials I started in the last post, I’ve now got a CSV file with the urls of the first editorial published in every edition of the SMH from 1913. Good-o, I thought, I can now start harvesting and analysing some content.

But then ensued a crisis of faith. The whole point of this exercise was to be able to build up some comparisons  – between newspapers, between states, between the city and the bush. But the process of actually finding the editorials seemed beset with difficulties. Could the rules I developed for the SMH be applied elsewhere? Could I ever assemble a useful set of editorials without large amounts of human intervention? I decided to try a few quick experiments to see whether the whole project was worth pursuing.

I started with a few assumptions:

1. The first (and only the first) editorial in any issue is headed with the name of the newspaper.
2. Editorials are published on even numbered pages.
3. Editorials vary in length between about 100 and 1500 words.

These assumptions were based on my own experience as a long-time newspaper researcher and on some preliminary poking around. For example, when I looked at The Argus I noticed that editorials were typically followed by news summaries. Unfortunately, these are treated as a single article in Trove, resulting in large blocks of text that are only part editorial. By specifying an upper word limit I hoped to filter these sorts of articles out. Similarly, there are sometimes brief announcements or publication details headed with the name of the newspaper. The lower word limit was intended to exclude these.

The next step was to harvest every article from 1913 that was headed with the name of its publication. I created a script to generate a list of all the newspapers that published issues in 1913. Then I called my existing harvester to download all the matching articles and save the details to a series of CSV files — one CSV file per newspaper.

In the previous instalment of this series I created a script to check the CSV output of my harvester for missing or duplicate dates. I extended this to perform a series of tests on each article based on the assumptions above. First, I filtered out articles on odd-numbered pages, then articles that were too short or too long. Finally I checked the remainder for missing or duplicate issue dates.

The details of the articles in each category were written out to JSON files. Using these files and a bit of JQuery magic I could quickly build a simple web interface that allowed me to explore the results.

Summary details of each newspaper

You can browse the summary results for the full list of newspapers, or you can drill down to view the actual articles assigned to each category.

Full details

I’ll save the full analysis for the next post, but if you play around with the results you quickly notice a few things. First, letters to the editor often include the name of the newspaper! If you look at The Mercury, for example, you’ll notice I’ve identified 1057 potential editorials — most of which are letters. Fortunately they should be fairly easy to filter out. In most cases the ‘even numbers only’ assumption worked pretty well, and the word length filters did remove quite a lot of false positives. There are still plenty of problems, but I’m encouraged enough to continue. Yes, there will be a Part #4!

2011 was a busy year. It’s hard to believe that it was only February when I first posted about my experiments mining the contents of the Trove newspaper database. Since then I’ve developed a set of digital tools, organised THATCamp Canberra, given a series of presentations on the possibilities of digital history, pushed ahead with Invisible Australians, and tried to develop my own digital research program. Oh yes, and endeavoured to earn enough money to feed the kids and pay the mortgage…

It looks like 2012 could be even busier, so before I lose track completely, I thought I’d pull together some of the past year’s exploits for handy reference. So here’s (most of) my presentations for 2011…

8 June 2011 — ‘Confessions of an impatient historian’
Scholars’ Lab, University of Virginia

• slides
• podcast

18 August — ‘Digital history: new tools and techniques’
National Museum of Australia

• links in Zotero

24 August — ‘Hacking the archives’
Archival description in an online world, Recordkeeping Roundtable, Sydney

• report

5 September 2011 — Digital research methods
Cultural heritage students, University of Canberra

• links in Zotero

14 September 2011 — ‘Every story has a beginning’
Keynote presentation at the Indexing See Change Conference (Australian and New Zealand Society of Editors)

• full text
• presentation

13 November 2011 — ‘Digital history: new tools and techniques’
Dragontails 2011: 2nd Australasian conference on overseas Chinese history & heritage, Museum of Chinese Australian History, Melbourne

• slides

30 November 2011 — ‘It’s all about the stuff’
National Digital Forum, Wellington, New Zealand

• full text
• video

7 December 2011 — ‘An introduction to digital history’
Digital December, State Library of NSW

• links in Google Docs

Videos from NDF2011 are now available online. Here’s the movie version of my talk It’s all about the stuff. I seem to spend a lot of time in the shadows…

Back when I was looking at ‘When did the Great War become the First World War?‘ I promised a detailed post on how I constructed the graphs. But of course I got distracted. Then I started adding new features to the script and redesigning the graphs, so…

Anyway, the result is a rather neat little gizmo henceforth named QueryPic (I got a bit sick of ‘search summariser’ and ‘graph-maker thing’). The first version just harvested data and left all the graph-making to you. But QueryPic does it all! It harvests the data and makes the graph. Woohoo.

Here’s an example showing ‘drought’ versus ‘flood’:

QueryPic features

• Explore your Trove newspaper query over time in the form of a simple line graph.
• Interactive — click on a point to retrieve sample articles from that date.
• Combine data sources to compare queries.
• Choose your interval — plot by year or month.
• Switch views between total results and the proportion of all articles.

Running QueryPic

Yes, it’s a Python script and yes it runs on the command line. Let’s get that out of the way now. I don’t think I have the time and energy to develop cross-platform gui versions of all my tools. I’d rather spend the time adding new features or exploring new possibilities. Sorry, but until I have a wealthy benefactor or a technical support team, I think that’s the way it has to be. In any case, the code is all there – so build your own gui!

Actually, if I did have the time and energy I don’t think I’d build a standalone gui anyway. What would be much cooler would be a web service, where people could run, share and combine their queries. Social graph-making! A celebration of serendipity! A historical playground! Hmmm…

But for now there’s this python script. It’s dead easy to use. Starting from the beginning…

1. Do you have Python installed? If you have a Mac or Linux the answer is yes. Fire up a terminal and type ‘python -V’ — see, I told you. If you have Windows you can get a handy installer. Do it.
2. Get the source code. Just download this zip file and open it into a new folder.
3. Open a terminal and cd into the new folder.
4. Run ‘python do_totals.py [your Trove query]‘.
5. Watch in excitement as the script chugs away retrieving data from Trove.
6. Once the script is finished, go to the ‘graphs’ directory, where you’ll find your newly-created html page complete with fancy interactive graph.
7. Open the html page in the web browser of your choice.
8. Enjoy! Celebrate! Drink a toast in my honour!

Customising QueryPic

There are a number of optional arguments that you add to the command line to customise your results:

-n (or –name) [a query name]
Give a name to your query. The name is used to create filenames for the html and data files, it is also used in the legend of the graph. The default is to use the search keywords as the name.

-d (or –directory) [a directory path]
The full pathname of the directory/folder for your results. The default is a ‘graphs’ sub-directory in the current directory.

-g (or –graph) [a graph name]
Specify the name of the html file that’s created. This is useful for displaying multiple queries on a single graph. Just run QueryPic for each query, using the same graph name each time. The default is either the value specified by the -n parameter or a name derived from the search keywords.

-m (or –monthly)
Plot the query at monthly intervals. The default interval is a year.

What QueryPic actually does

QueryPic builds a simple visualisation of your search query in the Trove newspaper database. A list of search results is difficult to interpret and offers little context. QueryPic shows you the number of articles matching your query over time, enabling you reframe your questions, pursue hunches, or simply play around.

QueryPic takes your Trove newspaper query and looks for a date range. If it doesn’t find one, it assumes you want your graph to go from 1803 to 1954 (the complete contents of the newspaper database — except for the Women’s Weekly). QueryPic then strips out any date parameters from the query, so it can fire off the query within the start and end dates, at the specified date interval.

Date interval? In the previous version of this script you could only plot points at yearly intervals, so it was impossible to zoom in an see what might be happening over the span of a single year or two. But amazing advances in QueryPic technology mean you can now plot changes by month. Here for example is a new version of my Great War/First World War graph, focused on 1938–1946 and plotted at monthly intervals.

So for each interval within the date range QueryPic fires off a request to Trove. From the response it scrapes out the total number of results for that date. If the total is greater than zero, it then fires off a second request to find the total number of newspaper articles for that year. Your query results divided by the total number of articles gives the proportion of articles for that date matching your search query.

The number of results and the proportion are written to a javascript file, together with some other important information including the original query and the date the harvest was performed. Remember, the Trove newspapers database is always changing! QueryPic then grabs a copy of it’s own special html template and inserts a reference to this javascript file. For good measure, it also inserts a link to your original query. The file is saved under a new name, ready for you to open and explore.

The html file contains everything necessary to take your data and turn it into a graph. It does this using the HighCharts javascript library. Please note, that while licence conditions allow HighCharts to be redistributed as part of a non-commercial package, it is not free for commercial use. Check the HighCharts website for details.

Some examples

Plot ‘cat’ against ‘dog’ in a graph called ‘animals’:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"
python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"

Specify a directory for your results:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -d "/User/bill/Documents/graphs"

Plot results at monthly intervals:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat&fromyyyy=1920&toyyyy=1921" -m

Specify a name:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -n "Felines"