Back when I was looking at ‘When did the Great War become the First World War?‘ I promised a detailed post on how I constructed the graphs. But of course I got distracted. Then I started adding new features to the script and redesigning the graphs, so…

Anyway, the result is a rather neat little gizmo henceforth named QueryPic (I got a bit sick of ‘search summariser’ and ‘graph-maker thing’). The first version just harvested data and left all the graph-making to you. But QueryPic does it all! It harvests the data and makes the graph. Woohoo.

Here’s an example showing ‘drought’ versus ‘flood’:

QueryPic features

• Explore your Trove newspaper query over time in the form of a simple line graph.
• Interactive — click on a point to retrieve sample articles from that date.
• Combine data sources to compare queries.
• Choose your interval — plot by year or month.
• Switch views between total results and the proportion of all articles.

Running QueryPic

Yes, it’s a Python script and yes it runs on the command line. Let’s get that out of the way now. I don’t think I have the time and energy to develop cross-platform gui versions of all my tools. I’d rather spend the time adding new features or exploring new possibilities. Sorry, but until I have a wealthy benefactor or a technical support team, I think that’s the way it has to be. In any case, the code is all there – so build your own gui!

Actually, if I did have the time and energy I don’t think I’d build a standalone gui anyway. What would be much cooler would be a web service, where people could run, share and combine their queries. Social graph-making! A celebration of serendipity! A historical playground! Hmmm…

But for now there’s this python script. It’s dead easy to use. Starting from the beginning…

1. Do you have Python installed? If you have a Mac or Linux the answer is yes. Fire up a terminal and type ‘python -V’ — see, I told you. If you have Windows you can get a handy installer. Do it.
2. Get the source code. Just download this zip file and open it into a new folder.
3. Open a terminal and cd into the new folder.
4. Run ‘python do_totals.py [your Trove query]‘.
5. Watch in excitement as the script chugs away retrieving data from Trove.
6. Once the script is finished, go to the ‘graphs’ directory, where you’ll find your newly-created html page complete with fancy interactive graph.
7. Open the html page in the web browser of your choice.
8. Enjoy! Celebrate! Drink a toast in my honour!

Customising QueryPic

There are a number of optional arguments that you add to the command line to customise your results:

-n (or –name) [a query name]
Give a name to your query. The name is used to create filenames for the html and data files, it is also used in the legend of the graph. The default is to use the search keywords as the name.

-d (or –directory) [a directory path]
The full pathname of the directory/folder for your results. The default is a ‘graphs’ sub-directory in the current directory.

-g (or –graph) [a graph name]
Specify the name of the html file that’s created. This is useful for displaying multiple queries on a single graph. Just run QueryPic for each query, using the same graph name each time. The default is either the value specified by the -n parameter or a name derived from the search keywords.

-m (or –monthly)
Plot the query at monthly intervals. The default interval is a year.

What QueryPic actually does

QueryPic builds a simple visualisation of your search query in the Trove newspaper database. A list of search results is difficult to interpret and offers little context. QueryPic shows you the number of articles matching your query over time, enabling you reframe your questions, pursue hunches, or simply play around.

QueryPic takes your Trove newspaper query and looks for a date range. If it doesn’t find one, it assumes you want your graph to go from 1803 to 1954 (the complete contents of the newspaper database — except for the Women’s Weekly). QueryPic then strips out any date parameters from the query, so it can fire off the query within the start and end dates, at the specified date interval.

Date interval? In the previous version of this script you could only plot points at yearly intervals, so it was impossible to zoom in an see what might be happening over the span of a single year or two. But amazing advances in QueryPic technology mean you can now plot changes by month. Here for example is a new version of my Great War/First World War graph, focused on 1938–1946 and plotted at monthly intervals.

So for each interval within the date range QueryPic fires off a request to Trove. From the response it scrapes out the total number of results for that date. If the total is greater than zero, it then fires off a second request to find the total number of newspaper articles for that year. Your query results divided by the total number of articles gives the proportion of articles for that date matching your search query.

The number of results and the proportion are written to a javascript file, together with some other important information including the original query and the date the harvest was performed. Remember, the Trove newspapers database is always changing! QueryPic then grabs a copy of it’s own special html template and inserts a reference to this javascript file. For good measure, it also inserts a link to your original query. The file is saved under a new name, ready for you to open and explore.

The html file contains everything necessary to take your data and turn it into a graph. It does this using the HighCharts javascript library. Please note, that while licence conditions allow HighCharts to be redistributed as part of a non-commercial package, it is not free for commercial use. Check the HighCharts website for details.

Some examples

Plot ‘cat’ against ‘dog’ in a graph called ‘animals’:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"
python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"

Specify a directory for your results:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -d "/User/bill/Documents/graphs"

Plot results at monthly intervals:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat&fromyyyy=1920&toyyyy=1921" -m

Specify a name:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -n "Felines"

As I explained in the first of this series, I’m documenting my efforts to extract every editorial published in the Sydney Morning Herald in 1913 from the Trove newspaper database. It’s an experiment both in text mining and historical writing — an attempt to put the method up front.

While I didn’t think there was anything very thrilling in the first instalment, recording my thoughts and assumptions in this way has already proved useful. In a comment, Owen Stephens noted that his attempt to reproduce my search query produced fewer results. After a little bit of poking around I realised that the fulltext modifier, which I often use to switch off fuzzy matching, counteracts the ‘search headings only’ flag. So my query was returning results that had the string ‘The Sydney Morning Herald’ anywhere in the article.

Try it for yourself.

Here’s my original query — searching for fulltext:”The Sydney Morning Herald” in headings only (supposedly). You’ll notice that it returns 335 results and it’s clear from a quick scan that a number are false positives (they don’t follow the pattern for editorials).

Here’s Owen’s query — searching for “The Sydney Morning Herald” in headings only. It returns 294 results, without any obvious false positives.

So my attempt to disable fuzzy matching actually produced a less accurate result! Weird.

Actually, I think one important benefit of this sort of text mining is that it helps you understand how the search engines you’re using actually work. Once you start poking and prodding, the idiosyncrasies start to emerge.

Anyway, I harvested Owen’s cleaner result set and opened up the resulting csv file. As it seemed in Trove, there we’re very few false positives. Indeed there were only two articles that didn’t seem to follow the standard editorial format, and these were notes added to the editorial page. On the other hand, there were obviously about 20 editorials missing. I could have manually worked through the csv file to identify the missing dates, but I thought I’d try to create some tools that would do the work for me.

What I wanted was the details of the first editorial in every edition of the newspaper in 1913 — so there should be one, and only one, article for each day on which the newspaper was published. I needed a tool that would analyse the csv file and do two things:

• identify dates that occur multiple times (false positive alert!)
• identify dates that are absent from the result set (missing in action!)

The resulting code is all on GitHub if you want follow along. I wrote a Python script that opens up the csv file, extracts all the date strings, converts them to datetime objects and then saves them to a list. Once that’s done it’s pretty easy to loop through and find duplicates:

def find_duplicates(list):
    '''
    Check a list for suplicate values.
    Returns a list of the duplicates.
    '''
    seen = set()
    duplicates = []
    for item in list:
        if item in seen:
            duplicates.append(item)
        seen.add(item)
    return duplicates

Finding missing dates was a little more complicated, but Google came to the rescue with some handy code samples. All I had to do was set a start and end date (in this case 1 January 1913 and 31 December 1913) and create a timedelta object equal to a day. Then it’s just a matter of adding the timedelta to the start date, comparing the new date to the dates extracted from the csv file, and continuing on until you hit the end. If the new date isn’t in the csv file, then it gets added to the missing list.

if year:
        start_date = datetime.date(year, 1, 1)
        end_date = datetime.date(year, 12, 31)
    else:
        start_date = article_dates[0]
        end_date = article_dates[-1]
    one_day = datetime.timedelta(days=1)
    this_day = start_date
    # Loop through each day in specified period to see if there's an article
    # If not, add to the missing_dates list.
    while this_day <= end_date:
        if this_day.weekday() not in exclude: #exclude Sunday
            if this_day not in article_dates:
                missing_dates.append(this_day)
        this_day += one_day

I’ve tried to make the code as reusable as possible, so you can either supply a year, or the script will read start and end dates from the csv file itself.

All that left me with two more lists of dates: ‘duplicates’ and ‘missing’. At first I just wrote these out to a text file, but then I decided it would be useful to write the results to an html page. That way I could add links that would take me to the actual issue within Trove, helping me to quickly find the missing editorial.

Unfortunately there’s no direct way to go from a date to an issue — you first need to find the issue identifier. How do you do this? If you dig around in the code beneath the page for each newspaper title, you’ll find that the ajax interface pulls in a json file with issue information. You can access this through a url like: http://trove.nla.gov.au/ndp/del/titlesOverDates/[year]/[month]. Here’s an example for January 1913.

The json includes all issues for all titles in the specified month. So you then have to loop through to find a specific title and day. Once you have the issue identifier you can just attach it to a url:

def get_issue_url(date, title_id):
    '''
    Gets the issue url given a title and date.
    '''
    year, month, day = date.timetuple()[:3]
    url = 'http://trove.nla.gov.au/ndp/del/titlesOverDates/%s/%02d' % (year, month)
    issues = json.load(urllib2.urlopen(url))
    for issue in issues:
        if issue['t'] == title_id and int(issue['p']) == day:
            issue_id = issue['iss']
    return 'http://trove.nla.gov.au/ndp/del/issue/%s' % issue_id

My results file with links to Trove

Finally, to save myself having to cut and paste the missing dates back into the csv file, I added a few lines to write them in automatically.

So now I have a handy little html page, complete with dates and links, that I’m working through to find all the missing editorials. All I need for the next stage are the urls for the editorial and the page on which it’s published. I’m just cutting and pasting these from the citation box in Trove into the csv file. Once this is done I can start trying to find all the editorials.

PS: I noted in my first post that one benefit in finding the editorials was that the main news articles usually appeared on the page after the editorials. I’ve been thinking some more about ways to identify ‘major’ news stories. Word length perhaps? But not always. Hmmm, but major stories do seem to be published at the top of the page. After a bit more poking around in the code I found that there’s a ‘y value’ assigned to each article that indicates its position on the page. So if I harvest all the articles on the page after the editorials and then rank them by their y values? Interesting…

In many of the presentations I’ve given in recent times I’ve managed to include a question raised by Tim Hitchcock in his chapter in The Virtual Representation of the Past. Tim asks:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

The idea of turning archival systems on their head to expose the people rather than the bureaucracy is what motivates Kate Bagnall and I in our attempts to make the Invisible Australians project into a reality.

Invisible Australians aims to liberate the lives of those who suffered under the restrictions of the White Australia Policy from the rich archival holdings of the National Archives of Australia and elsewhere.

We always knew that the portrait photographs, included on a range of government documents, would provide a compelling perspective on these lives, but we weren’t quite sure how we were going to extract them. Up until last weekend, I’d assumed that we’d develop a crowdsourcing tool that contributors would use to mark-up the photos.

Now I’m not so sure.

In the space of a couple of days I’ve extracted over 7,000 photographs and built an application to browse them — here is the real face of White Australia…

How did I do it? Paul Hagon, at the National Library of Australia, gave a presentation last year in which he explored the possibilities of facial detection in developing access to photographic collections. The idea lodged in my brain somewhere and a few days ago I started to poke around looking to see how practical it might be for Invisible Australians.

It didn’t take long to find a python script that used the OpenCV library to detect faces in photographs. I tried the script on a few of the NAA documents and was impressed — there were a few false positives, but the faces were being found!

So then the excitement kicked in. I modified the script so that instead of just finding the coordinates of faces it would enlarge the selected area by 50px on each side and then crop the image. This did a great job of extracting the portraits. I tweaked a few of the settings as well to try and reduce the number of false positives. Eventually, I developed a two-pass system that repeated the detection process after the image had been cropped and it’s contrast adjusted. This seemed to weed out a few more errors. You can find the code on GitHub.

Once the script was working I had to assemble the documents. I already had a basic harvester that would retrieve both the file metadata and digitised images for any series in the NAA database. Acting on Kate’s advice, I pointed it at series ST84/1 and downloaded 12,502 page images.

All I then had to do was loop the facial detection script over the images. Simple! The only problem was that my 3-year-old laptop wasn’t quite up to the task. As it’s CPU temperature rose and rose, I was forced to employ a special high-tech cooling system.

Keeping my laptop alive...

But after running for several hours, my faithful old laptop finally worked it’s way through all the documents. The result was a directory full of 11,170 cropped images.

The results

There were still quite a lot of false positives and so I simply worked my way through the files, manually deleting the errors. I ended up with 7,247 photos of people. That’s a strike rate of nearly 65% which seems pretty good. The classifier, which does the actual facial detection, was probably trained on conventional photographs rather than on the mixed-format documents I was feeding it.

Then it was just a matter of building a web app to display the portraits. I used Django for the backend work of managing the metadata and delivering the content, while the interface was built using a combination or Isotope, Infinite Scroll and FancyBox.

It’s important to note that the portraits provide a way of exploring the records themselves. If you click on a face you see a copy of the document from which the photo was extracted. A link is provided to examine the full context of the image in RecordSearch. This is not just an exhibition, it’s a finding aid.

What next? There are many more of these documents to be harvested and processed (and many more still yet to be digitised). I will be adding more series as I can (though I might have to wait until I can afford a new computer!). I’d also like to explore the possibilities of facial or object detection a bit more. Could I train my own classifier? Could I detect handprints, or even classify the type of form?

In the meantime, I think our experimental browser helps us to understand why the Invisible Australians project is so important — you look at their faces and you simply want to know more. Who are they? What were their lives like?

UPDATE: For more on the photos and the issues they raise, see Kate Bagnall’s posts over at the Tiger’s Mouth.

Townsville Daily Bulletin, 9 December 1939

I’m interested in time — in the way we imagine, manipulate, experience and describe time, particularly in the service of ideas such as ‘progress’.

This was one of the themes of Atomic Wonderland, but beyond constructing a few case studies it’s not all that easy to study. Or at least it wasn’t. Now projects such as Victorian Books are showing how we can explore the changing weights of ideas across times and cultures by analysing the contents of large textual collections.

Returning visitors will be probably be aware of my own experiments mining the contents of the National Library of Australia’s digitised newspapers database, available through Trove. So far I’ve focused on the development of generic tools and techniques, but I thought it would be interesting to apply these to my study of ‘progress’. Happily the NLA agreed and have awarded me a Harold White Fellowship for 2012 to do just that. Yippee!

I’ll be taking up the fellowship in February, but in preparation I’ve started to develop a few little sketches that prod at our fondness for periodisation. Labels such as ‘the Roaring Twenties’, ‘the Great Depression’ or even ‘the First World War’ are so familiar that we sometimes forget that they themselves have a history.

To begin with I decided to examine the question of when ‘the Great War’ became ‘the First World War’. At some point we realised that the Great War was not the final act in a centuries-long drama of European jealousy and jostling, but the first in a series of global conflicts. Can newspapers tell us when?

I already had a script that would generate a basic time series from a Trove query string. It simply takes the query, fires off a separate search for each year and grabs the number of matching articles. If the number of matches is more than zero, it also retrieves the total number of articles for that year and calculates the proportion matching the query. The results are saved in a json file which can be easily visualised using something like HighCharts. The original script needed a few tweaks to streamline the process, but I’ll describe these in detail in my next post.

For this experiment I constructed two queries. The first simply searched for the phrase ‘the great war‘ between 1900 and 1954. The second was a bit more complicated — it searched for any of the phrases ‘first world war’, ‘world war one’, ‘world war 1′ or ‘world war i’ across the same period. I fed the queries to my script and after a bit of ker-chugging, whirring and clunking I ended up with a graph.

Click to view the full interactive graph.

The result is not really surprising. As you can see on the full graph, the two lines cross late in 1941. With German victories across Europe and North Africa, the opening of the Eastern Front and, finally, the Japanese attack on Pearl Harbour, 1941 seems to make sense. But it’s interesting to see this reflected so clearly in such a rough and ready analysis.

What is perhaps more intriguing is the huge spike in 1939. Of course it makes sense that people would be referring back to the Great War as the prospect of a new conflict loomed, but it does make you wonder about the context of these discussions and how they might have developed as war edged closer.

Notable too are the earlier blips in the First World War count — the first centred on 1916 and the second on 1935. The peak in 1916 is actually due to the tags and comments added by Trove users. The standard ‘search everything’ option in Trove includes these as well as the text of the articles themselves. By using other search options you can choose to exclude the tags that match your query, but that seems rather messy. It would be nicer if Trove gave you the option of ignoring these matches from the start.

The West Australian, 24 May 1935

The second blip is a bit more interesting. By clicking on the graph and exploring the results from Trove, you can see that it’s due to the screening of a documentary film called ‘The First World War‘. The film used archival footage drawn from a number of nations and was based on Laurence Stalling’s book The First World War: A Photographic History. As one newspaper article noted: ‘this picture presents war, stripped of its gaudy trappings, and fearful in its grim reality’.

By way of comparison I tried a similar query using the Google Books Ngram viewer. The crossover point seems a little later, but of course books take longer to publish than newspapers. There is, however, no peak in 1939 for ‘the Great War’ — at least not if you use the combined ‘English’ corpus. If you examine the British-English and American-English corpora separately it’s a rather different story. Querying the British-English corpus produces something much closer to our Trove graph, complete with a spike around 1939. Again, this is only as we’d expect given the lesser significance of the First World War in American history.

This is, of course, only a sketch — something to prompt new questions or suggest avenues for attack. It’s made me want to find out a bit more about the nature of discussions in 1939, so I’ve fired up my Trove Newspaper Harvester and downloaded the text of all 6,582 articles from 1939 that include the phrase ‘the Great War’. More about that soon…

One of the advantages of building something yourself is that if you’re not happy with it you can tweak, change, modify and adapt until you are. But one of the disadvantages is that sometimes you get so caught up in all the tweaking, changing and adapting that you overlook a much simpler solution.

So I had a harvester that could save the publication details and content of all the newspaper articles in a search on Trove. But the warm glow of self-satisfaction quickly began to fade as I started to think about how I wanted to use the content I was harvesting.

The harvester saved the text of articles organised in directories by newspaper title. This seemed to make sense. It meant that you could easily analyse and compare the content of different newspapers. But what if you wanted to examine changes over time? In that case it’d be much easier if the articles were organised by year — then I could just pull out the a folder from a particular year, feed it to VoyeurTools, and start tracking the trends.

There ensued some minor tinkering. As a result, you can now you can pass an additional option to the harvest script, telling it whether to save the article texts and pdfs in directories by year or newspaper. Simply set the ‘zip-directory-structure’ option in harvest.ini to either ‘title’ or ‘year’. If you’re using the command-line you can use the ‘-d’ flag to set your preference. Easy.

But that set me wondering whether it might be possible to generate an overview, showing the number of articles matching a search over time. So I started on a modification of my harvest script that did just that — cycling through the search results, adding up the numbers. It wasn’t until I ran the new script for the first time that I realised there was a much simpler alternative.

All I needed to do was repeat the search for each year in the search span and grab the total results value from the page. D’uh…

So instead of sending hundreds or perhaps thousands of requests to Trove, all I needed was one for each year. From there it was easy and soon I had my first graph.

My first graph: Chinese in Australia (The Chinese Australian expert in my house predicted the 1888 peak.)

I was pretty pleased with that, but of course the raw numbers of articles on their own are rather misleading. The more interesting question was what proportion of the total number of articles for that year the search represents. Another quick tweak and I was grabbing the overall totals and calculating the proportions.

Total numbers versus proportions -- Chinese in Australia #2

At this point I invited my Twitter followers to suggest some possible topics — you can see the results on Flickr.

But what do the peaks and troughs represent? I wanted to use the graphs as a way of exploring the content itself. This was possible as I’d saved the data as JSON and used jqPlot to create the graphs in an ordinary HTML page. Courtesy of some clever hooks in the backend of jqPlot I could capture the value of any point as it was clicked. That gave me the year, so all I had to do was combine this with the search keyword values and send off a request to Trove.

So now instead of just looking at the graphs, you could explore them.

Explore -- Chinese in Australia #3

Perhaps you’re wondering how I managed to pull the Trove results into the page? Just a bit of simple AJAX magic combined with my own unofficial Trove API. (More about that in the next exciting installment!)

I’ve created a little gallery of graphs to explore. I’m still open to suggestions!

The code for gathering the data is all on Bitbucket, so start building your own. Just run the ‘do_totals.py’ script in the bin directory from the command line. The script takes two flags:

• -q (–query) the url of your Trove search (compulsory)
• -f (–filename) the path and filename for your data file (don’t include an extension)

The script will create a javascript file containing two JSON objects, ‘totals’ and ‘ratios’. These can then be fed to jqPlot. View the source of one of my interactive graphs to see how.

Of course it would be really nice to create a web service where people could create, share, compare and combine their graphs — but that might have to await a generous benefactor…

]]> http://discontents.com.au/shed/experiments/mining-the-treasures-of-trove-part-2/feed 9 http://discontents.com.au/shed/mining-the-treasures-of-trove-part-1 http://discontents.com.au/shed/mining-the-treasures-of-trove-part-1#comments Mon, 07 Feb 2011 15:07:10 +0000 tim http://discontents.com.au/?p=1088 Some time ago a well-meaning optometrist told me I had the eyes of a 60 year-old. I lay the blame for this premature ocular degeneration upon the many tiring hours I spent squinting at the screens of dodgy microfilm readers. Newspapers were a major source of my PhD research, and back then that meant learning [...]]]>

Some time ago a well-meaning optometrist told me I had the eyes of a 60 year-old. I lay the blame for this premature ocular degeneration upon the many tiring hours I spent squinting at the screens of dodgy microfilm readers. Newspapers were a major source of my PhD research, and back then that meant learning a little too much about films, spools and lenses. Not to mention the unending struggle to capture and hold the best machines.

Now it’s different. Instead of spending some weeks, as I did, sampling the Australian Womens Weekly in the hope of finding relevant articles, I can go to the National Library’s Australian Newspapers database in Trove and do a keyword search. Easy. The eyesight of future historians is safe.

But ready access to millions of newspaper articles across 150 years brings new challenges. Used to coaxing evidence from a meager array of sources, historians now, as Dan Cohen notes, have to ‘grapple with abundance’. How do we use and understand our new documentary riches?

Fortunately there are a growing array of tools to help. Zotero helps us manage our sources. Voyeur Tools brings sophisticated text analysis techniques within the grasp of all. And where the tools we need do not exist, we can make them.

So that’s what I did.

I’m interested in the way we talk about the weather. Wouldn’t it be good, I was thinking a few weeks back, if I could harvest the content of newspaper articles about weather or climate and start to analyse it — looking for patterns and shifts, mapping correlations or divergences against the actual climatic record.

Well… why not?

There’s currently no API that allows you to access Trove in this way, though one is apparently under development. But being the impatient sod that I am, I’d already built most of the parts I needed. Some time ago I wrote a screen scraper to query the newspaper database and extract article details from the returned HTML. This scraper is used in the History Wall to display random newspaper articles. It’s also sitting behind my little Headline Roulette game.

I’ve been continuing to improve and refine the scraper and recently used it to create my own API to Trove hosted on Google’s AppEngine. I’ll be posting some more details about this soon. And yes, I also developed a Zotero translator for the newspapers site, which I promise to finish off!

So to make a harvester, all I needed to do was run my scraper over all the results in a search and save them in some useful form. It took me less than an hour to develop a working prototype. Since then I’ve been adding a few bells and whistles…

The other night I harvested about 1400 articles that included the phrase climate change. The harvester had saved the text content of the articles in a zip file, so I uploaded it to Voyeur Tools. Here’s a simple word cloud:

(corpus)

Or what about the ‘atomic age’ seen through major Australian newspapers in 1945-46?

(corpus)

Hmmm, this is fun…

It’s harvest time

But all you really want to know is how to do it, right? So after that overlong introduction, here’s everything you need to know.

What does the harvester do?

You feed the harvester the url of a search you’ve constructed in Trove. The harvester then loops through all the results pages, extracting the article details. These details are saved in a CSV (comma separated values) file that you should be able to open as a spreadsheet or import into a database. The fields in the CSV file currently are:

• article id
• article title
• article url
• newspaper
• newspaper life dates and location
• newspaper id
• issue date
• page reference
• page url
• number of user corrections to the OCR output
• text of the article (including paragraph breaks)

In addition to the CSV file, the harvester can create two other data files for you. The first is a zip file that contains the text of all the articles, organised by newspaper and article. The internal structure of the zip is something like this:

[newspaper1 id]-[newspaper1 title]/
     [article1 id]-[article 1 issue date]-p[article1 page reference].txt
     [article2 id]-[article 2 issue date]-p[article2 page reference].txt
[newspaper2 id]-[newspaper2 title]/
     [article3 id]-[article 3 issue date]-p[article3 page reference].txt
     [article4 id]-[article 4 issue date]-p[article4 page reference].txt

For example:

35-The-Sydney-Morning-Herald/
     29765619-Friday-24-May-1946-p5.txt
     29763575-Friday-10-May-1946-p2.txt

Why is this useful? Once you have the texts organised in this format you can start feeding them to text-analysis programs. Voyeur Tools makes it easy, but there are other options like Mallet or NLTK. (Read about my first attempts at using NLTK over at NMA Labs.) As well as simple word frequencies and collocations you might want to investigate the possibilities of entity extraction, topic modelling or sentiment analysis.

The harvester also gives you the option of downloading a PDF version of every article. These are also saved in a zip file for convenience, with the same structure as above. Of course, if you’re harvesting a large number of articles this zip file might get very big.

Quick start (for the impatient)

If you just want to dive straight in here’s what you need to do:

1. If you don’t have it already, install Python (v.2.7 is recommended, but other versions should work ok)
2. Download my Trove Newspapers code
3. Unzip the TroveNewspapers file and put the contents somewhere handy.
4. Navigate to the trovenewspapers/bin directory.
5. Open the file ‘harvest.ini’ with a text editor (Notepad will do).
6. Change the default settings of ‘harvest.ini’ as instructed.
7. Save ‘harvest.ini’.
8. Find and run ‘do_harvest.py’.
9. Sit back and watch as your harvest chugs away.

The boring details (for the cautious)

Install Python

My scraper and harvester are written in Python, so you’ll need to have it installed on your system. If you’re working in Linux you should already have it. Downloads are available for all other platforms. For example, if you’re in Windows just download and run the Windows x86 MSI Installer.

Download my code

Now download my Trove Newspapers code. It’s avaiable as a zip file, so unzip it and put the contents somewhere you can find it again. The contents look something like this:

trovenewspapers/
     data/
     harvests/
     __init.py__
     harvest.py
     retrieve.py
     utilities.py
     LICENSE.txt

We’ll talk about the bin directory shortly. The data directory contains information about the newspaper holdings available through Trove. This is used by the scraper. If you ever want to update this data, you can use the save_titles function in utilities.py, but it’s not important for the harvester.

The harvests directory is empty, but if you start a harvest without specifying an output location, this is where it’ll end up.

BeautifulSoup is an extremely useful Python library for screen scraping. The scraper relies heavily on it, so I’ve included a copy in the package for your convenience.

The two files that do all the work are harvest.py and retrieve.py. Open them up in an editor and have a look if you’re interested. The scraper logic is all in retrieve.py, while harvest.py builds and runs the harvester.

But if all you want to do is start harvesting you can ignore all this and head straight to the bin top-level directory. Here you’ll find two files, do_harvest.py and harvest.ini. You’ll also find a README file which contains another version of these instructions and some added documentation.

Set your harvest options

Open up harvest.ini in any old text editor. You’ll see it contains some instructions and a series of configuration options with default values. If you run a harvest with out changing the defaults you’ll generate a fascinating set of 28 articles that contain the phrase ‘Inclement Wragge’.

The options you can set are:

1. query — the url of your Trove search
2. filename — where you want to save the CSV file
3. include-text — do you want to save the texts in a zip file (yes or no)?
4. include-pdf — do you want to save pdfs of the articles in a zip file (yes or no)?
5. start — the result number to start at (leave at 0 for a new harvest)

At this point you need to think — what do I actually want to harvest? Head over to the advanced search page for the newspapers database and start playing with the options until you get the results you want. Try to be as precise as possible — you don’t want to download lots of irrelevant articles.

Once you’re happy with your search, just copy the url in your browser’s location box. This url contains all the search parameters the harvester needs to find and process your results. Just paste the complete url into harvest.ini next to the ‘query’ option.

Set the filename option to tell the harvester where to save your CSV file. The harvester will use the filename you supply to build the filenames for the zip files (if you want them). If you don’t include a path the files will be saved in the bin directory. If you don’t set a filename, the harvester will create a default name — trove-newspapers-[timestamp].csv.

The ‘include-text’ and ‘include-pdf’ options should be pretty obvious. Set them to ‘yes’ if you want to save texts and pdfs, or ‘no’ if you don’t.

The ‘start’ option allows you to start your harvest at someplace other than the beginning of your results set. This is useful if your harvest is interrupted for any reason (more below). Just set it to the result number you want to start at.

Once your options are set, just save harvest.ini. It’s launch time!

Start your harvest

Remember the do_harvest.py file? It contains a little script that reads your configuration settings in harvest.ini and sends them off to the harvester. So to get things going all you need to do is run do_harvest.py.

How you actually do this depends a bit on your operating system and its settings. If you’re on Windows, then the python installation program should have told the OS to treat any file with a .py extension as a Python script. So you should just be able to double click it.

On Linux, the easiest way is to open up a terminal, cd to the bin directory and type ‘python do_harvest.py’.

That should be it. The script will let you know what’s going on, listing the articles as it processes them. Enjoy!

For lovers of the command line

The do_harvest script can also be run from the command line, with the various options supplied as arguments:

• -q (or –query) [full url of Trove newspapers search].
• -f (or –filename) [file and path name for the CSV output].
• -t (or –text) Create a zip file containing the text of articles.
• -p (or –pdf) Create a zip file containing pdfs of articles.
• -s (or –start) The result number to start at.

For example:

python do_harvest.py -q http://trove.nla.gov.au/newspaper/result?exactPhrase=inclement+wragge -f /home/wragge/trove-output.csv -t -p


Command line arguments will override any of the settings in harvest.ini.

If you’re using Windows you’ll need to make sure that the location of your Python
installation is included in your Windows path variable.

If something goes wrong

If there are problems at the Trove end, the harvester will take a little 10 second nap before trying again. It’ll do this 10 times before it finally gives up. Just before it dies, the script will write some details out to an error file ([your filename]_error.txt), including some instructions on what to do next.

This error file will include the number of the last completed record. Simply insert this as the ‘start’ value in harvest.ini (or include on the command line with the -s flag) and run do_harvest.py again. The harvester will spring back into life.

You’ll probably have a duplicate row in your CSV file at the point where the harvest failed, but that’s easy to delete.

What’s next?

Please have a go and let me know how you fare. You can add comments here, or raise issues over at my Bitbucket repository.

I’m thinking about building a little GUI version if there’s enough interest, and I have a few other improvements in mind.

I’ll be posting more about my adventures hacking Trove, and also about my efforts to analyse the results of my harvests (hence the part 1).

One of the things that’s keeping me busy at the moment is THATCamp Canberra. Yes, I got sick of missing out on all the THATCamp fun happening elsewhere and decided we should have our own.

THATCamp Canberra is a user-generated unconference on the digital humanities. It’ll be held at the University of Canberra on 28–29 August. We’re getting a great mix of applications and I’m really looking forward to learning about what’s going on around Australia.

Applications close on 23 July, so get yours in soon!

Some of you may have noticed that my Hacking a research project post featured a file from the National Archives of Australia embedded as a Cooliris widget. Huh? To jog your memory, here it is again:

These certificates allowed non-white Australians travelling overseas to re-enter the country. NAA: ST84/1, 1906/21-30

No, it’s not just an image, it’s a little 3D wall. You can pan and zoom to your heart’s content. You can enlarge an image, view fullscreen — you can even share an image via Twitter. Fun for all the family!

Regular viewers will recall my previous encounters with CoolIris — Archives in 3D and CoolIris enabled scrapbook — but these relied on having the CoolIris plugin installed. The embeddable Flash version wouldn’t work when the images were coming from the NAA because it upset Flash’s cross-domain settings.

So how did I get it to work? For various other projects I’ve been playing with simple image proxies using Python and Django, so I just applied the same principles. The image proxy makes it seem as if the images are coming from a local source, thus keeping Flash happy. Hurrah!

I’ve added a few little tweaks, so you can now view any digitised file in the National Archives of Australia in a CoolIris wall. Just go the the file browser page and enter a barcode. Even better you can install a bookmarklet. Just drag this link to your bookmarks bar (or save as a favourite) — View on wall. Then go to an item page in RecordSearch and click on the bookmarklet for 3D magic.

If you want to share a link to a file displayed in the 3D file browser, just use a url of the form:

http://wraggelabs.com/recordsearch/wall/[barcode]

— where [barcode] is fairly obviously the barcode of the file you want to view. For example:

• http://wraggelabs.com/recordsearch/wall/3445411/

If you want to embed one of the mini-walls in your blog post it’s easy. Just go to the CoolIris Express site and create your own wall. When it asks you for content source, click on ‘Media RSS’ and then in the ‘Feed URL’ box put:

http://wraggelabs.com/recordsearch/rss/[barcode]

– where [barcode] is… well, you know…

I think this a pretty interesting way to view, browse and navigate digitised files. Using Flash, rather than a browser plugin makes it more accessible, but I’d still rather have something based on open software and standards. I think it won’t be too long before we see something similar using Canvas and Javascript. That’ll be really exciting.

Amongst the holdings of the National Archives of Australia are some of the most visually arresting documents you’ll see — thousands and thousands of forms from the early decades of the twentieth century, each with a portrait photograph and palm print, each documenting the movements of a non-white resident. Along with many other certificates, regulations, correspondence and case files, these forms are part of the massive bureaucratic legacy of the White Australia Policy.

These certificates allowed non-white Australians travelling overseas to re-enter the country. NAA: ST84/1, 1906/21-30

But these are more than just interesting looking pieces of paper, they are snapshots of people’s lives. The forms capture data about an individual’s place of birth, physical characteristics and more. Over time a person might have submitted several of these forms, so by bringing them together we could trace their history, we could map their journeys — we could even watch them age.

The system which sought to render non-whites invisible has captured and preserved the outlines of their lives. By extracting and linking this data we could build a picture of another Australia, an Australia in which non-white residents lived, loved, struggled and succeeded, despite the impositions of a repressive regime.

I talked about these records at the AAHC conference last year, inspired in part by Tim Hitchcock’s chapter in the Virtual Representation of the Past. Tim Hitchcock argues that technology can allow us to restructure archives, looking beyond institutional hierarchies to the lives of individuals contained within:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

I don’t know, but I’d like to find out.

During my AAHC talk, Dave Lester suggested that the extraction of data from these forms might make a good crowdsourcing project. It’s a great idea. As you can see, the data is generally well-structured and legible, it should be possible to construct a simple series of forms that would allow volunteers to transcribe the data. The next stage would be to try and match identities across forms. That’s more complicated, but projects such as Tim Hitchcock’s London Lives show how users can construct identities by connecting a range of historical documents.

Then there are connections to resources outside of the archives — photographs, local histories, newspapers, genealogies, cemetery registers and more. By keeping our system open and extensible, and by working with others to help them expose their information in standard ways, it should be possible to develop the framework for an evolving mesh of biographical data.

So, how do we get started? This is the point when you usually have to start thinking about money — how can I fund this? In Australia that generally means a journey into the arcane world of the Australian Research Council. The ARC suffers from all the problems of a peer-reviewed system, but added to this is a rather antiquated notion of what research is.

In the rules covering each of the main schemes it’s clearly stated that the ‘compilation of data’ and the ‘development of research aids or tools’ are not supported. I spend part of my life working for the Australian National Data Service, an organisation that seeks to highlight how the sharing and reuse of data can open up new research possibilities. The ARC, however, seems to think that data has little value beyond its original research context.

Of course you can still mount a case for such activities. Applicants for a ‘Discovery’ grant can argue that data creation is integral to their project and provide details of the ‘specific research questions to be addressed’. But what if you don’t yet know what the questions are? Part of the point of a project such as this is to try and find out what questions we are able to ask. Until we start to compile, link and explore the data, the ‘specific research questions’ will be little more than convenient fictions, dreamt up to satisfy the prodding of peer reviewers.

Tom Scheinfeldt wrote a fantastic blog post recently, responding to concerns about the failure of many digital humanities projects to make arguments or answer questions. Drawing examples from the history of science, Tom argues:

we need to make room for both kinds of digital humanities, the kind that seeks to make arguments and answer questions now and the kind that builds tools and resources with questions in mind, but only in the back of its mind and only for later. We need time to experiment and even… time to play.

The ARC does not fund play.

You might imagine that the ARC’s infrastructure funding scheme would offer more hope for a project such as this. And yes, there are many worthy projects involving databases and online tools that have been supported in this way (and I have benefited from some of them!). But it seems that in the minds of research funders infrastructure is always BIG. Grants start at $150,000, and applications are expected to involve multiple institutional partners. Projects have to be scaled up to fit the ARC’s definition of infrastructure, often resulting in complex, lumbering, long-term projects whose products are out of date by the time of their release.

There is no room in our current infrastructure models for agile, innovative, user-focused digital toolmakers seeking small amounts to experiment with apps, prototypes, datasets or visualisations. I often look with envy upon the US National Endowment for the Humanities Digital Humanities Start-Up Grants.

In any case, neither I nor my partner in this endeavour, Kate Bagnall (@baibi), are currently in academic positions, so our chances of gaining any sort of research funding are next to none. We have the expertise — Kate has spent many years researching Australian-Chinese families and knows the records back-to-front, while I just can’t help playing with biographical data — but is that enough? How can you mount an ongoing research project without institutional support, research funding and the various badges and signifiers of academic authority?

I don’t know that either, but I have some ideas.

Mrs Ah Yin Pak Chong. NAA: ST84/1, 1907/321-330

I didn’t manage to get a contribution together for Dan Cohen and Tom Scheinfeldt’s crowdsourced-in-a-week book, Hacking the Academy, but watching the process from afar I did begin to wonder about how we might hack the way we build and run major research projects. This is what I have in mind:

• To strip down the large, lumbering beasts and design projects that are modular and opportunistic — able to grow quickly when resources allow, to bolt on related projects, to absorb existing tools.
• To follow the data freely across technological and institutional boundaries, developing open networks that invite participation and use.
• To develop a floating pool of collaborators, both inside and outside of academia, who are able to come and go, contributing whatever and whenever they can.
• To make everything public, accessible and standards-compliant, so that even if the project stalls it could be picked up and developed by someone else.

Most of all I just want to be able to do it. I don’t want to second-guess the ARC. I don’t want to spend months negotiating with potential partners or begging for an institutional home. I want to build, experiment and play. I want to make a start.

So that’s what we’re going to do.

We have a topic, plenty of raw materials, some basic principles and the beginnings of a plan. We even have a name — Invisible Australians: Living under the White Australia Policy.

As the project develops, I’ll be blogging here about some of the technical stuff, while Kate will be exploring the content over at the tiger’s mouth. I hope to have a prototype of the transcription tool ready to demo at THATCamp Canberra, while Kate is already at work putting together guides on using the records and developing an Omeka site that follows a number of Chinese-Australian families through the archives.

Can we hack together a major research project? Let’s find out.

The trained guinea pigs in the Wragge Labs bunker have been churning out all sorts of stuff in the last few months, and I’m way behind in my attempts to document their activities. So this is a bit of a catch-up post to try and commit a few pertinent details to the collective memory bank before they are lost forever in the sleep-deprived fog of day-to-day existence.

Identity upgrades

There have been a number of major improvements to Wragge’s Identity Browser. Regular viewers will recall that the Identity Browser is built on top of the People Australia SRU interface. You might not realise, however, that People Australia contains details of many organisations as well as people. We can only be thankful that it wasn’t called Entity Australia.

The first version of my Identity Browser only searched for people, but now all your corporate-entity-identification needs are also met, with only a few minor changes to the interface so-beloved by numerous generations of identity seekers. To be specific, through the wonders of drop-down technology you can choose whether you want to search for a person or an organisation. Or not. You can also just ignore that and search for everything and get back sensible results anyway. It’s your choice. Or not.

Gaze in awe at the power of my dropdown

Ah pattern matching… there are few phrases so redolent of warm summer days, hidden pleasures, and the subtle delights of wildcard characters. The People Australia SRU interface was sadly lacking in the pattern matching department, but this has now been rectified. So now you mix your stems and asterixes with wild abandon. Searching for ‘Curtin, J*’ will now retrieve all those Curtins whose names begin with ‘J’. Amazing isn’t it?

Astonishing too is the fact that the accompanying ‘Identify me!’ bookmarklet continues to function with nary a murmur of protest. There is, however, a little bit of cleverness built-in to enhance your bookmarklet experience. If the text that you highlight has a comma in it, the Identity Browser will conclude that you’re feeding it the name of a person – ie Surname, Firstname – and will treat the Firstname as a stem. So if you highlight ‘Whitlam, G’ and click on the bookmarklet, the Identity Browser will be kick-started into life, searching for everything that matches surname equals ‘Whitlam’ and firstname is like ‘G*’. If there’s no comma – ie firstname secondname – then it heads off to look for either a person whose surname equals ‘secondname’ and whose firstname is like ‘firstname*’, or an organisation whose name includes both ‘firstname’ and ‘secondname’. Got all that?

Basically the idea was to try and provide some sensible defaults so you really don’t have to think about it too much.

I have it in my head to prepare a long and rapturous homage to the wonders of machine tags. With their sly semantic ways and easy-going nature, they offer some exciting possibilities not just for user-generated content, but user-generated meanings and user-generated relationships. But for the full, ripe pleasure of that post you will have to wait another day, for now I shall simply say that as well as RDFa, the Identity Browser provides automagically-generated machine tags.

Where might you use them? Flickr’s a good place to start. Try identifying the subjects and creators of Flickr photos. At the NSW Reference and Information Services Group Seminar the other day I challenged those in attendance to go forth and machine tag. Already more than 100 machine tags have been added to Flickr using my Identity Browser. Expect to hear more about the Great Flickr Machine Tag Challenge soon…

One more thing… try adding ‘.rdf’ on to the end of an identity record – eg http://wraggelabs.com/identities/person/612109.rdf. Just an experiment at the moment…

More machine tag love

One night on Twitter, @lifeasdaddy pointed out that someone had started using fragments of urls from the NLA newspapers site as tags in the Powerhouse Museum’s collection database. In the conversation that ensued with @sebchan and others, I suggested that the PHM could encourage this sort of rich tagging by supporting machine tags, with all their wonderful juicy semantic goodness The guinea pigs got excited as well, and before I knew it, they’d constructed a little Semweb Helper app.

The Semweb Helper comes with its very own custom-tailored bookmarklet. If you find an article on the NLA newspapers site that you’d like to point to, just click on the bookmarklet and marvel as a range of useful machine tags are automagically generated. Then you just pick the appropriate tag, copy and paste et voila – instant semantic gratification.

Try out the Semweb Helper

It’s a very simple little app, and really just a demonstration of how semantic web technologies might be made available to the masses. It was also the first time the guinea pigs had been allowed to play with the Google Apps Engine.

Who am I?

This short catch-up post has become something quite long and rambling. Did I mention that I’m sleep-deprived? Anyway, a recent addition to the Wragge Labs range of lifestyle accessories is ‘Who am I?’ – a simple little game that is something like a cross between hangman and Wheel of Fortune. Choosing a person at random from People Australia and the Australian Dictionary of Biography, ‘Who am I?’ tests your powers of logic, stamina and historical guesstimation.

Your challenge is to figure out the surname of the mystery historical personage. To help you there are a series of clues, such as their birthplace and known associates. With each guess you also see a little bit more of their portrait. But beware! For ten wrong guesses are all that are permitted to any so brave as to enter upon this quest. Not eleven or twelve, but ten and ten only. To ignore this limit is to invite ridicule and disdain – do so at your peril.

Play Who am I?

‘Who am I’ builds upon some work I’ve been doing for the National Museum of Australia – looking at ways of mashing together various types of date-identified data. As part of that project I’ve built a series of APIs and have scraped, pummelled and munged data from a variety of sources.

What’s the point? I wonder this myself sometimes, particularly after I fling such things off into the aethernet and hear naught but a rare retweet. I am, after all, only in it for the glory, oh and the money of course. (Hmmm, I must look again at that business plan.) The point is twofold: first to highlight possibilities for the re-use and remixing of cultural data; second, to play with game-based models for discovery and exploration of cultural resources; and… err… thirdly just to try building something a little different.

Of course, if you like ‘Who am I?’ you will probably also want to try Headline Roulette…

Headline Roulette Reprieve

At the end of our last instalment, the future of Headline Roulette seemed in dire peril. Changes to the National Library of Australia web site threatened its very existence. Did it have a future? Could it survive? And did anybody care?

As we pick up the story oblivion looms. The feared changes are confirmed, but just as all seems lost… is it? Could it be? Yes, an advanced search facility is added to the newspapers site within Trove. Sensing this may be their only opportunity, the guinea pigs leap into action, building a new screen-scraper, saving Headline Roulette from doom, and setting the world upon the path to a safer, happier future.

In short, Headline Roulette will live on… so enjoy.

Handing out some presents

My head is easily turned by flattery and praise. Yes, I really am so shallow and so vain. But this means that if people say nice things to me, I’m inclined to give them presents.

As well as doing exciting things in the web 2.0 realm for the PROV, @asaletourneau leaves nice comments on this blog. So he earned himself a present. It’s not much, but I built a userscript that displays photos from the PROV site in a neat little slideshow (it’s the non-3D javascript version of CoolIris). Install Greasemonkey, get the userscript and try it out (just do a search, then click on the ‘Browse as slideshow’ button’).

PROV transport photos in a pretty slideshow

The State Library of NSW, or more specifically @ellenforsyth, also earned my favour by inviting me to rave on about Linked Data at the afore-mentioned NSW RISG seminar. As a result, I added support for the SLNSW photo collections to my Flickr Context Harvester userscript. Well… it’s the thought that counts, right? Once again – install Greasemonkey, get the userscript and then try it out.

The Flickr Context Harvester in action

And coming up…

Stay tuned for more on the Great Flickr Machine Tag Challenge, screencasts demonstrating my Identity Browser, some playing with relationships, and much much more. But right now the squirming baby on my lap needs a nappy change…

Did I mention that I’m sleep deprived?