In many of the presentations I’ve given in recent times I’ve managed to include a question raised by Tim Hitchcock in his chapter in The Virtual Representation of the Past. Tim asks:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

The idea of turning archival systems on their head to expose the people rather than the bureaucracy is what motivates Kate Bagnall and I in our attempts to make the Invisible Australians project into a reality.

Invisible Australians aims to liberate the lives of those who suffered under the restrictions of the White Australia Policy from the rich archival holdings of the National Archives of Australia and elsewhere.

We always knew that the portrait photographs, included on a range of government documents, would provide a compelling perspective on these lives, but we weren’t quite sure how we were going to extract them. Up until last weekend, I’d assumed that we’d develop a crowdsourcing tool that contributors would use to mark-up the photos.

Now I’m not so sure.

In the space of a couple of days I’ve extracted over 7,000 photographs and built an application to browse them — here is the real face of White Australia…

How did I do it? Paul Hagon, at the National Library of Australia, gave a presentation last year in which he explored the possibilities of facial detection in developing access to photographic collections. The idea lodged in my brain somewhere and a few days ago I started to poke around looking to see how practical it might be for Invisible Australians.

It didn’t take long to find a python script that used the OpenCV library to detect faces in photographs. I tried the script on a few of the NAA documents and was impressed — there were a few false positives, but the faces were being found!

So then the excitement kicked in. I modified the script so that instead of just finding the coordinates of faces it would enlarge the selected area by 50px on each side and then crop the image. This did a great job of extracting the portraits. I tweaked a few of the settings as well to try and reduce the number of false positives. Eventually, I developed a two-pass system that repeated the detection process after the image had been cropped and it’s contrast adjusted. This seemed to weed out a few more errors. You can find the code on GitHub.

Once the script was working I had to assemble the documents. I already had a basic harvester that would retrieve both the file metadata and digitised images for any series in the NAA database. Acting on Kate’s advice, I pointed it at series ST84/1 and downloaded 12,502 page images.

All I then had to do was loop the facial detection script over the images. Simple! The only problem was that my 3-year-old laptop wasn’t quite up to the task. As it’s CPU temperature rose and rose, I was forced to employ a special high-tech cooling system.

Keeping my laptop alive...

But after running for several hours, my faithful old laptop finally worked it’s way through all the documents. The result was a directory full of 11,170 cropped images.

The results

There were still quite a lot of false positives and so I simply worked my way through the files, manually deleting the errors. I ended up with 7,247 photos of people. That’s a strike rate of nearly 65% which seems pretty good. The classifier, which does the actual facial detection, was probably trained on conventional photographs rather than on the mixed-format documents I was feeding it.

Then it was just a matter of building a web app to display the portraits. I used Django for the backend work of managing the metadata and delivering the content, while the interface was built using a combination or Isotope, Infinite Scroll and FancyBox.

It’s important to note that the portraits provide a way of exploring the records themselves. If you click on a face you see a copy of the document from which the photo was extracted. A link is provided to examine the full context of the image in RecordSearch. This is not just an exhibition, it’s a finding aid.

What next? There are many more of these documents to be harvested and processed (and many more still yet to be digitised). I will be adding more series as I can (though I might have to wait until I can afford a new computer!). I’d also like to explore the possibilities of facial or object detection a bit more. Could I train my own classifier? Could I detect handprints, or even classify the type of form?

In the meantime, I think our experimental browser helps us to understand why the Invisible Australians project is so important — you look at their faces and you simply want to know more. Who are they? What were their lives like?

UPDATE: For more on the photos and the issues they raise, see Kate Bagnall’s posts over at the Tiger’s Mouth.

I’ve been doing a fair bit of coding in recent weeks and I thought I’d better write a few details down before I forget about them.

As previously noted, I’ve been gathering together various historical data sets for a project at the National Museum of Australia. One resource that I was keen on including was the fantastic Australian Newspapers project at the National Library of Australia. What I had in mind was being able to give a sense of context to any historical event by calling up the headlines for that particular time.

Unfortunately there’s no API for the newspapers project (or Trove in general), though apparently it’s in the works. So I had to reverse engineer the advanced search page to work out the various query options, and then build a screen scraper to harvest the results. I played around with the search options a bit to fine tune the results, finally deciding to limit them to ‘news’ articles with more than 1000 words. Annoyingly, only 10 results are returned at a time.

I had hoped to parse the results as xml, but a rogue <br> tag broke the XHTML, so I fell back on Beautiful Soup – a Python module that makes screen scraping considerably easier by tidying up HTML structures. After than it was pretty straightforward. Soon I had my own Python module to query the newspapers database and process the results.

The next step was to use the module to build a simple API that would let us quickly grab a set of headlines for a particular date and place. Django and Piston made this easy. To see headlines from Victoria on 1 January 1901, for example:

http://wraggelabs.com/api/newspapers/1901-01-01/nsw/

That was pretty cool and it started me thinking about what else I might do with the data. At first I was planning some sort of browser, like my Population Browser, but that seemed a bit boring. So I decided to create a simple game that grabbed a random headline and asked you to try and guess the date. After further refinement I decided to impose a limit of 10 guesses, with ‘higher’ or ‘lower’ prompts to get you moving in the right direction. Yes, basically it was a rip-off of The Price is Right – but an interesting, ironic and historically engaged rip-off…

This required me to make a change to the API and Python module so that I could retrieve a random headline. Basically it just meant generating a query based on random values for the day, month, year and state. For the interface I once again delved into JQuery’s box of tricks. With all the kerfuffle about ChatRoulette in the media, the name seemed obvious – Wragge’s Headline Roulette was born.

Test your historical nous with Headline Roulette!

It’s a very simple little app, but a number of people have said how much fun it is. The bad news is that imminent changes to the NLA newspapers site are probably going to break it (at least in its current form). So enjoy it while you can. When the NLA makes an API available I might work on something a little more sophisticated.

Of course, the broader point is that there are a whole range of cultural materials out there waiting to be remixed and re-used in various forms. Get hacking…

For a project that I’m working on at the National Museum of Australia, I’ve started collecting various sources of date-identified data. Most recently I had a go at extracting historical population data from the Australian Bureau of Statistics.

The data can all be downloaded as .xls files, but they’re not simple, flat spreadsheets – they’re data cubes. As the name suggests, data cubes are organised along a number of dimensions. In the case of the population data it’s year, state and gender.

This means that you can’t just export the data to CSV and suck it into your database – first you’ve got to flatten the cube. No doubt there are other ways to do this, but I just wrote a simple python script. It uses xlrd to read from the spreadsheet, does a bit or reorganisation, then writes the output to a CSV file. The code, for what it’s worth, is available at Bitbucket.

Once I had the CSV file I just imported it into MySQL and used Django and Piston to build a basic API. So if you want to know the population of NSW in 1856, you just go to:

http://wraggelabs.com/api/json/population/nsw/1856/

The number of infant deaths in Tasmania in 1932:

http://wraggelabs.com/api/json/infantdeaths/tas/1932/

The number of female births in Australia in 1959:

http://wraggelabs.com/api/json/births/australia/females/1959/

I’m sure you get the picture. You can change the ‘json’ to ‘xml’ if you’d like another flavour of data.

The API in action - a simple population browser

With an API delivering JSON you can start playing around with all sorts of fun AJAX-y stuff. To demonstrate I built a simple population browser using JQuery. Just drag the slider!

With a bit of time to play over Christmas I had a go at applying some of the techniques described at ProgrammingHistorian to the ADB Online.  I thought it might be interesting to create some word clouds, both for what they could reveal about the content of the ADB, and to see what they had to offer as a way of improving access to the articles.

So I set about learning Python and was soon downloading and scraping the more than 10,000 articles that make up the ADB online.

My first tests revealed that the most frequent words in ADB articles were…

born and died

Who’d have thought it? In a biographical dictionary?

After further refining the stopwords list I started to generate some useful clouds. Finally after 147 minutes of processing time, I had a word cloud representing the content of all 16 volumes of the Australian Dictionary of Biography.

The complete ADB word cloud

The words in the cloud are linked back to the ADB’s own search engine, allowing the cloud to be used as a way of exploring the articles themselves.

It shows the top 200 words, but if you want to see the rest you can download the raw word frequency file (>1mb txt file).

What can you see? Amongst other things, John is obviously the most popular name, Sydney just edges out Melbourne as the most popular place, and burial beats cremation as the most common mode of dispatch. It’s fun to explore.

But of course this then set me wondering about how these frequencies might change with the development of the ADB and changes in its subjects. So I generated word clouds for each volume and for each chronological series.

Word clouds by volume

Word clouds by series

I even added some simple Javascript slideshows so you could watch the clouds evolve.

One of the most obvious features in the series clouds is the gradual disappearance of ‘land’. It’s one of the most prominent words in the first series, but gradually fades until it disappears completely in the last.

After this successful foray into the world of word clouds, I began to think about other ways of visualising the ADB’s content. Many of the articles have portrait images, wouldn’t it be interesting to use the images themselves as the entry point to the biographical articles?

I’d already been playing with CoolIris, so I decided to harvest all the portrait references and use them to create a 3D wall. The result is pretty spectacular.

ADB portrait browser

Some technical details about the clouds and the portrait browser follow, for those interested in such things…

Gathering your words

Conveniently for me, ProgrammingHistorian uses the Dictionary of Canadian Biography as its main example, so there was much code that I could just cut and paste carefully examine and utilise. As the examples show, it’s easy to grab a webpage and analyse its content on the fly. But I wanted to process more than 10,000 pages and I knew that I was unlikely to get it working the first time round, so I decided to download the files first and then work on them locally. PH provided a basic example, to which I added some error-handling and the necessary loops to cycle through the ADB files. Because I had a bit of inside knowledge I cheated and hard-coded the numbers of articles in each volume. If I hadn’t known this I would have had to scrape all the browse pages, pulling out the links and creating a list in individual ids – not hard, but a bit tedious. Anyway this is how it ended up:

# download_adb.py

import urllib2, time, os, sys
import dh
items = (565, 575, 607, 526, 614, 533, 543, 723, 737, 742, 737, 759, 755, 721, 703, 714, 694, 126)
if os.path.exists('adb') == 0: os.mkdir('adb')

for v in range(0,18):
    for i in range (1,(items[v]+1)):
        if v == 0:
            filename = 'AS1%04db.htm' % i
        else:
            filename = 'A%02d%04db.htm' % (v, i)
        if os.path.isfile('adb/' + filename) == 0:
            print 'Processing: ' + filename
            url = 'http://adbonline.anu.edu.au/biogs/' + filename
            try:
                response = urllib2.urlopen(url)
            except IOError, e:
                if hasattr(e, 'reason'):
                    print 'We failed to reach a server.'
                    print 'Reason: ', e.reason
                elif hasattr(e, 'code'):
                    print 'The server couldn\'t fulfill the request.'
                    print 'Error code: ', e.code
            else:
                html = response.read()
                f = open('adb/' + filename, 'w')
                f.write(html)
                f.close
                time.sleep(2)
        else:
            print "File already downloaded"
        sys.stdout.flush()

Learning to count

Before too long I had a directory full of about 11,000 little html files just waiting for me to begin my evil experiments. First I had to slice them up and pull out all the interesting bits. By examining the code of the pages I could see that the main content was inside a div with the id of ‘content’. Using the Beautiful Soup Python library, I was easily able to extract this div. But the content div also usually included a portrait image and a bibliography. Once again I dipped into Beautiful Soup to discard all the unwanted bits. The slicing and dicing went something like this:

    g = open(dir + '/' + file, 'r')
    html = g.read()
    g.close()
    soup = BeautifulSoup(html)
    imagediv = soup.findAll(id="imagebox")
    if len(imagediv) > 0 :
        imagediv[0].extract()
    heading = soup.findAll('h4')
    if len(heading) > 0:
        heading[0].extract()
    footer = soup.findAll(id="selectbib")
    paras = footer[0].findNextSiblings('p')
    for para in paras:
        para.extract()
    footer[0].extract()
    content = soup.findAll(id="content")

Now I had the text of the article to play with. Following the PH examples it wasn’t long before I could extract word-frequency tables from a few files at a time. However, when I tried to process all the articles from a particular volume it took a verrry long time. I fiddled a bit with the code and amazed myself by dramatically improving the performance. I replaced the wordListToFreqDict function provided by PH with my own modified version:

def wordListToFreqDict2(wordlist):
    worddict = dict.fromkeys(wordlist)
    wordfreq = [wordlist.count(p) for p in worddict.keys()]
    return dict(zip(worddict,wordfreq))

The worddict = dict.fromkeys(wordlist) line made all the difference, creating a list of unique words that could then be checked against the full word list. With this hack in place I was able to process a complete volume in a few minutes.

I was already using a list of stopwords provided by PH to exclude things such as ‘such’ , ‘as’ and ‘and’, but obviously a few additions were necessary. To the list of stopwords I added:

stopwords += ['january', 'february', 'march', 'april', 'may', 'june', 'july', 'august', 'september', 'october', 'november', 'december']
stopwords += ['new', 'south', 'wales', 'australia', 'australian', 'victoria', 'south', 'western', 'queensland', 'tasmania']
#stopwords += ['sydney', 'melbourne', 'brisbane', 'adelaide', 'perth', 'hobart']
stopwords += ['died', 'born', 'life', 'lived', 'married', 'father', 'wife', 'children', 'son', 'sons', 'daughter', 'daughters', 'brother', 'brothers']
stopwords += ['street', 'st', 'year', 'years', 'months', 'acre', 'acres', 'ha']
stopwords += ['e', 'm', 'b', 'c', 'w', 'j', 'd', 'n', 'f', 'g', 'h', 'i', 'ii', 'l', 'o', 'p', 'th', 'r', 't', 'u', 'r', 'nd']

The first two lines should be pretty obvious. As you can see, I originally excluded names of the capital cities, but then realised that you could watch Sydney and Melbourne battle it out for pre-eminence, so I excluded the exclusion. Also out were family relations and various other words that turned up in almost every article. Cleaning out all the non-alphabetical characters from the text had left a lot of orphaned letters that had once been things like £ signs, so I had to dispose of them as well.

The modules for actually generating the clouds were mostly just copied from PH with a few minor changes. My complete script is here:

# adb-text-count.py

import urllib2
import dh, os, sys, time
from BeautifulSoup import BeautifulSoup
start = time.time()
print "Started at: ", time.asctime(time.localtime(start))
dir = 'adb'
filelist = dh.getFileNames(dir)

f = open('wordlist.txt', 'w')
for file in filelist:
    print 'Processing ' + file
    sys.stdout.flush()
    g = open(dir + '/' + file, 'r')
    html = g.read()
    g.close()
    soup = BeautifulSoup(html)
    imagediv = soup.findAll(id="imagebox")
    if len(imagediv) > 0 :
        imagediv[0].extract()
    heading = soup.findAll('h4')
    if len(heading) > 0:
        heading[0].extract()
    footer = soup.findAll(id="selectbib")
    paras = footer[0].findNextSiblings('p')
    for para in paras:
        para.extract()
    footer[0].extract()
    content = soup.findAll(id="content")
    text = dh.stripTags(str(content[0]))
    fullwordlist = dh.stripNonAlpha(text.lower())
    wordlist = dh.removeStopwords(fullwordlist, dh.stopwords)
    f.write(" ".join(wordlist))
f.close
f = open('wordlist.txt')
words = f.read()
f.close
wordlist = words.split(" ")
dictionary = dh.wordListToFreqDict2(wordlist)
sorteddict = dh.sortFreqDict(dictionary)
f = open('wordfreqs.txt', 'w')
for s in sorteddict: f.write(str(s)+"\n")
f.close
print 'Dictionary created'
sys.stdout.flush()
# create tag cloud and open in Firefox
cloudsize = 200
maxfreq = sorteddict[0][0]
minfreq = sorteddict[cloudsize][0]
freqrange = maxfreq - minfreq
outstring = ''
resorteddict = dh.reSortFreqDictAlpha(sorteddict[:cloudsize])
print 'Creating cloud'
sys.stdout.flush()
for k in resorteddict:
    kfreq = k[0]
    klabel = k[1]
    klabel = dh.undecoratedHyperlink('http://adbonline.anu.edu.au/scripts/adbp-ent_search.php?ranktext=' + k[1] + '&search=Go!', k[1])
    scalingfactor = (kfreq - minfreq) / float(freqrange)
    outstring += ' ' + dh.scaledFontSizeSpan(klabel, scalingfactor) + ' '
dh.wrapStringInHTML("html-to-tag-cloud", dh.defaultCSSDiv(outstring), "Complete")
finish = time.time()
print "Finished at: ", time.asctime(time.localtime(finish))
print "Total time: ", finish - start

Biographies in 3D

To display all the portrait images in CoolIris I had to harvest all the image details and then write them to a Media RSS file for CoolIris to read.

Extracting the details of all the thumbnail versions of the portraits in the ADB was easy using Beautiful Soup. But I also need the paths to the larger versions of the portraits stored on the sites of the repositories that hold the originals. All of these sites present the images differently, so a different scraper was needed for each of them. As yet I’ve only included major libraries and archives – I may add some more if I get the time.

Once the paths to the thumbnails and large versions had been harvested, it was just a matter of writing the RSS feed. Actually, I created a series of RSS files, one for each volume, linked using ‘rel=previous’ and ‘rel=next’ attributes. This helped speed up the loading of the gallery. For what it’s worth, the complete code is here:

# adb-portraits.py

import socket, urllib2, urllib
import dh, os, sys, time, re
from BeautifulSoup import BeautifulSoup
# timeout in seconds
timeout = 20
socket.setdefaulttimeout(timeout)
start = time.time()
print "Started at: ", time.asctime(time.localtime(start))
dir = 'adb'
for i in range(8,18):
    if (i == 17): vol = "AS1"
    else: vol = "A%02d" % i
    filelist = dh.getFileNamesByVol2(dir, vol)
    f = open('adb-portraits-%s.rss' % i, 'w')
    f.write("<?xml version='1.0' encoding='utf-8' standalone='yes'?>\n")
    f.write("<rss version='2.0' xmlns:media='http://search.yahoo.com/mrss/' xmlns:atom='http://www.w3.org/2005/Atom'>\n")
    f.write("<channel>\n")
    f.write("\n")
    f.write("<description>Portraits of individuals included in the Australian Dictionary of Biography</description>\n")
    f.write ("
<link>http://www.adb.online.anu.edu.au</link>\n")
    if (i > 1):
        f.write ("<atom:link rel='previous' href='adb-portraits-%s.rss' />" % (i-1))
    if (i < 17):
        f.write ("<atom:link rel='next' href='adb-portraits-%s.rss' />" % (i+1))
    for file in filelist:
        print str(file)
        sys.stdout.flush()
        g = open(dir + '/' + file, 'r')
        html = g.read()
        g.close()
        #print html
        sys.stdout.flush()
        soup = BeautifulSoup(html)
        imagediv = soup.findAll(id="imagebox")
        if len(imagediv) > 0 :
            print "Found an image"
            sys.stdout.flush()
            links = imagediv[0].findAll('a')
            if len(links) > 1:
                link = urllib.unquote(links[(len(links)-1)]['href'][31:])
            else:
                link = urllib.unquote(links[0]['href'][31:])
            print link
            sys.stdout.flush()
            try:
                response = urllib2.urlopen(link)
            except IOError, e:
                if hasattr(e, 'reason'):
                    print 'We failed to reach a server.'
                    print 'Reason: ', e.reason
                elif hasattr(e, 'code'):
                    print 'The server couldn\'t fulfill the request.'
                    print 'Error code: ', e.code
            else:
                id = str(file)[:7]
                thumbnail = 'http://www.adb.online.anu.edu.au' + imagediv[0].img['src'].lstrip('.')
                # print thumbnail
                title = imagediv[0].p.contents[0].split(',')[0].strip().replace(' - ', '-')
                title = title.encode('utf-8')
                print "Processing: " + title
                sys.stdout.flush()
                html = response.read()
                imgsoup = BeautifulSoup(html)
                if (link.find('sl.nsw') > -1):
                    if (link.find('ebindshow.pl') == -1): # Not thumbnail pages - see John Bingle
                        if (html.find('Higher quality image') != -1):
                            img = imgsoup.findAll(alt="Higher quality image")[0].parent['href'].split('?')[1]
                            #img = imgsoup.td.a['href'].split('?')[1]
                        else:
                            img = imgsoup.table.findAll('tr')[2].img['src']
                        repository = "State Library of NSW"
                elif (link.find('slv.vic') > -1):
                    img = imgsoup.findAll(id='ImageDisplay')[0].img['src']
                    repository = "State Library of Victoria"
                elif (link.find('slsa.sa') > -1):
                    img = imgsoup.findAll('td')[1].img['src']
                    img = link[:link.rfind('/')+1] + img
                    repository = "State Library of SA"
                elif (link.find('nla.gov') > -1):
                    img = link + '-v'
                    repository = "National Library of Australia"
                elif (link.find('naa.gov') > -1):
                    barcode = link[link.rfind('=')+1:]
                    img = "http://naa16.naa.gov.au/rs_images/ShowImage.php?B=%s&#038;T=P" % barcode
                    repository = "National Archives of Australia"
                elif (link.find('territorystories.nt.gov') > -1):
                    img = imgsoup.table.img['src']
                    repository = "Northern Territory Library"
                elif (link.find('statelibrary.tas.gov') > -1):
                    if (html.find('No matches were found') == -1):
                        img =imgsoup.blockquote.img['src']
                        repository = "State Library of Tasmania"
                elif (link.find('slq.qld.gov') > -1):
                    img = imgsoup.findAll(attrs={"class":"pictureback"})[0].a['onclick']
                    #img = img[img.find('http'):img.find(]
                    img = re.search('http://[\w\d\/\.]*.jpg', img).group()
                    repository = "State Library of Queensland"
                if (len(img) > 0):
                    f.write("<item>\n")
                    f.write("<guid isPermaLink='false'>%s</guid>\n" % id)
                    f.write("\n" % (title, repository))
                    f.write("
<link>http://www.adb.online.anu.edu.au/biogs/%sb.htm</link>\n" % id)
                    f.write("<media:thumbnail url='%s' />\n" % thumbnail.replace('&#038;','&amp;'))
                    f.write("<media:content url='%s' type='image/jpeg' />\n" % img.replace('&#038;','&amp;'))
                    f.write("</item>\n")
                    f.flush()
                    print "Success!"
                    sys.stdout.flush()
                img = ""
    f.write("</channel>\n")
    f.write("</rss>\n")
    f.close()