The trained guinea pigs in the Wragge Labs bunker have been churning out all sorts of stuff in the last few months, and I’m way behind in my attempts to document their activities. So this is a bit of a catch-up post to try and commit a few pertinent details to the collective memory bank before they are lost forever in the sleep-deprived fog of day-to-day existence.

Identity upgrades

There have been a number of major improvements to Wragge’s Identity Browser. Regular viewers will recall that the Identity Browser is built on top of the People Australia SRU interface. You might not realise, however, that People Australia contains details of many organisations as well as people. We can only be thankful that it wasn’t called Entity Australia.

The first version of my Identity Browser only searched for people, but now all your corporate-entity-identification needs are also met, with only a few minor changes to the interface so-beloved by numerous generations of identity seekers. To be specific, through the wonders of drop-down technology you can choose whether you want to search for a person or an organisation. Or not. You can also just ignore that and search for everything and get back sensible results anyway. It’s your choice. Or not.

Gaze in awe at the power of my dropdown

Ah pattern matching… there are few phrases so redolent of warm summer days, hidden pleasures, and the subtle delights of wildcard characters. The People Australia SRU interface was sadly lacking in the pattern matching department, but this has now been rectified. So now you mix your stems and asterixes with wild abandon. Searching for ‘Curtin, J*’ will now retrieve all those Curtins whose names begin with ‘J’. Amazing isn’t it?

Astonishing too is the fact that the accompanying ‘Identify me!’ bookmarklet continues to function with nary a murmur of protest. There is, however, a little bit of cleverness built-in to enhance your bookmarklet experience. If the text that you highlight has a comma in it, the Identity Browser will conclude that you’re feeding it the name of a person – ie Surname, Firstname – and will treat the Firstname as a stem. So if you highlight ‘Whitlam, G’ and click on the bookmarklet, the Identity Browser will be kick-started into life, searching for everything that matches surname equals ‘Whitlam’ and firstname is like ‘G*’. If there’s no comma – ie firstname secondname – then it heads off to look for either a person whose surname equals ’secondname’ and whose firstname is like ‘firstname*’, or an organisation whose name includes both ‘firstname’ and ’secondname’. Got all that?

Basically the idea was to try and provide some sensible defaults so you really don’t have to think about it too much.

I have it in my head to prepare a long and rapturous homage to the wonders of machine tags. With their sly semantic ways and easy-going nature, they offer some exciting possibilities not just for user-generated content, but user-generated meanings and user-generated relationships. But for the full, ripe pleasure of that post you will have to wait another day, for now I shall simply say that as well as RDFa, the Identity Browser provides automagically-generated machine tags.

Where might you use them? Flickr’s a good place to start. Try identifying the subjects and creators of Flickr photos. At the NSW Reference and Information Services Group Seminar the other day I challenged those in attendance to go forth and machine tag. Already more than 100 machine tags have been added to Flickr using my Identity Browser. Expect to hear more about the Great Flickr Machine Tag Challenge soon…

One more thing… try adding ‘.rdf’ on to the end of an identity record – eg http://wraggelabs.com/identities/person/612109.rdf. Just an experiment at the moment…

More machine tag love

One night on Twitter, @lifeasdaddy pointed out that someone had started using fragments of urls from the NLA newspapers site as tags in the Powerhouse Museum’s collection database. In the conversation that ensued with @sebchan and others, I suggested that the PHM could encourage this sort of rich tagging by supporting machine tags, with all their wonderful juicy semantic goodness The guinea pigs got excited as well, and before I knew it, they’d constructed a little Semweb Helper app.

The Semweb Helper comes with its very own custom-tailored bookmarklet. If you find an article on the NLA newspapers site that you’d like to point to, just click on the bookmarklet and marvel as a range of useful machine tags are automagically generated. Then you just pick the appropriate tag, copy and paste et voila – instant semantic gratification.

Try out the Semweb Helper

It’s a very simple little app, and really just a demonstration of how semantic web technologies might be made available to the masses. It was also the first time the guinea pigs had been allowed to play with the Google Apps Engine.

Who am I?

This short catch-up post has become something quite long and rambling. Did I mention that I’m sleep-deprived? Anyway, a recent addition to the Wragge Labs range of lifestyle accessories is ‘Who am I?’ – a simple little game that is something like a cross between hangman and Wheel of Fortune. Choosing a person at random from People Australia and the Australian Dictionary of Biography, ‘Who am I?’ tests your powers of logic, stamina and historical guesstimation.

Your challenge is to figure out the surname of the mystery historical personage. To help you there are a series of clues, such as their birthplace and known associates. With each guess you also see a little bit more of their portrait. But beware! For ten wrong guesses are all that are permitted to any so brave as to enter upon this quest. Not eleven or twelve, but ten and ten only. To ignore this limit is to invite ridicule and disdain – do so at your peril.

Play Who am I?

‘Who am I’ builds upon some work I’ve been doing for the National Museum of Australia – looking at ways of mashing together various types of date-identified data. As part of that project I’ve built a series of APIs and have scraped, pummelled and munged data from a variety of sources.

What’s the point? I wonder this myself sometimes, particularly after I fling such things off into the aethernet and hear naught but a rare retweet. I am, after all, only in it for the glory, oh and the money of course. (Hmmm, I must look again at that business plan.) The point is twofold: first to highlight possibilities for the re-use and remixing of cultural data; second, to play with game-based models for discovery and exploration of cultural resources; and… err… thirdly just to try building something a little different.

Of course, if you like ‘Who am I?’ you will probably also want to try Headline Roulette…

Headline Roulette Reprieve

At the end of our last instalment, the future of Headline Roulette seemed in dire peril. Changes to the National Library of Australia web site threatened its very existence. Did it have a future? Could it survive? And did anybody care?

As we pick up the story oblivion looms. The feared changes are confirmed, but just as all seems lost… is it? Could it be? Yes, an advanced search facility is added to the newspapers site within Trove. Sensing this may be their only opportunity, the guinea pigs leap into action, building a new screen-scraper, saving Headline Roulette from doom, and setting the world upon the path to a safer, happier future.

In short, Headline Roulette will live on… so enjoy.

Handing out some presents

My head is easily turned by flattery and praise. Yes, I really am so shallow and so vain. But this means that if people say nice things to me, I’m inclined to give them presents.

As well as doing exciting things in the web 2.0 realm for the PROV, @asaletourneau leaves nice comments on this blog. So he earned himself a present. It’s not much, but I built a userscript that displays photos from the PROV site in a neat little slideshow (it’s the non-3D javascript version of CoolIris). Install Greasemonkey, get the userscript and try it out (just do a search, then click on the ‘Browse as slideshow’ button’).

PROV transport photos in a pretty slideshow

The State Library of NSW, or more specifically @ellenforsyth, also earned my favour by inviting me to rave on about Linked Data at the afore-mentioned NSW RISG seminar. As a result, I added support for the SLNSW photo collections to my Flickr Context Harvester userscript. Well… it’s the thought that counts, right? Once again – install Greasemonkey, get the userscript and then try it out.

The Flickr Context Harvester in action

And coming up…

Stay tuned for more on the Great Flickr Machine Tag Challenge, screencasts demonstrating my Identity Browser, some playing with relationships, and much much more. But right now the squirming baby on my lap needs a nappy change…

Did I mention that I’m sleep deprived?

With a bit of time to play over Christmas I had a go at applying some of the techniques described at ProgrammingHistorian to the ADB Online.  I thought it might be interesting to create some word clouds, both for what they could reveal about the content of the ADB, and to see what they had to offer as a way of improving access to the articles.

So I set about learning Python and was soon downloading and scraping the more than 10,000 articles that make up the ADB online.

My first tests revealed that the most frequent words in ADB articles were…

born and died

Who’d have thought it? In a biographical dictionary?

After further refining the stopwords list I started to generate some useful clouds. Finally after 147 minutes of processing time, I had a word cloud representing the content of all 16 volumes of the Australian Dictionary of Biography.

The complete ADB word cloud

The words in the cloud are linked back to the ADB’s own search engine, allowing the cloud to be used as a way of exploring the articles themselves.

It shows the top 200 words, but if you want to see the rest you can download the raw word frequency file (>1mb txt file).

What can you see? Amongst other things, John is obviously the most popular name, Sydney just edges out Melbourne as the most popular place, and burial beats cremation as the most common mode of dispatch. It’s fun to explore.

But of course this then set me wondering about how these frequencies might change with the development of the ADB and changes in its subjects. So I generated word clouds for each volume and for each chronological series.

Word clouds by volume

Word clouds by series

I even added some simple Javascript slideshows so you could watch the clouds evolve.

One of the most obvious features in the series clouds is the gradual disappearance of ‘land’. It’s one of the most prominent words in the first series, but gradually fades until it disappears completely in the last.

After this successful foray into the world of word clouds, I began to think about other ways of visualising the ADB’s content. Many of the articles have portrait images, wouldn’t it be interesting to use the images themselves as the entry point to the biographical articles?

I’d already been playing with CoolIris, so I decided to harvest all the portrait references and use them to create a 3D wall. The result is pretty spectacular.

ADB portrait browser

Some technical details about the clouds and the portrait browser follow, for those interested in such things…

Gathering your words

Conveniently for me, ProgrammingHistorian uses the Dictionary of Canadian Biography as its main example, so there was much code that I could just cut and paste carefully examine and utilise. As the examples show, it’s easy to grab a webpage and analyse its content on the fly. But I wanted to process more than 10,000 pages and I knew that I was unlikely to get it working the first time round, so I decided to download the files first and then work on them locally. PH provided a basic example, to which I added some error-handling and the necessary loops to cycle through the ADB files. Because I had a bit of inside knowledge I cheated and hard-coded the numbers of articles in each volume. If I hadn’t known this I would have had to scrape all the browse pages, pulling out the links and creating a list in individual ids – not hard, but a bit tedious. Anyway this is how it ended up:

# download_adb.py

import urllib2, time, os, sys
import dh
items = (565, 575, 607, 526, 614, 533, 543, 723, 737, 742, 737, 759, 755, 721, 703, 714, 694, 126)
if os.path.exists('adb') == 0: os.mkdir('adb')

for v in range(0,18):
    for i in range (1,(items[v]+1)):
        if v == 0:
            filename = 'AS1%04db.htm' % i
        else:
            filename = 'A%02d%04db.htm' % (v, i)
        if os.path.isfile('adb/' + filename) == 0:
            print 'Processing: ' + filename
            url = 'http://adbonline.anu.edu.au/biogs/' + filename
            try:
                response = urllib2.urlopen(url)
            except IOError, e:
                if hasattr(e, 'reason'):
                    print 'We failed to reach a server.'
                    print 'Reason: ', e.reason
                elif hasattr(e, 'code'):
                    print 'The server couldn\'t fulfill the request.'
                    print 'Error code: ', e.code
            else:
                html = response.read()
                f = open('adb/' + filename, 'w')
                f.write(html)
                f.close
                time.sleep(2)
        else:
            print "File already downloaded"
        sys.stdout.flush()

Learning to count

Before too long I had a directory full of about 11,000 little html files just waiting for me to begin my evil experiments. First I had to slice them up and pull out all the interesting bits. By examining the code of the pages I could see that the main content was inside a div with the id of ‘content’. Using the Beautiful Soup Python library, I was easily able to extract this div. But the content div also usually included a portrait image and a bibliography. Once again I dipped into Beautiful Soup to discard all the unwanted bits. The slicing and dicing went something like this:

    g = open(dir + '/' + file, 'r')
    html = g.read()
    g.close()
    soup = BeautifulSoup(html)
    imagediv = soup.findAll(id="imagebox")
    if len(imagediv) > 0 :
        imagediv[0].extract()
    heading = soup.findAll('h4')
    if len(heading) > 0:
        heading[0].extract()
    footer = soup.findAll(id="selectbib")
    paras = footer[0].findNextSiblings('p')
    for para in paras:
        para.extract()
    footer[0].extract()
    content = soup.findAll(id="content")

Now I had the text of the article to play with. Following the PH examples it wasn’t long before I could extract word-frequency tables from a few files at a time. However, when I tried to process all the articles from a particular volume it took a verrry long time. I fiddled a bit with the code and amazed myself by dramatically improving the performance. I replaced the wordListToFreqDict function provided by PH with my own modified version:

def wordListToFreqDict2(wordlist):
    worddict = dict.fromkeys(wordlist)
    wordfreq = [wordlist.count(p) for p in worddict.keys()]
    return dict(zip(worddict,wordfreq))

The worddict = dict.fromkeys(wordlist) line made all the difference, creating a list of unique words that could then be checked against the full word list. With this hack in place I was able to process a complete volume in a few minutes.

I was already using a list of stopwords provided by PH to exclude things such as ’such’ , ‘as’ and ‘and’, but obviously a few additions were necessary. To the list of stopwords I added:

stopwords += ['january', 'february', 'march', 'april', 'may', 'june', 'july', 'august', 'september', 'october', 'november', 'december']
stopwords += ['new', 'south', 'wales', 'australia', 'australian', 'victoria', 'south', 'western', 'queensland', 'tasmania']
#stopwords += ['sydney', 'melbourne', 'brisbane', 'adelaide', 'perth', 'hobart']
stopwords += ['died', 'born', 'life', 'lived', 'married', 'father', 'wife', 'children', 'son', 'sons', 'daughter', 'daughters', 'brother', 'brothers']
stopwords += ['street', 'st', 'year', 'years', 'months', 'acre', 'acres', 'ha']
stopwords += ['e', 'm', 'b', 'c', 'w', 'j', 'd', 'n', 'f', 'g', 'h', 'i', 'ii', 'l', 'o', 'p', 'th', 'r', 't', 'u', 'r', 'nd']

The first two lines should be pretty obvious. As you can see, I originally excluded names of the capital cities, but then realised that you could watch Sydney and Melbourne battle it out for pre-eminence, so I excluded the exclusion. Also out were family relations and various other words that turned up in almost every article. Cleaning out all the non-alphabetical characters from the text had left a lot of orphaned letters that had once been things like £ signs, so I had to dispose of them as well.

The modules for actually generating the clouds were mostly just copied from PH with a few minor changes. My complete script is here:

# adb-text-count.py

import urllib2
import dh, os, sys, time
from BeautifulSoup import BeautifulSoup
start = time.time()
print "Started at: ", time.asctime(time.localtime(start))
dir = 'adb'
filelist = dh.getFileNames(dir)

f = open('wordlist.txt', 'w')
for file in filelist:
    print 'Processing ' + file
    sys.stdout.flush()
    g = open(dir + '/' + file, 'r')
    html = g.read()
    g.close()
    soup = BeautifulSoup(html)
    imagediv = soup.findAll(id="imagebox")
    if len(imagediv) > 0 :
        imagediv[0].extract()
    heading = soup.findAll('h4')
    if len(heading) > 0:
        heading[0].extract()
    footer = soup.findAll(id="selectbib")
    paras = footer[0].findNextSiblings('p')
    for para in paras:
        para.extract()
    footer[0].extract()
    content = soup.findAll(id="content")
    text = dh.stripTags(str(content[0]))
    fullwordlist = dh.stripNonAlpha(text.lower())
    wordlist = dh.removeStopwords(fullwordlist, dh.stopwords)
    f.write(" ".join(wordlist))
f.close
f = open('wordlist.txt')
words = f.read()
f.close
wordlist = words.split(" ")
dictionary = dh.wordListToFreqDict2(wordlist)
sorteddict = dh.sortFreqDict(dictionary)
f = open('wordfreqs.txt', 'w')
for s in sorteddict: f.write(str(s)+"\n")
f.close
print 'Dictionary created'
sys.stdout.flush()
# create tag cloud and open in Firefox
cloudsize = 200
maxfreq = sorteddict[0][0]
minfreq = sorteddict[cloudsize][0]
freqrange = maxfreq - minfreq
outstring = ''
resorteddict = dh.reSortFreqDictAlpha(sorteddict[:cloudsize])
print 'Creating cloud'
sys.stdout.flush()
for k in resorteddict:
    kfreq = k[0]
    klabel = k[1]
    klabel = dh.undecoratedHyperlink('http://adbonline.anu.edu.au/scripts/adbp-ent_search.php?ranktext=' + k[1] + '&search=Go!', k[1])
    scalingfactor = (kfreq - minfreq) / float(freqrange)
    outstring += ' ' + dh.scaledFontSizeSpan(klabel, scalingfactor) + ' '
dh.wrapStringInHTML("html-to-tag-cloud", dh.defaultCSSDiv(outstring), "Complete")
finish = time.time()
print "Finished at: ", time.asctime(time.localtime(finish))
print "Total time: ", finish - start

Biographies in 3D

To display all the portrait images in CoolIris I had to harvest all the image details and then write them to a Media RSS file for CoolIris to read.

Extracting the details of all the thumbnail versions of the portraits in the ADB was easy using Beautiful Soup. But I also need the paths to the larger versions of the portraits stored on the sites of the repositories that hold the originals. All of these sites present the images differently, so a different scraper was needed for each of them. As yet I’ve only included major libraries and archives – I may add some more if I get the time.

Once the paths to the thumbnails and large versions had been harvested, it was just a matter of writing the RSS feed. Actually, I created a series of RSS files, one for each volume, linked using ‘rel=previous’ and ‘rel=next’ attributes. This helped speed up the loading of the gallery. For what it’s worth, the complete code is here:

# adb-portraits.py

import socket, urllib2, urllib
import dh, os, sys, time, re
from BeautifulSoup import BeautifulSoup
# timeout in seconds
timeout = 20
socket.setdefaulttimeout(timeout)
start = time.time()
print "Started at: ", time.asctime(time.localtime(start))
dir = 'adb'
for i in range(8,18):
    if (i == 17): vol = "AS1"
    else: vol = "A%02d" % i
    filelist = dh.getFileNamesByVol2(dir, vol)
    f = open('adb-portraits-%s.rss' % i, 'w')
    f.write("<?xml version='1.0' encoding='utf-8' standalone='yes'?>\n")
    f.write("<rss version='2.0' xmlns:media='http://search.yahoo.com/mrss/' xmlns:atom='http://www.w3.org/2005/Atom'>\n")
    f.write("<channel>\n")
    f.write("\n")
    f.write("<description>Portraits of individuals included in the Australian Dictionary of Biography</description>\n")
    f.write ("
<link>http://www.adb.online.anu.edu.au</link>\n")
    if (i > 1):
        f.write ("<atom:link rel='previous' href='adb-portraits-%s.rss' />" % (i-1))
    if (i < 17):
        f.write ("<atom:link rel='next' href='adb-portraits-%s.rss' />" % (i+1))
    for file in filelist:
        print str(file)
        sys.stdout.flush()
        g = open(dir + '/' + file, 'r')
        html = g.read()
        g.close()
        #print html
        sys.stdout.flush()
        soup = BeautifulSoup(html)
        imagediv = soup.findAll(id="imagebox")
        if len(imagediv) > 0 :
            print "Found an image"
            sys.stdout.flush()
            links = imagediv[0].findAll('a')
            if len(links) > 1:
                link = urllib.unquote(links[(len(links)-1)]['href'][31:])
            else:
                link = urllib.unquote(links[0]['href'][31:])
            print link
            sys.stdout.flush()
            try:
                response = urllib2.urlopen(link)
            except IOError, e:
                if hasattr(e, 'reason'):
                    print 'We failed to reach a server.'
                    print 'Reason: ', e.reason
                elif hasattr(e, 'code'):
                    print 'The server couldn\'t fulfill the request.'
                    print 'Error code: ', e.code
            else:
                id = str(file)[:7]
                thumbnail = 'http://www.adb.online.anu.edu.au' + imagediv[0].img['src'].lstrip('.')
                # print thumbnail
                title = imagediv[0].p.contents[0].split(',')[0].strip().replace(' - ', '-')
                title = title.encode('utf-8')
                print "Processing: " + title
                sys.stdout.flush()
                html = response.read()
                imgsoup = BeautifulSoup(html)
                if (link.find('sl.nsw') > -1):
                    if (link.find('ebindshow.pl') == -1): # Not thumbnail pages - see John Bingle
                        if (html.find('Higher quality image') != -1):
                            img = imgsoup.findAll(alt="Higher quality image")[0].parent['href'].split('?')[1]
                            #img = imgsoup.td.a['href'].split('?')[1]
                        else:
                            img = imgsoup.table.findAll('tr')[2].img['src']
                        repository = "State Library of NSW"
                elif (link.find('slv.vic') > -1):
                    img = imgsoup.findAll(id='ImageDisplay')[0].img['src']
                    repository = "State Library of Victoria"
                elif (link.find('slsa.sa') > -1):
                    img = imgsoup.findAll('td')[1].img['src']
                    img = link[:link.rfind('/')+1] + img
                    repository = "State Library of SA"
                elif (link.find('nla.gov') > -1):
                    img = link + '-v'
                    repository = "National Library of Australia"
                elif (link.find('naa.gov') > -1):
                    barcode = link[link.rfind('=')+1:]
                    img = "http://naa16.naa.gov.au/rs_images/ShowImage.php?B=%s&#038;T=P" % barcode
                    repository = "National Archives of Australia"
                elif (link.find('territorystories.nt.gov') > -1):
                    img = imgsoup.table.img['src']
                    repository = "Northern Territory Library"
                elif (link.find('statelibrary.tas.gov') > -1):
                    if (html.find('No matches were found') == -1):
                        img =imgsoup.blockquote.img['src']
                        repository = "State Library of Tasmania"
                elif (link.find('slq.qld.gov') > -1):
                    img = imgsoup.findAll(attrs={"class":"pictureback"})[0].a['onclick']
                    #img = img[img.find('http'):img.find(]
                    img = re.search('http://[\w\d\/\.]*.jpg', img).group()
                    repository = "State Library of Queensland"
                if (len(img) > 0):
                    f.write("<item>\n")
                    f.write("<guid isPermaLink='false'>%s</guid>\n" % id)
                    f.write("\n" % (title, repository))
                    f.write("
<link>http://www.adb.online.anu.edu.au/biogs/%sb.htm</link>\n" % id)
                    f.write("<media:thumbnail url='%s' />\n" % thumbnail.replace('&#038;','&amp;'))
                    f.write("<media:content url='%s' type='image/jpeg' />\n" % img.replace('&#038;','&amp;'))
                    f.write("</item>\n")
                    f.flush()
                    print "Success!"
                    sys.stdout.flush()
                img = ""
    f.write("</channel>\n")
    f.write("</rss>\n")
    f.close()