Harvesting context #1: Flickr comments

tim — Sun, 23 Aug 2009 23:57:52 +0000

Instead of idly waiting for visitors to stumble over their holdings on some lonely information by-way, archives are starting to push their content out into the bustling metropolis of the social web. They are going where the people are. Photographic collections, in particular, are gaining new lives and new audiences thanks to Flickr.

But that’s only part of the story. Released into the wild, these photos are slowly picking up the habits of the locals. They are making friends, building connections, even speaking with new accents and dialects. Commented, tagged, organised, linked – they are building new contexts for themselves outside of the cloying control of archival descriptive systems.

Unfortunately it seems there is often a chasm between the old lives of the photos, documented in databases and finding aids, and their new post-institutional careers. This is a pity because the new contexts they are gathering can help us both understand and find them. What can we do to overcome this divide? How could finding aids harvest and display the user-generated content that aggregates around collection items living in the outside world?

The good news is that the tools to start doing this already exist – Flickr has a powerful API that makes it easy to extract photo metadata. Time for a bit of experimenting…

The first result is a userscript that displays Flickr comments in a number of collection databases. Just install it and then try it out:

National Archives of Australia Photosearch – try it!
State Records NSW Photo Investigator – try it!
National Archives and Records Administration ARC – try it!

Flickr comments in PhotoSearch

Gory details follow…

So to begin with I thought I’d just harvest comments from Flickr and display them within existing collection interfaces. As before (here and here), Greasemonkey was my tool of choice for hacking finding aids. The plan was to trigger a Greasemonkey script when you arrive at a photo in a collection database, the script would then:

extract a unique identifier for the photo that could be used to find it in Flickr
send off a request through the Flickr API to see if the photo was there
if so, then fire off another request to retrieve any comments
format the comments and insert them at a suitable point in the DOM of the database page

Easy! Obviously for the script to work there needed to be a way of connecting entries in the database with photos on Flickr. In practice this means that the photos need to be described at item level, and that a unique identifier needs to be used somewhere in the description of the photo both on Flickr and in the collection database.

Any archive that meets these criteria is a candidate for inclusion. Only three pieces of information are necessary:

the institution’s Flickr id
an expression to extract the identifier from the database page
an expression to identify the point on the database page at which the comments should be inserted

The expressions could use XPath or regular expressions – whatever it takes to find the desired elements. I’m using JQuery, so that makes selecting elements a lot easier. For example, NARA ARC includes the item identifier in a div with the class ‘arcID’, so I just select that element using JQuery and then use regex matching to pull out the number:

this.identifier = $('.arcID').text().match(/ARC Identifier (\d+)/i)[1];

To start with I’ve included the databases of three institutions:

the National Archives of Australia’s PhotoSearch database
State Records of NSW’s Photo Investigator
the US National Archives and Records Administration’s Archival Research Catalog

This is the code to save the settings for each institution:



if (document.location.href.match(/naa.gov.au\/scripts\/PhotoSearchItemDetail.asp/i)) {

this.name = 'NAA';

this.identifier = document.location.href.match(/M=0&B=(\d+)/)[1];

this.flickrId = '24849862@N08';

this.position = 'table:last';

} else if (document.location.href.match(/records.nsw.gov.au\/asp\/photosearch\/photo\.asp\?/i)) {

this.name = 'StateRecordsNSW';

this.identifier = document.location.href.match(/photo\.asp\?([\d\w_]+)/i)[1];

this.flickrId = '27331537@N06';

this.position = 'table:first';

} else if (document.location.href.match(/arcweb.archives.gov\/arc\/action\/ShowFullRecord|arcweb.archives.gov\/arc\/action\/ExternalIdSearch/i)) {

this.name = 'NARA';

this.identifier = $('.arcID').text().match(/ARC Identifier (\d+)/i)[1];

this.flickrId = '35740357@N03';

this.position = '.genPad:first';

}

From there it’s just a matter of building the calls to the API using Greasemonkey’s built-in GM_xmlhttpRequest method. Once the comments are retrieved, they’re given some basic formatting and inserted at the point in the DOM identified by the siteDetails.position property. Once again, JQuery greatly simplifies all the DOM manipulation. If there are no comments then a suitable message is inserted together with a link to the photo in Flickr. Finally some CSS is added to prettify it all a little bit.

You can view the full code on the Userscripts site.

Of course, it would be good to have this sort of stuff happening on the server side. In fact, with a few small modifications, this script could just be dropped into the code of any of the collection databases I’ve used. But in the meantime, Greasemonkey gives us a chance to play around with some of the possibilities – to start thinking about what finding aids might be like.

So what’s next? I’d like to do some playing around with tags and locations, perhaps using them to suggest related photos. I’ve also just realised that Flickr machine tags allow semantic markup… hmmm…

If you have any suggestions for databases to add to this script – let me know!

Cooliris-enabled scrapbook

tim — Mon, 09 Feb 2009 12:08:08 +0000

There’s more 3D goodness for you to enjoy now that the Mapping our Anzacs scrapbook is Cooliris-enabled. If you have Cooliris installed, you’ll notice that the Cooliris icon on your browser toolbar lights up when you visit the site. Just click on the icon to browse all the photos posted to the scrapbook on a glorious 3D wall.

Scrapbook posts in 3D

(If you don’t have Cooliris then go and get it. It can be used both in Internet Explorer and Firefox, though you’ll probably need to have admin rights to install for IE.)

Having given the 3D treatment to digitised files from the National Archives of Australia and portrait images from the Australian Dictionary of Biography, it wasn’t too hard to do. The scrapbook is a Tumblr site and the api makes it easy to extract all the photos. So I created a php file to gather all the details and then write them to a media-rss file. Then it was just a matter of inserting a link to it in the scrapbook.

Code follows:



if ($_GET['start']) {

$start = $_GET['start'];

} else {

$start = 0;

}

$url = "http://our-anzacs.tumblr.com/api/read?start=$start&num=50&type=photo&filter=text";

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $url);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

curl_setopt($ch, CURLOPT_FAILONERROR, true);

curl_setopt($ch, CURLOPT_TIMEOUT, 20);

$result = curl_exec($ch);

if (!$result) {

echo "cURL error number:" .curl_errno($ch);

echo "cURL error:" . curl_error($ch);

exit;

}

curl_close($ch);

$dom = new DOMDocument();

@$dom->loadHTML($result);

$xpath = new DOMXPath($dom);

$attrs = $xpath->evaluate("//posts/@total[1]");

foreach ($attrs as $attr) {

$total = $attr->nodeValue;

}

$num_pages = ceil($total/50);

$start_next = $start+50;

$start_previous = $start-50;
echo "n";

echo "n";

echo "n";

echo "n";

echo "Photos posted to the Mapping our Anzacs scrapbookn";

echo "
http://our-anzacs.tumblr.comn";

if ($start_previous >= 0) {

echo "";

}

if ($start_next <= $total) {

echo "";

}

$posts = $xpath->evaluate("//post/@id");

foreach ($posts as $post) {

$id = $post->nodeValue;

$url = "http://our-anzacs.tumblr.com/post/$id";

$photos = $xpath->evaluate("//post[@id='$id']/photo-url[@max-width='500']/text()");

foreach ($photos as $photo) {

$photo_500 = $photo->nodeValue;

}

$photos = $xpath->evaluate("//post[@id='$id']/photo-url[@max-width='250']/text()");

foreach ($photos as $photo) {

$photo_250 = $photo->nodeValue;

}

$nodes = $xpath->evaluate("//post[@id='$id']/photo-caption/text()");

foreach ($nodes as $node) {

$caption = $node->nodeValue;

preg_match("/View details fors+([ws,-]*)/", $caption, $matches);

$names = explode(", ", $matches[1]);

$name = "$names[1] $names[0]";

}

echo "n";

echo "$idn";

echo "n";

echo "
$urln";

echo "n";

echo "n";

echo "n";

}

echo "n";

echo "n";

?>

discontents » photos

Harvesting context #1: Flickr comments

Cooliris-enabled scrapbook