Skip to content

shoebox
digital humanities

Extracting editorials #3

By my own criteria I’ve already failed… I started this series of posts with the intention of documenting the process of finding and extracting editorials as I was actually doing the work. But here I am about to describe some work I finished a few weeks back. Oh well…

In my previous instalments (here and here), I focused on the Sydney Morning Herald. Having continued the hunt for missing editorials I started in the last post, I’ve now got a CSV file with the urls of the first editorial published in every edition of the SMH from 1913. Good-o, I thought, I can now start harvesting and analysing some content.

But then ensued a crisis of faith. The whole point of this exercise was to be able to build up some comparisons  – between newspapers, between states, between the city and the bush. But the process of actually finding the editorials seemed beset with difficulties. Could the rules I developed for the SMH be applied elsewhere? Could I ever assemble a useful set of editorials without large amounts of human intervention? I decided to try a few quick experiments to see whether the whole project was worth pursuing.

I started with a few assumptions:

  1. The first (and only the first) editorial in any issue is headed with the name of the newspaper.
  2. Editorials are published on even numbered pages.
  3. Editorials vary in length between about 100 and 1500 words.

These assumptions were based on my own experience as a long-time newspaper researcher and on some preliminary poking around. For example, when I looked at The Argus I noticed that editorials were typically followed by news summaries. Unfortunately, these are treated as a single article in Trove, resulting in large blocks of text that are only part editorial. By specifying an upper word limit I hoped to filter these sorts of articles out. Similarly, there are sometimes brief announcements or publication details headed with the name of the newspaper. The lower word limit was intended to exclude these.

The next step was to harvest every article from 1913 that was headed with the name of its publication. I created a script to generate a list of all the newspapers that published issues in 1913. Then I called my existing harvester to download all the matching articles and save the details to a series of CSV files — one CSV file per newspaper.

In the previous instalment of this series I created a script to check the CSV output of my harvester for missing or duplicate dates. I extended this to perform a series of tests on each article based on the assumptions above. First, I filtered out articles on odd-numbered pages, then articles that were too short or too long. Finally I checked the remainder for missing or duplicate issue dates.

The details of the articles in each category were written out to JSON files. Using these files and a bit of JQuery magic I could quickly build a simple web interface that allowed me to explore the results.

Summary details of each newspaper

You can browse the summary results for the full list of newspapers, or you can drill down to view the actual articles assigned to each category.

Full details

I’ll save the full analysis for the next post, but if you play around with the results you quickly notice a few things. First, letters to the editor often include the name of the newspaper! If you look at The Mercury, for example, you’ll notice I’ve identified 1057 potential editorials — most of which are letters. Fortunately they should be fairly easy to filter out. In most cases the ‘even numbers only’ assumption worked pretty well, and the word length filters did remove quite a lot of false positives. There are still plenty of problems, but I’m encouraged enough to continue. Yes, there will be a Part #4!

 

Posted 20 February 2012 † tim § experiments ‡ Comments (0) ° Tagged: , ,

2011 — the year of little sleep

2011 was a busy year. It’s hard to believe that it was only February when I first posted about my experiments mining the contents of the Trove newspaper database. Since then I’ve developed a set of digital tools, organised THATCamp Canberra, given a series of presentations on the possibilities of digital history, pushed ahead with Invisible Australians, and tried to develop my own digital research program. Oh yes, and endeavoured to earn enough money to feed the kids and pay the mortgage…

It looks like 2012 could be even busier, so before I lose track completely, I thought I’d pull together some of the past year’s exploits for handy reference. So here’s (most of) my presentations for 2011…

8 June 2011 — ‘Confessions of an impatient historian’
Scholars’ Lab, University of Virginia

18 August — ‘Digital history: new tools and techniques’
National Museum of Australia

24 August — ‘Hacking the archives’
Archival description in an online world, Recordkeeping Roundtable, Sydney

5 September 2011 — Digital research methods
Cultural heritage students, University of Canberra

14 September 2011 — ‘Every story has a beginning’
Keynote presentation at the Indexing See Change Conference (Australian and New Zealand Society of Editors)

13 November 2011 — ‘Digital history: new tools and techniques’
Dragontails 2011: 2nd Australasian conference on overseas Chinese history & heritage, Museum of Chinese Australian History, Melbourne

30 November 2011 — ‘It’s all about the stuff’
National Digital Forum, Wellington, New Zealand

7 December 2011 — ‘An introduction to digital history’
Digital December, State Library of NSW

Posted 24 January 2012 † tim § conference presentations ‡ Comments (0) °

It’s all about the stuff — the movie

Videos from NDF2011 are now available online. Here’s the movie version of my talk It’s all about the stuff. I seem to spend a lot of time in the shadows…

Posted 23 January 2012 † tim § conference presentations ‡ Comments (0) ° Tagged: , ,

QueryPic

Back when I was looking at ‘When did the Great War become the First World War?‘ I promised a detailed post on how I constructed the graphs. But of course I got distracted. Then I started adding new features to the script and redesigning the graphs, so…

Anyway, the result is a rather neat little gizmo henceforth named QueryPic (I got a bit sick of ‘search summariser’ and ‘graph-maker thing’). The first version just harvested data and left all the graph-making to you. But QueryPic does it all! It harvests the data and makes the graph. Woohoo.

Here’s an example showing ‘drought’ versus ‘flood’:

QueryPic features

  • Explore your Trove newspaper query over time in the form of a simple line graph.
  • Interactive — click on a point to retrieve sample articles from that date.
  • Combine data sources to compare queries.
  • Choose your interval — plot by year or month.
  • Switch views between total results and the proportion of all articles.

Running QueryPic

Yes, it’s a Python script and yes it runs on the command line. Let’s get that out of the way now. I don’t think I have the time and energy to develop cross-platform gui versions of all my tools. I’d rather spend the time adding new features or exploring new possibilities. Sorry, but until I have a wealthy benefactor or a technical support team, I think that’s the way it has to be. In any case, the code is all there – so build your own gui!

Actually, if I did have the time and energy I don’t think I’d build a standalone gui anyway. What would be much cooler would be a web service, where people could run, share and combine their queries. Social graph-making! A celebration of serendipity! A historical playground! Hmmm…

But for now there’s this python script. It’s dead easy to use. Starting from the beginning…

  1. Do you have Python installed? If you have a Mac or Linux the answer is yes. Fire up a terminal and type ‘python -V’ — see, I told you. If you have Windows you can get a handy installer. Do it.
  2. Get the source code. Just download this zip file and open it into a new folder.
  3. Open a terminal and cd into the new folder.
  4. Run ‘python do_totals.py [your Trove query]‘.
  5. Watch in excitement as the script chugs away retrieving data from Trove.
  6. Once the script is finished, go to the ‘graphs’ directory, where you’ll find your newly-created html page complete with fancy interactive graph.
  7. Open the html page in the web browser of your choice.
  8. Enjoy! Celebrate! Drink a toast in my honour!

Customising QueryPic

There are a number of optional arguments that you add to the command line to customise your results:

-n (or –name) [a query name]
Give a name to your query. The name is used to create filenames for the html and data files, it is also used in the legend of the graph. The default is to use the search keywords as the name.

-d (or –directory) [a directory path]
The full pathname of the directory/folder for your results. The default is a ‘graphs’ sub-directory in the current directory.

-g (or –graph) [a graph name]
Specify the name of the html file that’s created. This is useful for displaying multiple queries on a single graph. Just run QueryPic for each query, using the same graph name each time. The default is either the value specified by the -n parameter or a name derived from the search keywords.

-m (or –monthly)
Plot the query at monthly intervals. The default interval is a year.

What QueryPic actually does

QueryPic builds a simple visualisation of your search query in the Trove newspaper database. A list of search results is difficult to interpret and offers little context. QueryPic shows you the number of articles matching your query over time, enabling you reframe your questions, pursue hunches, or simply play around.

QueryPic takes your Trove newspaper query and looks for a date range. If it doesn’t find one, it assumes you want your graph to go from 1803 to 1954 (the complete contents of the newspaper database — except for the Women’s Weekly). QueryPic then strips out any date parameters from the query, so it can fire off the query within the start and end dates, at the specified date interval.

Date interval? In the previous version of this script you could only plot points at yearly intervals, so it was impossible to zoom in an see what might be happening over the span of a single year or two. But amazing advances in QueryPic technology mean you can now plot changes by month. Here for example is a new version of my Great War/First World War graph, focused on 1938–1946 and plotted at monthly intervals.

So for each interval within the date range QueryPic fires off a request to Trove. From the response it scrapes out the total number of results for that date. If the total is greater than zero, it then fires off a second request to find the total number of newspaper articles for that year. Your query results divided by the total number of articles gives the proportion of articles for that date matching your search query.

The number of results and the proportion are written to a javascript file, together with some other important information including the original query and the date the harvest was performed. Remember, the Trove newspapers database is always changing! QueryPic then grabs a copy of it’s own special html template and inserts a reference to this javascript file. For good measure, it also inserts a link to your original query. The file is saved under a new name, ready for you to open and explore.

The html file contains everything necessary to take your data and turn it into a graph. It does this using the HighCharts javascript library. Please note, that while licence conditions allow HighCharts to be redistributed as part of a non-commercial package, it is not free for commercial use. Check the HighCharts website for details.

Some examples

Plot ‘cat’ against ‘dog’ in a graph called ‘animals’:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"
python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -g "animals"

Specify a directory for your results:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -d "/User/bill/Documents/graphs"

Plot results at monthly intervals:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat&fromyyyy=1920&toyyyy=1921" -m

Specify a name:

python do_totals.py "http://trove.nla.gov.au/newspaper/result?q=cat" -n "Felines"
Posted 01 January 2012 † tim § hacks ‡ Comments (0) ° Tagged: , ,

Extracting editorials #2

As I explained in the first of this series, I’m documenting my efforts to extract every editorial published in the Sydney Morning Herald in 1913 from the Trove newspaper database. It’s an experiment both in text mining and historical writing — an attempt to put the method up front.

While I didn’t think there was anything very thrilling in the first instalment, recording my thoughts and assumptions in this way has already proved useful. In a comment, Owen Stephens noted that his attempt to reproduce my search query produced fewer results. After a little bit of poking around I realised that the fulltext modifier, which I often use to switch off fuzzy matching, counteracts the ‘search headings only’ flag. So my query was returning results that had the string ‘The Sydney Morning Herald’ anywhere in the article.

Try it for yourself.

Here’s my original query — searching for fulltext:”The Sydney Morning Herald” in headings only (supposedly). You’ll notice that it returns 335 results and it’s clear from a quick scan that a number are false positives (they don’t follow the pattern for editorials).

Here’s Owen’s query — searching for “The Sydney Morning Herald” in headings only. It returns 294 results, without any obvious false positives.

So my attempt to disable fuzzy matching actually produced a less accurate result! Weird.

Actually, I think one important benefit of this sort of text mining is that it helps you understand how the search engines you’re using actually work. Once you start poking and prodding, the idiosyncrasies start to emerge.

Anyway, I harvested Owen’s cleaner result set and opened up the resulting csv file. As it seemed in Trove, there we’re very few false positives. Indeed there were only two articles that didn’t seem to follow the standard editorial format, and these were notes added to the editorial page. On the other hand, there were obviously about 20 editorials missing. I could have manually worked through the csv file to identify the missing dates, but I thought I’d try to create some tools that would do the work for me.

What I wanted was the details of the first editorial in every edition of the newspaper in 1913 — so there should be one, and only one, article for each day on which the newspaper was published. I needed a tool that would analyse the csv file and do two things:

  • identify dates that occur multiple times (false positive alert!)
  • identify dates that are absent from the result set (missing in action!)

The resulting code is all on GitHub if you want follow along. I wrote a Python script that opens up the csv file, extracts all the date strings, converts them to datetime objects and then saves them to a list. Once that’s done it’s pretty easy to loop through and find duplicates:

def find_duplicates(list):
    '''
    Check a list for suplicate values.
    Returns a list of the duplicates.
    '''
    seen = set()
    duplicates = []
    for item in list:
        if item in seen:
            duplicates.append(item)
        seen.add(item)
    return duplicates

Finding missing dates was a little more complicated, but Google came to the rescue with some handy code samples. All I had to do was set a start and end date (in this case 1 January 1913 and 31 December 1913) and create a timedelta object equal to a day. Then it’s just a matter of adding the timedelta to the start date, comparing the new date to the dates extracted from the csv file, and continuing on until you hit the end. If the new date isn’t in the csv file, then it gets added to the missing list.

if year:
        start_date = datetime.date(year, 1, 1)
        end_date = datetime.date(year, 12, 31)
    else:
        start_date = article_dates[0]
        end_date = article_dates[-1]
    one_day = datetime.timedelta(days=1)
    this_day = start_date
    # Loop through each day in specified period to see if there's an article
    # If not, add to the missing_dates list.
    while this_day <= end_date:
        if this_day.weekday() not in exclude: #exclude Sunday
            if this_day not in article_dates:
                missing_dates.append(this_day)
        this_day += one_day

I’ve tried to make the code as reusable as possible, so you can either supply a year, or the script will read start and end dates from the csv file itself.

All that left me with two more lists of dates: ‘duplicates’ and ‘missing’. At first I just wrote these out to a text file, but then I decided it would be useful to write the results to an html page. That way I could add links that would take me to the actual issue within Trove, helping me to quickly find the missing editorial.

Unfortunately there’s no direct way to go from a date to an issue — you first need to find the issue identifier. How do you do this? If you dig around in the code beneath the page for each newspaper title, you’ll find that the ajax interface pulls in a json file with issue information. You can access this through a url like: http://trove.nla.gov.au/ndp/del/titlesOverDates/[year]/[month]. Here’s an example for January 1913.

The json includes all issues for all titles in the specified month. So you then have to loop through to find a specific title and day. Once you have the issue identifier you can just attach it to a url:

def get_issue_url(date, title_id):
    '''
    Gets the issue url given a title and date.
    '''
    year, month, day = date.timetuple()[:3]
    url = 'http://trove.nla.gov.au/ndp/del/titlesOverDates/%s/%02d' % (year, month)
    issues = json.load(urllib2.urlopen(url))
    for issue in issues:
        if issue['t'] == title_id and int(issue['p']) == day:
            issue_id = issue['iss']
    return 'http://trove.nla.gov.au/ndp/del/issue/%s' % issue_id

My results file with links to Trove

Finally, to save myself having to cut and paste the missing dates back into the csv file, I added a few lines to write them in automatically.

So now I have a handy little html page, complete with dates and links, that I’m working through to find all the missing editorials. All I need for the next stage are the urls for the editorial and the page on which it’s published. I’m just cutting and pasting these from the citation box in Trove into the csv file. Once this is done I can start trying to find all the editorials.

PS: I noted in my first post that one benefit in finding the editorials was that the main news articles usually appeared on the page after the editorials. I’ve been thinking some more about ways to identify ‘major’ news stories. Word length perhaps? But not always. Hmmm, but major stories do seem to be published at the top of the page. After a bit more poking around in the code I found that there’s a ‘y value’ assigned to each article that indicates its position on the page. So if I harvest all the articles on the page after the editorials and then rank them by their y values? Interesting…

Posted 19 December 2011 † tim § experiments § hacks ‡ Comments (0) ° Tagged: , ,

It’s all about the stuff: collections, interfaces, power and people

This is the full version of a paper I presented at the National Digital Forum, 30 November 2011.

In 1901, one of the first acts of the Commonwealth of Australia was to create a system of exclusion and control designed to keep the newly-formed nation ‘white’. But White Australia was always a myth. As well as the Indigenous population, there were already many thousands of people classified as ‘non-white‘ living in Australia — most were Chinese, but there were also Japanese, Indians, Syrians and Indonesians.

Here are some of them…

The real face of White Australia

The administration of what became known as the White Australia Policy created a huge volume of records, much of which is still preserved within the National Archives of Australia. These photographs are attached to certificates that non-white residents needed to get back into the country if they decided to travel overseas. There are thousands upon thousands of these certificates in the Archives. Thousands of certificates representing thousands of lives — all monitored and controlled.

But is is too easy to see these people as the powerless victims of a repressive system. There were many acts of resistance. Some argued against the need to be identified ‘just like a criminal’. Others exercised control over their representation, submitting formal studio portraits instead of mug shots.

Most commonly and most powerfully, people resisted the policy simply by going ahead and living rich and productive lives.

My partner, Kate Bagnall, is helping to rewrite Australian-Chinese history by overthrowing the stereotype of the culturally isolated Chinese man living a lonely, meagre existence surrounded by gambling and opium dens. By mining the available records, by reading against the grain of contemporary reports and by working with family historians, Kate is documenting their intimate lives — their wives, their lovers, their families and descendants — the sorts of relationships that sent a shudder through the edifice of White Australia. Power can be reclaimed in many subtle and subversive ways.

‘The real face of White Australia’ is an experiment. It uses facial detection to technology to find and extract the photographs from digital copies of the original certificates made available through the National Archives of Australia’s collection database. The photographs you see here come from just one series, ST84/1. There’s no API to the collection so I reverse-engineered the web interface to create a script that would harvest the item metadata and download copies of all the digitised images. There are 2,756 files in this series. On the day I harvested the metadata, 347 of those files had been digitised, comprising 12,502 images. It took a few hours, but I just ran my script and soon I had a copy of all of this in my local database.

Then came the exciting part. Using a facial detection script I found through Google and an open source computer vision library, I started experimenting with ways of extracting the photos. After a few tweaks I had something that worked pretty well, so I pointed my aging laptop at the 12,502 images and watched anxiously as the CPU temperature rose and rose. It took a few emergency cooling measures, but the laptop survived and I had a folder containing 11,170 cropped images. About a third of these weren’t actually faces, but it was easy to manually remove the false positives, leaving 7,247 photos.

These photos. These people.

With my database fully primed and loaded it was just a matter of creating a simple web interface using Django for the backend and Isotope (a jQuery plugin) at the front. Both are open source projects. All together, from idea to interface, it took a bit more than a weekend to create, and most of that was waiting for the harvesting and facial detection scripts to complete. It would be silly to say it was easy, but I would say that it wasn’t hard.

What we ended up with was a new way of seeing and understanding the records — not as the remnants of bureaucratic processes, but as windows onto the lives of people. All the faces are linked to copies of the original certificates and back to the collection database of the National Archives. So this is also a finding aid. A finding aid that brings the people to the front.

According to Margaret Hedstrom the archival interface ‘is a site where power is negotiated and exercised’. Whether in a reading room or online, finding aids or collection databases are ‘neither neutral nor transparent’, but the product of ‘conscious design decisions’. We would like to think that this interface gives some power back to the people within the records. Their photographs challenge us to do something, to think something, to feel something. We cannot escape their discomfiting gaze.

But this interface represents another subtle shift in power. We could create it without any explicit assistance or involvement by the National Archives itself. Simply by putting part of the collection online, they provided us with the opportunity to develop a resource that both extends and critiques the existing collection database. Interfaces to cultural heritage collections are no longer controlled solely by cultural heritage institutions.

It’s these two aspects of the power of interfaces that I want to focus on today.

There are a growing number of examples where the records created by repressive or discriminatory regimes have, in Eric Ketelaar’s words, ‘become instruments of empowerment and liberation, salvation and freedom’. Nazi records of assets confiscated during the Holocaust have been used to inform processes of restitution and reparation. Government records have helped members of Australia’s Stolen Generations trace family members. Descendants of inmates incarcerated by American colonial authorities in what was the world’s largest leprosy colony in the Philippines, have embraced the administrative record as an affirmation of their own heritage and survival. Records can find new meanings. Power can be reclaimed.

Technology can help. Tim Hitchcock has described how something as simple as keyword searching can turn archives on their heads. Recordkeeping systems tend to reflect the structures and power relations of the organisations that create them. The ‘hierarchical and institutional nature of most archives’, Hitchcock argues, ‘contains an ideological component which is sucked in with every dust-filled breath’. But digitisation and keyword searching free us from having to follow the well-worn paths of institutional power. We can find people and follow their lives against the flow of bureaucratic convenience. We can gain a wholly new perspective on the workings of society. ‘What changes’, Hitchcock asks, ‘when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?’

Projects such as Unknown no longer may help us answer that question.

Unknown no longer

It’s aiming to extract the names and biographical details of slaves from the 8 million manuscript documents held by the Virginia Historical Society. The documents include court records, receipts, wills and inventories. Here is a page from the ‘Inventory of Negroes at Berry Plain Plantation, King George County, Virginia’ for 1855, listing names, occupations and valuations.

Tim Hitchcock is one of the directors of London Lives a project that similarly seeks to find the people in 240,000 manuscript pages documenting the lives of plebeian Londoners in the 17th century.

London Lives

More than three million names have already been extracted from the records of courts, workhouses, hospitals and other institutions. Work is continuing to link these names together, to merge these various shards of identity and piece together the experiences of London’s poorest inhabitants.

Remember me from the US Holocaust Memorial Museum is working with photographs taken by relief agencies in the aftermath of World War Two. The photographs are of displaced children who survived the Holocaust but were separated from families. What happened to them? The project is seeking public help to identify and trace the children.

Remember me

These are all projects about finding people. Finding the oppressed, the vulnerable, the displaced, the marginalized and the poor and giving them their place in history. This is what Kate and I hope to do with Invisible Australians, the broader project of which our faces experiment is part.

Invisible Australians

‘Invisible Australians’ aims to extract more than just photographs. We want to record and aggregate the biographical data contained within the records of the White Australia Policy — to extract the data and rebuild identities.

But we want to do more, we want to link these identities up with with other records, with the research of family and local historians, with cemetery registers and family trees, with newspaper articles and databases we don’t even know about yet. We want to find people, families and communities.

It’s ridiculously ambitious and totally unfunded. But it is possible.

The most exciting part of online technology is the power it gives to people to pursue their passions. As with the faces, we don’t need the help of the National Archives. We need the records to be digitized, but that’s happening anyway and we can afford to be patient. Most of the tools we need already exist, and are free. In the past 12 months, for example, there have been a number of open source tools released for crowd-sourced transcription of manuscript records.

People with passions, people with dreams, people who are just annoyed and impatient, don’t have to wait for cultural institutions to create exactly what they need. They can take what’s on offer and change it.

Interfaces can be modified. It is amazingly easy to write a script that will change the way a web page looks and behaves in your browser. I was frustrated by the standard interface to digitized files in the National Archives of Australia’s Recordsearch database — so I changed it.

Before and after

Not only did make it look a bit nicer, I added new functions. My script lets you print a whole file or a range of pages and display the entire contents of the file on a pretty cool 3d wall.

I’ve shared this script, and a few other Recordsearch enhancements. Anyone can install them with a click and use them.

Wragge Labs Emporium

Interfaces are sites of power and we can claim some of that power for ourselves. Online technologies not only free us from the having to brave the physical intimidation of the reading room, they free us up to engage with the records in new ways. The archivist-on-duty would probably not be pleased if I pulled out some scissors and started snipping photos out of certificates. Or if I pulled a file apart and pasted it’s contents on the wall. But online we are free to experiment.

The power of cultural heritage organisations is perhaps expressed most forcefully in their ability to control the arrangement and description of their collections. ‘Every representation, every model of description, is biased’, note Verne Harris and Wendy Duff, ‘because it reflects a particular world-view and is constructed to meet specific purposes’. Archives, libraries and museums are already starting to share this power, by allowing tagging, or seeking public assistance with description through crowd sourcing projects. But most of the these activities still happen within spaces created and curated by the institutions themselves. Our cathedrals of culture might be opening their doors and inviting the public to participate in their ceremonies, but that doesn’t make them bazaars. The architecture stills speaks of authority.

In any case, people already have a space where they can explore and enrich collections — it’s called the internet.

It would be great to see cultural institutions doing more to watch, understand and support what people are doing with collections in their own spaces — following them as they pursue their passions, rather than thinking of ways to motivate them.

A quick example… You might have heard of Zotero, it’s an open source project that lets you capture, annotate and organize your research materials.

Zotero

One cool thing about Zotero is that you can build and contribute little screen scrapers, called translators, that let Zotero extract structured data from any old collection database. You might not be surprised to learn that I’ve created a translator for Recordsearch. Another cool thing about Zotero is that you can share the stuff that you collect in public groups.

Invisible Australians Zotero group

Put those two cool things together and what do you have? Well to me they spell out user generated finding aids — parallel collection databases created by researchers simply pursuing their own passions.

Linked Open Data greatly increases opportunities for collection description to leak into the wider web. If objects and documents are identified with a unique URL, then anyone can can make and publish statements about them in machine-readable form. These statements can then be aggregated and explored. Initiatives such as the Open Annotation Collaboration will hasten the development of these shared descriptive and interpretative layers around our cultural collections.

And of course all this descriptive and interpretative work can be harvested back to enhance existing collection databases. We could start doing it now — though I will spare you today my rant about the possibilities of mining footnotes.

As well as exploring the possibilities of user-generated content, cultural institutions are starting to open up their collection data for re-use. APIs are great (though Linked Open Data is better), and New Zealand is lucky to have an organisation like DigitalNZ which just gets it. People can and will make cool things with your stuff.

But again, we don’t have to wait for everything to be delivered in a convenient, machine-readable form. If it’s on the web anybody can scrape, harvest and experiment.

You probably all know about the National Library of Australia’s newspaper digitisation project — it’s building a magnificent resource. But I wanted to do more than just find articles. I wanted to explore and analyze their content on a large scale. So I built a screen scraper to extract structured data from search results, and then used the scraper to  power a series of tools. I have a harvester that lets you download an entire results set — hundreds or thousands of articles — with metadata neatly packaged for further analysis.

Harvester

Or what about a script that graphs the occurrence of search terms over time, and allows you to ask questions like When did the Great War become the First World War?.

When did the Great War become the First World War?

In the end I got a bit carried away and built my own public API to the Trove newspaper database.

Unofficial Trove newspapers API

I think it’s important to note that the tools I developed were guided by the types of questions I wanted to ask. While we should welcome APIs and celebrate their possibilities, we should also remain critical. APIs are interfaces, they too embed power relations. Every API has an argument. What questions do they let us ask? What questions do they prevent us from asking?

Even as we move from the age of lumbering, slow-witted data silos into the rapidly-evolving realms of Linked Open Data, we have to constantly question the models we make of the world. Ontologies and vocabularies are culturally determined and historically specific. Yes, they too are interfaces, complete with their own distributions of power and authority. But we can revisit and change them. And we can relate our new models to our old models, capturing complex, long-term shifts in the way we think about the world. That’s incredibly exciting.

All of this hacking, harvesting, questioning, enriching and meaning-making makes me think about the possibilities of grassroots leadership. Online technologies enable people to take cultural institutions into unexpected realms. They can build their own interfaces, ask their own questions, determine their own needs — they can point the way instead of simply waiting to be served.

You might wonder what the National Library of Australia thinks of my various scrapers and harvesters. I can’t speak for them, but I can say that they’ve awarded me a fellowship to explore further the possibilities of text-mining in their newspaper database.

The idea of grassroots leadership brings me back to the title of this talk — ‘It’s all about the stuff’. It seems to me that we tend to model the interactions between cultural institutions and the public as transactions. The public are ‘clients’, ‘patrons’, ‘users’ or ‘visitors’. But the sorts of things I’ve been talking about today give us a chance to put the collections themselves squarely at the centre of our thoughts and actions. Instead of concentrating on the relationship between the institution and the public, we can can focus on the relationship we both have with the collections.

It’s all about the stuff.

It’s all about the respect and responsibility we both have for our collections.

It’s all about the respect and responsibility we both have for people like this.

 

 

Posted 01 December 2011 † tim § archives § conference presentations ‡ Comments (4) ° Tagged: , , , , , ,

Extracting editorials #1

In their chapter in Writing History in the Digital Age, Trevor Owens and Fred Gibbs encourage historians to write about the ways they work with data — to document their methods, their working assumptions, their dead ends and their discoveries. It’s an important argument and one that makes me wonder again about forms of publication that might integrate narrative, methods and sources.

In the meantime though we have blogs. My problem is that I’m easily bored so by the time I get to the end of a project or experiment I’m already thinking about the next one. Going back and trying to write things up seems a bit of a chore (which is why I’m always way behind in my blog writing). Also leaving the writing to the end means that I tend to take shortcuts — leaving out some of the ‘boring’ procedural stuff or the ‘stupid’ ideas that just didn’t work.

But Trevor and Fred’s chapter has made me think I should be a bit more diligent, so as I start a new series of text-mining experiments I’ve decided to write things up as I’m doing them. So be warned, this could get messy…

So what do I want to do? You might not be surprised to learn that it’s another Trove newspaper database experiment. I want to see if I can harvest newspaper editorials over a certain period and then analyse these to build up a picture of what issues, events or ideas were perceived as important. As I’m currently looking at ways of harvesting digital sources relating to 1913 for an exhibition being developed by the National Museum of Australia, I’m going to start by focusing on 1913.

But editorials are opinion pieces, wouldn’t it be better to harvest ‘news’ articles?

First of all, I’m thinking that editorials will be fairly easy to identify and extract — there’s no real way in Trove to separate out current news from other sorts of articles. Secondly, I’m assuming that the issues that make it into editorials have some importance attached to them. Attached by whom, you may well ask — whose voice is being represented in the editorial? This is an important question and I’m thinking that it could be explored in interesting ways by harvesting editorials from a range of papers and regions. Thirdly, finding the editorials might actually help me find the major news articles, simply because in this period the main news stories were often on the page after the editorials.

So how do I find them? Looking at the Sydney Morning Herald for 1913, you can see that the editorials follow a regular pattern:

  • the first editorial is always headed with the name of the paper and the date, followed by the title
  • subsequent editorials that follow have a title but no subtitle (most other types of articles have a subtitle)
  • editorials are published on an even-numbered page, usually about half way through the newspaper

To check this I conducted a search for articles including ‘The Sydney Morning Herald’ in their title. The search returns 335 results. Of course we’d expect there to be 312 (6 x 52), but it looks like there’s quite a few false positives and some days missing altogether (presumably due to OCR errors). You can see there’s a fair bit of consistency in the pages that editorials appear on, but it doesn’t quite seem consistent enough to rely on. So I’ve decided that as a first step I’ll harvest all the articles from this query. I’ll then do some manual cleaning to remove the articles that aren’t editorials and try and identify and retrieve the missing days.

Remember, this won’t give me all the editorials, only the first editorial from each day. To get all the editorials, I’ll have to write a new script that will take this first result set, retrieve all the articles from the editorial page and then try to work out which of the articles are editorials — they should be the ones that come after the first editorial and have no subtitle. Or that’s the theory.

I’ve harvested the query. You can view the spreadsheet on Google Docs if you feel so moved.

[After I wrote the sentence above I checked the CSV file properly and realised I'd stuffed up. There's a bit of a bug in my harvester that means if the query string you use includes a start value, the harvester wil retrieve the same page of results over and over again... I really need to fix that. :-) I'm now running it again. You wanted warts and all, right?]

[After I wrote the paragraph above I checked my new harvest and realised I'd stuffed up again. There were only half as many results as there should have been! So I poked around and realised some recent changes I'd made to the harvest script meant I was only getting odd numbered results (I was incrementing the row value twice). A lesson in what happens when you do this stuff late at night... Trying again. ]

I’m not sure when I’ll have time to do the cleaning. But hey folks this is what research is like for people like me who have to try and fit it in around the edges of their lives. You can expect posts to come in sudden bursts and then dry up altogether for long periods as other priorities intrude.

 

Posted 21 November 2011 † tim § ‡ Comments (5) ° Tagged: , , ,

An infrastructure wishlist

I have problems with the idea of infrastructure, particularly that of the e-research variety. It seems like we always end up talking about huge amounts of money and multi-institutional partnerships. It just doesn’t seem like a great model for innovation. As I’ve previously argued, I’d like to see something more like the funding schemes offered by the NEH Office for Digital Humanities. Encourage people with ideas, don’t just reward the good networkers. Build tools and apis, not portals and platforms.

Of course I’d still like to see the digital humanities well represented in the list of Virtual Laboratories and eResearch Tools currently under consideration by NeCTAR. It’s time the digital research needs of the humanities were properly recognised. There are lots of possibilities, most of which we can’t yet envisage, but as I was asked what I would like to see as part of a Virtual Laboratory I had a go at setting down a few brief ideas. For what it’s worth, here’s my e-research infrastructure wishlist…

Grappling with abundance

Traditional historical research is often based on a presumed scarcity of resources — the skill is in tracking down the sources. But large digital collections, like the Trove newspapers database, change this — you now have to make sense of the sheer volume of material. Digital history, through techniques such as text-mining and visualisation, offer a way of using these new riches effectively. We need to ensure that investments in digitisation are accompanied by evolutions in scholarly practice.

Understanding what’s not online

At the same time, it must be recognised that large quantities of our cultural heritage are not available in digital form. For example, only about 10% of the holdings of the National Archives of Australia are described in their collection database, and only a small proportion of these are digitised. Easy online access could foster a certain circularity in historical research where only ‘known’ resources are consulted. We need to develop tools and visualisations that reveal the valleys as well as the mountaintops — identifying the holes in our research fabric.

Critical engagement

More generally, we need to foster critical engagement with the tools and assumptions of digital research. Federated searching sounds great, but as scholars we need to expose the assumptions implicit in any such tool. What is being federated, from where, how is relevance being determined etc? Humanities e-research infrastructure should have built-in levels of reflexivity that enable scholars to understand the limits and assumptions of their digital research. Every algorithm contains an argument.

Documenting change

The resources we build are arguments with are subject to change. The Trove newspapers database, for example, is constantly adding new titles and articles, while users are improving the text transcriptions. Any analysis based on the holdings of this database needs to explicitly recognise this. At the very least the tools we have need to be able to generate time-stamped citations. It would be even better if we could capture a snapshot of the data to accompany our analyses. Perhaps there are possibilities for using something like the Memento project to ensure that the temporal context of humanities research is adequately documented.

Show your working out

Scholarly publication in history, and the humanities generally, tends to present a finished product. But as we delve further into digital research the research processes themselves will be equally important both for fostering critical engagement with tools and methods and for enabling others to reproduce or extend the research. We need easy ways for researchers to expose their working out (subject to whatever access controls they think appropriate). It should be possible to save a series of steps – search, analysis, visualisation etc as modules for sharing and re-use.

Follow your nose

Search needs to be complemented by rich, exploratory environments that encourage browsing, enable you to follow relationships, and foster serendipitous discovery. The problem with many collections is knowing enough about what’s in them to frame a useful search. Browsing, though a variety of interfaces — people, maps, events, record types, physical proximity — overcomes this problem. As more cultural institutions make use of Linked Open Data and shared identifiers — such as People Australia, Geonames or the Powerhouse Object Thesaurus — the possibilities for navigating this rich contextual space will increase.

Citation

We need to develop better models for embedding rich citations within scholarly research — citations that describe not only the resource in structured, machine-readable forms, but also relevant relationships. This will link research directly to resources, making scholarly outputs a means of resource discovery, and enabling resource databases to re-use the scholarly research to enhance their own descriptions and finding aids.

Constructing narratives

Moving beyond simple citation, we need better ways of exposing the structures of people, events, places and things that are referenced in our narratives. Linked Open Data provides a model, but we need tools to make it simple and examples to make it obvious.

Posted 08 November 2011 † tim § ‡ Comments (4) °

When did the ‘Great War’ become the ‘First World War’?

Townsville Daily Bulletin, 9 December 1939

I’m interested in time — in the way we imagine, manipulate, experience and describe time, particularly in the service of ideas such as ‘progress’.

This was one of the themes of Atomic Wonderland, but beyond constructing a few case studies it’s not all that easy to study. Or at least it wasn’t. Now projects such as Victorian Books are showing how we can explore the changing weights of ideas across times and cultures by analysing the contents of large textual collections.

Returning visitors will be probably be aware of my own experiments mining the contents of the National Library of Australia’s digitised newspapers database, available through Trove. So far I’ve focused on the development of generic tools and techniques, but I thought it would be interesting to apply these to my study of ‘progress’. Happily the NLA agreed and have awarded me a Harold White Fellowship for 2012 to do just that. Yippee!

I’ll be taking up the fellowship in February, but in preparation I’ve started to develop a few little sketches that prod at our fondness for periodisation. Labels such as ‘the Roaring Twenties’, ‘the Great Depression’ or even ‘the First World War’ are so familiar that we sometimes forget that they themselves have a history.

To begin with I decided to examine the question of when ‘the Great War’ became ‘the First World War’. At some point we realised that the Great War was not the final act in a centuries-long drama of European jealousy and jostling, but the first in a series of global conflicts. Can newspapers tell us when?

I already had a script that would generate a basic time series from a Trove query string. It simply takes the query, fires off a separate search for each year and grabs the number of matching articles. If the number of matches is more than zero, it also retrieves the total number of articles for that year and calculates the proportion matching the query. The results are saved in a json file which can be easily visualised using something like HighCharts. The original script needed a few tweaks to streamline the process, but I’ll describe these in detail in my next post.

For this experiment I constructed two queries. The first simply searched for the phrase ‘the great war‘ between 1900 and 1954. The second was a bit more complicated — it searched for any of the phrases ‘first world war’, ‘world war one’, ‘world war 1′ or ‘world war i’ across the same period. I fed the queries to my script and after a bit of ker-chugging, whirring and clunking I ended up with a graph.

Click to view the full interactive graph.

The result is not really surprising. As you can see on the full graph, the two lines cross late in 1941. With German victories across Europe and North Africa, the opening of the Eastern Front and, finally, the Japanese attack on Pearl Harbour, 1941 seems to make sense. But it’s interesting to see this reflected so clearly in such a rough and ready analysis.

What is perhaps more intriguing is the huge spike in 1939. Of course it makes sense that people would be referring back to the Great War as the prospect of a new conflict loomed, but it does make you wonder about the context of these discussions and how they might have developed as war edged closer.

Notable too are the earlier blips in the First World War count — the first centred on 1916 and the second on 1935. The peak in 1916 is actually due to the tags and comments added by Trove users. The standard ‘search everything’ option in Trove includes these as well as the text of the articles themselves. By using other search options you can choose to exclude the tags that match your query, but that seems rather messy. It would be nicer if Trove gave you the option of ignoring these matches from the start.

The West Australian, 24 May 1935

The second blip is a bit more interesting. By clicking on the graph and exploring the results from Trove, you can see that it’s due to the screening of a documentary film called ‘The First World War‘. The film used archival footage drawn from a number of nations and was based on Laurence Stalling’s book The First World War: A Photographic History. As one newspaper article noted: ‘this picture presents war, stripped of its gaudy trappings, and fearful in its grim reality’.

By way of comparison I tried a similar query using the Google Books Ngram viewer. The crossover point seems a little later, but of course books take longer to publish than newspapers. There is, however, no peak in 1939 for ‘the Great War’ — at least not if you use the combined ‘English’ corpus. If you examine the British-English and American-English corpora separately it’s a rather different story. Querying the British-English corpus produces something much closer to our Trove graph, complete with a spike around 1939. Again, this is only as we’d expect given the lesser significance of the First World War in American history.

This is, of course, only a sketch — something to prompt new questions or suggest avenues for attack. It’s made me want to find out a bit more about the nature of discussions in 1939, so I’ve fired up my Trove Newspaper Harvester and downloaded the text of all 6,582 articles from 1939 that include the phrase ‘the Great War’. More about that soon…

Posted 29 August 2011 † tim § experiments ‡ Comments (4) ° Tagged: ,

Some exhibition magic with Zotero and Omeka

Letter from CKR Kilby (‘Parkwood’, Hall FCT)

One of my most exciting archival discoveries was a cache of letters written by farmers from across NSW in 1938. Seeking to marshal support for the long-range weather forecaster Inigo Jones, The Land newspaper asked it's readers to send their opinions of the 'weather prophet'. And they did. One hundred and two letters were received, and duly forwarded to the Minister for the Interior. These 102 letters now reside in the National Archives of Australia.

I tell the full story of the newspaper’s campaign in Inigo Jones: The Weather Prophet. but I’ve always wanted to do something more with the letters. Inspiration finally arrived this year when a conference on rural media was organised to mark the The Land‘s centenary. I decided to create a little exhibition using the letters and, in doing so, provide myself with a platform for further research.

Step one: assembling the letters with Zotero

I’d already asked for the file containing the letters to be digitised, so images of all of the letters were available through RecordSearch. Zotero ships with a RecordSearch translator that I wrote some years ago, so 102 clicks later, I’d captured both the metadata and all of the images into my own database.

The Zotero translator saves individual pages with a generic title, so I then had to do some manual editing. I also had partial transcripts of the letters which I’d created during my original research. I simply cut and pasted the transcripts into a note field in Zotero and tidied them up. Pretty soon my collection was complete.

Step two: creating an exhibition with Omeka

The Weather Prophets

I actually started building the exhibition during an Omeka Bootcamp session I ran at THATCamp Melbourne. Starting with nothing but a LAMP server and my Zotero library, I had the basics of site up within the hour.

It was simply a matter of installing Omeka, adding the Zotero import plugin, creating an API key in my Zotero profile, and then pointing the Zotero import plugin to my collection. (The API key is necessary if you want to download files, such as my images.)  With a press of a button the internets started chugging away and pretty soon all the letters were in Omeka. Hey presto — instant exhibition!

With the framework built I could start to play a bit. First I installed Omeka’s geolocation plugin and started mapping where the letters had come from. Google’s geocoder did a pretty good job of finding many of the small country towns, but I also made use of Geoscience Australia’s gazetteer. In the end, I managed to geolocate all but one letter. Hey presto — a map!

I thought the pre-installed ‘Summer’ theme suited my exhibition, but I did make a few minor tweaks here and there. This included adding some RDFa into the list of references (which were also imported from Zotero).

Step three: going deeper with Voyeur Tools

Word cloud of the letters created by Voyeur Tools

The transcripts of the letters were all available on the site, but I thought it would be interesting to analyse their content a bit more systematically. To do this, I exported the collection from Zotero and wrote a few lines of Python code to save each transcript in a separate text file, named after the letter’s author. I could then zip up the transcripts and feed them to Voyeur Tools.

Most of the letters are pretty short, so there’s a limited amount to be gleaned from them in isolation. Voyeur Tools did, however, make it easy for me to explore further the prevalence of the phrase ‘on the right track‘.

What will be more interesting will be to compare the letters to other bodies of text about Inigo Jones, weather forecasting and rural life. I’m hoping to start mining some of this sort of material from the Trove newspapers database.

Next steps?

Using Zotero , Omeka and Voyeur Tools it was quick and easy to build my own little research hub. It’s still a work-in-progress of course — that’s the point! Now I can continue to assemble and analyse my sources, while giving my work a public face. Magic!

Posted 21 June 2011 † tim § weather ‡ Comments (3) ° Tagged: , , , , ,

Confessions of an impatient historian

Here’s the slides from the talk I gave recently in the Scholars’ Lab at the University of Virginia. Thanks to everyone who came and to the Scholars’ Lab for their hospitality.

A podcast version is now available.

View on SlideShare
Posted 16 June 2011 † tim § speeches ‡ Comments (0) ° Tagged: , , ,

Mining the treasures of Trove (part 2)

One of the advantages of building something yourself is that if you’re not happy with it you can tweak, change, modify and adapt until you are. But one of the disadvantages is that sometimes you get so caught up in all the tweaking, changing and adapting that you overlook a much simpler solution.

So I had a harvester that could save the publication details and content of all the newspaper articles in a search on Trove. But the warm glow of self-satisfaction quickly began to fade as I started to think about how I wanted to use the content I was harvesting.

The harvester saved the text of articles organised in directories by newspaper title. This seemed to make sense. It meant that you could easily analyse and compare the content of different newspapers. But what if you wanted to examine changes over time? In that case it’d be much easier if the articles were organised by year — then I could just pull out the a folder from a particular year, feed it to VoyeurTools, and start tracking the trends.

There ensued some minor tinkering. As a result, you can now you can pass an additional option to the harvest script, telling it whether to save the article texts and pdfs in directories by year or newspaper. Simply set the ‘zip-directory-structure’ option in harvest.ini to either ‘title’ or ‘year’. If you’re using the command-line you can use the ‘-d’ flag to set your preference. Easy.

But that set me wondering whether it might be possible to generate an overview, showing the number of articles matching a search over time. So I started on a modification of my harvest script that did just that — cycling through the search results, adding up the numbers. It wasn’t until I ran the new script for the first time that I realised there was a much simpler alternative.

All I needed to do was repeat the search for each year in the search span and grab the total results value from the page. D’uh…

So instead of sending hundreds or perhaps thousands of requests to Trove, all I needed was one for each year. From there it was easy and soon I had my first graph.

My first graph: Chinese in Australia (The Chinese Australian expert in my house predicted the 1888 peak.)

I was pretty pleased with that, but of course the raw numbers of articles on their own are rather misleading. The more interesting question was what proportion of the total number of articles for that year the search represents. Another quick tweak and I was grabbing the overall totals and calculating the proportions.

Total numbers versus proportions -- Chinese in Australia #2

At this point I invited my Twitter followers to suggest some possible topics — you can see the results on Flickr.

But what do the peaks and troughs represent? I wanted to use the graphs as a way of exploring the content itself. This was possible as I’d saved the data as JSON and used jqPlot to create the graphs in an ordinary HTML page. Courtesy of some clever hooks in the backend of jqPlot I could capture the value of any point as it was clicked. That gave me the year, so all I had to do was combine this with the search keyword values and send off a request to Trove.

So now instead of just looking at the graphs, you could explore them.

Explore -- Chinese in Australia #3

Perhaps you’re wondering how I managed to pull the Trove results into the page? Just a bit of simple AJAX magic combined with my own unofficial Trove API. (More about that in the next exciting installment!)

I’ve created a little gallery of graphs to explore. I’m still open to suggestions!

The code for gathering the data is all on Bitbucket, so start building your own. Just run the ‘do_totals.py’ script in the bin directory from the command line. The script takes two flags:

  • -q (–query) the url of your Trove search (compulsory)
  • -f (–filename) the path and filename for your data file (don’t include an extension)

The script will create a javascript file containing two JSON objects, ‘totals’ and ‘ratios’. These can then be fed to jqPlot. View the source of one of my interactive graphs to see how.

Of course it would be really nice to create a web service where people could create, share, compare and combine their graphs — but that might have to await a generous benefactor…

Posted 06 March 2011 † tim § experiments ‡ Comments (9) ° Tagged: , , ,

Inside the big tent

I seem to have been repeating myself a lot lately.

Bethany Nowviskie’s recent visit to our shores prompted much interesting discussion about the possibility of establishing a local digital humanities association. Not for the first time I expressed my hope that any such association would actively involve those of us who eke out an intellectual existence beyond the safe harbour of academe. Organising THATCamp Canberra made it clear to me — digital humanists are everywhere.

Reassuringly everyone seemed to agree, but still I worry. Bethany’s excellent post reflecting on her experiences in Australia and New Zealand prompted me to ponder once more the dangers of boundary-setting. ‘While we might belong to a society I suspect that our sense of belonging is more likely to be found at a THATCamp or in the collegial embrace of Twitter’, I suggested.

It seems I just can’t help myself. Craig Bellamy blogged the other day about the need for a research-led digital humanities, asking ‘inconvenient’ questions of our society and culture. It’s a useful and important argument, but it triggered another bout of nervousness. I submitted a comment to Craig’s blog, but it seems higher powers (or more likely the spam filter) decided my comments were best kept from public view. What I said was:

Nicely said, though I’m a bit worried about the distinction between ‘service’ and ‘research’. I’m with Tom Scheinfeldt in thinking that toolmaking (and service building) can precede the articulation of research questions and yet be an important strand in digital humanities research. Our enthusiastic and sometime playful embrace of new ‘conveniences’ can and should lead us in ‘inconvenient’ directions, but where does the service end and the research begin?

Similarly, in thinking of the new organisation (and reflecting my own prejudices), I worry that such a formulation downplays the work of people in libraries, archives and museums who fight long and hard to create ‘conveniences’ for researchers, who have little institutional support for open-ended research, but whose efforts nonetheless are informed by a rich understanding of the research process. I think we have a lot to gain by inviting such people to join us in the ‘big tent’ of digital humanities.

Bethany’s vision of the ‘big tent’ and her desire to find ways to support the careers of hybrid scholar/developers is inspiring. Tom Scheinfeldt’s post (mentioned above) and his more recent talk on ‘Stuff digital humanists like‘ also offer much-needed validation to an institutional fringe dweller like myself. Tom notes:

Innovation in digital humanities frequently comes from the edges of the scholarly community rather than from its center—small institutions and even individual actors with few resources are able to make important innovations.

But while celebrating my inner hybrid and enjoying the freedom that comes from living on the innovative edges, still I can’t shake a nagging feeling of illegitimacy, as if someone’s about to tap me on the shoulder and point the way out of the tent.

Perhaps it’s the talk of a ‘professional’ society — as if being a digital humanist is a job. For me it’s not. No matter what paid work I have, I’m a historian. Yep, I’m a historian who likes building digital tools for other historians to use. I’m a historian who pays the bills developing websites, editing other people’s content, or wrangling metadata. But I’m still a historian — it’s just who I am. So the ‘big tent’ for me is not so much about forging a career as finding a home.

Perhaps it’s insecurity born of jealousy or resentment. My partner Kate Bagnall and I are both historians, but it’s something we have to fit in around the edges of paid employment outside of academia. As we write articles, do research, organise conferences, referee articles and advise graduate students — all in our ‘spare’ time — it’s a little hard not be exasperated by the manoeuvres of academic game players, carving out their illustrious careers. (I’m sure you know the type.) But it’s a life we chose, and, despite its challenges, we find ways to keep both mind and body nourished.

Perhaps it’s a reflection of my own state of mind as I ponder yet another change of employment status in the coming year.

Or perhaps all this nervousness and worry is a sign that there’s more work to do. When I complained on Twitter about not being able to get to THATCamp, somebody suggested I organise my own. So I did. I want more than an association. I want a CHNM or a Scholars’ Lab. I want an Office for the Digital Humanities dispensing start-up grants for innovative digital projects. I want a new research funding system. I want THATCamps every other week.

I’m a historian and I like to build things.

Posted 22 December 2010 † tim § ‡ Comments (2) °