Entering the web of data

[view the presentation...] [view the triples...]

Keynote delivered at the annual conference of the Australia and New Zealand Society of Indexers, 14 September 2011.

This is me.

Today, Wednesday, 14 September 2011, I’m honoured to be able to join you here in the luxurious surrounds of the Brighton Savoy Hotel for the ‘Indexing See Change‘ conference. This is an event, a moment in history; we can pinpoint ourselves, this gathering, both in time and in space.

If we do that, if we move outside the moment and position ourselves on a timeline or a map, interesting things start to happen. Connections emerge.

Here we are at number 150, The Esplanade, in Brighton. A bit over a kilometre away is the stately villa, Kamesburgh. For many years Kamesburgh was also known as the Anzac Hostel — a refuge for permanently-incapacitated World War One veterans.

The Anzac Hostel opened on 5 July 1919. Here it is draped in its patriotic finery, from the collections of the Australian War Memorial. According to the caption, the Anzac Hostel was ‘a home, not an institute’.

Also amongst the War Memorial’s holdings is a wheeled bed that was used at the hostel. This particular bed was apparently occupied by one man, Albert Ward, for forty-three years.

Death notice for Alexander Kelley. Argus, 29 January 1944.

It was probably in a bed just like this that Alexander Dewar Kelley passed away on 27 January 1944. Alexander Kelley was cremated, and his remains interred amongst the roses at what is now called the Springvale Botanical Cemetery. Not far from my own grandparents.

Alexander Kelley spent close to half his life in the Anzac Hostel. Like many young men, he bravely answered his nation’s call to arms, but returned from war much changed. We can follow Alex’s war through his service record, easily-accessible through the website ‘Mapping Our Anzacs‘.

Alex was a coach painter who enlisted in the AIF in January 1916. Within a year he was in France. In May 1917 he suffered a gunshot wound to the head, but was able to rejoin his unit in August. Less than a month later though, he was wounded again, this time more severely. For Alex the war was over, and he was shipped back to Australia in May 1918.

‘Mapping Our Anzacs’ includes a scrapbook feature through which visitors to the site can attach notes or photographs to a service record. Amongst the the many thousands of postings is a fragment from a diary, found tucked inside the bible of Alexander Kelley’s mother. The diary entry reads simply: ‘Alex arrived from Front. Wet day. Saw him at “Caulfield”.’

Alex had survived and had returned to his family. This was a day to remember. But there was sadness too, for Alex was not the same young man who had left for the battlefields of Europe. In the diary fragment, ‘Caulfield’ is enclosed in inverted commas, indicating perhaps that the reunion took place, not in the suburb, but in the Caulfield rehabilitation hospital. Alexander Kelley was wounded in the face, hands and legs. He was left blind in both eyes and his right leg was amputated. He would live the remainder of his life a little over a kilometre away from here at the Anzac Hostel.

This is just one story. There are over 375,000 World War One service records held by the National Archives of Australia. How can we hope to understand a number like that? How can we hope to imagine the war’s impact on families, on communities?

‘Mapping Our Anzacs’ uses familiar Google maps to display the places of birth and enlistment recorded in many of those service records. But technical limitations make it impossible to display all the places at once. You can, however, take the same data and open it in Google Earth. If you then zoom in on Victoria, you see something like this.

Mapping Our Anzacs data viewed in Google Earth.

Each marker represents a place where a service person was born or enlisted. It’s impossible to read, of course, but that’s the point. There is so little blank space. As you zoom further, more markers appear, more place names resolve. It’s simple, but it’s powerful. They came from everywhere. From the smallest village to the biggest city; nowhere was untouched.

The ‘Mapping Our Anzacs’ scrapbook offers another perspective. It’s possible to extract the images posted to the scrapbook and present them on a 3D wall. Amidst an assortment of memorabilia, there are faces. Not places, or records — this is a wall of people.

Mapping Our Anzacs Scrapbook photos viewed through CoolIris

It’s worth noting too that like the markers on the maps, these faces link back to the actual service records. So they’re not just a new way of seeing the collection, they’re a new way of exploring it.

But the records don’t stand in isolation, they themselves have a context. A couple of years ago, Mitchell Whitelaw from the University of Canberra, undertook a project called ‘The Visible Archive‘ to investigate ways of visualising the holdings of the National Archives of Australia. Have you ever wondered what 360km worth of records looks like?

The collections of the NAA visualised by Mitchell's Series Browser.

This represents the holdings of the National Archives. Files within the archives are organised into series, and each square in this image represents a single series — there are about 60,000 of them. Naturally the size of the square gives an indication of the size of the series itself. It’s a fascinating and strangely beautiful picture.

It’s easy enough to pick out the World War One service records — Series B2455. In the interactive version of Mitchell’s series browser you can click on a box and display links between series, as well as other series created by the same government agency. Again, it’s not just a way of seeing the collection, but a means of exploring and interpreting it. As Mitchell says:

Visualisation enables us to literally show everything, to display large volumes of data in a way that reveals patterns and communicates context, but also provides access to the fine grain of individual elements.

But we can also employ such techniques to ask new kinds of questions. Can you imagine how Alexander Kelley and the other inhabitants of the Anzac Hostel must have felt in 1939? They had lost so much in the Great War, the ‘war to end all wars’, and yet within their own lifetime it was all happening again. More young men were answering the call, more lives were going to be destroyed.

There must have been a dreadful, disheartening moment when Australians realised that the Great War was not an end, but a beginning — the first in a series of devastating global conflicts. At some point the ‘Great War’ became the ‘First World War’, but when?

When did the 'Great War' become the 'First World War'?

What is perhaps more intriguing is the dramatic peak in the occurrence of ‘the great war’ in 1939. It’s no surprise that the looming threat of a new conflict would provoke comment and comparisons, but it does make you wonder about the context of those discussions and how they might have changed as the reality of war edged closer.

To start exploring this I’ve harvested the content of the 6,600 articles from 1939 that included the phrase ‘the great war’. Using an online text analysis service called VoyeurTools I can quickly generate a picture of their contents.

This simple visualisation shows us the relative frequencies of words within the articles. It doesn’t reveal any great mysteries, but it does suggest some possibilities for further prodding. The prevalence of ‘time’ and ‘new’, for example — might these help us understand the shift in perspective from one war to the next? We can follow this up by browsing the different contexts in which the words were used.

But what actually is it that we’re actually searching? We know that Trove includes newspapers from 1803 to 1954, but if we’re really going to analyse shifting words and ideas it’s important to have a clear picture of the sources of those words.

Something like this perhaps. This graph shows the holdings of the Trove newspaper database on 4 August 2011, organised by state. You can see, for example, that if you’re searching on a topic between the 1920s and 1940s you’re probably likely to get more results from Queensland than anywhere else.

So starting from our location here, today, we can make connections across time and space. We can pull back and look at the big picture, or dive in and examine the fabric of a single life. Through the web we can build and explore a rich and complex contextual network.

It’s an exciting time to be a cultural data hacker. We now have a growing range of tools and technologies available for extracting interesting data from a wide variety of sources, both structured and unstructured.

The ‘Visible Archive’ project started with well-structured data, courtesy of Peter Scott, the developer of the Series System — the descriptive framework used by many Australian archives. But we’re rarely so lucky.

Even when the data starts off in nicely-organised fields in a database there’s no guarantee that that’s how it’s going to be delivered to our web browser. In order to extract the data from my Trove graphs, for example, I had to write a little program called a ‘screen scraper‘ to identify and save the important metadata elements from the raw web page itself.

Where there are no subject keywords we can infer them using techniques such as topic modelling. Where there are no access points we can identify people, organisations, places and events using special tools developed for named entity extraction. Where there are no common identifiers across datasets we can employ record linkage technologies to find possible connections.

We can count words, we can identify parts of speech, we can formulate a measure of the similarity of any two pieces of text. Once we have some useful data we can manipulate and enrich it. Place names can be geolocated — you simply send your place name off to a web service and get back its latitude and longitude.

Increasingly these sorts of tools are becoming accessible to anyone. For historians they offer a means of wrestling with rapidly-growing bulk of source material that is becoming available in digital form. How do you make use of 5 million digitised books, 50 million newspaper articles or the complete archive of every public message ever sent on Twitter?

The digital historian Dan Cohen has noted:

These computational methods which allow us to find patterns, determine relationships, categorize documents, and extract information from massive corpuses, will form the basis for new tools for research in the humanities and other disciplines in the coming decade.

Dan is involved in a number of interesting projects investigating the possibilities of these techniques — often grouped together under the heading ‘text mining’. One of these projects, ‘With Criminal Intent‘, is looking to see what patterns can be drawn out of the digitised proceedings of criminal trials held at the Old Bailey from 1645 to 1913. That’s 197,745 trials, in case you were wondering.

Here’s one of their visualisations showing how the length of trials varies over time. Much to the surprise of the research team, this graph suggests a dramatic shift in legal practice around 1825 — defendants started pleading guilty!

A visualisation by the With Criminal Intent project showing changing trial lengths.

Rather than falter under the growing weight of digital sources, these technologies can actually thrive. The more raw material available, the more chance there is to observe and track new patterns. As digitisation continues apace will we ever reach the point when history can simply be read from a graph?

There are some researchers at Harvard who seem to think that’s where we’re heading. Borrowing liberally from the store of scientific metaphors they have staked out the new field of ‘culturomics‘. By mining massive digital resources, like Google’s scanned books, they hope to map the ‘cultural genome’ that would enable us to follow the evolution of language and culture.

But there’s something quite barren in this ambition. I prefer the vision of digital humanist Stephen Ramsay, who commented in regard to the ‘With Criminal Intent’ project:

The Old Bailey, like the Naked City, has eight million stories. Accessing those stories involves understanding trial length, numbers of instances of poisoning, and rates of bigamy. But being stories, they find their more salient expression in the weightier motifs of the human condition: justice, revenge, dishonor, loss, trial. This is what the humanities are about. This is the only reason for an historian to fire up Mathematica or for a student trained in French literature to get into Java.

Ultimately it’s the stories that nourish, anger, inspire and depress us. The closely-packed map of places recorded in World War I service records is so powerful because we know that under each marker are men, women, families, communities — each with their own story. These new technologies offer new perspectives, they raise new questions, and they challenge us with new contexts to explore and understand. But there is still space for stories and perhaps we can use them to give our stories new life and depth.

This is another World War One service record. It belongs to Charlie Allen. Charlie enlisted three times in the AIF and was discharged on medical grounds each time. It seems he had a problem with his ankle.

Charlie’s service record notes a tattoo, proclaiming his love for ‘Maud Gordon’. He married Maud in Sydney in 1917 and had two daughters soon after.

Charlie survived the war without further injury, but was not so lucky in peace. On 11 March 1938, Charlie was crushed to death between two railway cars. The accident happened at the Bunnerong Power Station, only a short distance from his home in Matraville. He was buried nearby in the Botany Cemetery.

We also know quite a bit about Charlie’s early life. Why? Because Charlie’s father was Chinese and he was therefore categorised as a ‘half-caste’, as someone who was not white, and therefore fell under the restrictions imposed by the White Australia Policy.

Charlie was born in Sydney in 1896. His mother was Frances Allen (sometime sweet shop owner and brothel keeper), his father Charlie Gum (a buyer for Wing On company). Charlie was raised by his mother, but in 1909, at the age of 13, he was taken to China by his father.

NAA: ST84/1, 1909/22/41-50

This certificate granted Charlie an exemption to the Dictation Test. Without it, he may not have been allowed back into the country.

Every time one of many thousands of non-Europeans resident in Australia sought to travel overseas and return home again they needed one of these certificates.

Charlie’s father returned to Sydney, leaving him in China. He lived with relatives in the town of Shekki (inland from Hong Kong). Charlie was naturally homesick, but had no means of getting back to Australia. He wrote to his mother in 1910:

Do try and bring me home every minute I think of you and long for a piece of bread and butter this tucker is not doing me well.

His mother wrote to the Prime Minister Billy Hughes in an attempt to enlist government help but to no avail. Charlie finally returned to Australia in 1915.

Despite this experience, Charlie visited China again in 1922 for 7 months. Once again carrying papers to grant him re-entry to the country of his birth.

These fragments of Charlie’s life have been assembled by my partner, Kate Bagnall, a historian of Chinese-Australia. They are remarkable, and yet not so, because there are many thousands of stories like Charlie’s contained within the voluminous records generated by the administration of the White Australia Policy.

We’re all of course familiar with the general outlines of the White Australia Policy, and the way it underpinned conceptions of Australia as a nation in the first half of the 20th century.

But what we sometimes forget is that it was also a massive bureaucratic exercise.

Forms and certificates were printed, issued, used and filed. Regulations were modified, guidelines were distributed and administering officers were managed and advised. Individual cases were reviewed, policy was changed and new forms and certificates were printed, issued, used and filed…

Much of this system is now preserved in the National Archives.

You can get a idea of the range of material available from a case study Kate has prepared focusing on the efforts of Poon Gooey, a successful businessman in Horsham, to keep his wife and family in Australia.

If we look again at Charlie’s certificate from 1909 we can see that it contains a lot of interesting structured data:

• name
• place of birth
• age
• height
• destination
• date of departure
• name of ship

We estimate that there are probably about 50,000 of these forms remaining in the Archives, and then there’s case files and a variety of other government documents.

Wouldn’t it be great if we could extract this structured data. If we could piece together the slivers of identity that remain within the Archives and give people back their lives.

This is the dream of Invisible Australians, a project Kate and I are trying to turn into a reality. Our aim is to build systems that will enable this data to be extracted, aggregated, shared and connected — whether to a family tree, a cemetery record, or another document in another archive.

Imagine being able to navigate the network of lives, families and relationships. To follow their journeys, to share their tragedies, to celebrate their small victories against a repressive system.

Imagine being able to watch them age.

We tend to assume that new technologies require us to change, to adapt. But sometimes they can take advantage of our strengths. Mitchell Whitelaw is interested in finding out what happens when you take large cultural datasets and try to ‘show everything’. Such an approach, he suggests, takes advantage of the raw processing power of computers, while giving us space to do what we’re good at — finding patterns, making connections, crafting meanings.

The History Wall tries to create a similar sort of space. The History Wall brings together material from a range of different sources — newspaper articles from Trove, biographies from the Australian Dictionary of Biography, records from a database of NSW convicts, population statistics, collection items from the National Museum of Australia — you can pretty much plug anything in as long as it has a date attached to it.

Irish History Wall

For a particular year, the Wall retrieves a random sample from the available sources, jumbles everything up and then throws it onto the screen. As a result, no two views of the Wall are ever quite the same. This is not a traditional exhibition. There is no curator controlling the content or designing the structure. It’s ephemeral, it’s serendipitous — instead of relying on an authorial voice to smooth over the gaps and transitions, it leaves open the cracks and allows new contexts to seep in and around each item.

As the pioneering digital historian Edward Ayers noted:

even isolated and inert pieces of evidence — a list, a letter, a map, a picture — can assume new and unimagined meanings when placed in juxtaposition with other fragments.

This is not an absence of narrative, but an opportunity for narration. Edward Ayers suggests that we’re actually quite comfortable filling in blanks and untwisting timelines:

Humans, presented with pieces of information about people, put things into the form of a story. They need not be simple stories, for we know how to deal with unexplained lapses of time, flashbacks, and overlapping narratives. We know how to imagine, infer, things happening at the same time in different places. Film and television train all of us at early ages to weave strands of narrative out of intentional (if carefully constructed) confusion and to take pleasure in that weaving.

And so I can show you a death notice, or a certificate and you will take those fragments, those isolated data points and you will construct a story — you will see the person behind them, you will imagine their life. It’s what we do. We’re good at it.

Computers on the other hand will just see data.

In her ode in praise of humanities data, digital humanist Amanda French wonders whether we always need to crunch our data into abstract, pliable forms:

What I wonder is whether instead we can begin with the data, or with a datum, and simply watch for what it may tell us, even if what it tells us is simply a story.

Yes we can. And we should teach computers how to do it as well. Not because we want them to take over. Not because they can necessarily do it faster or better. But because they can help us share, preserve and connect those stories.

Let’s think again about the array of documents that Kate has assembled to piece together the story of Charles Allen. How can you share this sort of material? Typically you’d ‘write it up’. You’d capture the story behind the data and commit it to words. The documents would then become evidence — points of connection between your text and the historical record.

So in order to share the meanings of these documents we remove them from the context of the person’s life and marshal them as allies to proclaim the authenticity of our rendering. Wouldn’t it be better if we could tell the story, but maintain within our texts the direct connections between sources and subject?

What we need is a data framework that sits beneath the text, identifying people, dates and places, and defining relationships between them and our documentary sources. A framework that computers could understand and interpret, so that if they saw something they knew was a placename they could head off and look for other people associated with that place. Instead of just presenting our research we’d be creating a whole series of points of connection, discovery and aggregation.

Sounds a bit far-fetched? Well it’s not. We have it already — it’s called the Semantic Web.

The Semantic Web exposes the structures that are implicit in our web pages and our texts in ways that computers can understand. The Linked Data movement takes the basic ideas of the Semantic Web and turns them into a collaborative activity. You share vocabularies, so that other people (and computers) know when you’re talking about the same sorts of things. You share identifiers, so that other people (and computers) know that you’re talking about a specific person, place, object or whatever.

Linked Data is Storytelling 101 for computers. It doesn’t have the full richness, complexity and nuance that we invest in our narratives, but it does at least help computers to fit all the bits together in meaningful ways. And if we talk nice to them, then they can apply their newly-acquired interpretative skills to the things that they’re already good at — like searching, aggregating, or generating the sorts of big pictures that enable us to explore the contexts of our stories.

This is why we’ve always imagined Invisible Australians to be something more than an online database. We want to provide points of connection that other people can build into their own stories. But to do that we have to pay attention to things like vocabulary management and authority control, we have to construct web addresses that are not going to break every time we upgrade our software. We have to think about the sorts of things we’re talking about — not just people, but government agencies, legislation, certificates, and correspondence. How do we describe these entities and what sorts of relationships do they have?

And of course we need to expose all these structures so that we can say, these things are people, these are events, these are places and these are documents.

Or perhaps, to introduce Alexander Kelley.

Or remember Charles Allen.

You might be wondering why we don’t just leave it all to the computers themselves. Didn’t I just talk about all the exciting new tools and techniques that enable us to analyse the structures of texts? Perhaps we should just wait for the Culturomics guys to solve all the problems.

But who defines the problems?

Our postmodern sensibilities encourage a suspicion of neutrality. Labels like ‘the new museology’ or Archives 2.0 reflect an awareness that the way we describe and arrange our collections is itself culturally-determined. It’s not just a matter of what our descriptive systems show, but what they hide.

Tim Hitchcock, another member of the ‘With Criminal Intent’ team, has described how online technologies can change the way we access archives. Instead of being forced to navigate the hierarchical structures that archives impose on records, which in turn tend to reflect the workings of the institutions that created the records, we can directly find the people whose lives were regulated, influenced, shaped or controlled by the policies of those institutions.

Instead of merely hearing ‘the institutional voice… in all its stentorian splendour’, he says, we can listen in to ‘the quieter tones uttered by the individual’.

This reminds us that search boxes, along with other digital tools, themselves embody arguments. There are assumptions built into their code about what is relevant, what is significant, what is necessary.

We can build our own tools of course, and we can critique other people’s algorithms. But what if we just want to collect and share stories?

Linked Data gives us a way to present an alternative to Google’s version of the world. We can argue back against the search engines, defining our own criteria for relevance, and building our own discovery networks.

Changing the way we access resources changes the sorts of stories we can tell. Tim Hitchcock asks:

What happens when institutions and archives are ‘decentred’ in favour of the individual? What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

Perhaps the invisible become visible.

Wordle's interpretation of the Australian Constitution

If you haven’t played with Wordle yet, you should. Feed it your latest article, your thesis, your blog and see what emerges from the cloud.

Some months ago I wordled the Australian Constitution (as you do). Wordle’s expert legal analysis offers a fairly positive assessment of our federal system, suggesting that Commonwealth and state powers are fairly well balanced. Who needs a High Court when you can just count words?

[Contains many broken links – included for historical interest only!]

what is there to know about archives?

In this age of virtual wonders, it seems that our past is rushing towards us. New communication technologies promise greatly improved access to Australia’s cultural heritage. The previous government had hoped to lead us along the aisles of our own “Electronic Smithsonian”, according to its 1995 statement, Innovate Australia [HREF 2]:

…school children will be able over the Internet to read the diaries of Cook and Bligh, Burke and Wills, stories of the Royal Flying Doctor Service in outback Australia, and see the works of Rover Thomas and Arthur Boyd.

In rather less expansive terms, the current government plans a National Cultural Network [HREF 3] that will “simplify and enhance the communication and exchange of cultural and heritage resources, information and ideas”. But where will the material be coming from to fill the virtual display cases? Government statements often point to “libraries, museums and galleries”, but what about archives? Of course we’re meant to assume that archives are somewhere amongst the “cultural and heritage organisations”, and anyway the major libraries collect archival material like diaries, letters and manuscripts. But consigning archives to the ranks of fellow-travellers in the information putsch, means that little attention is given to their specific needs and their unique potential. We will have no strategies for ensuring that appropriate forms of access are developed. Instead of delving deeply into our “vast cultural resources” we may simply skim the top, presenting only the familiar in a new digital guise. Instead of an “Electronic Smithsonian” we might end up with an “Electronic Disneyland”. This paper will examine how the World Wide Web might be used to avoid this by facilitating access to Australia’s archival resources – providing pathways for exploring our collective memory.

Archives are not dusty, boring warehouses – they are treasure troves. The many voices which constitute the Australian experience can still be heard in archives. This is not history in its sifted, digested and pre-packaged form, this is memory – the surviving fragments of our lives and our work. This wealth of material is scattered around the country in over four hundred repositories of various sizes and types. They may be attached to schools, churches, businesses, or local councils, and staffed by part-time or volunteer archivists. They are all around. Unfortunately, however, diaries and letters do not photograph as well as artefacts, and you certainly can’t use them on TV, so unless Einstein’s notebook or Jackie Kennedy’s shopping list happens to be up for sale, archives don’t make the news.

Moreover, archives are not always easy to use. You can walk into a library and expect to have a fairly good idea about how things work and what you expect to find there. Archives tend to lack this consistency in the way their collections are described and managed. Also, of course, the holdings of an archive will tend to be unique – you don’t just buy a standard set from your local reseller! So how do you know what is held where? The National Library of Australia [HREF 4] has developed a Guide to Collections of Manuscripts Relating to Australia (1990), available on microfiche, that provides basic details of around 6,0000 non-government records collections. The Guide has been issued in 20 parts over the past 30 years, and relies on cooperating repositories to supply information about their collections. Subject guides, such as the Australian Science Archive Project’s Register of the Archives of Science in Australia (RASA) [HREF 5] (see also McCarthy, 1991) have also been developed, but information remains fragmented and incomplete.

The Archives Act 1983 [HREF 6] proclaimed that Australian Archives [HREF 7] would be responsible for maintaining a National Register of Archives, but the proposed nature of this Register has never been articulated. In the meantime, the Internet has arrived in Australia, bringing with it a range of new possibilities for sharing information. Here the evil spectre of standards emerges to block our way. “You cannot proceed”, it shrieks, “without the magic words… and the appropriate descriptive standards”. But this spectre is more of fear than substance. In the 1980s it was thought that the development of a National Register, then conceived as some sort of centralised database, was dependent on the implementation of standards for archival description. However, as Chris Hurley (1994), Australia’s descriptive standards guru, has pointed out, the situation has changed. Client-server architecture and data interchange standards such as Z39.50 have eliminated the need for specific archival standards for sharing information. Descriptive standards are important in ensuring that the data is worth sharing, but not for delivering it over a network.

There is no doubt that some sort of national system providing information on archival sources is now possible. The question is, how do we go about it? When confronted with a situation like this, there is a tendency to look for the BIG SOLUTION – that all-encompassing, highly-complex, very-expensive, monolithic system that is usually out of date by the time it begins to operate (if it ever does). The “Deep Thought” approach, which seeks the ultimate answers to everything, may reflect humanity’s unending quest for meaning, but what happens in the meantime? The committees form, meet, report, re-form, re-meet, re-report…. – all the while the technology changes and the users miss out.

I think it helps to keep in mind what the WWW does best - linking. Old information can be recycled into a new context by the establishment of appropriate links. Work can proceed in a piecemeal and distributed fashion, and still at a later stage be integrated into a larger-scale resource. With some basic cooperation, very little work need be wasted. You don’t need to have a fully-formulated grand plan to make a start. The WWW is not an all-or-nothing kind of place. You can experiment. You can try out systems, if they work you can scale them up, if not you can try something new. It’s not like developing a mainframe database, or pressing a CD-ROM – it’s a journey, not an end-point.

The planned development of AMIS [HREF 8] (Australian Museums Information System) is instructive here. Rather than remain stalled by the problem of standards, AMIS is proceeding in a staged fashion. In a first phase, information is being collected about Australian museums for access through a centralised database, with links to institutional WWW pages where they exist. But this phase will be used to explore the ways in which free-ranging access to the museums’ own native information systems might be provided to users. Solutions will be found along the way.

There are other aspects of the AMIS project that are worth noting. First, it is inclusive, recognising that significant collections aren’t just held by the big museums. AMIS is also scalable, allowing institutions to increase their participation gradually as their technical capabilities develop. Third, AMIS is open, reliant on general standards such as Z39.50 [HREF 9] and SGML [HREF 10]. This will allow it to take advantage of further software developments, particularly in indexing and navigation. These are important principles which should be kept in mind as we attempt to develop access to our archival resources.

Before we go too far, we need to think about the types of information on archives that can be made available. Essentially, there are four levels of information, each providing more detail, taking us closer and closer to the original records:

1. Contact details – Where are the repositories?
2. Summaries of their holdings – What’s in the repositories?
3. Finding Aids – What’s really in the repositories?
4. Items from the collections – Why don’t you show me?

where are the repositories?

Where are Australia’s archival repositories? How can you contact them? What facilities do they offer? One way of improving access to Australia’s archival resources is to make this sort of basic information more freely available, and one simple way of achieving this is to encourage archives to publish information about themselves and their holdings on the WWW. It seems an obvious step, but too often the obvious is overlooked. Such information might not make much impact on the average Australian, but it will at least provide intrepid researchers with a few more signposts. More importantly it will build a level of interest, knowledge, and resources amongst the archival community so that more ambitious projects can be designed and undertaken.

A number of Australian archives have already established WWW sites [HREF 11]. As you would expect, these tend to be associated with government, universities and the large public libraries – organisations that already have a significant Internet presence. One notable exception, however, is the Benedictine Community of New Norcia [HREF 12]. Sometimes, one enthusiastic individual is all that is required in a small organisation to establish a WWW presence, but mechanisms need to be in place to nurture and direct their enthusiasm, to share skills and information, and to provide access to the necessary facilities.

The government’s proposed National Cultural Network may be one source of support, promising that “a total of $10 million over three years will be made available for the provision of hardware and software for cultural institutions, together with training in new technologies and assistance in the digitisation of their collections”. In the current atmosphere of cost-slashing, it is unclear to what extent, and in what way, this program will be pursued. However, the Department of Communications and the Arts [HREF 13] in conjunction with the Ngapartji Co-operative Multimedia Centre [HREF 14] has already established a set of tutorials [HREF 15] aimed at assisting cultural and heritage organisations to develop on-line resources.

Already, some archival organisations have cast open their servers to embrace their peers. Australian Archives has established the Archives of Australia site [HREF 16], with links to state archives, and information on professional bodies. It also hosts a site for the Sydney City Council Archives [HREF 17]. ASAP [HREF 18] (the Australian Science Archives Project) currently hosts the Basser Library [HREF 19] of the Australian Academy of Science , and the Science, Technology and Medicine Archives (STAMA) Special Interest Group [HREF 20] of the Australian Society of Archivists (ASA). It also runs email lists for STAMA and the ASA (the Aus-Archivists list [HREF 21]). The University of Sydney Archives [HREF 22], meanwhile, provides a home for the University and College Archives Special Interest Group [HREF 23] of the ASA, and the International Council on Archives, Provisional Section of University and Research Institution Archives [HREF 24].

In order to overcome unequal access to the Internet, such cooperative activities will need to be extended and coordinated, presumably by the peak representative bodies – the Australian Council of Archives, and the Australian Society of Archivists [HREF 25]. Continued education in the use and development of WWW resources will also important, as innovative approaches to the presentation and dissemination of archival information are more likely to come from archivists than IT specialists. The broader the base of experience the better.

What sort of material is likely to be published? A 1995 study of archival WWW sites around the world found that there was a great diversity in the types of information being made available (Wallace, 1995). Similarly, the content of Australian sites varies widely, from a single page of contact information, to newsletters, finding aids and collection lists. Such lack of consistency fills some minds with dread, but why worry? Organisations will naturally have their own needs and priorities. The important thing is not that they follow some pre-determined model, but that they share information about their plans and approaches. The focus should be on commonality, not difference, for it is on the common ground of shared knowledge and experience that the seeds of new collaborative projects will sprout.

An alternative to this “let a hundred flowers bloom” approach is the establishment of a centralised directory, providing information on Australian archival repositories whether or not they are on the WWW. But this is no “either/or” situation, for the WWW enables the two approaches to be pursued in a highly complimentary fashion. The centralised directory enables us to ensure that non-Internet-equipped repositories are not pushed off the edges of the cultural map, while the existence of individual WWW sites backs up the directory by providing more detailed information on repositories. The two fundamentally different approaches become part of one integrated network of archival information.

Not only is this possible, it’s been done! Back in 1992, in the days of printed books, the Australian Society of Archivists (1992) published a Directory of Archives in Australia. This contained contact details, information on facilities and access, summaries of holdings etc, for more than four hundred archival repositories around the country. ASAP realised that this could provide the basis for a valuable WWW resource, and with the assistance of the ASA, it set about the conversion process. The WWW edition of the Directory of Archives in Australia [HREF 26] is now available. In addition to the information contained in the printed version, the WWW edition provides URLs and email addresses where available. A publication which was quickly become out of date, has been updated and given new life as a gateway to Australian archival resources on the WWW.

This has some interesting parallels with AMIS, the first stage of which is a national online directory of Australian museums, linked to the museums’ own WWW sites where they exist. However, AMIS aims to go further, for they are also gathering information about the collection management systems of museums around the country, in order to provide integrated access to a wide range of museum databases:

When fully developed AMIS will enable users to access information about all Australian museums, from objects in their collection to their exhibition programs, in one search. For example, a user researching gold mining in 19th century Victoria or planning a tour of south east Queensland museums will be able, through AMIS, to search the entire national network for relevant information. (“Introducing the Australian Museums Information System” [HREF 27])

Ideally the same could be done for archives, but how?

what’s in the repositories?

Knowing where repositories are is useful, but most often we’re more interested in what’s in them – in what collections they hold. The next stage in developing access to our public memory is to investigate how to make such information available. Once again, let’s start simple – what resources are already out there? The Directory of Archives in Australia itself provides a broad coverage of archival holdings. Most entries include information in the fields “Acquisition Focus” and “Major Holdings”, which can be queried using a fielded search interface [HREF 28]. To use the AMIS example, a search on “gold mining” in the “Major Holdings” field reveals that the James Cook University Archives holds “Charters Towers Gold Mining Records: Accounts 1895-1940″. This is a useful beginning, but being dependent on the repositories’ own identification of their “major” collections, it provides only a highly select sample of archival holdings.

The contents of the National Library’s Guide to Manuscripts Relating to Australia are available through the Australian Bibliographic Network (ABN) [HREF 29], which does have an Internet gateway. The summaries of archival collections have been encoded in MARC-AMC (Machine Readable Cataloguing – Archive and Manuscript Control) format and added to the general bibliographic database. The idea of manuscript collections and published works being embedded within the same information space is, in many ways, an attractive one. It follows the model of RLIN [HREF 30] in the USA, which includes over 500,000 summaries of archival collections in its massive bibliographic database. Practically, however, the archival content of ABN is almost invisible, and the pricing structure makes browsing difficult. ABN itself is in the process of being made-over as part of the much-vaunted World 1 service [HREF 31], and it is possible that the archival summaries will be made accessible as a separate database. Other means of delivering the contents of the Guide, or some version thereof, directly through the WWW are also being considered.

The only other national guide available on the WWW is ASAP’s Bright Sparcs [HREF 32], the on-line incarnation of its Register of Archives in Australia. Bright Sparcs is a subject-oriented guide, providing information on over 1300 collections around the country relating to science and technology. You just look up the name of a scientist, or a scientific field, and you are presented with a list of repositories with relevant material and summaries of the collections. But Bright Sparcs is more, as you will see.

A number of individual institutions provide information about their holdings. In particular, Australian Archives has developed an on-line version of its RINSE database [HREF 33], which provides detailed information on about “7,400 agencies, 570 people, 64,700 series and 100 organisations”. This huge amount of information is accessed by a series of alphabetical indices to functions, agencies, persons and organisations. This can be fun to browse, but those unfamiliar with Australian Archives’ system may find it rather daunting. There is currently no search facility available. Other institutions such as the National Library [HREF 34], the Basser Library [HREF 35], the University of Melbourne Archives [HREF 36], and the University of Sydney Archives [HREF 37], provide lists of their holdings. The National Library is the only institution that provides database access to collection information, through its OPAC [HREF 38]. As with the Guide, MARC-AMC format records are used to provide summaries of many of its manuscript collections.

More repositories are likely to provide access of some sort to their collection databases, and no doubt it would be possible to undertake some AMIS-like project to weld their collection information systems together through the use of a common standard, such as Z39.50. This would need to be accompanied by dedicated funds to assist smaller archives to computerise their holdings, as has been done in the museums sector. But there is a more fundamental question – what would we really achieve? We would still only have access to collection names or summaries – a few paragraphs, at best, attempting to describe hundreds, perhaps thousands of files. AMIS can aim to provide item-level descriptions, but to achieve the same for archives, we must move down another layer

what’s really in the repositories?

Collections within collections within collections… An archival repository doesn’t think about its holdings as just so many pieces of paper. Each document, each file exists as part of a collection, usually defined by the person or agency that created it – its provenance. The meaning and significance of a file comes not just from its own contents, but also from its context within a collection. Such contextual information – the collection’s provenance, arrangement, and contents – is usually recorded in a finding aid.

So far we’ve been considering making information available about the range of collections in Australia – their location, their date range, their subject area. This may well provide the researcher with enough information to know that a trip to a particular repository will be worthwhile, but the information of most use is the detailed collection listings provided by the finding aids themselves. Collection summaries provide us with a catalogue entry, telling us where to find the material and what to expect, just as if you were looking up a book in a library catalogue. But the finding aid is the equivalent of the book’s contents, introduction and index. It moves us another step closer to the actual contents of the collection.

A team led by Daniel Pitti [HREF 39] at the University of California, Berkeley, has been working for some years to develop a standard for on-line finding aids. After studying various options, the project settled on SGML as the most effective way of provided structured, distributed access to this potentially huge resource. An Encoded Finding-Aid Definition (EAD)has been developed and is currently in alpha release, with trials being undertaken at a number of institutions.

The developers of the EAD foresee an integrated system where the MARC encoded collection summaries, currently on RLIN, are linked to the SGML-encoded finding aids, which themselves may be linked to digitised copies of the collection contents.(Pitti, 1995 [HREF 40]) But this does not mean that the provision of summary information must necessarily precede the encoding of finding aids. The establishment of separate systems for summary and detailed information has more to do with the pervasiveness of the bibliographic mentality and the limitations of past software, than it does about descriptive needs.

Of course, summaries are useful as a first-level guide. But what are they summaries of? They are summaries of information contained within finding aids. Once finding aids are encoded in a standard format, it should be straightforward to extract the summary information as required. We don’t need two systems, just two (or more) ways of viewing the data held within the one system of encoded finding aids.

This does not mean, however, that we should dump all existing efforts and await the EAD revolution. As I have argued, the WWW is a platform where little information need be lost. Fragmentation is merely a link waiting to be made. Resources such as the Guide to Manuscripts will remain valuable, and it is to be hoped that the records in ABN (or its successor), like those in RLIN, will be linked to finding aids as they come on-line. The point is, that as we come to survey our pathways, we should be conscious not just of what has been done, but of what can be done. We need to remain open to opportunities.

In any case, we can’t expect the EAD to be implemented quickly and uniformly, particularly if we wish to remain sensitive to our principles of inclusivity and scalability. The conversion and marking-up of finding aids from a variety of software systems (if in digital form at all) to SGML, and delivery of these to users both internally and externally will not be a trivial matter. If the barriers for entry are too high, repositories will be inclined to postpone their involvement, perhaps indefinitely. We need to provide a few steps along the way.

The first step is the establishment of a centralised listing of finding aids on the WWW. You can’t get much more simple than a hyperlinked listing, but even this will be of significant benefit to users. Moreover, by linking to finding aids in whatever form they are published, be it text or hypertext, you are ensuring that repositories will gain some reward for any effort they make towards the digitisation of their finding aids. Such a listing [HREF 41] has already been established as part of the Directory of Archives in Australia. Currently over 260 finding aids are listed.

Building on this base, it is relatively easy to use a system such as Harvest [HREF 42] to build an index and simple search interface to all of the finding aids available on-line. This too has been done [HREF 43]. Two simple steps have significantly improved access to our archival resources, and this, as they say, is just the beginning. Finding aids will continue to be added, and as the numbers increase, the listings and indexes can be distributed across a number of sites to ease the administrative burden.

It sounds wonderful, but when you come to view the results of a search process under such a system, you quickly realise the value of standards. Without some standardisation across finding aids, it is impossible to be sure that the search will return information in a meaningful way. That doesn’t mean that it’s EAD or nothing. A few simple standards will add considerable functionality – for example, ensuring that the collection name is included in the <TITLE></TITLE> field of a HTML finding aid! From there, you could agree on the use of the HTML <META> tag to provide values for certain fields, such as “Collection Name”, “Repository Name”, “Start Date”, “End Date” etc. These could be tailored to fit in with other WWW metadata initiatives such as the Dublin Core Elements. Information in the <META> tag can be read by indexing programs, enabling structured queries, and standardised result formats.

Education and assistance will accompany this standardisation by stealth, equipping all members of the archival community with the means to participate as their own resources allow. Tools and methodologies will be developed and shared, moving us all towards some EAD-based network. Our aim should be a system with levels of compliance, allowing as many repositories as possible to play a part, while providing a clear pathway for development through standardisation. Instead of waiting for the full implementation of the BIG SOLUTION, we can find our answers along the way – building expertise, and remaining open to future developments. Rather than asking users to wait while the “perfect” system is built, we can invite users to be part of the process, giving them the power to choose to use finding aids of varying levels of compliance. The WWW gives us the power to consolidate our pathways over time, to see them develop from barely-blazed trails to well-trod thoroughfares. We don’t have to wait for the freeway developers

show or tell?

No doubt some of you are wondering by now, ‘Why bother with all this finding aid stuff? Why not just digitise the archives themselves?’ Hopefully one answer will come quickly upon reflection. As I have said, the meaning of individual items is dependent on their context within a collection. Finding aids provide this context. Without them, individual digitised items would no longer be records to inform our understanding of society, they would be mere souvenirs – digital ephemera.

The other reasons are more mundane. Digitisation is expensive and time-consuming. Given that many repositories struggle along with funding barely adequate to cover the creation of appropriate finding aids, it is unrealistic to expect wholesale digitisation. Digitisation of complete collections will only occur, in the near future, in circumstances where the expense can be justified. This may be the case in regard to certain high-use collections in major repositories, or for preservation purposes as an alternative to microfilm. Active business records requiring easy or distributed access are already being digitised.

Otherwise, digitisation will occur for exhibition or demonstration purposes – to give users a greater understanding of the nature of archival sources. Australian Archives, for example, provides images of some significant documents [HREF 44] from its collections. Hopefully digitisation will be used to provide a more rounded picture of the circumstances surrounding an archival collection. A fragment of voice or handwriting perhaps, to give us some feel for the people involved, or a photograph or film clip to help us imagine their life and times. Funding will be the key. It is no coincidence that the only major project in Australia involving the digitisation of a manuscript collection – the Banks CD-ROM project (Anemaat, 1995) – has its own dedicated funds, through a long-established trust.

But the issue of what to digitise and when, raises another more fundamental question – where should our pathways end

the end of the road?

The strategies I have outlined for accessing archives on the WWW are unremarkable, perhaps even conservative. They simply involve developing skills and resources within the archival community, taking advantage of existing resources, and moving gradually to integrate and standardise activities. The main point is that we will learn by doing, not by waiting. We have the opportunity to provide a new level of access to this country’s archival resources, to build and signpost the pathways that will open the vaults of our public memory. But still, this is not enough. Even as we follow these pathways doorways appear to the left and right. To stay on our route is to keep within the realm of archives, to open one of the doorways is to explore the connections that might be made between archives and…

The WWW offers us the chance to move beyond the disciplinary boundaries that separate archives from libraries, documents from images, history from its sources. Our archival resources can be integrated into the broader information environment through the construction of multi-layered virtual spaces that encourage exploration for research, education, or even entertainment. Within these spaces, archival information would be interlinked with images, biographical stories, news items, bibliographical data, exhibitions, and historical articles – information sources both on the net and outside.

Bright Sparcs [HREF 31] is an attempt to develop such a space centred on the history of Australian science and technology. Bright Sparcs emerged from our database, the Register of the Archives of Science in Australia, but has been transformed by life on the WWW. The original database provided access to information on over 2,000 Australian scientists from the 18^th century to the present. Brief biographical notes were provided together with a summary of any known archival sources. Into this broad framework other resources have been woven, including a bibliographical register of Australian physicists to 1945 [HREF 45], and the biographical memoirs [HREF 46] of Fellows of the Australian Academy of Science. All of these resources are fully-linked – each provides a gateway onto the others. An archival finding aid has become a space inhabited by figures from Australia’s scientific past.

Bright Sparcs will continue to grow. We will keep adding and linking, building layer upon layer of this information environment. New resources will be interwoven. New gateways will be constructed – thematic exhibitions tailored to particular audiences. We will encourage contributions to Bright Sparcs, and assist individuals and organisations to make their material available over the WWW. Each addition will enrich this environment, adding flesh to the original database, until Bright Sparcs’ archival underpinnings seem no more than lines spoken in a grand, unfolding theatre – part of the story, not abstracted “collections”.

Daniel Pitti envisages a ‘comprehensive archive, library, research, and publishing environment that can provide and orderly, civilized space for scholarly communication.’ (Pitti, 1995 [HREF 39]) I would hope for something more open and spontaneous – a do-it-yourself museum where people could use the information space provided by archives and other cultural institutions to tell their own stories, to add to our public memory, by drawing upon it. Hopefully the National Cultural Network will encourage the development of such free-ranging cultural landscapes, but it can only do so if the custodians of our heritage are prepared to investigate the possibilities.

On the WWW we have the opportunity to give archives back their context, to embed them in stories about the people, places and events that brought about their creation. Our aim should be to create an infinite number of pathways through this collective memory – opening our culture for exploration and greater participation, not simply arranging our institutions in a virtual showcase.

REFERENCES

Anemaat, Louise (1995), “The ‘Banks on CD-ROM’ project at the State Library of New South Wales”, Archives & Manuscripts, vol. 23, no. 2.
Hurley, Chris (1994), “Data, Systems, Mangement and Standardisation”, Archives & Manuscripts, vol. 22, no. 2, pp. 338-359.
McCarthy, Gavan (1991), Guide to the Archives of Science in Australia – Records of Individuals, D.W.Thorpe, Melbourne, 1991.
National Library of Australia (1990), Guide to Collections of Manuscripts Relating to Australia: A Selective Union List. Supplements were issued in 1990, 1992, 1994 and 1995.
Pitti, Daniel (1995), “Settling the Digital Frontier: The Future of Scholarly Communication in the Humanities”, paper presented at the Berkeley Finding Aid Conference, April 1995 [HREF 46].
Wallace, David A. (1995), “Archival Repositories on the WWW – Preliminary Survey and Analysis”, Archives and Museum Informatics, vol. 9, no. 2, pp. 150-175.