Closed Access

Presented at DHA 2016 in Hobart. Follow my research. View the slides.

A good type

In January 1954, the economic historian Ernst Boehm requested access to Commonwealth government records as part of his research into the depressions of the 1890s and 1930s. The request was forwarded from the Archives Office to the Prime Minister’s Department where it was noted ‘Boehm is I believe a good type + will agree to the usual conditions’.[1]

This sort of character assessment was important because there were few formal guidelines around access to government records. The Commonwealth Archives Committee reported in 1955 that the use of government archives was ‘seriously hampered by the lack of an access policy’.[2] The Prime Minister’s Department noted that requests for access were ‘treated on an “ad hoc” basis’.[3] Prospective researchers first needed to establish their ‘bone fides’. Archives staff would then attempt to identify records of interest and submit them to the relevant government departments for clearance. The Archives Committee noted that the ‘inevitable uncertainties and delays inherent in this system’ often left researchers ‘puzzled and disappointed’.[4]

In the years that followed, interdepartmental agreements and Cabinet decisions gradually refined the process, but it would be another thirty years before the public’s right to access was enshrined in legislation.

Right to Access

In 1955, the Cold War was near its height. The Royal Commission on Espionage was grilling public servants, scientists, journalists and others in its hunt for Communist infiltrators. Amidst the hysteria about ‘secrets’ and spies the Commonwealth Archives Committee emphasised the ‘moral obligation’ of democratic governments to make the records of their activities available for research. ‘In doing so’, the Committee argued, ‘they make a significant contribution to the continuing search for truth which is an essential part of the continuing democratic process’.[5]

The idea that the public had a ‘right’ to access government records did not go unchallenged, however. In a meeting with the visiting American archivist Theodore Schellenberg the previous year, a representative of the Department of Defence had admitted that he ‘found it difficult to accept the concept that the general public had a right of access to public records’.[6]

But the obligations of war and empire actually pushed Australia towards a more liberal policy. In the 1950s, Australia agencies tended to follow British precedent, applying a version of the 50 year rule. Given that the Commonwealth of Australia was founded in 1900, a 50 year embargo didn’t leave a lot of history to write. In 1966, however, the UK government opened access to records through to 1922 – encouraging research into World War I and its aftermath. Australia had little choice but to follow their example. As John Gorton noted in his submission to Cabinet, ‘we should not impose greater restrictions on our own scholars than are imposed in Britain’.[7] When the UK shifted to a 30 year rule a few years later, Australia again adjusted its policy.

In 1972 the Whitlam Labor government brought with it a professed commitment to ‘open government’. Access to archives started to be linked to broader questions around ‘freedom of information’. It took a few attempts, but the Archives Act was finally passed in 1983. In 2010 ammendments reduced the closed period from thirty to twenty years.

The public interest

Under the Archives Act (in its current form) the public has a right to access Commonwealth records more than twenty years old. But this right has limits. In 1954, Schellenberg described the ‘public interest’ as a balance between a liberal access regime and the protections necessary to shield both governments and individuals from the dangers of over exposure.[8] As access policy evolved, so did the articulation of these limits.

Protections for national security and the privacy of individuals have been long accepted as grounds to withhold records from public access. But the scope and meaning of such exceptions has shifted. In 1966, restrictions on ‘the release of papers relating to the personal expenditure of the Governor-General’ were provided as an example of how individuals were being protected from ‘embarrassment or distress’.[9] Similarly it was assumed that records ‘commenting on the person of the Sovereign’ would naturally be withheld.[10]

But the core of these exemptions remained and was codified within section 33 of Archives Act. Section 33(1)(a), for example, allows records to be withheld from access if they could ‘cause damage to the security, defence, or international relations of the Commonwealth’. Section 33(1)(g) exempts records that might result in ‘the unreasonable disclosure of information relating to the personal affairs of any person’.

The ad hoc approach of the 1950s has been replaced by a well-documented bureaucratic process with clearly defined outcomes. Under the Act a researcher can ask for a record to be examined for public access. The contents of the file is then assessed against the list of defined exemptions and an access status is assigned – Open, Open with exception, or Closed (sometimes closed files are described as ‘withheld’).

Statistics on the number of files examined and their access outcomes are reported annually. The vast majority are opened to the public. But some are not.

Open Access

What do we mean by ‘open access’? Back in 1964 the Interdepartmental Committee on Commonwealth Archives defined ‘open access’ as referring to documents ‘which have been cleared for consultation by any member of the public’.[11] Open could only really be understood by invoking its limits – open was not closed.

I’ve shared this potted history of Australian archival policy because it continues to shape the research that we do and what we can know.

These shifting policies and priorities have shaped the data now available online through RecordSearch, the National Archives of Australia’s database. As digital scholars we should recognise that there’s nothing natural or inevitable about these systems and their contents. They are deeply and profoundly historical.

But recognising this is both an obligation and opportunity, because we can view such systems as archaeological sites and dig back through the layers of data. We can meet the history of policy, with an analysis of practice as it has been captured in the descriptions, dates and decisions.

The access status of each item in RecordSearch is recorded and displayed – ‘Not yet examined’, ‘Open’, ‘Open with Exception’, or ‘Closed’. If a file is ‘closed’ or ‘open with exception’ the reasons for the restriction are provided along with the date when the decision was made. This is interesting and important data. But while you can filter your search by status, you can’t specify a reason or date. So I can browse a list of ‘closed’ files, but there’s no way of getting an overview of why they’re closed.

Until now.

Last year, as an experiment, I screen-scraped RecordSearch to capture the details of files that had an access status of closed. On the 1 January this year, I repeated the harvest with a bit more care.

I now have a dataset of 14,370 closed files. These are the files we’re not allowed to see.

Closed Access

You can explore them online using Closed Access, a public site I developed to play with the data.

You can view the aggregated metadata from a number of different angles – by the reason for exemption, by record series, by the age of the contents, and by the date of the decision. You can drill down from any of these views to a list of closed files. And of course you can search – want to know which files were closed in May 1989 for national security reasons? ClosedAccess can tell you!

Given that none of the files are actually publicly available, it’s possibly the most frustrating search interface yet devised. However, you can always ask for a file’s access status to be re-examined.

ClosedAccess is really more of a workbench than a discovery interface, with additional experiments documented in my open research notebook.

There for example, you can read about TF-IDF analysis of file titles, or my failed attempts to reconcile the statistics reported in the National Archives’ annual reports with the data from RecordSearch.

Reasons

So, why have these files been closed? Here’s the list of reasons.

The first thing to note is while there are 17 exemptions defined by the Archives Act, there are 28 reasons for closing files cited in RecordSearch. Some of these extra reasons relate to specific classes of records mentioned elsewhere in the legislation, while others, such as ‘MAKE YOUR SELECTION’ tell us something about the design of the data entry interface.

It’s no surprise that sections 33(1)(a) and 33(1)(g) of the Act are the most frequently cited exceptions. As I’ve noted, these relate to national security and individual privacy – longstanding grounds for withholding access.

More interestingly, these exemptions appear very frequently in combination with others. The DH standard network graph illustrates their centrality. For example of the 425 closed records citing 33(1)(b) – which refers to confidential information supplied by another government – 399 (94%) also cite 33(1)(a). This sort of pattern, combined with the low level of use for many of the exemptions, suggests that the whole legislative framework could be simplified – how many reasons to close files do we really need?

What’s more surprising are the two highly-cited reasons at the bottom of this graph, neither of which are defined in the legislation or anywhere that I’ve been able to find on the National Archives website. They’re ‘Pre access recorder’ and ‘Withheld pending advice’.

Pre-access recorder

According to the brief explanatory note in RecordSearch, ‘Pre access recorder’ denotes ‘non standard reasons for restriction’ – not very illuminating. But using ClosedAccess you can dig into the metadata to get an idea of what this actually means.

The average age of files citing ‘Pre access recorder’ is 75 years, significantly older than the overall average of 58 years. Indeed, if you look at the dates on which decisions were made about these files, you’ll see that most of these were closed before 1983 when the Archives Act was introduced. So it seems these files were examined and withheld from public access under the less formal arrangements that existed prior to the legislation.

This was (sort of) confirmed by the National Archives who told me that yes they were ‘made prior to the introduction of the Archives Act’, but they were also decisions ‘where the reason for restriction didn’t map to one of the reasons for restriction set out in the Act’. I still don’t quite understand this.

If you take all the file titles and throw them into Voyant you’ll see the word ‘staff’ figures prominently, reflecting the fact that many of them are personnel files. Surely these might have mapped to the privacy provisions of section 33(1)(g)?

From there things get even more complicated. Just as in quantum theory, by observing the phenomena of pre-access recorder, I have changed it. As a result of my inquiry, which the Archives seemed to interpret as a criticism of their metadata, they changed the status of most the ‘Pre access recorder’ files from ‘closed’ to ‘not yet examined’. They’re not any more open, but they no longer show up in a search for closed files. Fortunately, I’ve preserved their historical status in Closed Access. I’m planning to repeat my harvest on the 1 January each year, tracking newly closed files as well as those whose status has changed. So next year I’ll be able to check up on the ‘Pre access recorder’ files to see what’s really happened to them.

Withheld pending advice

Withheld pending advice is easier to interpret. The files have been referred back to the government agencies that created or controlled them to check whether they really can be unleashed on the public.

This process is known and feared by researchers because of the long delays it can create. In 2014 Philip Dorling wrote in the Canberra Times about a file from 1932 relating to Australia’s annexation of Antarctica which languished with the Department of Foreign Affairs and Trade for five years before being released in a redacted form.[12]

Delays in access examination are well-known, but the scale and impact aren’t obvious.

I think that part of the problem I was having in reconciling access examination data from annual reports was that the Archives doesn’t include the ‘Withheld pending advice’ files in its statistics. Perhaps this is fair enough – the access examination process isn’t complete. On the other hand, under the Archives Act if examination requests aren’t completed within 90 days then access is assumed to have been refused. Are they less closed because they may yet be opened? I’ve asked the National Archives what the status of these files really is, and how this affects the public’s right to access.

In the meantime we can use Closed Access to see the scale of the backlog and indulge in a bit of agency shaming. You can easily search for ‘Withheld pending advice’ files and order them by the date they were closed.

Ignoring the one file with the title ‘unknown’ that has been waiting 18 years for a decision (I suspect it’s been mislaid), there’s still a file that’s been waiting for 9 years, and 9 files that have been waiting for 7 years. Indeed, you can limit your search to files that have been waiting more than three years for a decision – there’s 1467 of them.

The Defence department gets the longest in limbo award for the 9 year delay, but DFAT is the overall champion with series A1838 containing almost 1500 files awaiting departmental determinations – 54 of which have been waiting more than five years. Too bad if you’re doing a PhD. Researchers are still being left ‘puzzled and disappointed’.

It’s important to make these processes visible because it’s all to easy to forget that the default status for any Commonwealth government file more than 20 years old is ‘open’. The reasons for closing files defined under the Archives Act are called exemptions for this very reason.

Philip Dorling suggested that agencies are becoming more conservative in their decisions. Once I’ve harvested a few year’s worth of closed files, I’m hoping I’ll be able to trace the fate of those poor souls currently ‘withheld pending advice’. But there are other measures of conservatism we can start to explore.

You’d think that the older a file was the more likely it was not to be closed. So what happens when we look at the gap between the date a decision was made to close a file and the date of it’s latest content? I’ve added an experimental view to Closed Access where you can explore the files than seem to have provoked the most conservative approach.

Strangely, at the top of this list are the proceedings of the Senate from 1903. But in fact, no decision was made to close this file. It belongs to a category of records called ‘Parliament Class A’. Regulations drawn up under the Archives Act simply declare these records as ‘exempt’. You need the permission of Parliament to access them.

It’s a reminder that at least we do have the metadata for closed files. The intelligence agencies hold unknown quantities of files where access to the recordkeeping systems themselves is restricted. We don’t know what we don’t know.

I’m hoping to explore these gaps further, but for now if you’d like a copy of the 300,000 digitised pages contained within publicly accessible ASIO files let me know.

No magic

It’s important to recognise all of this as the workings of a system. Digging through the layers of policy and metadata helps us to see that there is nothing intrinsically dangerous in a closed file. A judgement has been made against a list of historically determined criteria.

But it’s also important to resist the temptation to claim privileged access to the truth through our mastery of such systems. Next time you see words like ‘recently released documents’ in a book or article, remember that access examination is a process without magic or wonder. We need to remain critical of the process without adding layers of mystification – without succumbing to the cultural power of the secret.

This work is licensed under a Creative Commons Attribution 4.0 International License.