Presented at the National Digital Forum, 13 October 2015, Wellington.

23andMe is a US-based company that will, for a fee, analyse your genetic ancestry. Provide some saliva and they’ll send you a summary that compares your DNA with 31 identified populations across the world. It seems that this sort of genetic analysis is the next big thing in family history. You may have noticed that Ancestry offers a similar service.

23andMe, however, also provides an Application Programming Interface, or API, so developers can create third party apps that do cool things with your DNA. Don’t worry — you do have to provide permission. Facebook won’t automatically start recommending friends based on your genes. Well… not yet.

But it didn’t take long for the ethical boundaries of this sort of service to be tested. One developer created a Genetic Access Control authentication system. Using it, online sites or services could restrict access to people who had a specific genetic make-up.

The developer’s API access was quickly revoked with 23andMe noting that their terms of use prohibit applications that ‘contain, display or promote hate’.

Four years ago I stood on this stage describing a project I was working on with Kate Bagnall called Invisible Australians. We were (and are) trying to encourage use of the National Archives of Australia’s collection of records that document the workings of the White Australia Policy in confronting detail.

As an experiment I’d downloaded thousands of images from the Archives’ collection database. Most were certificates used in the control of ‘non-white’ immigration — visually compelling documents that include both portrait photographs and handprints.

I ran a facial detection script over the images to extract the portraits and created an online resource we called The Real Face of White Australia. For a weekend project it’s had a significant impact around the digital humanities world.

Some people criticised us because they thought we were selecting records based on race. This was just a misunderstanding of the records and the technology we were using. Even if I’d wanted to, I wouldn’t have had a clue back then about how to categorise portraits by race.

Now I do.

I can sign up for an account with a service like Face++ and use their API to analyse an image of a person’s face to determine both race and gender.

After a bit of a break because of — you know — LIFE, Kate and I are getting back to Invisible Australians. In 2011 I had about 12,000 images from one series in the National Archives. I’ve now harvested almost 160,000 images from 22 series.

But the intervening years have also brought changes in the Australian government’s treatment of asylum seekers, greater power for security agencies, and the normalisation of electronic surveillance.

When the White Australia Policy was implemented, portrait photographs and fingerprints were the latest in crime-fighting technology. Just over a month ago the Australian government announced its ‘newest national security weapon’ — a national facial recognition system to be known henceforth as ‘The Capability’. This system would assist authorities in ‘putting a name to the face of terror suspects, murderers, and armed robbers’.

Tools that help identify faces can offer powerful new means of discovery and analysis within the holdings of our cultural institutions. But can those of use who work with these tools avoid engaging with broader systems of surveillance, categorisation and control? For Kate and me the parallels are too strong — history is not just about the past.

I’m talking today about two related technologies — facial detection and facial recognition.

Facial detection simply tells you if there’s a face in an image. It’s the technology that draws boxes around faces when you’re taking photos and it’s pretty efficient and well established. This is the technology I used to create our wall of faces.

Basic detection is now being supplemented by algorithms that examine the shape of facial features and the characteristics of things like skin to estimate gender, age and race. They can also tell you whether the person is wearing glasses and the quality of their smile.

Facial recognition, on the other hand, first detects faces within an image and then searches for those faces within an existing set of previously identified images. The Capability, for example, plans to take an image and look for matches across a series of interlinked databases, such as passports and driving licences. It’s also what Facebook does when it tags people in photos you share.

Facial recognition is a lot trickier than detection, but last year FaceBook announced that its DeepFace system had reached a level of accuracy similar to humans. Not to be outdone, Google claimed its FaceNet technology had pushed the bar even higher, reporting an accuracy of over 99%. How this translates to real-world applications such as The Capability is not clear, but we can assume that security agencies are heading down the same path.

Machines have struggled to match humans in finding and recognising faces because faces are so important in simply being human. Faces connect us to our social world.

But, as cultural institutions know, faces can also connect us through time. Looking into the eyes of a person, no matter how far removed through history or culture, affects us. We do not merely read, we feel. This, I think, is where The Real Face of White Australia gains its emotional power.

Last year in another experiment I started extracting faces from Trove’s millions of digitised newspaper articles. The images are of a much lower quality than photographs of course, but still… there’s that feeling of connection.

Facial detection is one example of a broader class of computer vision operations known as feature detection. You can train your computer to find all manner of patterns and shapes in your images. This includes cats and bananas, as well the components of a face — eyes, noses and mouths.

So of course I wondered, what would a newspaper discovery interface based on eyes look like?

Eyes on the past has been variously described as both beautiful and creepy — something of which I’m rather proud. It was another weekend project — an experimental intervention rather than a practical tool. By clicking on eyes and faces you can find your way to newspaper articles, but that’s not really the point.

I was hoping to say something about the fragility of our connections to the past — we glimpse past lives through tiny cracks in the walls of time. These moments may be fleeting, but they can also be full of meaning.

So I kept harvesting faces. I’ve now got about 6000 from the 1880s through to 1913. If you’d like to play, the complete set is available for download from Figshare.

I also built my own face API to encourage further experimentation. While services like Face++ have APIs that take your face and pull it apart, mine just gives you random faces from the past. That’s all.

Most recently, I used my collection of faces to create a Twitter bot called the Vintage Face Depot. Tweet a picture of yourself to the bot and it will send you back a new version of yourself, in which your face is overlaid with a random visage from my extensive range of vintage faces.

Tweaking the transparency means the faces start to blend — you are neither yourself, nor them, but someone new. Each face replacement also comes gift wrapped with a link to the original newspaper article.

The Vintage Face Depot tells you nothing about yourself. I built it at about the same time as Microsoft launched their How-Old bot that uses machine learning to estimate your age. Face Depot does nothing clever, and yet sometimes the results are uncanny, even unsettling. Microsoft might be able to tell you how old you are, but Face Depot asks who you are and pushes you in the direction of a past life, linked merely through chance.

Of course the next obvious step is to feed to the results of Vintage Face Depot to the How-Old bot.

Or to Face++’s API.

In a similar vein, the developer Kirk Kaiser has been pitting neural network against neural network — altering images of himself using Google’s DeepDream and uploading them to Facebook for DeepFace to analyse and tag.

Digital artists like Adam Harvey have reverse-engineered facial detection algorithms to devise anti-surveillance fashion styles. He shows how you can use make up to disrupt key regions, such as where your nose and eyes intersect, and render your face invisible. From face to ‘anti-face’.

While none of these interventions provide a detailed critique of state surveillance, they do highlight the constructed nature of these technologies. By fucking with their parameters we understand better how they work.

But what’s the role of cultural heritage organisations in all of this?

Libraries are already leading the way in supporting online privacy. But leaving aside the whole ‘living in a surveillance state’ thing for a moment, these technologies don’t just find faces, they reduce us to a set of external characteristics. We become what they can measure.

Researchers are currently investigating how facial detection systems can be used to identify depression. The aims are worthy of course, but it’s not hard to imagine how, like 23andMe, such systems could be used to discriminate rather than support. Other studies have explored whether human observers can tell if you’re gay or prone to infidelity by looking at your face. Anyone remember phrenology?

With measurement comes the power to categorise and control. These are technologies that enable us to be judged at a distance — to be identified as threat or a sales opportunity just by the way we look.

Facial recognition takes this further — not only can we be reduced to a set of externally-verifiable measurements, but these measurements are assumed to constitute our identity.

Ok, so running Face++’s API across photographic collections to identify pictures of women seems like it could be a useful thing to do. But we also know that the male/female binary is hopelessly inadequate in describing who we are. Identifiers are not the same as identities.

Cultural heritage data is gloriously messy. Even as we try and wrangle it to fit our systems we recognise its resistance as something profoundly human. Against the power of surveillance — both for security and sales — we have the opportunity, the obligation, to celebrate this complexity. To deny the meaning of measurement.

You cannot know me from my face.

My identity cannot be captured in your database.

Let’s use facial detection to enrich our metadata, but let’s also work with artists, developers, and activists to challenge the technology’s embedded assumptions about the ‘perfect face’.

Four years ago I showed you our wall of faces. A few weeks ago I took those 7000 photos and ran them through a program that averaged the facial features.

I expected to be critical. I expected to be annoyed. But instead of seeing some algorithmically-generated nonsense, I just saw a person.

There’s power in that.

Faces offer an instant connection to history, reminding us that the past is full of people. People like us, but different. People with their own lives and stories. People we might only know through a picture, a few documentary fragments, or a newspaper article.

Eyes on the Past is an experimental interface, built in a weekend. I’m exploring whether faces can provide a way to explore more than 120 million newspaper articles available on Trove.

This collection of tweets tells the story of its development.

There’s some details about the software used in the site’s about page. You can view the harvest/detection and the website code on GitHub.

In many of the presentations I’ve given in recent times I’ve managed to include a question raised by Tim Hitchcock in his chapter in The Virtual Representation of the Past. Tim asks:

What changes when we examine the world through the collected fragments of knowledge that we can recover about a single person, reorganised as a biographical narrative, rather than as part of an archival system?

The idea of turning archival systems on their head to expose the people rather than the bureaucracy is what motivates Kate Bagnall and I in our attempts to make the Invisible Australians project into a reality.

Invisible Australians aims to liberate the lives of those who suffered under the restrictions of the White Australia Policy from the rich archival holdings of the National Archives of Australia and elsewhere.

We always knew that the portrait photographs, included on a range of government documents, would provide a compelling perspective on these lives, but we weren’t quite sure how we were going to extract them. Up until last weekend, I’d assumed that we’d develop a crowdsourcing tool that contributors would use to mark-up the photos.

Now I’m not so sure.

In the space of a couple of days I’ve extracted over 7,000 photographs and built an application to browse them — here is the real face of White Australia…

How did I do it? Paul Hagon, at the National Library of Australia, gave a presentation last year in which he explored the possibilities of facial detection in developing access to photographic collections. The idea lodged in my brain somewhere and a few days ago I started to poke around looking to see how practical it might be for Invisible Australians.

It didn’t take long to find a python script that used the OpenCV library to detect faces in photographs. I tried the script on a few of the NAA documents and was impressed — there were a few false positives, but the faces were being found!

So then the excitement kicked in. I modified the script so that instead of just finding the coordinates of faces it would enlarge the selected area by 50px on each side and then crop the image. This did a great job of extracting the portraits. I tweaked a few of the settings as well to try and reduce the number of false positives. Eventually, I developed a two-pass system that repeated the detection process after the image had been cropped and it’s contrast adjusted. This seemed to weed out a few more errors. You can find the code on GitHub.

Once the script was working I had to assemble the documents. I already had a basic harvester that would retrieve both the file metadata and digitised images for any series in the NAA database. Acting on Kate’s advice, I pointed it at series ST84/1 and downloaded 12,502 page images.

All I then had to do was loop the facial detection script over the images. Simple! The only problem was that my 3-year-old laptop wasn’t quite up to the task. As it’s CPU temperature rose and rose, I was forced to employ a special high-tech cooling system.

But after running for several hours, my faithful old laptop finally worked it’s way through all the documents. The result was a directory full of 11,170 cropped images.

There were still quite a lot of false positives and so I simply worked my way through the files, manually deleting the errors. I ended up with 7,247 photos of people. That’s a strike rate of nearly 65% which seems pretty good. The classifier, which does the actual facial detection, was probably trained on conventional photographs rather than on the mixed-format documents I was feeding it.

Then it was just a matter of building a web app to display the portraits. I used Django for the backend work of managing the metadata and delivering the content, while the interface was built using a combination or Isotope, Infinite Scroll and FancyBox.

It’s important to note that the portraits provide a way of exploring the records themselves. If you click on a face you see a copy of the document from which the photo was extracted. A link is provided to examine the full context of the image in RecordSearch. This is not just an exhibition, it’s a finding aid.

What next? There are many more of these documents to be harvested and processed (and many more still yet to be digitised). I will be adding more series as I can (though I might have to wait until I can afford a new computer!). I’d also like to explore the possibilities of facial or object detection a bit more. Could I train my own classifier? Could I detect handprints, or even classify the type of form?

In the meantime, I think our experimental browser helps us to understand why the Invisible Australians project is so important — you look at their faces and you simply want to know more. Who are they? What were their lives like?

UPDATE: For more on the photos and the issues they raise, see Kate Bagnall’s posts over at the Tiger’s Mouth.