Yep, there’s a new version of QueryPic.

About 18 months ago I created a little Python script to visualise search results in Trove’s collection of digitised newspapers. After a bit more tweaking. I christened it QueryPic. People started to use it. It was even reviewed in the Journal of Digital Humanities. With the release of the Trove API earlier this year I rewrote the whole thing in Javascript and let it loose on the web. People could make graphs without having to download any code or fire up the command line. Anyone could play.

And now?

The latest version lets you save your QueryPics. As new features go it’s not very revolutionary. But it meant another significant shift. From Python script, to web page, to web app. The Javascript-enabled interface now connects to a Django-powered backend. Save a graph and you can access it via a lovely, short, persistent url (like this). It’s as much a platform as a tool. But to be persistent, the urls need to work for ummm… a long time. Is this even possible for a project that has no funding and a support team of one?

I don’t know.

My enthusiasm for making tools is punctuated by regular bouts of doubt and disillusionment. With millions of dollars being spent on industrial-strength digital research infrastructure why should I devote my evenings to hand-crafting pretty little widgets like QueryPic?

My grandfather made this brass dish. He owned an engineering workshop and forge. My dad was a draftsman, engineer and builder. My mum made fine dresses in the fashion houses of Melbourne. I make things too. It’s what I do. It took me quite a few years to work this out. Years spent wondering why I felt out of place in academia. I’m also a historian, so I research and I write, but without some time to tinker, well… I’m just not happy. Making things is not separate — for me it’s all part of being a historian. I make things that let people connect to the past in different ways. And along the way I learn.

And by people I mean people. Just last week I took part in an online question and answer session organised by Inside History Magazine. It was a lot of fun. Amidst the questioning I unveiled the latest version of QueryPic. Considerable excitement ensued. QueryPic graphs are starting to be included in research publications, but anyone can make and understand them. Local and family historians are enthusiastic users of digital technologies and I’m excited to see them playing around with tools that I’ve made. I want to create things that other people use. Things that help them, and sometimes surprise them.

QueryPic has graduated from WraggeLabs to dhistory — my platform for digital history research. There it joins The Front Page and Archives Viewer. As usual, I have big plans. Are they practical? Probably not. Are they sustainable? I doubt it. Will I keeping making things anyway? Of course.

So please accept this gift. I made it for you. I hope you find it useful.

QueryPic — explore digitised newspapers from Australia & New Zealand.

http://dhistory.org/querypic/

Features include:

• Save and date-stamp your graphs with persistent urls — perfect for citing and sharing
• Copy and paste query urls from Trove or Digital NZ, or connect automatically with a handy bookmarklet
• Easily regenerate saved graphs to draw in updated data
• Explore QueryPics created by others — use them as the starting point for your own visualisations
• Combine any number of queries, either from Australia or New Zealand
• Click on the graphs to preview matching articles

All this and more documented on QueryPic’s extensive help page. Code on Github.

It seems a bit late to be introducing the newest version of QueryPic. Folks are already using it to explore the contents of digitised newspapers made available through Trove and Papers Past. Some, like the National Library of New Zealand, Andrew S. Bowman and the Carnamah Historical Society are already blogging about it. But I suppose I’d better document a few things…

As I noted in my post about QueryPicNZ (yes I now have a rather confusing proliferation of QueryPics), I was waiting for the Trove API to become public. Last week I noticed a little ‘API’ link pop up in the Trove footer and so I set to work…

My original version of QueryPic (recently reviewed in the Journal of the Digital Humanities) used a series of Python scripts to harvest and scrape content from the Trove web pages. This meant that you had to download the scripts and be code-confident enough to run them in a terminal. It’s still a useful tool and I’ll be updating it as well, but I wanted to create something quicker and simpler that encouraged people to explore and play.

The latest version of QueryPic (QueryPic+, QueryPic Web, QueryPic 2.0?) simply runs in your browser. It uses JQuery to grab data on the fly from the Trove and DigitalNZ APIs. Like previous versions, it uses the HighCharts library to turn the data into pretty graphs.

What does it do? It’s really pretty basic. QueryPic just displays the number of articles matching your search query over time. By default, these are displayed as a proportion of the total articles available for that year, but a dropdown field lets you switch to view the raw numbers. It’s simple, but it’s also remarkably evocative, suggestive and fun. Just try it!

Why stop at just one query? To compare frequency patterns you can add as many as you like. Just keep entering new words or phrases.

If you notice an interesting peak or trough you can just click on it and another API request will be fired off to retrieve the first 20 matching articles. So it’s also a new way of exploring the newspaper databases themselves.

There are plenty of limitations — not all newspapers are digitised, for example, and the quality of the OCR is patchy. The National Library of New Zealand’s post does a great job summing up a number of issues relating to Papers Past. It’s not magic, it’s not perfect, but is it useful? I think so.

Tasks for the future:

• Create some sort of backend that makes it easy to save , share and cite your query data. The ‘share’ link just regenerates the graph which, of course, might change as new articles are added to the databases.
• Make it possible to add more complex queries — I want to keep the interface simple, so I’ll probably create a bookmarklet to take any Trove or Papers Past query and display it using QueryPic.
• As I mentioned over at the WraggeLabs Emporium, I intend to rewrite my various Trove tools to work with the new API. This will include the classic Python version of QueryPic. I still think it’s useful for harvesting your own data.

You may have noticed I have a bit on an interest in exploring ways of using digitised historical newspapers. In the last year or so I’ve spent a lot of time scraping, mining, processing and visualising content from the Trove collection of digitised Australian newspapers. But what about other countries?

Recently I was invited to a digital history workshop organised by Sydney Shep (@nzsydney) at the Victoria University of Wellington. In between sessions I started to play with the DigitalNZ API guided by Chris McDowall (@fogonwater). In anticipation of the forthcoming Trove API I’d already done a bit of work converting QueryPic to run in the browser. It didn’t take long to adapt this to work with New Zealand newspapers available through Papers Past.

So presenting for your enjoyment and education… QueryPicNZ.

Like QueryPic, the New Zealand version graphs newspaper search results over time. But thanks to the DigitalNZ API it has a number of advantages:

• it runs in your browser — no need to download or run any scripts
• results appear almost instantly
• easy to combine queries — just search on a new word or phrase
• easy to remove queries — just use the ‘Clear last’ button
• easy to share — just copy the provided link or use the Tweet button

It’s limited to simple word or phrase searches at the moment, but eventually I’ll add the ability to process more sophisticated queries. I also want to add a way of saving, sharing and citing graphs. For now the ‘share’ link simply regenerates the graph, so if the content has changed the result could well be different.

The code is available on GitHub.

Ultimately, I want to combine Trove and Papers Past so that you can query and combine content from either Australia or New Zealand… perhaps even other countries?