Comments on: Mining the treasures of Trove (part 2)

By: discontents - For you, with all best wishes…

discontents - For you, with all best wishes… — Wed, 26 Sep 2012 14:24:16 +0000

[…] 18 months ago I created a little Python script to visualise search results in Trove’s collection of digitised newspapers. After a bit more […]

By: discontents - QueryPic

discontents - QueryPic — Sat, 31 Dec 2011 15:22:00 +0000

[…] QueryPic (I got a bit sick of ‘search summariser’ and ‘graph-maker thing’). The first version just harvested data and left all the graph-making to you. But QueryPic does it all! It harvests the […]

By: discontents - When did the ‘Great War’ become the ‘First World War’?

Mon, 29 Aug 2011 13:42:01 +0000

[…] already had a script that would generate a basic time series from a Trove query string. It simply takes the query, fires […]

By: tim

tim — Wed, 10 Aug 2011 06:58:59 +0000

Josh,

There may well be simpler ways, but it seems to me you could script something by looking at what’s going on behind the scenes on the ‘Find an issue’ page. If you have a poke around you’ll see that there’s a very useful JSON call to http://trove.nla.gov.au/ndp/del/titlesOverDates/{year}/{month}. For example to find out all the issues from January 1901:

http://trove.nla.gov.au/ndp/del/titlesOverDates/1901/01

Once you have that, you can pull out all the issues of the title that you’re after using the approprate title id – ’13’ for the Argus. You’ll then have a list of issue numbers. Calling an issue number (eg http://trove.nla.gov.au/ndp/del/issue/26777) seems to return the first page of that issue. I suppose you could assume that the second page’s id will simply be the first page’s id + 1, but I don’t know how safe that is. A safer approach would simply be to grab the id for the page you want from the select page drop down. Once you have the id for the page you want it’s an easy matter to download the pdf using a url of the form: http://trove.nla.gov.au/ndp/imageservice/nla.news-page{page id}/print

All of this would be easily scriptable. You’d just need to feed your script the years to harvest and it would grab all the relevant pdfs for you.

Hope this helps.

Cheers, Tim

By: Josh Cockfield

Josh Cockfield — Wed, 10 Aug 2011 06:22:55 +0000

Hi Tim,

What you’re doing with TROVE is very similar to what we’d like to be doing. We are running a citizen science program called OzDocs, where people help extract climate histroy information from sources like TROVE. So sometimes we need to pull out bulk images or text to hand over to volunteers to sort through. Your Harvester looks good for that but I have a bit of a problem with one task that we are working on and was wondering if you had any ideas. We want to digitise all the pressure table published in the Argus in 1850s. However when we do a key word search we get less than 50% of what is really there. I guess that is because of OCR. Is there a way we could select dates or dates ranges and just pull the best quality image possible of page 2 of every edition within that range?

Thanks,

Josh

By: Delving Deeper into Trove « Stumbling Through the Future

Delving Deeper into Trove « Stumbling Through the Future — Sat, 07 May 2011 06:00:06 +0000

[…] The first step was to identify and collect the documents that I wanted to analyse. I used Sherratt’s “harvester” to collect all newspaper articles containing the phrase “secular education”. The search I constructed in Trove returned over 7,800 articles. It would take a ridiculously long time to click on each article, download it and save it. This became a feasible exercise with Sherratt’s tool. After I had run it, the articles were all saved as text files and could be found in subdirectories according to the year they were published. The file names included the name of the newspaper, the date and page where the article was found. Wonderful! I encourage you to have a go using the harvester. It requires a bit of work but Tim Sherratt has given good instructions in his posts, Mining the Treasures of Trove (Part 1), and Part 2. […]

By: The Inner Geek is Emerging | Stumbling Through the Past

The Inner Geek is Emerging | Stumbling Through the Past — Thu, 05 May 2011 10:34:53 +0000

[…] post about analysing information held by Trove, this post explains how the tools created by Wragge can prove a boon for […]

By: Know Your Database « Stumbling Through the Future

Know Your Database « Stumbling Through the Future — Wed, 04 May 2011 04:08:23 +0000

[…] want to do this yourself check out Wragge’s posts, Mining the Treasures of Trove (Part 1) and (Part 2). Firstly let’s look at Wragge’s graph of a topic that I have been writing about this […]

By: Know Your Database | Stumbling Through the Past

Know Your Database | Stumbling Through the Past — Sun, 17 Apr 2011 08:18:14 +0000

By: Airminded · More THATCamp thoughts

Airminded · More THATCamp thoughts — Sat, 26 Mar 2011 11:20:05 +0000

[…] won't release an official API is a bit mysterious.) He uses that to scrape Trove to do searches and display results which aren't possible with the interface offered by the NLA, such as plotting the frequency of […]