Comments on: Extracting editorials #1 http://discontents.com.au/extracting-editorials-1/ working for the triumph of content over form, ideas over control, people over systems Fri, 03 Nov 2017 05:29:17 +0000 hourly 1 https://wordpress.org/?v=4.8.17 By: discontents - Extracting editorials #3 http://discontents.com.au/extracting-editorials-1/#comment-172 Sun, 19 Feb 2012 23:11:58 +0000 http://discontents.com.au/?p=1462#comment-172 […] my previous instalments (here and here), I focused on the Sydney Morning Herald. Having continued the hunt for missing […]

]]>
By: discontents - Extracting editorials #2 http://discontents.com.au/extracting-editorials-1/#comment-171 Mon, 19 Dec 2011 13:18:57 +0000 http://discontents.com.au/?p=1462#comment-171 […] recording my thoughts and assumptions in this way has already proved useful. In a comment, Owen Stephens noted that his attempt to reproduce my search query produced fewer results. After a little bit of poking […]

]]>
By: Craig Bellamy http://discontents.com.au/extracting-editorials-1/#comment-170 Thu, 24 Nov 2011 02:36:12 +0000 http://discontents.com.au/?p=1462#comment-170 Hi Tim, this looks like an interesting first stage. But what are you trying to find out from the editorials? I know of someone who did some similar work; ‘mapping’ arguments in Op Ed articles. I’ll try and find it.

]]>
By: Owen Stephens http://discontents.com.au/extracting-editorials-1/#comment-169 Tue, 22 Nov 2011 09:16:42 +0000 http://discontents.com.au/?p=1462#comment-169 Glad it was helpful 🙂

If you can’t do an exact phrase search in just the headings, you could always add a quick check into the harvester (or whatever post-processing after the harvester has run) to do a check for the exact phrase using a regular expression – I’ve had to resort to this in the past when dealing with imprecise search interfaces.

]]>
By: tim http://discontents.com.au/extracting-editorials-1/#comment-168 Mon, 21 Nov 2011 22:14:10 +0000 http://discontents.com.au/?p=1462#comment-168 Thanks Owen, that’s very useful. In fact you’ve alerted me to another oddity in the way Trove’s search works. The difference between your query and mine is that I use the ‘fulltext’ modifier on the phrase. Fulltext switches off fuzzy matching (stemming etc) — I’ve gotten into the habit of using it because usually I’m looking for exact strings. However, it seems that the fulltext modifier overrides the ‘Search headings only’ option in the advanced search and so looks for matches in the whole article (perhaps that’s why it’s called ‘fulltext’???). This behaviour doesn’t seem to be documented. I wonder if that means there’s no way of searching for exact phrases in just the headings? I think I’ll post something to the Trove forums and see if I can get clarification. Anyway, that explains the false positives I was getting! Your search is cleaner, so I’ll start again and harvest that.

]]>
By: Owen Stephens http://discontents.com.au/extracting-editorials-1/#comment-167 Mon, 21 Nov 2011 15:34:14 +0000 http://discontents.com.au/?p=1462#comment-167 I hope you have time to keep up blogging your process as well as your results – always interesting.

I don’t know if any of this is helpful but I had some questions/thoughts as I read the above:

A) I’ve tried to recreate your search but get different results depending on the approach I take – and I’ve not yet managed to recreate your exact search!

For example I note that this search http://bit.ly/tphXME gets 294 hits (less than yours), but includes for example the editorial from the 2nd January which your search omits for some reason.

I also found variance depending on whether I used the date range in the advanced search, specified the data via the faceted filtering

B) Length of article looks like it might be another possible filter criteria – it seems on relatively brief inspection that editorials are longer than articles

C) How do you tell an article is actually an Editorial if reading the paper? It seems impossible to actually tell systematically? (is it just a matter of ‘guessing’ when you read it?)

D) I’m not familiar with how editorials would be structured, but I notice that there can be multiple ‘sections’ within an Editorial column with a single heading. For example this editorial http://trove.nla.gov.au/ndp/del/article/15388548 is headed ‘Traffic Growth’, but that is actually only the first part of the article. There is then a double rule, followed by (I think) 11 ‘sketches’ – short paras on different topics which are headed only by a short title followed by an em-dash (I think – could be en-dash?). Not sure how much this matters once you get into the textual analysis stage – but it seems a shame these aren’t separated out into articles.

Just dumping some thoughts really – sorry if not helpful!

]]>