Excel spreadsheet of candidates and their constituencies

How to scrape with OutwitHub

Data scraping is one of those data journalism phrases that I encountered and thought – yep, never going to be able to do that. But it sounds a lot more scary than it is.

In this how-to guide I’ll run through how to do a really simple scrape using the free version of Outwit Hub.


In honour of 420 (unofficial national marijuana day in America), I wrote a piece about GE 2015 candidates from the Cannabis is Safer Than Alcohol (CISTA) Party in order to practise/refresh my scraping skills.

I will be walking you through the scraping process I went through to get the following data from the CISTA website:

a) Candidate name

b) Candidate constituency

I then mapped the candidates according to where they were standing for election, using mapping tool CartoDB.

What does Outwit Hub do?

At its most basic, Outwit Hub retrieves text from between two ‘markers’ defined by you, the user. It delves into the source code of a web-page and recovers the data you want.

Step 1: Download OutWit Hub
The free version limits you to scraping 100 rows of data, but that should definitely be more than enough.

Step 2: Open up OutWit Hub, copy and paste the CISTA candidate web page into the URL bar at the top of the application. This will also show the source code of that particular web page.


Step 3: Click on scrapers, create new scraper

Step 4: For the first line, under the ‘Description’ column, put ‘Candidate name’. This line will be where we’ll pull out the candidate name.

Step 5: Go to ‘marker before’, and put in: <div class=”col-md-3″>

Step 6: Go to ‘marker after’, and put in: </h3>

Step 7: For the second line, under the ‘Description column, put ‘Constituency’. This will be where we’ll pull out the candidate’s constituency that they’re standing in.

Step 8: Go to ‘marker before’, and put in: </a>

Step 9: Go to ‘marker after’, and put in: </p>

A photo probably illustrates the process better than me just writing about it. So — your screen should now look something like this:

OutWit Hub showing before and after markers

Step 10: Click ‘execute’

Congratulations! You have successfully scraped using OutWit Hub.

OutWit Hub showing what final scrape should look like

I still find it a bit touch and go in terms of deciding exactly which bits of source code to use as markers. In this case, you can see that the scraper pulled out one result (the top one) that also fit into the before and after markers we specified – it’s not an exact science, but you can delete outliers.

You can now export your results (click ‘Export’).

Excel spreadsheet of candidates and their constituencies

"Mellieha Bay beach Malta 1" by Karelj - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Mellieha_Bay_beach_Malta_1.jpg#/media/File:Mellieha_Bay_beach_Malta_1.jpg

Budget 2015: Why are we spending so much on overseas winter fuel allowance?

On Wednesday, Chancellor George Osborne presented his Budget to parliament.

While new policies like the increase in the personal tax allowance (or setting aside £1m to commemorate the Battle of Agincourt) grabbed the headlines, it was a one-liner on page 71 that caught my attention – ‘Winter Fuel Payments: overseas eligibility’.

Screenshot Budget 2015 showing savings in winter fuel allowance

This came under the category of ‘Measures announced at Autumn Statement 2013’, i.e. to be implemented in the 2015 Budget. It lays out the amount of money which will go back to the government through the cutting of overseas winter fuel payment – a projected total of +£85m from fiscal years 2015/2016 to 2019/2020, according to the Budget.

All well and good – but this begs the question: How much were we actually spending on overseas winter fuel payment in the first place?

It’s (not?) always sunny in the south of Europe

Turns out, it’s not so much the total amount spent that is so surprising, but the European Economic Area (EEA) countries – and Switzerland – in which it is spent. Pensioners who live overseas could qualify for winter fuel allowance payments irregardless, as it turns out, of how hot or cold their chosen country actually was…

For the fiscal year 2013/14, a total of £21,736,000 was spent on overseas winter fuel payments to British pensioners living in the EEA and Switzerland.

Overseas Winter Fuel Payments - fiscal year 2013/14
Get the data: Winter Fuel Payment statistics (Gov.uk)

Of this total, Spain topped the chart, with over £8million spent in the year. Fair enough you might say, some parts of Spain maybe can get quite cold in Winter.

However fourth-highest on the list is the freezing country of… Cyprus. In the fiscal year 2013/14, over £1million (£1,432,000 to be exact) was spent keeping British pensioners warm.

Cyprus does not get that cold. In the winter months, an average temperature ranges from a freezing 12ºC to 14ºC. To put this in perspective, this is like summer in most parts of the United Kingdom.

Other countries not exactly famous for their freezing winters include Malta (average winter temperatures of between 10ºC – 15ºC), Greece and Portugal.

If you’re looking for areas of spending to save on, overseas winter fuel allowance is probably as good a place as any to start. After all, we could have commemorated the Battle of Agincourt 21 times over if it had been cut a year earlier.