Boundaries, not barriers

Choroleth map of Scotland

I wrote some recent articles about the state of open data in Scotland. Those highlighted the poor current provision and set out some thoughts on how to improve the situation. This post is about a concrete example of the impact of government doing things poorly.

Ennui: a great spur to experimentation

As the Christmas ticked by I started to get restless. Rather than watch a third rerun of Elf, I decided I wanted to practice some new skills in mapping data: specifically how to make Choropleth Maps. Rather than slavishly follow some online tutorials and show unemployment per US state, I thought it would be more interesting to plot some data for Scotland’s 32 local authorities.

Where to get the council boundaries?

If you search Google for “boundary data Scottish Local Authorities”  you will be taken to this page on the website. It is titled “Scottish Local Authority Areas”  and the description explains the background to local government boundaries in Scotland. The publisher of the data is the Scottish Government Spatial Data Infrastructure (SDI). Had I started on their home page, which is far from user-friendly, and filtered and searched, I would have eventually been taken back to the page on the data portal.

The latter page offers a link to “Download via OS OpenData” which sounds encouraging.

Download via OS Open Data
Download via OS Open Data

This takes you to a page headed, alarmingly, “Order OS Open Data.” After some lengthy text (which warns that DVDs will take about 28 days to arrive but that downloads will normally arrive within an hour), there then follows a list of fifteen data sets to choose. The Boundary Line option looked most appropriate after reading descriptions.

This was described as being in a proprietary ERSI shapefile format, and being 754Mb of files, with another version in the also proprietary Mapinfo format. Importantly, there was no option for downloading data for Scotland only, which I wanted. In order to download it, I had to give some minimal details, and complete a captcha. On completion, I got the message, “Your email containing download links may take up to 2 hours to arrive.”

There was a very welcome message at the foot of the page: “OS OpenData products are free under the Open Government Licence.” This linked not to the usual National Archives definition, but to a page on the OS site itself with some extra, but non-onerous reminders.

Once the link arrived (actually within a few minutes) I then clicked to download the data as a Zip file. Thankfully, I have a reasonably fast connection, and within a few minutes I received and unzipped twelve sets of 4 files each, which now took up 1.13GB on my hard drive.

Partial directory listing of downloaded files
Partial directory listing of downloaded files

Two sets of files looked relevant: scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. I couldn’t work out what the differences were in these, and it wasn’t clear why Wales data is also bundled with Scotland – but these looked useful.

Wrong data in the wrong format

My first challenge was that I didn’t want Shapefiles, but these were the only thing on offer, it appeared. The tutorials I was going to follow and adapt used a library called Folium, which called for data as GeoJson, which is a neutral, lightweight and human readable file format.

I needed to find a way to check the contents of the Shapefiles: were they even the ones I wanted? If so, then perhaps I could convert them in some way.

To check the shapefile contents, I settled on a library called GeoPandas. One after the other I loaded scotland_and_wales_region.shp and scotland_and_wales_const_region.shp. After viewing the data in tabular form, I could see that these are not what I was looking for.

So, I searched again on the Scottish Spatial Infrastructure and found this page. It has a Download link at the top right. I must have missed that.

SSI Download Link
SSI Download Link

But when you click on Download it  turns out to be a download of the metadata associated with the data, not the data files. Clicking Download link via OS Open Data, further down page, takes you back to the very same link, above.

I did further searching. It appeared that the Scottish Local Government Boundary Commission offered data for wards within councils but not the councils’ own boundaries themselves. For admin boundaries, there were links to OS’ Boundary Line site where I was confronted by same choices as earlier.

Eventually, through frustration I started to check the others of the twelve previously-downloaded Boundary Line data sets and found there was a shape file called “district_borough_unitary_region.shp” On inspection in GeoPandas it appeared that this was what I needed – despite Scottish Local Authorities being neither districts nor boroughs – except that it contained all local authority boundaries for the UK – some 380 (not just the 32 that I needed).

Converting the data

Having downloaded the data I then had to find a way to convert it from Shapefile to Geojson (adapting some code I had discovered on StackOverflow) then subset the data to throw away almost 350 of the 380 boundaries. This was a two stage process: use a conversion script to read in Shapefiles, process and spit out Geojson; write some code to read in the Geojson, covert it to a python dictionary, match elements against a list of Scottish LAs, then write the subset of boundaries back out as a geojson text file.

Code to convert shapefiles to geojson
Code to convert shapefiles to geojson

Using the Geojson to create a choropleth map

I’ll spare the details here, but I then spent many, many hours trying to get the Geojson which I had generated to work with the Folium library. Eventually it dawned on me that while the converted Geojson looked ok, in fact it was not correct. The conversion routine was not producing the correct Geojson.

Another source

Having returned to this about 10 days after my first attempts, and done more hunting around (surely someone else had tried to use Scottish LAs as geojson!) I discovered that Martin Crowley had republished on Github boundaries for UK Administrations as Geojson. This was something that had intended to do for myself later, once I had working conversions, since the OGL licence permits republishing with accreditation.

Had I had access to these two weeks ago, I could have used them. With the Scottish data downloaded as Geojson, producing a simple choropleth map as a test took less than ten minutes!

Choropleth of Scottish councils. Data Source: Contains OS data © Crown copyright and database right (2019)
Choropleth of Scottish councils. Data Source: Contains OS data © Crown copyright and database right (2019)

While there is some tidying to do on the scale of the key, and the shading, the general principle works very well. I will share the code for this in a future post.

Some questions

There is something decidedly user-unfriendly about the SDI approach which is reflective of the Scottish public sector at large when it comes to open data. This raises some specific, and some general questions.

  1. Why can’t the Scottish Government’s SDI team publish data themselves, as the OGL facilitates, rather than have a reliance on OS publishing?
  2. Why are boundary data, and by the looks of it other geographic data, published as ESRI GIS shapefiles or Mapinfo formats rather than the generally more-useable, and much-smaller, GeoJson format?
  3. Why can’t we have Scottish (and English, and Welsh) authority boundaries as individual downloads, rather than bundled as UK-level data, forcing the developer to download unnecessary files? I ended up with 1.13GB (and 48 files) of data instead of a single 8.1MB Scottish geojson file.
  4. What engagement with the wider data science / open community have SDI team done to establish how their data could be useful, useable and used?
  5. How do we, as the broader Open Data community share or signpost resources? Is it all down to government? Should we actively and routinely push things to Google Dataset Search? Had there been a place for me to look, then I would have found the GitHub repo of council boundaries in minutes, and been done in time to see the second half of Elf!

And finally

I am always up for a conversation about how we make open data work as it should in Scotland. If you want to make the right things happen, and need advice, or guidance, for your organisation, business or community, then we can help you. Please get in touch. You can find me here or here or fill in this contact form and we will respond promptly.

Are Twitter closing the door on the educational sector?

Sorry - we are closed

Changes to Twitter’s terms of use for developers mean that universities, online tutors, and authors of instructional textbooks may no longer be able to teach students how to mine Twitter data in the way that they have done for several years. So how will the next generation learn how to use Twitter data? Ian Watt is concerned. 

If you’ve read any book on gathering Twitter data via the API it  is highly likely that you will get a fairly standard set of instructions, such as these from Python Social Media Analytics by Chatterjee and Krystyanczuk [PacktPub]:

  1. Create a Twitter account or use your existing one.
  2. Go to and log in with your account.
  3. Click on Create your app and submit your phone number. A valid phone number is required for the verification process. You can use your mobile phone number for one account only….

That’s how I learned it from books, and that’s how it was taught in our Data Science Course, but it appears that access to that method is now closed.

As computing science faculties around the world start to welcome a new intake of students, they need to face up to a difficult period ahead.

The Party’s Over

Towards the end of July 2018 I was completing my MSc Data Science project, a semester-long piece of work which relied heavily on access to Twitter’s API for its data. When I went to add an application to my Twitter account I got a nasty surprise!

If you visit now, you will find that you cannot create a new App unless you have a full developer account.

This was announced on 24th July, 2018,  and the developer documentation was updated at the same time.

“Starting July 24th, 2018, anyone who wants to create a new Twitter app will need to have an approved developer account. You can apply for a developer account at Once your application has been approved, you’ll be able to create new apps on”

It should be noted that pre-existing apps, created under existing accounts using the former method, will still work for now, but unless you are the owner of an approved developer account you will no longer be able to create new apps, and so will not be able to get authorisation tokens to run your new project.

As an experiment, I applied to have my account upgraded to a Developer one at the end of July to see how long it would take. As of today, 27th August, I am still waiting.

Why does it matter?

This change creates real challenges to Computer Science departments in universities and colleges world-wide.  If you teach big data, social media analytics, data mining, data science or similar, here are a bunch of questions for you:

  • How are your students going to learn how to use Twitter data now?
  • How will your R and Python Data Science courses teach how to use data which is no longer readily available to students?
  • If you continue to teach this, how long will it take students to obtain Developer accounts?
    • How can you guarantee that they will get them?
    • And what it they don’t?

And for developers, there are further issues ahead. The number of apps you can register are being reduced and the rate limiting is getting tighter in the next couple of weeks.

I understand why Twitter is doing this, and I respect their attempts to tackle real issues by removing bots, fake accounts etc. but, like with all big decisions, it does appear that there are unexpected consquences.

A final call to action

Who, in the academic community, has faced up to these issues? Is there a back-channel to Twitter to ensure that students can still be taught to use Twitter data responsibly? Who is lobbying Twitter on behalf of the educational sector?

We all need to raise these issues and ensure continuity of data science education.

Header photo by Tim Mossholder on Unsplash