Open Data Scotland – a nudge from OD Camp?

Drawnalism captures open data sketch notes

Over the first weekend of November 2018, just over 100 people congregated in Aberdeen to attend the UK Open Data Camp. We’d pushed hard to bring it to Scotland, and specifically Aberdeen, for the first time. The event, the sixth of its type, which follows an unconference model where the attendees set the agenda, has previously taken place in England, Wales and Northern Ireland.

I’m not going to go through what we did over the weekend, you can find plenty of that here and here. There are links to all 44 sessions which took places on this Google doc, and many of those have collaborative notes taken during the sessions.

Instead this is a reflective piece, seeking to understand what OD Camp can show us about the state of Open Data in Scotland and beyond.

Who was there?

Of the 100+ attendees, including camp-makers, we estimate that about 40 were from the public sector. Getting exact numbers is hard – people register in their own name, with their own email addresses, but we think that is a good guess.

While this sounds good, during the pitching session on the first day Rory Gianni asked a question: “Hands up who is here from the Scottish public sector?” Two people’s hands went up out of 100+. Each were from local authorities, Aberdeen and Perth city councils, and a third (also from Aberdeen) joined later on Saturday.

This is really concerning and shows the gulf between what Scotland could, or rather must, be doing and what is actually happening.

The Scottish Public Sector

It is estimated that the Scottish Civil Service encompasses 16,000+ officers. It encompasses 33 directorates,  nine executive agencies  and around 90 Non-departmental public bodies (NDPBs) plus other odds and ends such as the Crown Office and Procurator Fiscal Service.

Then we have 14 health boards, 32 local authorities, 32 Joint Health and Social Care Partnerships and so on.

All of these should be producing open data.

Reality

Sadly, we are very far from that. Few are of any scale or quality. I’ve written about this extensively in the past including in this blog post and its successor post.

So, if we use attendance by the Scottish public sector, at a free-to-attend event which was arranged for them on their very doorstep, as a barometer of commitment to open data, it is clear that something is rotten in the state of Denmark Scotland.

Three weeks on

Since the event, I’ve reached out to the Scottish Government through two channels. I contacted the Roger Halliday, the Chief Statistician, the senior civil servant with a responsibility for Open data, and responded to a Twitter contact from Kate Forbes, the minister for Public Finance and Digital Economy.

I then had an hour-long conversation with Roger and two of his colleagues. This was a very positive discussion. I took away that there is a genuine commitment to doing things better, underpinned by a realism about capacity and capability to widely deliver publication and engagement with the wider OD community. I have agreed to be part of a round table meeting on OD to be held in the new year – and have expressed a commitment to assist in any way needed to improve things.

Meanwhile

Ironically, in the midst of this three week period, the Scottish Government published its Open Government action plan. This emerged on 14th November and is open for feedback until 27th November. So, if you are quick, you can respond to that – and I encourage you to do so. While this certainly seeks to move things in the right direction in terms of openness and transparency, it is extremely light on open data and committed actions to address some of the issues which I have raised.

My next blog post will be a copy of the feedback which I provide, and on which I am currently working.

And finally

When I started drafting this post I was in a very negative frame of mind as regards the Scottish Open Data scene – and particularly in terms of the public sector. In the intervening period, I  launched the Scottish Open Data Action group on Twitter. The thinking  behind this was to get together a group of activists to swell the public voice beyond mine and that of ODI Aberdeen.

Given the way things are moving on with the Scottish Government and the positive engagement that has begun, the group, which is in its infancy, may not be needed as a vocal pressure group. Instead we could be a supportive external panel who provide expertise and encouragement as needed. Who knows – let’s see!

Are Twitter closing the door on the educational sector?

Sorry - we are closed

Changes to Twitter’s terms of use for developers mean that universities, online tutors, and authors of instructional textbooks may no longer be able to teach students how to mine Twitter data in the way that they have done for several years. So how will the next generation learn how to use Twitter data? Ian Watt is concerned. 

If you’ve read any book on gathering Twitter data via the API it  is highly likely that you will get a fairly standard set of instructions, such as these from Python Social Media Analytics by Chatterjee and Krystyanczuk [PacktPub]:

  1. Create a Twitter account or use your existing one.
  2. Go to https://apps.twitter.com/ and log in with your account.
  3. Click on Create your app and submit your phone number. A valid phone number is required for the verification process. You can use your mobile phone number for one account only….

That’s how I learned it from books, and that’s how it was taught in our Data Science Course, but it appears that access to that method is now closed.

As computing science faculties around the world start to welcome a new intake of students, they need to face up to a difficult period ahead.

The Party’s Over

Towards the end of July 2018 I was completing my MSc Data Science project, a semester-long piece of work which relied heavily on access to Twitter’s API for its data. When I went to add an application to my Twitter account I got a nasty surprise!

If you visit https://apps.twitter.com/ now, you will find that you cannot create a new App unless you have a full developer account.

This was announced on 24th July, 2018,  and the developer documentation was updated at the same time.

“Starting July 24th, 2018, anyone who wants to create a new Twitter app will need to have an approved developer account. You can apply for a developer account at developer.twitter.com. Once your application has been approved, you’ll be able to create new apps on developer.twitter.com.”

It should be noted that pre-existing apps, created under existing accounts using the former method, will still work for now, but unless you are the owner of an approved developer account you will no longer be able to create new apps, and so will not be able to get authorisation tokens to run your new project.

As an experiment, I applied to have my account upgraded to a Developer one at the end of July to see how long it would take. As of today, 27th August, I am still waiting.

Why does it matter?

This change creates real challenges to Computer Science departments in universities and colleges world-wide.  If you teach big data, social media analytics, data mining, data science or similar, here are a bunch of questions for you:

  • How are your students going to learn how to use Twitter data now?
  • How will your R and Python Data Science courses teach how to use data which is no longer readily available to students?
  • If you continue to teach this, how long will it take students to obtain Developer accounts?
    • How can you guarantee that they will get them?
    • And what it they don’t?

And for developers, there are further issues ahead. The number of apps you can register are being reduced and the rate limiting is getting tighter in the next couple of weeks.

I understand why Twitter is doing this, and I respect their attempts to tackle real issues by removing bots, fake accounts etc. but, like with all big decisions, it does appear that there are unexpected consquences.

A final call to action

Who, in the academic community, has faced up to these issues? Is there a back-channel to Twitter to ensure that students can still be taught to use Twitter data responsibly? Who is lobbying Twitter on behalf of the educational sector?

We all need to raise these issues and ensure continuity of data science education.

Header photo by Tim Mossholder on Unsplash