An Open Digital Iraq Inquiry

The internet opens up parts of our democracies other media cannot reach. The Obama team is merrily using internet as a can opener for information too dense to be penetrated by the ADD-afflicted traditional media.

The Obama approach is similar to the UK Power of Information work begun a few years ago. No less than Sir Tim Berners-Lee himself is now employed to help the UK government open up its data to web analysis. This geek democracy is a powerful new force.

The recent expenses fiasco for UK Members of Parliament (aka moat-gate or scamalot) has changed expectations about the way information is published. The information Parliament published was so opaque that The Guardian website ran a brilliant exercise to crowd source basic information about what was in the hundreds of thousands of documents put online. People in their thousands helped put in the 'metadata' that should have been there in the first place. This is back to front -- the cost of the opacity should be born by the obfuscator.

In the UK I am wondering how we can apply geek democracy to the upcoming Iraq inquiry. Sir John Chilcott's inquiry is in its planning stages. The inquiry will probably receive huge quantities of scanned documents in electronic form. The 1996 Scott inquiry into arms to Iraq received 130,000 documents. Sir John Chilcot refers to his impending challenge in a letter to Prime Minister Gordon Brown:

One important point which has not received much public notice so far is that examining and analysing the very large body of existing documentary evidence, stretching over seven or more years, will necessarily occupy a significant part of the time available to the Inquiry, especially in the early stages.

Documents like this one from a 2003 inquiry can't really be computer read or scanned into proper text. But as a member of the public I would want to be able to Google these documents in depth. If the Inquiry just puts these documents online their treasure trove of information could end up like a cupboard full of tins with no labels. So it's vital for each document to have an electronic index of what is in it -- to put the label on the tin.

But the hard pressed Inquiry team shouldn't have to do that. The Inquiry could insist that all evidence supplied comes with proper metadata. That is information that tells you precisely what is in the document -- like an index to a book -- so that you can work out whether to read it. Then the Inquiry and when the documents are published, all of us can search for what is in them.

What other things could the Inquiry do to maximize the potential of geek democracy? In general, the Inquiry should presume that other people will reuse all the information it publishes and design their web service accordingly. To do this they should take advice from a panel of folk with experience of publishing for reuse or scraping information that wasn't intended to be reused. Berners-Lee's superb note on publishing government data should be instructive. In the spirit of the Power of Information work the Inquiry could provide a simple API to its data to allow other to present it how they want.

There are other simple things the Inquiry could do. For instance, publishing non-sensitive evidence information raw as soon as it is received, not at the end of the process after a lot of fettling. Talking about government data in general, Berner's-Lee says:

'Put the data up where it is: join it together later.'

A daily transcript could be posted up copyright free or under a permissive creative commons license. Live video and audio feeds from the Inquiry could be archived online free of copyright in near real time. This would give the public a route into the evidence independent of the broadcasters and their ante-diluvian copyright and selective editing regimes. Ideally the transcript and the video and audio could be linked together -- i.e. click in the transcript to see or listen to the words. As well as this rich content, the Inquiry team could make sure their web site has a low graphics or text-only version so it can be read in countries like Iraq easily as well as caching or mirroring the site in the Middle East.

Lord Hutton's 2003/4 judicial inquiry in the death of David Kelly was a model of open-ness and transparency, unusual in Britain at that time. Freedom of Information is still in its early days in Britain and daily transcripts live video to rolling news, documents made available online were a novelty in 2003 and remarkable for a formal judicial inquiry.

Technology has moved on a lot since 2003. The MP's expenses fiasco showed how not to use the web to publish large quantities of information in which there is intense public interest. Sir John Chilcot is not constrained by being a judicial inquiry and has a wealth of good will towards him. There is an opportunity for him to follow the Obama model and redefine open-ness and transparency using the internet.

This article is written in an entirely personal capacity and based upon a more detailed blog post.