Digital records sensitivity review
The National Archives and The University of Glasgow have recently been successful in applying for funding from The University of Glasgow’s Knowledge Exchange fund. The project will see collaboration between the Humanities Advanced Technology and Information Institute (HATII), Glasgow’s School of Physics and Astronomy, the School of Computing Science and The National Archives to address the issue of sensitivity review for digital records.
Government departments are obliged by the Public Records and Freedom of Information Acts to select, review for sensitivity, and transfer public records to The National Archives. The selection and review is conducted by the government department concerned based on advice and guidance published by The National Archives; the outcome is that about 5 per cent of records are selected for permanent preservation and release to the public. In the present paper format these are largely the familiar files held together with Treasury tags.
There are clearly defined exemptions to this release where the information to be preserved has continuing sensitivities (ranging from personal data to material related to national security), and must be legitimately withheld from public view for an extended period of time. While the review of paper records for sensitivities is a long established and well understood practice, the move to digital records transforms the landscape from one in which well structured paper files are replaced by documents held as single instantiations in electronic systems which may or may not be aggregated into something approaching a file in the paper world.
The result is that digital records are much more granular than paper ones. Just like grains of sands on the sea shore. This will inevitably render established procedures for sensitivity review both redundant and so expensive that departments may consider themselves unable to comply with their statutory obligations. Digital records are arriving in large volumes from bodies such as the 2012 London Olympic Games (London Organising Committee of the Olympic and Paralympic Games and Olympic Delivery Authority). The first born-digital records from departments will be made available by The National Archives within the next ten years following the implementation of the 20 year rule.
Recent work at The National Archives on the subject of ‘Parsimonious Preservation’ 1 has confirmed that the much-discussed ‘black hole’ in our history, wrought by obsolete digital formats, is largely fictitious. Far more will survive than anyone predicted. This however presents a significant emerging risk, which this research addresses directly.
The risk is that the increased cost and complexity of the traditional review process will lead to unacceptable delays in the transfer of the digital record or precautionary exemptions being applied to large swaths of records damaging public accountability and transparency and making it difficult for contemporary history to be written. Further research is therefore urgently needed to determine the most effective and efficient methods for the sensitivity review of digital records before transmission. This is a problem for the whole archival sector.
The University of Glasgow’s School of Computing Science is a world-leader in information retrieval and has developed the Terrier Platform for this purpose. Building on their work, this project will develop a prototype decision-support system to implement and develop new review methods. The system will be evaluated on real records with real sensitivities, which will inform, support and improve the ability of government departments and in fact any organisation which wishes to make contemporary records publicly available to review digital records.
Tim Gollins, Head of Digital Preservation, will be seconded to the University of Glasgow to provide one of the primary research resources to the project. Interviews will be held with the relevant teams at The National Archives, such as FOI and Knowledge Information Management and with teams at the Foreign and Commonwealth Office and Ministry of Defence faced with the practical problems ‘at the coal face’ to provide sources of expertise and experience in sensitivity review and possible approaches and methods. Although from one perspective this is a technical problem, from another it is a juridical issue that lies at the heart of the The National Archives’ mission to protect the ‘rule of law’. The team will seek advice from the Lord Chancellor’s Advisory Council on National Records and Archives and other experts in the field in developing in this new modus operandi.
This research is central to one of The National Archives’ strategic research priorities around developing advanced tools for selection and sensitivity review. We are thrilled to be a partner in this project as it addresses a pressing need, sees the transfer of research into practical solutions and fits well within our research goals.