The National Archives and The University of Glasgow have recently been successful in applying for funding from The University of Glasgow’s Knowledge Exchange fund. The project will see collaboration between the Humanities Advanced Technology and Information Institute (HATII), Glasgow’s School of Physics and Astronomy, the School of Computing Science and The National Archives to address the issue of sensitivity review for digital records.
A server room at The National Archives
Government departments are obliged by the Public Records and Freedom of Information Acts to select, review for sensitivity, and transfer public records to The National Archives. The selection and review is conducted by the government department concerned based on advice and guidance published by The National Archives; the outcome is that about 5 per cent of records are selected for permanent preservation and release to the public. In the present paper format these are largely the familiar files held together with Treasury tags.
There are clearly defined exemptions to this release where the information to be preserved has continuing sensitivities (ranging from personal data to material related to national security), and must be legitimately withheld from public view for an extended period of time. While the review of paper records for sensitivities is a long established and well understood practice, the move to digital records transforms the landscape from one in which well structured paper files are replaced by documents held as single instantiations in electronic systems which may or may not be aggregated into something approaching a file in the paper world.
The result is that digital records are much more granular than paper ones. Just like grains of sands on the sea shore. This will inevitably render established procedures for sensitivity review both redundant and so expensive that departments may consider themselves unable to comply with their statutory obligations. Digital records are arriving in large volumes from bodies such as the 2012 London Olympic Games (London Organising Committee of the Olympic and Paralympic Games and Olympic Delivery Authority). The first born-digital records from departments will be made available by The National Archives within the next ten years following the implementation of the 20 year rule.
Recent work at The National Archives on the subject of ‘Parsimonious Preservation’ 1 has confirmed that the much-discussed ‘black hole’ in our history, wrought by obsolete digital formats, is largely fictitious. Far more will survive than anyone predicted. This however presents a significant emerging risk, which this research addresses directly.
The risk is that the increased cost and complexity of the traditional review process will lead to unacceptable delays in the transfer of the digital record or precautionary exemptions being applied to large swaths of records damaging public accountability and transparency and making it difficult for contemporary history to be written. Further research is therefore urgently needed to determine the most effective and efficient methods for the sensitivity review of digital records before transmission. This is a problem for the whole archival sector.
The University of Glasgow’s School of Computing Science is a world-leader in information retrieval and has developed the Terrier Platform for this purpose. Building on their work, this project will develop a prototype decision-support system to implement and develop new review methods. The system will be evaluated on real records with real sensitivities, which will inform, support and improve the ability of government departments and in fact any organisation which wishes to make contemporary records publicly available to review digital records.
Tim Gollins, Head of Digital Preservation, will be seconded to the University of Glasgow to provide one of the primary research resources to the project. Interviews will be held with the relevant teams at The National Archives, such as FOI and Knowledge Information Management and with teams at the Foreign and Commonwealth Office and Ministry of Defence faced with the practical problems ‘at the coal face’ to provide sources of expertise and experience in sensitivity review and possible approaches and methods. Although from one perspective this is a technical problem, from another it is a juridical issue that lies at the heart of the The National Archives’ mission to protect the ‘rule of law’. The team will seek advice from the Lord Chancellor’s Advisory Council on National Records and Archives and other experts in the field in developing in this new modus operandi.
This research is central to one of The National Archives’ strategic research priorities around developing advanced tools for selection and sensitivity review. We are thrilled to be a partner in this project as it addresses a pressing need, sees the transfer of research into practical solutions and fits well within our research goals.
You refer to “Government departments are obliged by the Public Records and Freedom of Information Acts to select, review for sensitivity, and transfer public records to The National Archives” but this is not always true. For example Treasury have outsourced their records management work to Iron Mountain who would have little knowledge of the sensitivity except in a few specific areas, like data protection. In fact the legislation has no teeth and no penalties if departments do not comply and this has led to a number of departments including Treasury not complying with the Public Records Acts.
I think you underestimate the amount of material to be kept whilst it may be an average of 5%, departments like Treasury have been around 40% but that is now rising because of the inability to work out which is the master set, given that around 15 copies of major documents were sent around (including the 5 Treasury Ministers) and the decision over what to keep were much easier and clearer before the TNA guidelines were changed. Why are the teams at Treasury/Iron Mountain not being talked to about the sensitivity of digital records?.
The question is why cannot the digitised service be outsourced?, once it is no longer sensitive then the private sector can take over the work and in the long-term we are I am sure going to see TNA diminish and perhaps close down.
The Treasury / Iron Mountain project is focused on transferring paper records to The National Archives. All stakeholders work very closely together, including discussions about master sets, and The National Archives has provided training and support for both teams. The Treasury Project Management team remain responsible for ensuring sensitivity review is carried out appropriately.
The National Archives is working with all government departments to manage the transfer of any legacy records. The figures for records overdue for transfer can be seen in the Records Transfer Report on our transparency pages: http://www.nationalarchives.gov.uk/about/record-transfer-report.htm
The outsourcing of any transfer processes is a decision for each department to make individually and as we move closer to the transfer of digital records under The Public Records Act, The National Archives continues to work with departments and suppliers to ensure compliance and to understand issues such as digital sensitivity review. The National Archives remains, under the legislation, the government’s national archive for England, Wales and the United Kingdom.
TNA should have a role in outsourcing to ensure that the work is correctly outsourced and that it is functioning correctly, something which is clearly not working. If you take TPEC (the Treasury Public Expenditure Committee) there are more than one set of the committee papers whereas previously they were only kept in T 277 (Treasury: Committee Section: papers). The inaccuracies in Discovery since Iron Mountain/Treasury have run the project is so poor (they just copy the file covers!), I have corrected a number of them myself and there is one series where bilateral discussions with the Department of Health about 8 years before they existed!. Other examples are local government finance files said to have been transferred from another department, whereas they would never have been transferred there and in one case (for which Treasury have apologised) where it was said the work on the supervision of banks and building societies was transferred to an independent commission whereas the work remained in Treasury. There are so many sets of records missing and there has been a clear lack of transparency by TNA over what series there are to be transferred and the guidance by TNA is not only not working but is at variance with TNA guidance and in some case the files have been refiled and mis-filed, though at least Iron Mountain seems to have stopped ‘squaring-off’ the file covers!. There was a time when Treasury met the 30-year rule and even now we await their files on the Falklands War.