Speaking at Oxford on the occasion of the 800th anniversary of Magna Carta (Danziger and Gillingham, 2004), Lord Dyson, the then Master of the Rolls—the second most senior member of the judiciary, supported the opinion of his predecessor Lord Denning in more measured terms: “One tenet of Magna Carta that remains as valid now as it was in 1215 is its statement that justice shall be done by ‘the law of the land’. It is not surprising that our view of what the law of the land should be today differs markedly from what the barons thought it should be in 1215. But the principle that justice should be done according to the law of the land is as important today as it was in 1215. Establishing and preserving the rule of law is a vital pillar of our democratic system. To use the language of a later version of Magna Carta, justice must be determined according to ‘the due process of law’ ” (Dyson, 2015, para 8). The rule of law extends to everything and everybody from the legality of the declaration of war on Iraq, which another of Lord Dyson’s predecessors Lord Bingham considered illegal (Bingham, 2010: 120–129, 158), to a woman from Paisley complaining that she had been poisoned by a snail in a ginger beer bottle in 1932 (cited in Dyson, 2015, para 11; Chapman, 2010). No one can be above the rule of law as Lord Denning declared: “the foundation of the freedom of the individual against the arbitrary authority of the despot”, whether that be a democratically elected president or prime minister, an hereditary monarch or an autocrat. The Chinese authorities were so concerned about the implications of the charter that they blocked its display at Remnin University and the exhibition had to be moved to the British Residence (Middlehurst, 2015). This fundamental constitutional principle was taken seriously by Lord Neuberger, president of the Supreme Court, in his recent judgement in the case of the disclosure of the Prince of Wales letters that had been blocked by the Attorney General: “there is no clear or specific suggestion anywhere in the FOIA [Freedom of Information Act, 2000] that it is intended that a section 53 certificate [used by the Attorney General to block disclosure] should enable a member of the executive to override a judicial decision” (Neuberger, 2015a, para 90). The recent report of the Independent Commission on Freedom of Information has confirmed, despite fears to the contrary, that Section 53 be used sparingly and should remain open to judicial review—“Where a veto is exercised appeal rights [to the Information Commissioner] would fall away and a challenge to the exercise of the veto would be by way of judicial review to the High Court” [Independent Commission, 2016, recommendation 15: 40].

For the rule of law to operate there must be evidence, which can be corroborated and has not been obtained unfairly, for example, by bribes or torture in judicial processes, or fabricated by government or simply never recorded (Bingham, 2010). Record-keeping by government has always been in Lord Panmure’s words in a memorandum of 1855 “The great desiderata for the easy and efficient discharge of the duty of a public office is a simple and efficient system of registration of the papers of the department” (cited in Moss, 2012: 866). It has always had another function to draw a clear line between the back and front office and in so doing give essential protection to civil servants (Foster, 2005: 26). When the current Lord Chancellor, Michael Gove, in the Conservative Administration protests that the government has a duty of care to civil servants to protect them from vexatious Freedom of Information Act (FOIA) requests, he overlooks this fundamental aspect of good record-keeping (Graham, 2015). If information requested under FOIA is difficult to find and places an undue burden on civil servants then it can only mean it was not recorded effectively in the first place. As Sir Alan Beith, MP, Chairman of the Westminster Justice Committee puts it in 2012:

Evidence we have seen suggests that reducing the cost of FoI can be achieved if the way public authorities deal with requests is well-thought through.

Complaints about the cost of FoI will ring hollow when made by public authorities which have failed to invest the time and effort needed to create an efficient freedom of information scheme. (Beith, 2012)

Lloyd George, the British Prime Minister who was the architect of allied victory in 1918, recognizing the importance of record-keeping in a modern bureaucracy established the Treasury O&M department in 1919 solely to mandate record-keeping registries across government departments (Treasury, 1962-1963; Craig, 2002). It continued to function until the early 1970s when it was swept away in a misconceived modernization that failed to appreciate that accountability in the public sector is very different from the corporate sector. Despite fleeting attempts to emphasize the importance of good record-keeping in the digital age largely by records managers and archivists, there is now nothing equivalent to the O&M department and attempts to mandate good practice have largely failed unless there is a strong compliance driver, such as passport control or the issue of driving licences (for example, Knowledge Council, 2008). This is the cause for concern. The inadequacies of process can be seen clearly in the evidence presented to the Hutton Inquiry in 2003 (Moss, 2005) and, in more recent reports by Sir Alex Allan (Allan, 2014,2015)

It is not only for the judiciary, constitutional lawyers and those who police Freedom of Information(FOI) to ensure records in whatever form are kept that make it possible for the executive to be called to account; it is also the responsibility of the legislature that creates the laws in the first place. At the time of Magna Carta it was the barons, who formed the Royal Council, today it is the elected representatives and in Britain the unelected House of Lords. In the United Kingdom select committees, which are now free of party control, have launched searching enquiries into the exercise of executive power. In the wake of the loss of data by the UK HM Revenue and Customs, the Home Affairs Select Committee embarked on a detailed enquiry into the so-called surveillance society that threatens one of the essential tenets of Magna Carta—the freedom of the individual against the arbitrary power of the state. While they agree with a previous report for the Information Commissioner that surveillance could be seen as just a further stage in the long process of “modernity” and by so doing distanced itself from scare-mongering stories about intrusive monitoring of citizens, it called for greater accountability of data controllers and handlers and by extension better record-keeping (Home Affairs, 2008; Moss and Gollins, 2016).

When records are transferred to the archives in whatever form, they must be stored securely in such a fashion that they cannot be altered or tampered with and assurance can be given to third parties that a record is what it purports to be. In England the Master of the Rolls was so-called because he performed the function of the Keeper of the Rolls (records) from medieval times. It is not coincidental that the first Public Record Act of 1838 coincided with what in Weberian terms might be described “as progress towards efficient administration”, an essential characteristic of modernity (Cohen, 2013). The keeping of records is now delegated by legislation to The National Archives (TNA) at Kew, the National Records of Scotland and the Public Record Office of Northern Ireland. All national archives, even if they are funded by the state, now have a fiduciary and statutory responsibility to store securely the records of the law courts and of government. The same is true for all public bodies covered by FOIA. This principal is embodied in public records legislation in Britain’s former colonial possessions and is an essential bulwark against “the arbitrary power of the despot” and the right to a fair trial. In many countries it has to be resolutely defended.

Only with the advent of a scientific approach to history, another token of modernity pioneered in Germany, did archives become storehouses of evidence for historical research, where as Lord Acton put it: “Historic responsibility has to make up for the want of legal responsibility” (Acton, J. D- to Creighton, M.). The holding of government to account in the court of history is not fanciful and helps prevent the mythologizing of the past in pursuit of contemporary political advantage as happened twice in Germany in the last century. However, with an increasing emphasis on openness, much shorter closure periods than in Acton’s day and international efforts to stamp out corruption, records transferred to the archives are being used to right wrongs and abuse, such as the allegations of historic child abuse in the United Kingdom or endemic corruption in many African states. In a recent report on record-keeping in three Nordic countries, Estonia, Finland and Norway, Anne Thurston of the International Records Management Trust emphatically endorsed the crucial role of the archive in holding the evidence in the digital environment disclosed by due process which can be used to hold government to account—“The longstanding role of the national archives, as the agency with statutory responsibility for managing and protecting the evidentiary value of records, is ongoing” (Thurston, 2015: 5).

Under the now 20-year rule in the United Kingdom (15 in Scotland) records are transferred by government departments after review for sensitive content to TNA (1962–1963) in London under the terms of the Public Records Act (PRA) (1958) and FOIA 2000. PRA and FOIA applies to all public sector bodies wherever they deposit their records. There is separate legislation in the North of Ireland and Scotland (Public Records Act (Northern Ireland), 1923; Public Records (Scotland) Act (PR(S)A), 2011). Although responsibility for TNA was transferred to the Department of Culture Media and Sport (DCMS) by the incoming Conservative Administration in 2015, oversight of the transfer of government records remains in the hands of the Master of the Rolls, who chairs the Lord Chancellor’s Advisory Council on National Records and Archives (Advisory Council, 2016), which is a non-departmental public body (NDPD) independent of TNA. It is composed of unpaid lay members who advise the responsible minister on which records can remain closed to public access beyond the 20-year rule under the terms of the exemptions listed in the appendix to FOIA (FOIA, Chapter 36). These should not be confused with protective markings (top-secret, secret, restricted and so on) and range from national security (Section 24), to international relations (Section 27) to health and safety (Section 38) and personal information (Section 40). Some of the exemptions decay after 20 years, such as formation of government policy (Section 35). Departments select records for transfer with advice from TNA and then request the Advisory Council to confirm recommendations to close records for specific periods of time. The majority of closures are simply the redaction of names or sentences and sometimes the extraction of a piece from a file and only rarely the closure of a whole file. The longest closure periods are for health and safety and personal information that can extend to 125 years to protect the identity of individuals judged to be especially vulnerable, for example, informants and interpreters in a theatre of war.

In reaching its decisions the Advisory Council applies what is known as the “public interest test” or “override”—is it in the public interest that this information should be withheld or disclosed? As Carter and Bourise explain:

The “public interest” is an amorphous concept, which is typically not defined in access to information legislation. This flexibility is intentional. Legislators and policy makers recognise that the public interest will change over time and according to the circumstances of each situation. In the same way, the law does not try to define categorically what is “reasonable”. (Carter and Bouris, 2006, para 2.5)

The public interest was the basis of the judgement by Lord Bingham in the House of Lords in the well-known case to support the Serious Fraud Office in halting investigation of accusations against BAE systems on evidence that had been inadvertently released in TNA and made public by the Campaign Against the Arms Trade (Lords Judgment, 2008). The public interest is also the basis on which the veto under Section 53 is exercised “where the accountable person takes a different view [Independent Commission on Freedom of Information - report (2016)] of the public interest in disclosure” (Information Commission, recommendation 14: 38).

Although under FOIA some records enjoy an absolute exemption while held by departments, these do not apply except in certain specified cases (for example, the Geneva Convention) when records are transferred to the TNA where reasons for closure have to be clearly stated in the catalogue. It is possible to request access to closed records held by TNA under FOIA. These are mostly requests to view personal information as the subject can be shown to be dead. Where there is doubt requests are reviewed by panels of the Advisory Council convened periodically. Decisions could in theory be challenged in the Courts, but none have unlike FOIA requests to departments, such as the case involving the Prince of Wales correspondence with government departments (Neuberger, 2015a). Sir Alex Allan in his recent report on government record-keeping in the wake of the inadvertent release of two documents “by the Cabinet Office ... relating to the Indian operation at Sri Harmandir Sahib—also called the Golden Temple —in Amritsar in June 1984, raised questions about the ability of the Advisory Council to probe the sheer volume of material”—3,600 paper files alone in 2013–2014 (Allan, 2014). As a result the Cabinet Office has set up a challenge panel composed of senior civil servants to scrutinize all requests for closure before submission to the Advisory Council. It remains to be seen how this will work in practice. For its part the Advisory Council (2015) has set up a digital transfer working group to look at “the risks that digital brings and how these and the processes for handling digital materials will differ from those of paper” (Item 5).

The other important statutory function of the Advisory Council is to ensure that departments transfer records in a timely fashion to TNA and if they do not to seek an explanation. A good example is the recent controversy over the Foreign and Commonwealth Office’s (FCO) so-called “migrated archive” from former colonial possessions where—“In 2011–12 we [the Advisory Council] continued to take a close interest in the work of the Foreign and Commonwealth Office and The National Archives regarding the transfer of the Colonial Administration Files –also known as the Migrated Archive” (Advisory Council, 2013). In the alarmist press reporting about the “migrated archive” the Advisory Council was only referred to obliquely and not by name (Cobain, 2013). There was also passing reference to the resources needed to review such a mass of material for any sensitive content that was at times confused with de-classification. Selecting and reviewing files for transfer is time consuming. Although the FCO was given special funding to increase the number of reviewers from 12 to 26 (Engelhart, 2014), quite the reverse is true across government where the number of reviewers has been slashed because of cuts in expenditure.

The first step in the process is the selection or appraisal of records for transfer by the Departmental Record Officer. Appraisal guidance is available on the TNA Website (TNA, 2013). It assumes that there are robust record-keeping systems in departments and agencies. Despite claims to the contrary, it is still couched in the language of the Grigg Report of 1954 (Grigg, 1954) with considerable emphasis on records relating to policy and major initiatives—“Records of policy decisions that had a significant impact on UK or international events and conditions where department was the lead (for example, foreign policy, policing, asylum and other legal cases, deployment of troops, setting the budget, intervention on foreign exchanges)” (8). The guidance about the preservation of case files or particular instance papers (PIPs—referred to in the United States confusingly as PIIs) is vague, even though Grigg recommended that advice should be taken from the historical community. A PIP committee was set up to review recommendations for retention and disposal, but it was composed entirely of civil servants (Wilson Committee, 1981). One of the reasons that Grigg equivocated over PIPs was on grounds of space—“In 1965, the Keeper of Public Records raised the scary suggestion that if military service records were preserved, then the records of civil servants, teachers, miners and National Health Service (NHS) workers should also be kept” (Cantwell, 1991). This alarming prediction was followed by the Denning Committee on Legal Records, which reported the following year and recommended the destruction of huge swathes of legal processes, relieving “the Public Record Office alone ... of two hundred tons of records (occupying 15,000 feet of shelving” (Denning, 1966). The committee, which was again composed entirely of civil servants, argued that there was then a growing body of evidence “about what is going on” that lessoned the “relative importance of legal records as sources of historical interest”. They concluded this “can be satisfied by a few representative examples” (Denning, 1964: 8).

Grigg and Denning were writing in the context of a corporatist state that has largely been swept away by privatization and contracting out of services, many of which relate directly to individuals and where accountability is of concern. There was an attempt to revisit the question of PIPs by a committee chaired by Sir Duncan Wilson in 1975, but there was no satisfactory outcome, even though the committee was well aware that such records could now be analysed much more easily and effectively by the use of computers and there was growing interest in family history (Wilson Committee, 1981). Although PIPs are vital for righting wrongs, such as the Hillsborough disaster of 1989 when 96 people died or allegations of historic child abuse, and are the life blood of family history, policy for their selection and preservation still needs to be resolved (Hillsborough, 2012–2015). Christopher Price, a Labour MP who chaired the Education Select Committee, speaking in a House of Commons debate in 1985 on the government’s response to the Wilson Committee made an insightful observation long before computers are as sophisticated as they are today:

The Wilson committee talks of particular instance papers. That is important. The whole academic trend has changed since the Wilson committee and it is still changing quickly. Computers change the nature of academic research. Our academics could produce a new and tremendously useful insight into our immediate history if they could have a proper look at the particular instance papers. (Hansard, 1982, para 1135)

In much the same way as Grigg had proposed taking advice from historians about the preservation of PIPs, Wilson recommended the appointment of “sector panels—panels of acknowledged experts outside the Civil Service to co-operate with the civil servants, particularly in Departments, to ensure the right selection” (Hansard, 1982, para 1136). This was rejected on the grounds of cost. Although more PIPs are now retained, for example, the records of all those who served in the Home Guard during the Second World War, there is still not a consistent approach.

In the TNA guidance there is no reference to the critical constitutional function of the archive that will only grow with the introduction of the 20-year rule where records will be released that contain evidence where legal as well as historical responsibility will apply. It is axiomatic that records produced from the archives can be shown to be what they purported to be when they were deposited and that it can be shown that there is no possibility that they could have been interfered with when in archival custody (Jenkinson, 1980). The Nazi regime in Germany completely suborned the archive, making it compliant in their racial policies and misrepresentation of German history (Ernst, 1999). President Banda did much the same in Malawi, personally vetting applications for access with the connivance of the national archivist (Lihoma, 2012). The safeguard against political interference or manipulation by failure to transfer or unwarranted closures is the “rule of law” and in the British system the oversight of due process by the most senior civil lawyer in the judiciary. In these circumstances, it is questionable if departmental responsibility for TNA should have been transferred to DCMS from the Ministry of Justice, which suggests that government regards the archives as a cultural resource, presumably for family historians, rather than the repository of records by which it can be called to account.

The assumption is that the Grigg system will transfer seamlessly to the digital world without recognizing “the digital memorial shift from a culture of ‘scarcity’ to ‘post-scarcity’ ” (Hoskins, 2014), and in the belief that most records are stored in so-called Electronic Document Management Systems according to a “file plan” much as existed in paper registries. Despite regular TNA Information Audits designed to provide “government departments and public bodies with an independent and bespoke assessment of how well they are managing their information, and mitigating related risks” (TNA, 2016a), a second report by Sir Alex Allan on “Government digital records and archives” concluded:

Existing systems which require individual users to identify documents that should constitute official records, and then to save them into an EDRMS or corporate file plan, have not worked well. The processes have been burdensome and compliance poor. As a result, almost all departments have a mass of digital data stored on shared drives that is poorly organised and indexed. (Allan, 2015: 1)

If this is the case, there is every reason that the Advisory Council should scrutinize the Information Audits that is not at present part of its remit, particularly as the current Keeper asserts that: “it also helps ensure that the decisions and life of that organisation today become tomorrow’s publicly accessible record” (TNA, 2016b).

Although Allan recognized that this chaotic state of affairs presented a problem, much of the report is couched in terms of conventional records management with little reference to the impact the digital has made on the way business is conducted and to research, mostly in the United States but also by the UK Open Data Initiative, to interrogate such data using novel sense-making techniques, such as natural language progressing or visualization (for example, Prabhakaran and Rambow, 2014). These must be part of the solution in managing data that is being produced on an industrial scale in the digital environment against a background of tightening government expenditure. Such techniques accompanied by public and historical interest in a much wider spectrum of government than simply policy will result in demands for more records to be kept than before. As Arthur Lucas, a former member of the Advisory Council, pointed out at a recent conference on Threats to Openness in the Digital World at Northumbria University there is no external scrutiny of appraisal decisions by the Advisory Council, which he characterized as an “accountability gap”(Lucas, 2015). It was this gap that the Wilson Committee attempted to plug through an overarching committee of the Privy Council and departmental scrutiny panels. Appraisal decisions cannot simply be taken on the basis of use in the past or in the future or available space, but must also pass the public interest test (Carter and Bouris, 2006: 113–114). The recommendation of the Independent Commission on Freedom of Information that the destruction of information that has been requested should be punishable by fines rather than admonition reinforces the need for independent scrutiny of appraisal decisions and the publication of destruction schedules as under recent Scottish legislation (Independent Commission recommendation 3: 16). The publication of retention and destruction schedules is already a requirement of the new Scottish legislation under the supervision of the Keeper: “Section 1(2)(b)(iii) of the Act specifically requires a RMP to include provision about the archiving and destruction, or other disposal, of an authority’s public records” (PR(S)A, 2011; National Records of Scotland (NRS), 2011). Taken together this suggests that there is an urgent need for an updated PRA and may well be the result of the Independent Inquiry into Historic Child Sex Abuse, as was the PR(S)A the outcome of the Shaw Report (Shaw Report, 2007; Independent Inquiry, 2016).

Both the Westminster Justice Committee and the UK and Scottish Information Commissioners have robustly defended the right of access to information, admittedly citing FOIA rather than public records legislation but the meaning is the same—“the danger with open data is that the public only sees what the government chooses to make available—rather than the government providing data in response to specific queries”, and we could add by due process (Graham, 2015). Whereas Lord Bridges at the Cabinet Office in Westminster in announcing a review of FOIA claimed to the contrary:

We are committed to being the most transparent government in the world.

To deliver that goal we are opening up government to citizens by making it easier to access information and increase the volume available, with a record 20,000 datasets now on data.gov.uk, while protecting a private space for frank advice. (Bridges, 2015)

What he could have added that such releases comply with the EU Directive on the “re-use of public sector information”, which:

... focuses on the economic aspects of re-use of information rather than on the access of citizens to information. It encourages the Member States to make as much information available for re-use as possible. It addresses material held by public sector bodies in the Member States, at national, regional and local levels, such as ministries, state agencies, municipalities, as well as organisations funded for the most part by or under the control of public authorities (e.g. meteorological institutes). Since 2013 content held by museums, libraries and archives falls within the scope of application as well. (EU, 2015)

It comes then as no surprise that the bulk of the data Bridges referred to is aggregate transport data. Even if FOIA was amended as he wished which is not the conclusion the Independent Commission reached, records would still be released after 20 years under the PRA and it can be anticipated that any attempt to withhold information by the executive would result in complaints from the press and is likely to be quashed by the Supreme Court which is beginning under Lord Neuberger to flex its constitutional muscles (Neuberger, 2014). This simply reinforces the need to close the “accountability gap” by external scrutiny of appraisal and destruction policies by the Advisory Council in consultation with the user community. As in all decisions about what to keep and what to discard there will be tensions, particularly over PIPs especially in light of child abuse scandals. For the archive itself to become too engaged in appraisal decisions threatens to undermine its crucial fiduciary position, but nevertheless there must be some form of external scrutiny.

The release of data in any form into the public domain by whatever route demands scrutiny for sensitive content under the terms of the exemptions listed in FOIA and the data protection act. Even in the analogue world this is a time consuming and labour intensive process. It needs reviewers whose familiarity with the work of a department makes it easy for them to judge what might be considered to be sensitive. This is particularly true in the great departments of state where there is the most interest by historians, such as the FCO, the Ministry of Defence (MoD) and the Home Office. However, this is not always the case the records of some public bodies, such as local authorities and universities, review is left to the archivist or the person responsible for responding to FOI Enquiries. As well as the public interest test, reviewers need to take into account what information might be deemed already to be in the public domain. This is by no means straightforward. For example, if, as is the custom, counsel reads an extract from a brief in court, does it follow that the whole brief is in the public domain? Similarly if information in a record has been widely and accurately reported in the press or on the media, does this mean that corroborating information can safely by released, even if in so doing it breaches government protocol “neither to confirm nor deny” speculation (Carter and Bouris, 2006, para 3.12)? Opinions differs, the Information Commissioner favours release in the public interest, while the Advisory Council tends to take an opposite view. One of the obstacles is the lack of unambiguous case law with the exception of the findings of Information Commission tribunals (Hugenholtz, 2006).

The majority of records recommended for closure by departments and all public sector bodies are under sections 38 (health and safety) and 40 (personal information). They are normally closed for 100 years less the age of the individual if it is known and if it is not it is assumed to be sixteen. For minors closure can be for the full 100 years. There is very little understanding of what information about an individual is in the public domain, so the tendency is to close all records that contain personal information, even if some of it could be deemed to be in the public domain (Erdos, 2013). The fact that someone attended and graduated from a named university must be in the public domain as names of graduates and classes of degrees are published and can be verified by third parties. However, marks gained for individual classes will not be. The majority of roles and responsibilities held by individuals will be public and are often available in directories and official publications, but not necessarily all. The MoD, for example, does not make public the roles individuals occupy and will neither confirm nor deny press speculation. Likewise the security services will not divulge the names of agents. Where a name is deemed to be sensitive it can be redacted (blacked out) and the rest of the piece opened. However redaction is expensive and there are those who claim that redactions can easily be reversed with relatively few coordinates. David Thomas, a visiting professor at Northumbria, once requested the opening of a file relating to a fraud committed in Sussex. The file was opened but a huge amount of effort was put into redacting the name of the victim. However, since the victim’s address was given as West Dean House, it did not require a vast amount of research to identify the intended victim as the poet Edward James.

Following the revelations in Julian Assange’s WikiLeaks and Edward Snowden’s global surveillance disclosures, which provided evidence of the range and scale of information about individuals being harvested by the United States security services and being shared with her allies without the knowledge of citizens, there is worldwide concern about the privacy of personal data, particularly that which can be easily found on the Internet (Der Derian, 2014). The revelations confirmed anxieties about the extent and intrusion of the surveillance society. While they may not articulate their worry in terms of Magna Carta, it is its principles they echo. The executive may respond that such surveillance is necessary to meet the terrorist threat and no doubt it is, but that does not mean that it can exert such powers arbitrarily. The European Court of Human Rights accepts that individual states should be free to conduct such surveillance, but it insists that it must be in accordance with the law and be proportionate (Croft, 2015).

There is a minority who believe we live in a post-privacy age where nothing can be kept secret, so there is little point in registering any protest. On the contrary we can all claim to be “celebrities” (Auchard, 2007). The majority take an opposite view that the citizen has the right to know what information about them is being harvested and that measures should at least be taken to prevent the storage of such data for any longer than is necessary and that every effort should be made to safeguard the security of data held by Internet providers. Some even wish to exercise the right to be forgotten, in other words erased from the pages of history (Ghezzi et al., 2014). The United Kingdom Information Commissioner has taken such concerns seriously by encouraging a “privacy by design” approach in any new legislation and publishing a Privacy Impact Assessment Handbook (Information Commissioner, 2014). Taken together anxiety about personal data is leading to wholesale closure and reportedly destruction of records (Erdos, 2013). Lord Neuberger in a lecture in Singapore in 2015 had no doubt that, although privacy could be considered a fundamental human right, it was inevitably a “qualified right” when it comes into conflict with freedom of expression and law enforcement and national security. For him, it is the duty of the law to police the citadel of privacy against public intrusion, with three overarching principles, the boundary of privacy “is by no means clear”, the difficulty of distinguishing “friend from foe” and the “weaponry to assault and breach the citadel has become remarkably sophisticated and bewildering” (Neuberger, 2015b).

In the digital environment ubiquitous search engines make it possible to find details that could only be discovered serendipitously in the analogue. If as seems inevitable users want more records preserved, the likelihood of long closures or destruction becomes even greater. Closure of large swathes of data simply because they contain personal information threatens to negate accountability and transparency. One of the challenges of reviewing born public digital records is identifying possibly sensitive content if much of the content is held haphazardly as Sir Alex Allan (2015) suggests. Reviewers working through paper files can easily understand the context in which a document was created, but if the document is simply reviewed independent of any context it will be hard for reviewers to assess the degree of sensitivity unless ways can be found to overcome the problem (Moss and Gollins, 2016). What is required are computational tools that will identify elements that reviewers should examine for possible sensitivities, ranked in order of the severity of the risk of inappropriate disclosure. Research is ongoing at TNA, Northumbria and Glasgow universities, supported in part by Information Technology as a Utility (ITaaU), and in other parts of the world into the construction of such tools that will help resolve possibly the biggest obstacle in the transmission of born digital records in to the archive and so the public domain (ItaaU, 2016; TNA, 2016c).

Once records have been transmitted users have always needed ways to make sense of them through a variety of mechanisms, indexes, registers, calendars, catalogues and so on. Born digital records need different mechanisms, particularly if they lack structure. Search engines will help, but they will not be sufficient, other tools will be needed, for example graphical visualizations to unscramble E-mail threads and to exclude redundancies, natural language processing utilities to help identify the nature of relationships within power structures, statistical tools to analyse traffic and so on (Harvard, 2015; History Lab, 2016; Historical Network Research (HNR), 2016). Many of these are in course of development and raise wider concerns about how much should be kept and in what form to satisfy such novel forms of interrogation. There is the more generic problem of how such data is to be accessed. It comes in different formats with often radically different behaviours, these can be preserved by using emulation services so the user can experience the data in the same way as when it was created. Yale University is experimenting with the use of virtual libraries (Cochrane, 2014). What is critical about all these initiatives is that the objective is to uphold the principal, fundamental to a democratic society, that records that allow the executive to be called to account are transmitted to the archive and can readily be accessed and understood by the citizen.

The technology that is used to create, capture and discover information challenge accepted practice in the analogue world and provide the opportunity for unprecedented intrusion into privacy that must be policed by the law and the judiciary to defend “the individual against the arbitrary power of the state” in the words of Magna Carta (Neuberger, 2015a). The balance between the public and private interest is delicate and is left deliberately fuzzy so that it can shift to reflect changing circumstances, such as heightened terrorist threats or changes in public expectations when the state has been shown to have abused its power. Just as was the case 800 years ago, evidence must be preserved to support the rule of law, to guarantee a “fair trial”, to make it possible for the executive to be called to account and for wrongs to be redressed. As the custodian of the evidence all national archives, as Thurston states unambiguously, occupy a crucial fiduciary position in the constitution protected by the authority of the courts. Citizens must have confidence that records of government, however embarrassing or “disruptive” they may be, will eventually be open to public scrutiny in the archives and that the executive cannot impose arbitrary closures without being challenged in the courts. In many developing countries national archives protected by the “rule of law” is a bulwark against corruption, we do a disservice to the brave men and women who run them if we redefine the archive as a “cultural heritage asset”. As Lord Bingham reminded us “The Rule of Law cannot stop short at national boundaries because the problems we face in the world today—climate change, pollution, financial regulation, crime, immigration, do not stop short at national boundaries … .On acceptance of this lesson, it might be thought depend the peace and prosperity of the world. It is not a lesson which Magna Carta taught, but it is an extension of the principle which Magna Carta so memorably gave to posterity around the world” (Bingham, 2011: 13).

Additional information

How to cite this article: Moss M (2016) 800 years on can Magna Carta still disrupt the executive?. Palgrave Communications. 2:16049 doi: 10.1057/palcomms.2016.49.