Intelligence Oversight – Upcoming Challenges and Proposals

Sed quis custodiet ipso custodes? (Who will guard the guards?) – Juvenal (Satire VI, lines 347–348)

The Challenge

Under pressure from civil society seeking more democratic transparency of surveillance operations in the aftermath of the Snowden revelations, lawmakers and governments in most NATO- and other democratic countries will, in the near future, amend their national legal provisions to improve the democratic oversight, control and supervision of data collection, monitoring and surveillance activities by their law enforcement, secret intelligence and security services. However, national oversight authorities seem to increasingly face multiple organizational and technical problems in implementing these regulations.


On the other side, law enforcement, intelligence and security agencies find themselves confronted with even more operational, technical and bureaucratic challenges to meet the required compliance regulations imposed by their regulatory legal bodies. Meanwhile, facing new challenges in the terrorist, criminal and military fields, some practitioners and service chiefs are said to have already issued some warnings to their government and legal authorities that some of the planned legal restrictions may sensibly reduce their OSINT/SIGINT collection efficiency, preventing them from fulfilling vital missions.


In addition, the growing technological complexity and ubiquitous use of IT-based surveillance tools and technology by state and non-state intelligence and security actors operating in the wide realm of surveillance will continuously lead to even bigger volumes of data to be controlled, and, by the same token, augment the possibilities for data abuse and the circumvention of legal requirements.


Considering that all data is information, but not all information is intelligence, governments, parliaments, oversight authorities, secret intelligence and security agencies will have to find viable political, organizational, practical and technical solutions 

  • (1) to satisfy the need for more transparency by establishing a workable democratic oversight and effective control system governing  surveillance activities, and, at the same time,
  • (2) to ensure the ability of their law enforcement, intelligence and security agencies to carry out their core mission even more efficiently. Customer-oriented, legal, and compliant intelligence and security operations in conformity with national and international legal frameworks will be the result of a smart and effective oversight activity.


Lawmakers, governments, oversight bodies, intelligence and security agencies will have to make considerable efforts to find legal and organizational solutions to balance the citizen’s legitimate right to privacy and the justified state’s need for secrecy and security. At first glance, meeting that challenge seems to be contradictory and something like squaring the circle. 

The complex structure of intelligence oversight.
The complex structure of intelligence oversight.

Intelligence Governance

To establish national authorities responsible for the control and oversight of surveillance activities by law enforcement, secret intelligence and security agencies, governments in various states have adopted different approaches. The fundamental statutory question is whether

  • (1) to delegate democratic control to an oversight body external to the executive branch of government, like special commissions placed under the authority of parliament or a court (external oversight), 
  • (2) to leave it within the executive branch (internal oversight), such as the establishment of an Inspector General or special commissions, and oversight boards within government or the services, but with the obligation to report to parliament, or
  • (3) to choose a hybrid solution (external plus internal oversight bodies) with the legal prerogative on the side of parliament.


Generally, in democratic or “hybrid” (autocratic democracies) regimes, secret intelligence and security agencies are held accountable for their activities mostly by their national government (or ruling political party in hybrid regimes) but also by their parliament, particularly when it comes to the appropriation of budgets, and by civil society as a whole. However, accountability can only be claimed if scrutinous oversight procedures are established, providing, if it were, proof and evidence of lawlessness, organizational or operational abuse, mismanagement, budget and expenditure excess, etc. Moreover, weak intelligence oversight will automatically result in, and intensify activities of, independent investigative journalists and whistleblowers who, in the past, uncovered all sorts of deficiencies resulting in breathtaking intelligence scandals. 


Obviously, given the impenetrable “fence of secrecy” surrounding these agencies and their particular deontology and culture of secrecy, effective and informed control and oversight are impossible without the collaboration of those being controlled. That dilemma can only be mitigated by providing oversight bodies – external and internal – with a clear legal statute, competent experts, sufficient budget and powerful technology.


One important precondition for oversight scrutiny and efficiency is the political will to fully implement constitutional law and to elaborate clear-cut rules regulating the way, type and volume of data oversight bodies can access. Ideally, all data collected, stored and disseminated by the agencies should be made available to oversight bodies whenever need be. In the face of mushrooming data volumes, unprecedented complexity of technology, as well as intricate political and operational relations between national and international intelligence actors, only timely, direct and comprehensive access to critical data may allow for an on-time and effective democratic oversight.


While direct data access is less problematic for internal oversight entities, external oversight bodies, however, are still having considerable problems justifying direct access to sensitive raw data being collected, shared, and stored by the agencies. Confidentiality and security concerns on the part of intelligence practitioners and still existing legal gaps are characterizing the actual debate. In addition, in most countries concerned, there is considerable doubt on the part of the executive branch as to the reliability and security of the technical tools in use by oversight bodies, guaranteeing the trusted and secure handling of secret data.


Another general weakness consists in the interdependence between powerful or monopolistic industrial technology providers and intelligence actors and agencies. As a matter of fact, with open or tacit support of the scientific research community, all agencies must and will always try to acquire or trigger the development of the most sophisticated surveillance technology. Therefore, as a logical consequence, oversight bodies must be authorized to extend their review power to include corporate oversight as early as government contracts with corporate technology providers are being conceived, designed and negotiated. That approach would prevent agencies from falling into a dependency trap, making sure the resulting contract is compliant with all pertinent legal privacy regulations and includes provisions for built-in oversight even before it is signed and implemented, thus saving a lot of taxpayers’ money.

The Proposal – Traversals’ Intelligence Oversight Tool Kit

We at Traversals are convinced that there are several possibilities to facilitate and support efficient and innovative solutions to the most pressing oversight challenges by using our Intelligence Oversight Tool Kit. Traversals’ approach in developing and using intelligence oversight tools:

  1. The condition sine qua non is to analyze the legal requirements to find out what kind of limitation effects on intelligence data collection  – technical, operational and functional – do they cause or impose on the processes of data collection by the agencies concerned.
  2. Identifying what kind of tools would best attend to the needs of internal and external oversight bodies to comply with their mission of having to trace, control, sift, assess, and report a growing volume of multi-source data. Although extremely powerful, our tools are not designed and intended to replace the human oversight agent. Yet, the tools can be an intelligent assistant for oversight analysts and managers to cope with the extremely difficult task of ensuring compliance and accountability.
  3. Depending on the respective national legal framework and the type of agency concerned, and whether it is used by external or internal oversight bodies, the tool kit meets all major functional requirements. Due to its modularity, it can be adapted to the majority of compliance and oversight tasks, e.g. including a compliance audit tool, authentication tracker, identifier search, data track and trace modes, suspicious pattern detection and analysis, data deletion monitoring, contextual data analysis and visualization, data retention control, log file monitoring, etc.
  4. We are fully aware of the fact that some of the tools presented can work only within the framework of the pertinent legal, privacy and security provisions of the respective country. Therefore, our tool kit is highly adaptable to the specific situation of each agency or oversight body in every country.


Traversals’ Oversight Tool Kit includes software tools for all activities taking place in the  oversight relationship structure shown above to include the following types of functionalities:

  1. Track and tracing tools
  2. Filters for external and internal oversight
  3. Search tools
  4. Assessment tools
  5. Presentation and reporting tools

Tracing Tool 

The tracing tool will allow for the tracking and tracing of all data access and handling processes. With the agencies having every data set being protocolled, registered and accounted for, oversight bodies could have ad-hoc or, in some cases, even permanent online access to data (log files, registers and protocols). Thus, they are able to track and trace back every critical data set to its originator, author, releaser, destinator or modifier, and include other metadata. If required, content data, named entities, and other linguistic and numerical data can also be used as tracers.


The high number of different data sources, the constantly changing landscape of data sources and the unbelievable amount of data in the data sources to be checked require a solution that can cope with the requirements.


Our Federated Search was designed and conceived with exactly these requirements in mind. It allows the integration of different data sources, such as SQL, NoSQL or web services, transparent to the user. The Federated Search in combination with the Enterprise Knowledge Graph allows the tracing of singular data elements.

Oversight Filters

There are several types of filters required for oversight purposes: Internal and External Oversight Filters.

Internal Oversight Filters

Internal oversight entities verify the data minimization effect of the filters implemented by their agency at the collection level, following pertinent compliance rules. To ensure ad-hoc and random control audits and compliance checks to verify the efficiency of the filters implemented, direct and unhampered electronic access to all stored and disseminated pre- and post-collection data must be possible. Since the internal oversight bodies are an integral part of, and working inside, the organization, security aspects shouldn’t be an obstacle. Subject to verification are e.g. privacy data of persons or groups specially protected by law, such as lawyers, clerics, doctors, etc. Internal oversight entities should report directly to the agency director and coordinate with external oversight bodies.


The identification of personal data within collected metadata and the content plays also an essential role in the context of GDPR compliance. Especially in this area, we have been able to gain a lot of experience with our OSINT Intelligence Platform. These insights can be applied to the Oversight Tool Kit as well. The identification and GDPR-compliant handling of personal data is a key element of the Federated Search and can be used to estimate a compliance score relevant for the oversight agents.

External Oversight Filters

Following pertinent legal provisions, external oversight bodies design and apply their own filters as they verify the data minimization of the agency. They need to have direct access to post-collection data that is stored inside the agency and disseminated to third parties according to pertinent policy regulations. They should be able to compare their results with those delivered by the internal filters, thus verifying the accuracy of the filters used by internal oversight entities.

Oversight Search Tools

Oversight entities must be equipped with powerful search tools. Since their mission is to verify whether the agencies are compliant with the legally prescribed privacy and transparency rules, the search tools must allow for search in all types of files, metadata, databases, data storage facilities, registries, protocols and log files, etc.


All Big Data criteria (Volume, Velocity, Variety, Veracity and Value) are fulfilled when it comes to intelligence oversight. Often the complexity is projected purely on the data storage. The question of how to search in such data is also demanding. In the past 12 months, our solutions relied on the extremely flexible query language GraphQL, which powers our Federated Search. GraphQL allows graph filtering, geo-filtering, fuzzy searches, etc.

Oversight Assessment Tools

The function of assessment tools is to identify and present the deficiencies, violations and other rule-breaking activities incurred by the agencies or internal oversight entities. They include anomaly pattern analysis, deletion monitoring, intrusion and data access abuse, data dissemination violations to external third parties, suspicious data query and search behavior, etc.


Part of our Oversight Tool Kit is an event-triggered processing engine that includes a large set of various processing modules. This list can be extended with the latest state-of-the-art technology coming from open research and development.

Presentation and Reporting Tools

Oversight entities must be equipped with secure communications, and state-of-the-art presentation and reporting software to ensure a swift and transparent dissemination of the results obtained from their audits to their respective authorities and customers, such as parliament, government institutions and the judiciary.


Our Oversight Tool Kit includes a property-based access control that can be adapted to the already existing classification system of intelligence agencies. Users see only what they are allowed and cleared to see. In addition to that, all identified personal data is encrypted with a purpose-specific 256-bit AES key. Access to all data is tracked in an immutable audit log. These facts ensure multi-dimensional secure communication between data storage and the Oversight Tool Kit. All data to be checked is visualized in various analysis views, such as graphs, maps or dashboards.


Intelligence oversight is one of the main pillars of intelligence transparency, a necessary element of “making intelligence information publicly available in a manner that enhances public understanding of intelligence activities while continuing to protect information when disclosure would harm national security.” (ODNI, Principles of Intelligence Transparency – Implementation Plan, 2015).


Due to its vast experience with intelligence software applications, our Intelligence Oversight Tools will provide all types of oversight entities with tailor-made software tools to cope with all major challenges resulting from their difficult task.

Further Reading 

  • Article by Dr. Thorsten Wetzling, a privacy expert at the German foundation Stiftung Neue Verantwortung (SNV), 
  • Doctoral thesis by Dr. Annelie Siemsen, Protection of Personal Data by German Bundeswehr Soldiers working in Foreign Intelligence (Der Schutz personenbezogener Daten bei der Auslandsaufklärung durch Bundeswehrsoldaten), ISBN 978-3-428-85524-7(Print & E-Book), Schriften zum Öffentlichen Recht Bd. 1387;
  • Article by Peter Gill, Intelligence oversight and the challenge of surveillance corporatism, Intelligence and National Security, DOI: 10.1080/02684527.2020.1783875

Copyright © 2024, Traversals Analytics and Intelligence GmbH. All Rights Reserved.