|Assessment and Communication of Uncertainty in Intelligence to Support Decision-Making|
|System Analysis and Studies|
C2, decision making, intelligence, Risk assessment, riskuncertainty communication
In a wide variety of defence and security contexts, ranging from intelligence analysis to operational (e.g., C2, METOC) and capability planning to defence acquisitions, risks and uncertainties must be effectively assessed and just as effectively communicated in order to support sound decision-making and action. This is true of national and international (NATO) levels of operation.
Two broad problem areas include (a) the promulgation of multiple, inconsistent standards within and across nations and (b) the use of standards that are either fundamentally flawed in certain respects or that are poorly suited to the specific context in which they are applied. With the first problem, inconsistent standards can actually institutionalize miscommunication and foster confusion by, for instance, assigning different meanings to common terms in a lexicon, such as likely or low risk. At the SAS-ET-CR First Meeting in Paris (July 2014), ET members already reported examples of such inconsistencies in national standards. The second problem is multifaceted and can affect not only the communication of risk and uncertainty but also its assessment. Some examples of fundamental problems already noted by ET members in current standards include the conflation of probability terms with frequency or modal terms and the use of risk assessment methods that provide a very limited range of risk levels that do not differentiate between, e.g., high severity / low probability and low severity / high probability threats. Given that most risk and uncertainty standards establish only ordinal scales of risk and uncertainty, they invite fuzzy interpretations. For instance, a standard might specify that very likely is more probable than likely, but it is unclear what range of numerical probability either term is intended to convey. Moreover, we know from existing behavioural science research that the interpretation of such terms is highly variable across individuals and can also vary within individuals across contexts.
Finally, assessment and communication of risk and uncertainty is never an end in itself. In most instances the information communicated is meant to support effective decision-making. Yet very little is known about how decision-makers interpret and act on such information. Certainly the promulgation of standards is not enough to ensure effective use. Studies are needed to verify that standards for encoding risk and uncertainty levels lead to proper decoding by end users. Likewise, studies are needed to verify the degree to which different assessors relying on the same information and using the same standards reach the same assessments. That is, we know relatively little about the reliability of risk and uncertainty assessments made by human assessors, and we know roughly as little about their accuracy. These are the broad issues that motivate this proposal.
SAS-114/RTG-053 focuses on better understanding how to improve the assessment and communication of risk and uncertainty, as well as information attributes such as source reliability and information credibility. Although specific applications of such knowledge differ to some extent across task group members and the nations or organizations they represent, all members agree that such issues have a wide range of applicability and need to be more systematically and rigorously addressed. The task group’s objectives are motivated by an interest in ensuring that national and international standards effectively promote reliable, valid, accurate, usable and useful assessments of risk and uncertainty, and that such assessments are communicated in the ways that are clear, verifiable, and support the decision-making processes of the end-user. Significant progress can be made because there is already a vast scientific literature on human judgment under conditions of risk and uncertainty and, likewise, there is a well-established literature on the communication of risk and uncertainty.
Phase 1: Team members will collect existing standards or procedures for assessing and communicating risk or uncertainty from defence and security establishments and organizations within their countries. Phase 2: Team members and perhaps other scientists who are networked with our team will conduct original research to address a set of focused questions that will be determined by the participating members. Phase 3: The team will produce a final report that outlines the teams findings and makes evidence-based recommendations for future practice in the areas of assessment and communication of risk and uncertainty.