|
| |
| Activity title | Human Systems Integration for Meaningful Human Control over AI-based systems | Activity Reference | HFM-330 | Panel | HFM | Security Classification | NATO UNCLASSIFIED | Status | Active | Activity type | RTG | Start date | 2020-02-11T00:00:00Z | End date | 2024-02-10T00:00:00Z | Keywords | Artificial Intelligence, Autonomy, Human Systems Integration, HumanMachine Teaming, Meaningful Human Control | Background | This activity addresses an important issue identified in the Specialists Meeting SCI-296 on Autonomy from a System Perspective, held in May 2017 as part of the STO theme devoted to that topic. As noted in the SCI-296 TER, “in many or most cases, it is foreseen that ‘meaningful human control’ (MHC) will be mandated, necessitating the human to maintain awareness and ‘drill down’ on demand”. Responding to this need, the HFM Panel commissioned an exploratory team (HFM-178) to rapidly assess the area from a human-centric perspective. This team came to a consensus as to a working description of MHC, which is “Humans have the ability to make informed choices in sufficient time to influence AI-based systems in order to enable a desired effect or to prevent an undesired immediate or future effect on the environment”. This team also canvased MHC from several dimensions and settled on the need for a dedicated expert-heavy workshop (HFM-322) to formally unpack the most pressing influencing factors. The current proposed activity would serve to integrate several key research challenges emerging from HFM-178 and HFM-322, especially those combining humans, (technical) systems, organisation and behaviour.
The activity also builds upon work on human-autonomy teaming conducted in HFM-247 on “Human-Autonomy Teaming: Supporting Dynamically Adjustable Collaboration”. In this RTG, experts from 7 countries tracked technology activities, explored novel approaches such as human-autonomy “design patterns”, developed metrics for human-machine teaming, and prioritized key challenges for future research.
Since meaningful human control and appropriate levels of human judgement is deemed to be important for many kinds of automated and (semi)autonomous systems, the term “AI-based systems” is used to encompass all AI-based forms of automation and autonomy, for tasks that are either physical (e.g. robotics, autonomous sensors, Mine Countermeasures) or informational (e.g. big data analytics, logistics, decision support). Given the implications of MHC for the latter application domain, this TAP is also relevant for the STO theme “Big data and AI for military decision making”.
| Objectives | Our core objective is not to duplicate the ongoing efforts at the national and international level in the legalities and ethics of MHC. Rather, it is to learn from these ongoing discussions, apply a perspective to the problem squarely rooted in human factors and cognitive science understanding, and thus distil a set of practical human-centred guidelines to inform future NATO actions in this increasingly important area. We have made some initial attempts to distinguish between (at least) two (interrelated) types of human control: Meaningful Human Control, which aims at improving morally and legally acceptable behaviour and proper accountability, and Effective Human Control, which aims at improving performance and reducing risks. Though this attempt at clarification requires more discussion and debate, it serves as a useful starting point for this proposed RTG to explore further.
The scientific objectives with respect to these fields are:
1. Develop HSI guidelines for Human Control
- Investigate current guidelines from defense and other domains (e.g. bowtie model, iso-26262 in automotive) and assess them for applicability for HC (both MHC and EHC).
- Analyze current guidelines for various disciplines (e.g. legal, ethical, HF, engineering), and investigate how they can be integrated.
- Develop Team Design Patterns for HC specifying standardized ways of interaction.
- What does HC imply for various stakeholders (Policy makers, System acquirers, System designers, organizational users, end users, R&D community)
- Develop HSI principles and guidelines for HC to inform future NATO operations and acquisitions.
- Determine how HC can be embedded in an accreditation process to ensure that human accountability/control is maintained through information management, fusion, decision making and action processes
2. Measurement and assessment of Human Control
- Being largely situation-dependent, determine how examples and scenarios be used in the assessment of effective HC
- Investigate how to qualitatively and quantitatively assess MHC? Can critical thresholds be determined?
- Investigate what are individual measures of various aspects of human control and how can these be integrated?
- Explore how we can facilitate post-hoc analysis? E.g. can we use black box (as in airplane), traceability, provenance graphs, accountability chains, etc.?
- Understand how teaming performance metrics/guidelines apply to and inform MHC?
In case the complexity of a subject exceeds what can be studied within the scope of this RTG, this RTG might choose to spawn a new activity and later integrate its outcomes into a coherent whole. This is the third function of this RTG
3. Integration and development of other NATO activities
- Develop a model containing the main concepts such as trust, training, accountability, situation understanding.
- Develop or reuse reference use cases/ scenarios for various aspects of human control.
- As deemed necessary, develop new activities and act as central executive (i.e., the hub) to these interrelated activities (“the spokes”).
- Discover related groups (e.g. on V&V, AI/Big Data, C2, (operational) ethics), in order to share ideas and influence alignment where mutually beneficial. Act as critical review of guidelines in specialist meetings, e.g. act as red team to highlight human control issues, questionnaires.
| Topics | Topics and research questions to be covered include:
- Definition and scope of MHC; the range of applications to be considered. This is not just LAWS, but MHC emerges on the collective level. For example, an AI-based picture compilation system could have serious implications for MHC. AI-based logistics will likely be different.
- Analysis of the state of the art: which MHC methods are currently used in AI-based systems (fielded systems, prototypes, and concept demonstrations)? Which guidelines and methods can be imported from human factors, cognitive science, systems engineering and other fields.
- Identification and assessment of new & emerging interface and process methods for ensuring MHC.
- System specification methods for ensuring HC. What types of requirements can be included to ensure HC. How can (team) design patterns be applied to ensure HC?
- System validation methods for HC. How can HC be anticipated, assessed, and measured? What certification methods can be based upon these methods? Explore the potential of teaming performance metrics to inform validation.
- Training human-machine teams for meaningful HC. How can human-machine teams be accredited to work in accordance with MHC guidelines?
- Collective validation methods for HC. HC is a combination of human competence, an assured system, and the processes of interaction between the two. How can these be assessed collectively?
- How should the legal, political, and public perception aspects of MHC over AI-based systems be factored into the verification and validation of these systems?
| | Contact Panel Office |
| |
|
|
|
