RSS feed Add to your Facebook page Watch us on Youtube

Activity title

Meaningful Human Control of AI-based Systems: Key Characteristics, Influencing Factors and Design Considerations

Activity Reference

HFM-322

Panel

Human Factors and Medicine

Security Classification

PUBLIC RELEASE

Status

Planning

Activity type

RWS

Start date

2019

End date

2022

Keywords

artificial intelligence, C2, human factors, human machine teaming, machine learning, Meaningful Human Control, transparency

Background

This activity addresses an important issue identified in the Specialists Meeting SCI-296 on Autonomy from a System Perspective, held in May 2017 as part of the STO theme devoted to that topic. As noted in the SCI-296 TER, “in many or most cases, it is foreseen that ‘meaningful human control’ (MHC) will be mandated, necessitating the human to maintain awareness and ‘drill down’ on demand”. Responding to this need, the HFM Panel commissioned an exploratory team (HFM-178) to rapidly assess the area from a human-centric perspective. This team came to a consensus as to a working description of MHC, which is “Humans have the ability to make informed choices in sufficient time to influence AI-based systems in order to enable a desired effect or to prevent an undesired immediate or future effect on the environment”. This team also canvased MHC from several dimensions and settled on the need for a dedicated expert-heavy workshop to unpack the most pressing influencing factors. The current proposed activity integrates several research issues emerging from SCI-296, especially those combining humans, (technical) systems, organisation and behaviour. This activity also directly builds upon work on human-autonomy teaming conducted in HFM-247 on “Human-Autonomy Teaming: Supporting Dynamically Adjustable Collaboration”. In this RTG, experts from 7 countries tracked technology activities, explored novel approaches such as “design patterns”, developed metrics, and prioritized key challenges for future research. Other related NATO activities currently underway include IST-173, IST-157, IST-152, SET-263, SAS-143, SAS-ET-DV, ACT PoW - MUAAR. Since meaningful human control is deemed to be important for many kinds of automated and (semi)autonomous systems, the term “AI-based systems” is used to encompass all AI-based forms of automation and autonomy, for tasks that are either physical (e.g., unmanned platforms) or informational (e.g., big data analytics, decision support). Given the implications of MHC for the latter application domain, this TAP is also relevant for the STO theme “Big data and AI for military decision making”.

Objectives

HFM-ET-178 discussed several dimensions associated with MHC, and settled on a subset for increased investigation. These MHC “Themes” have become the organizing focus and objectives of this workshop. These themes are: • Organizational Considerations of MHC (including team training, agile C2 structures) • HF guidelines to achieve and maintain MHC for all NATO AI applications • Systems Engineering methods (including TEV&V for learning systems) to support MHC • Adversary tactics to counter/undermine MHC (methods and mitigations) • MHC for complex socio-technical system of systems (emergent properties, MHC propagation) • Legal, ethical, political, and public perception of MHC over AI-based systems

Topics

MHC spans many scientific topic areas across several disciplines. This workshop will especially focus on: • Human-autonomy interaction • Explainable Artificial Intelligence • Teaming research/ distributed intelligence/team structures and roles • Shared dynamic mental models, maintenance of common ground • Dynamic task allocation • Directability and predictability with AI • Observability/transparency and trust • Organizational influences and processes • Joint human-machine decision making/biases • Joint human-machine learning/training • Accountability and MHC • Ethics and morality • Complexity research

Contact Panel Office