RSS feed Add to your Facebook page Watch us on Youtube

Activity title

Automated Scene Understanding for Battlefield Awareness

Activity Reference



Sensors & Electronics Technology

Security Classification




Activity type


Start date


End date



Algorithms, Autonomy, Data Fusion, EOIR, Image Processing, Information Sensing


The state-of-the-art in digital video information extraction and exploitation has advanced rapidly over the past decade. It has been largely driven by the work of commercial industry as vendors seek to understand the vast amounts of video and image data that is traversing the internet. For these vendors, understanding the content and context of this data is the key to delivering targeted advertising and new services, such as self-driving cars. The key to achieving that understanding has been the simultaneous growth of deep learning techniques and the large sets of annotated data widely available thanks to social media. While NATO partners have collaborated on certain specific portions of the problem of information extraction, such as human detection and activity recognition, the general task of converting sensor data into actionable information remains unsolved. The successes of industry have not yet been leveraged to address the military problem because relevant military data does not exist in sufficient quantity and is typically not available or of interest to industry leaders. In order to transition these algorithms to military systems, robust data sets will need to be developed with multi-modal sensors, and scenarios and targets that are relevant to the military. Furthermore, commercially available algorithms should be studied and joint algorithm evaluations should be performed to better understand which approaches best address NATO partner requirements and which areas are most in need of additional research. Successful transition of these algorithms would allow NATO members to develop sensor systems that include processing to automatically condense sensor data into timely, actionable information for user consumption with minimal user interaction. This would drastically reduce user fatigue while increasing effectiveness and allowing users to take full advantage of the ongoing proliferation of sensors on the battlefield. Data collection, standardization and annotation efforts would both benefit from joint NATO cooperation, as would assessment and evaluation of the vibrant commercial industry in video content exploitation.


The initial objectives are to mutually develop a strategic plan for to investigate joint activities to develop standard, annotated data sets for training and evaluation of scene understanding algorithms as well as standard evaluation metrics to assess algorithm performance. Additional objectives include: developing a plan for researching established algorithms from industry for potential use in military scenarios; and assessing the potential for joint algorithm evaluations to compare results between nations and address gaps in the technology.


The effort will cover selection of standard data sets, formats, evaluation procedures and metrics to provide high-quality sensor data for algorithm development and performance assessment related to militarily relevant scenarios. The effort will cover using modelling and simulations to augment collected data sets to further improve algorithm training. The effort will cover conversion of open source and commercially available algorithms for training on data relevant to the military. Collected and simulated data will be discussed with a goal of sharing in subsequent tasks to participating nations for algorithm development and evaluation purposes.

Contact Panel Office