RSS feed LinkedIn Watch us on Youtube

Activity title

Content Based Real-time Analytics of Multi-media Streams

Activity Reference

IST-158 (AI2S)


Information Systems Technology

Security Classification



Awaiting Publication

Activity type


Start date


End date



content-based information, Data analytics, deep learning, imagery, machine learning, predictive analytics, text, video and imagery analysis, visual analytics


This technical activity proposal for a specialist meeting is an integral part of the NATO Information Systems and Technology (IST) panel Research Task Group (RTG), IST-RTG-144 on “Content-Based Multi-media Analytics (CBMA)”. Part of the working plan of this RTG is the organization of a specialist meeting with the RTG technical team and world leading practitioners and experts in this field plus NATO Nations military stakeholder representatives. This gathering of expertise will be able to provide constructive challenge and critique to the foundational work in CBMA of the RTG and its forward plans. In particular strengthen links with other research that could be leveraged and exploited, influence future direction and transition opportunities for exploitation as well as improve the technical quality and utility of the RTG outputs.


to debate and explore the problem spaces and solution spaces and utility of approaches in the following themes: 1. Capture and indexing of motion imagery: further investigate intelligent capturing and initial processing by sensor systems, to include initial video indexing and key frame information produced in audio and metadata entries. 2. Exploit imagery indexing through hierarchical methods using semantic identifiers and human evaluations of exploitation results. 3. Explore motion-based index generation to generate rapid and robust retrieval of context. Types of motion include background motion of static structures related with sensor flight, background motion generated by normal patterns such as traffic flow, an explosion and after effects at a location, etc. 4. Expand the Deep Learning approach for semantic video analytics through a semantic hierarchy of full motion video. Long term impact is the provision of optimal semantic information to users in rapid fashion while adapting to dynamically varying computational resources. 5. Explore the mechanisms by which text analysis results can be used to drive/exploit video and imagery indexing and retrieval. 6. Explore frameworks for optimizing multi-media analytics via systems engineering and architectural design concepts.


1. Capture and indexing of motion imagery 2. Information flow and analysis a. Sensors: data, metadata, context. b. Signal/data conditioning and processing: c. Object assessment 3. Situation and Impact assessment: Situation Inference and understanding, Scenario interpretation, Predictive Analytics, Advice on measures: 4. Hierarchical methods using semantic identifiers and human evaluations of exploitation results 5. Motion-based index generation to generate rapid and robust retrieval of context 6. AI/Machine Learning/Deep Learning: Higher performance automated processing of multi-media streams 7. Systems Engineering and Architectures: Open, modular, scalable and reconfigurable frameworks for optimizing processing and multi-media analytics

Contact Panel Office