STONewsArchive: Assessing and modelling the performance of digital night vision image fusion
Title:
Assessing and modelling the performance of digital night vision image fusion
Start_Publishing:
11/05/2021
Panel_Page:
SET
Page_ID:
3815
Main_Body_Multi:
U. Adomeit (Fraunhofer IOSB, GER), D. Bergstrom (FOI, SWE), C. Brook Jackson (4Sight Imaging, GBR), S. Landeau (DGA, FRA), I. Pedlar (dstl, GBR), V. Petrovic (4Sight Imaging, GBR), J. Reynolds (CCDC C5ISR NVESD, USA), S. Roy (DRDC, CAN), G. Swiathy (DGA, FRA).
Motivation
Thermal imagers (TI) and low light imagers (LLI; e.g. Night Vision Goggles) are essential for military operations at night. Both technologies possess complementing advantages and disadvantages. For example, LLI typically is hampered by dark areas (e.g. shadows) where its sensitivity and dynamic range are insufficient, but this has no effect on TI as it is independent of lighting conditions. On the other hand, TI has low spatial resolution and this together with the different scene representation hinders movement, which is no problem with LLI. Therefore, combining LLI and TI is obviously of interest to improve the overall performance of a night vision system. Current state of the art is optical fusion of the two bands, either in form of clip-on devices (e. g. Thermoteknix Systems Ltd ClipIR) or using devices of both types of detectors within one housing (e. g. L3 Insight Enhanced Night Vision Goggle). Such devices are typically advertised using images similar to Figure 1.
Figure 1 : Typical imagery showing the advantage of night-time VIS/LWIR image fusion. Compare the text for details.
The human to the right of the shed near the woods is easy to detect in the long wave infrared (LWIR, Figure 1 middle) but invisible in the visible (VIS, Figure 1 left) imagery. On the other hand, details, which are visible in the VIS, e.g. stars and contrast on the reflectance panels in the foreground, are missing in the LWIR. Fusing the images (Fusion, Figure 1 right) gives both. Expected replacement of image intensifiers by Si focal plane technologies has potential to improve night vision image fusion e.g. by using sophisticated fusion algorithms and image enhancement.
Unfortunately, performance assessment and modelling using such image fusion is an open problem that so far has resisted attempts to arrive at a general procedure. This is in large part caused by the fact that the benefit of a fusion algorithm is strongly dependent on the composition of the actual scenery and the specific military tasks, a complication it shares with other advanced image and signal processing algorithms. The objective of NATO SET-217 was to develop and provide common tools, methods and metrics for assessment of such a system. As image fusion for military applications is a large area, the focus lay on VIS/LWIR fusion as in Figure 1, night vision applications, rural and urban scenarios and a limited, representative number of algorithms. These limitations are in accordance with fused LLI and TI imaging systems currently being fielded by some of the NATO armed forces.
Main results
In its first phase, the RTG surveyed the state of the art in image fusion, available and releasable fusion imagery as well as metrics suitable for assessment of image fusion. In the second, larger phase, the RTG addressed the problem of performance assessment. Because the goal of image fusion is often to join information from two systems that are primarily designed for separate and distinct military tasks, an approach focused on how fused imagery allows both tasks to be performed simultaneously was developed. To cover a larger area the participating nations conducted field trials for different applications. In parallel, works on assessing image fusion in laboratory and by modelling were performed. Whereas these are classic approaches to the assessment problem, two new ones were also looked at using simulations and metrics from machine learning using corner point assessments.
Figure 2 : Imagery and results of a perception experiment for identification of weapons. The experiment Included the two original bands and 10 different fusion algorithms. Probability of Identification is given averaged over all assessed ranges.
Although SET-217 did not succeed in creating a stable approach for modelling and laboratory measurement of fusion systems, assessing the observer tasks (detection, recognition and identification) by perception experiments gave interesting results. As shown above and in Figure 1, in the currently fielded systems the LWIR is typically intended for detection purposes. Experiments proofed the advantage of this band for the task and that the advantage transfers in the fused imagery when using a suitable algorithm, even if the spatial resolution in the LWIR is distinctly lower than in the VIS. Unexpectedly, the same showed up when assessing identification tasks as shown in Figure 2. Overall, the perception experiments gave the following impression: LWIR is distinctly better than VIS and fusions are in-between these two. Fusion can decrease, but not improve the performance. From the algorithms used here, colour overlay (as used in todays fielded systems) proved to be one of the best and gives comparable performance to LWIR. It is still assumed, that the VIS band is needed for situational awareness and may also be helpful for identification of humans.
Although SET-217 did not succeed in creating a stable approach for modelling and laboratory measurement of fusion systems, assessing the observer tasks (detection, recognition and identification) by perception experiments gave interesting results. As shown above and in Figure 1, in the currently fielded systems the LWIR is typically intended for detection purposes. Experiments proofed the advantage of this band for the task and that the advantage transfers in the fused imagery when using a suitable algorithm, even if the spatial resolution in the LWIR is distinctly lower than in the VIS. Unexpectedly, the same showed up when assessing identification tasks as shown in Figure 2. Overall, the perception experiments gave the following impression: LWIR is distinctly better than VIS and fusions are in-between these two. Fusion can decrease, but not improve the performance. From the algorithms used here, colour overlay (as used in todays fielded systems) proved to be one of the best and gives comparable performance to LWIR. It is still assumed, that the VIS band is needed for situational awareness and may also be helpful for identification of humans.
Conclusion
1. Even by limiting the effort to a small part of the large image fusion topic, the strong scene and task dependency of the algorithms prevented deriving a general valid assessment procedure. Including the results of former groups (e. g. SET-140), it seems questionable if this will be possible at all.
2. The need of the VIS band e. g. for situational awareness is obvious, but impossible to prove using the classical range approaches. Additionally, creating situational awareness is the task for more and more imaging systems (e.g. night vision goggles especially with increased field of view, 360° systems for tanks and ships, force protection). Thus, procedures and quality parameters to assess imaging systems against this task should be developed.
Page_Intro:
Thermal imagers (TI) and low light imagers (LLI; e.g. Night Vision Goggles) are essential for military operations at night. Both technologies possess complementing advantages and disadvantages. For example, LLI typically is hampered by dark areas (e.g. shadows) where its sensitivity and dynamic range are insufficient, but this has no effect on TI as it is independent of lighting conditions.
HomePageImage:
2021-set-217.png
HomePageBodyText:
U. Adomeit (Fraunhofer IOSB, GER), D. Bergstrom (FOI, SWE), C. Brook Jackson (4Sight Imaging, GBR), S. Landeau (DGA, FRA), I. Pedlar (dstl, GBR), V. Petrovic (4Sight Imaging, GBR), J. Reynolds (CCDC C5ISR NVESD, USA), S. Roy (DRDC, CAN), G. Swiathy (DGA, FRA).
Motivation
Thermal imagers (TI) and low light imagers (LLI; e.g. Night Vision Goggles) are essential for military operations at night. Both technologies possess complementing advantages and disadvantages. For example, LLI typically is hampered by dark areas (e.g. shadows) where its sensitivity and dynamic range are insufficient, but this has no effect on TI as it is independent of lighting conditions. On the other hand, TI has low spatial resolution and this together with the different scene representation hinders movement, which is no problem with LLI. Therefore, combining LLI and TI is obviously of interest to improve the overall performance of a night vision system. Current state of the art is optical fusion of the two bands, either in form of clip-on devices (e. g. Thermoteknix Systems Ltd ClipIR) or using devices of both types of detectors within one housing (e. g. L3 Insight Enhanced Night Vision Goggle). Such devices are typically advertised using images similar to Figure 1.
Figure 1 : Typical imagery showing the advantage of night-time VIS/LWIR image fusion. Compare the text for details.
The human to the right of the shed near the woods is easy to detect in the long wave infrared (LWIR, Figure 1 middle) but invisible in the visible (VIS, Figure 1 left) imagery. On the other hand, details, which are visible in the VIS, e.g. stars and contrast on the reflectance panels in the foreground, are missing in the LWIR. Fusing the images (Fusion, Figure 1 right) gives both. Expected replacement of image intensifiers by Si focal plane technologies has potential to improve night vision image fusion e.g. by using sophisticated fusion algorithms and image enhancement.
Unfortunately, performance assessment and modelling using such image fusion is an open problem that so far has resisted attempts to arrive at a general procedure. This is in large part caused by the fact that the benefit of a fusion algorithm is strongly dependent on the composition of the actual scenery and the specific military tasks, a complication it shares with other advanced image and signal processing algorithms. The objective of NATO SET-217 was to develop and provide common tools, methods and metrics for assessment of such a system. As image fusion for military applications is a large area, the focus lay on VIS/LWIR fusion as in Figure 1, night vision applications, rural and urban scenarios and a limited, representative number of algorithms. These limitations are in accordance with fused LLI and TI imaging systems currently being fielded by some of the NATO armed forces.
Main results
In its first phase, the RTG surveyed the state of the art in image fusion, available and releasable fusion imagery as well as metrics suitable for assessment of image fusion. In the second, larger phase, the RTG addressed the problem of performance assessment. Because the goal of image fusion is often to join information from two systems that are primarily designed for separate and distinct military tasks, an approach focused on how fused imagery allows both tasks to be performed simultaneously was developed. To cover a larger area the participating nations conducted field trials for different applications. In parallel, works on assessing image fusion in laboratory and by modelling were performed. Whereas these are classic approaches to the assessment problem, two new ones were also looked at using simulations and metrics from machine learning using corner point assessments.
Figure 2 : Imagery and results of a perception experiment for identification of weapons. The experiment Included the two original bands and 10 different fusion algorithms. Probability of Identification is given averaged over all assessed ranges.
Although SET-217 did not succeed in creating a stable approach for modelling and laboratory measurement of fusion systems, assessing the observer tasks (detection, recognition and identification) by perception experiments gave interesting results. As shown above and in Figure 1, in the currently fielded systems the LWIR is typically intended for detection purposes. Experiments proofed the advantage of this band for the task and that the advantage transfers in the fused imagery when using a suitable algorithm, even if the spatial resolution in the LWIR is distinctly lower than in the VIS. Unexpectedly, the same showed up when assessing identification tasks as shown in Figure 2. Overall, the perception experiments gave the following impression: LWIR is distinctly better than VIS and fusions are in-between these two. Fusion can decrease, but not improve the performance. From the algorithms used here, colour overlay (as used in todays fielded systems) proved to be one of the best and gives comparable performance to LWIR. It is still assumed, that the VIS band is needed for situational awareness and may also be helpful for identification of humans.
Although SET-217 did not succeed in creating a stable approach for modelling and laboratory measurement of fusion systems, assessing the observer tasks (detection, recognition and identification) by perception experiments gave interesting results. As shown above and in Figure 1, in the currently fielded systems the LWIR is typically intended for detection purposes. Experiments proofed the advantage of this band for the task and that the advantage transfers in the fused imagery when using a suitable algorithm, even if the spatial resolution in the LWIR is distinctly lower than in the VIS. Unexpectedly, the same showed up when assessing identification tasks as shown in Figure 2. Overall, the perception experiments gave the following impression: LWIR is distinctly better than VIS and fusions are in-between these two. Fusion can decrease, but not improve the performance. From the algorithms used here, colour overlay (as used in todays fielded systems) proved to be one of the best and gives comparable performance to LWIR. It is still assumed, that the VIS band is needed for situational awareness and may also be helpful for identification of humans.
Conclusion
1. Even by limiting the effort to a small part of the large image fusion topic, the strong scene and task dependency of the algorithms prevented deriving a general valid assessment procedure. Including the results of former groups (e. g. SET-140), it seems questionable if this will be possible at all.
2. The need of the VIS band e. g. for situational awareness is obvious, but impossible to prove using the classical range approaches. Additionally, creating situational awareness is the task for more and more imaging systems (e.g. night vision goggles especially with increased field of view, 360° systems for tanks and ships, force protection). Thus, procedures and quality parameters to assess imaging systems against this task should be developed.
Created at 11/05/2021 16:47 by ad.rodes
Last modified at 11/05/2021 16:51 by ad.rodes
Go back to list
Home(NATO STO)