A new approach to crop RGB and IR videos for image fusion in human action recognition dataset

In recent years, human action recognition has received much attention from researchers. But one of the main challenges is extracting effective features. To implement human action recognition models usually, IR or RGB videos are used however none of them contain complete information and rich details of the scene, although it is possible to extract suitable features by having both data at once. One of the methods that can be used to achieve this goal is fusing two types of video data, which helps us having salient features from IR videos and rich details from RGB videos. Most human action recognition video types are not aligned, which is one of the most important conditions for fusion. In this study, we used the NTU RGB+D dataset, which is one of the largest datasets in human action recognition, and proposed a method to crop identical windows from two types of IR and RGB video data types so that we can fuse videos and therefore extracting robust features and improving data. The performance of the proposed method was evaluated by fusing videos and studying the results using different metrics such as EN, MI, SSIM, and MS-SSIM which shows the effectiveness of the method.

کلیدواژه ها:

Human action recognition ، Image fusion ، Inverse problems