Multi-Speaker Noise Reduction through Audio-Visual Fusion and Disentanglement

Recognizing speech from multiple simultaneous speakers is critical for real-world speech applications but suffers from overlapping acoustic noise. This paper proposes a multi-view audio-visual deep learning architecture to reduce noise and improve speech recognition accuracy in multi-talker settings using visual speech cues. A visual processing front-end disentangles blended mouth movements into individual speaker representations using adversarial training. These representations are fused with beamformed audio encodings through temporally synchronized co-attention. Experiments demonstrate significant noise reduction and increased recognition accuracy compared to audio-only methods. The model generalizes well to varying speaker counts and unseen combinations. This audio-visually fused framework enables deploying robust multi-speaker speech recognition without requiring per-speaker training.

کلیدواژه ها:

Multi-Speaker ، Noise ، Audio-Visual Fusion ، Disentanglement