Recent advancements in audio generation have been spurred by the evolution of large-scale deep learning models and expansive datasets.
However, the task of video-to-audio (V2A) generation continues to be a challenge, principally because of the intricate relationship
between the high-dimensional visual and auditory data, and the challenges associated with temporal synchronization. In this study,
we introduce FoleyGen, an open-domain V2A generation system built on a language modeling paradigm. FoleyGen leverages an
off-the-shelf neural audio codec for bidirectional conversion between waveforms and discrete tokens. The generation of audio tokens
is facilitated by a single Transformer model, which is conditioned on visual features extracted from a visual encoder. A prevalent
problem in V2A generation is the misalignment of generated audio with the visible actions in the video. To address this, we explore
three novel visual attention mechanisms. We further undertake an exhaustive evaluation of multiple visual encoders, each pretrained
on either single-modal or multi-modal tasks. The experimental results on VGGSound dataset show that our proposed FoleyGen outperforms
previous systems across all objective metrics and human evaluations.
