TY - JOUR
AU - Tulyakov, Sergey
AB - Abstract:We propose AV-Link, a unified framework for Video-to-Audio (A2V) and Audio-to-Video (A2V) generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal conditioning. The key to our framework is a Fusion Block that facilitates bidirectional information exchange between video and audio diffusion models through temporally-aligned self attention operations. Unlike prior work that uses dedicated models for A2V and V2A tasks and relies on pretrained feature extractors, AV-Link achieves both tasks in a single self-contained framework, directly leveraging features obtained by the complementary modality (i.e. video features to generate audio, or audio features to generate video). Extensive automatic and subjective evaluations demonstrate that our method achieves a substantial improvement in audio-video synchronization, outperforming more expensive baselines such as the MovieGen video-to-audio model.
TI - AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation
JF - Electrical Engineering and Systems Science
DO - 10.48550/arxiv.2412.15191
DA - 2025-03-10
UR - https://www.deepdyve.com/lp/arxiv-cornell-university/av-link-temporally-aligned-diffusion-features-for-cross-modal-audio-Unzj3MegS5
VL - 2025
IS - 2412
DP - DeepDyve
ER -