Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Joint embeddings with multimodal cues for video-text retrieval

Joint embeddings with multimodal cues for video-text retrieval For multimedia applications, constructing a joint representation that could carry information for multiple modalities could be very conducive for downstream use cases. In this paper, we study how to effectively utilize available multimodal cues from videos in learning joint representations for the cross-modal video-text retrieval task. Existing hand-labeled video-text datasets are often very limited by their size considering the enormous amount of diversity the visual world contains. This makes it extremely difficult to develop a robust video-text retrieval system based on deep neural network models. In this regard, we propose a framework that simultaneously utilizes multimodal visual cues by a “mixture of experts” approach for retrieval. We conduct extensive experiments to verify that our system is able to boost the performance of the retrieval task compared to the state of the art. In addition, we propose a modified pairwise ranking loss function in training the embedding and study the effect of various loss functions. Experiments on two benchmark datasets show that our approach yields significant gain compared to the state of the art. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Multimedia Information Retrieval Springer Journals

Joint embeddings with multimodal cues for video-text retrieval

Download PDF
16 pages

Loading next page...
 
/lp/springer-journals/joint-embeddings-with-multimodal-cues-for-video-text-retrieval-MpbYCOmtHq
Publisher
Springer Journals
Copyright
Copyright © 2019 by Springer-Verlag London Ltd., part of Springer Nature
Subject
Computer Science; Multimedia Information Systems; Information Storage and Retrieval; Information Systems Applications (incl.Internet); Data Mining and Knowledge Discovery; Image Processing and Computer Vision; Database Management
ISSN
2192-6611
eISSN
2192-662X
DOI
10.1007/s13735-018-00166-3
Publisher site
See Article on Publisher Site

Abstract

For multimedia applications, constructing a joint representation that could carry information for multiple modalities could be very conducive for downstream use cases. In this paper, we study how to effectively utilize available multimodal cues from videos in learning joint representations for the cross-modal video-text retrieval task. Existing hand-labeled video-text datasets are often very limited by their size considering the enormous amount of diversity the visual world contains. This makes it extremely difficult to develop a robust video-text retrieval system based on deep neural network models. In this regard, we propose a framework that simultaneously utilizes multimodal visual cues by a “mixture of experts” approach for retrieval. We conduct extensive experiments to verify that our system is able to boost the performance of the retrieval task compared to the state of the art. In addition, we propose a modified pairwise ranking loss function in training the embedding and study the effect of various loss functions. Experiments on two benchmark datasets show that our approach yields significant gain compared to the state of the art.

Journal

International Journal of Multimedia Information Retrieval – Springer Journals

Published: Jan 12, 2019

References

  • An ensemble classification-based approach applied to retinal blood vessel segmentation
    Fraz, MM; Remagnino, P; Hoppe, A; Uyyanonvara, B; Rudnicka, AR; Owen, CG; Barman, SA
  • A multi-view embedding space for modeling internet images, tags, and their semantics
    Gong, Y; Ke, Q; Isard, M; Lazebnik, S
  • Canonical correlation analysis: an overview with application to learning methods
    Hardoon, DR; Szedmak, S; Shawe-Taylor, J
  • Framing image description as a ranking task: data, models and evaluation metrics
    Hodosh, M; Young, P; Hockenmaier, J
  • Ensemble based systems in decision making
    Polikar, R
  • Bootstrap inspired techniques in computational intelligence: ensemble of classifiers, incremental learning, data fusion and missing features
    Polikar, R
  • A crossmodal approach to multimodal fusion in video hyperlinking
    Vukotić, V; Raymond, C; Gravier, G
  • Places: a 10 million image database for scene recognition
    Zhou, B; Lapedriza, A; Khosla, A; Oliva, A; Torralba, A