Extracting semantic knowledge from web context for multimedia IR: a taxonomy, survey and challenges

Extracting semantic knowledge from web context for multimedia IR: a taxonomy, survey and challenges Since its invention, the Web has evolved into the largest multimedia repository that has ever existed. This evolution is a direct result of the explosion of user-generated content, explained by the wide adoption of social network platforms. The vast amount of multimedia content requires effective management and retrieval techniques. Nevertheless, Web multimedia retrieval is a complex task because users commonly express their information needs in semantic terms, but expect multimedia content in return. This dissociation between semantics and content of multimedia is known as the semantic gap. To solve this, researchers are looking beyond content-based or text-based approaches, integrating novel data sources. New data sources can consist of any type of data extracted from the context of multimedia documents, defined as the data that is not part of the raw content of a multimedia file. The Web is an extraordinary source of context data, which can be found in explicit or implicit relation to multimedia objects, such as surrounding text, tags, hyperlinks, and even in relevance-feedback. Recent advances in Web multimedia retrieval have shown that context data has great potential to bridge the semantic gap. In this article, we present the first comprehensive survey of context-based approaches for multimedia information retrieval on the Web. We introduce a data-driven taxonomy, which we then use in our literature review of the most emblematic and important approaches that use context-based data. In addition, we identify important challenges and opportunities, which had not been previously addressed in this area. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Multimedia Tools and Applications Springer Journals

Extracting semantic knowledge from web context for multimedia IR: a taxonomy, survey and challenges

Download PDF
37 pages
Read Article

Loading next page...
 
/lp/springer_journal/extracting-semantic-knowledge-from-web-context-for-multimedia-ir-a-UmiVasg8fu
Publisher
Springer Journals
Copyright
Copyright © 2017 by Springer Science+Business Media, LLC
Subject
Computer Science; Multimedia Information Systems; Computer Communication Networks; Data Structures, Cryptology and Information Theory; Special Purpose and Application-Based Systems
ISSN
1380-7501
eISSN
1573-7721
DOI
10.1007/s11042-017-4997-y
Publisher site
See Article on Publisher Site

Abstract

Since its invention, the Web has evolved into the largest multimedia repository that has ever existed. This evolution is a direct result of the explosion of user-generated content, explained by the wide adoption of social network platforms. The vast amount of multimedia content requires effective management and retrieval techniques. Nevertheless, Web multimedia retrieval is a complex task because users commonly express their information needs in semantic terms, but expect multimedia content in return. This dissociation between semantics and content of multimedia is known as the semantic gap. To solve this, researchers are looking beyond content-based or text-based approaches, integrating novel data sources. New data sources can consist of any type of data extracted from the context of multimedia documents, defined as the data that is not part of the raw content of a multimedia file. The Web is an extraordinary source of context data, which can be found in explicit or implicit relation to multimedia objects, such as surrounding text, tags, hyperlinks, and even in relevance-feedback. Recent advances in Web multimedia retrieval have shown that context data has great potential to bridge the semantic gap. In this article, we present the first comprehensive survey of context-based approaches for multimedia information retrieval on the Web. We introduce a data-driven taxonomy, which we then use in our literature review of the most emblematic and important approaches that use context-based data. In addition, we identify important challenges and opportunities, which had not been previously addressed in this area.

Journal

Multimedia Tools and Applications – Springer Journals

Published: Jul 25, 2017

References

  • Modeling annotated data
    Blei, DM; Jordan, MI
  • Composite retrieval of heterogeneous web search
    Bota, H; Zhou, K; Jose, JM; Lalmas, M
  • Reprint of: the anatomy of a large-scale hypertextual web search engine
    Brin, S; Page, L
  • Proceedings of the IEEE workshop on content-based access of image and video libraries, CBAIVL ’98
    Cascia, ML; Sethi, S; Sclaroff, S
  • Web mining for web image retrieval
    Chen, Z; Wenyin, L; Zhang, F; Li, M; Zhang, H
  • Proceedings of the 18th international conference on multimedia, MM ’10
    Chen, Y; Yu, N; Luo, B; Chen, X
  • Web media semantic concept retrieval via tag removal and model fusion
    Chen, C; Zhu, Q; Lin, L; Shyu, ML
  • Proceedings of the 3rd ACM multimedia workshop on geotagging and its applications in multimedia, geoMM ’14
    Choi, J; Thomee, B; Friedland, G; Cao, L; Ni, K; Borth, D; Elizalde, B; Gottlieb, L; Carrano, C; Pearce, R; Poland, D
  • Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, SIGIR ’07
    Craswell, N; Szummer, M
  • Image retrieval: ideas, influences, and trends of the new age
    Datta, R; Joshi, D; Li, J; Wang, J
  • Information extraction from multimedia web documents: an open-source platform and testbed
    Dupplaw, DP; Matthews, M; Johansson, R; Boato, G; Costanzo, A; Fontani, M; Minack, E; Demidova, E; Blanco, R; Griffiths, T; Lewis, P; Hare, J; Moschitti, A
  • Query processing in spatial-query-by-sketch
    Egenhofer, MJ
  • A survey of audio-based music classification and annotation
    Fu, Z; Lu, G; Ting, KM; Zhang, D
  • Visual-textual joint relevance learning for tag-based social image search
    Gao, Y; Wang, M; Zha, ZJ; Shen, J; Li, X; Wu, X
  • Proceedings of the 3rd international conference on multimedia, MULTIMEDIA ’95
    Ghias, A; Logan, J; Chamberlin, D; Smith, BC
  • CLEF (Online working notes/labs/workshop)
    Gilbert, A; Piras, L; Wang, J; Yan, F; Dellandrea, E; Gaizauskas, R; Villegas, M; Mikolajczyk, K
  • Proceedings of the 20th ACM international conference on multimedia, MM ’12
    Hanjalic, A; Kofler, C; Larson, M
  • Proceedings of the 35th international conference on research and development in information retrieval, SIGIR ’12
    Hauff, C; Houben, GJ
  • Proceedings of the 23rd international conference on World Wide Web, WWW ’14
    He, X; Kan, MY; Xie, P; Chen, X
  • A survey on visual content-based video indexing and retrieval
    Hu, W; Xie, N; Li, L; Zeng, X; Maybank, S
  • Mediaeval 2014 workshop
    Ionescu, B; Popescu, A; Lupu, M; Gınsca, AL; Müller, H
  • Result diversification in social image retrieval: a benchmarking framework
    Ionescu, B; Popescu, A; Radu, AL; Müller, H
  • Proceedings of the 20th international conference on World Wide Web, WWW ’11
    Jain, V; Varma, M
  • Proceedings of the 22nd ACM international conference on multimedia, MM ’14
    Jia, Y; Shelhamer, E; Donahue, J; Karayev, S; Long, J; Girshick, R; Guadarrama, S; Darrell, T
  • Proceedings of the 5th ACM on international conference on multimedia retrieval, ICMR ’15
    Jiang, L; Yu, SI; Meng, D; Mitamura, T; Hauptmann, AG
  • Proceedings of the 21st international conference on information and knowledge management, CIKM ’12
    Kamath, KY; Caverlee, J
  • Proceedings of the 7th ACM conference on recommender systems, recsys ’13
    Kaminskas, M; Ricci, F; Schedl, M
  • Proceedings of the 20th international conference on knowledge discovery and data mining, KDD ’14
    Kannan, A; Baker, S; Ramnath, K; Fiss, J; Lin, D; Vanderwende, L; Ansary, R; Kapoor, A; Ke, Q; Uyttendaele, M; Wang, XJ; Zhang, L
  • Image retrieval from the world wide web: issues, techniques, and systems
    Kherfi, ML; Ziou, D; Bernardi, A
  • Trust, distrust and lack of confidence of users in online social media-sharing communities
    Kim, YA; Ahmad, MA
  • A survey of music similarity and recommendation from music context data
    Knees, P; Schedl, M
  • User intent in multimedia search: A survey of the state of the art and future challenges
    Kofler, C; Larson, M; Hanjalic, A
  • Intelligent social media indexing and sharing using an adaptive indexing search engine
    Leung, CHC; Chan, AWS; Milani, A; Liu, J; Li, Y
  • Content-based multimedia information retrieval: State of the art and challenges
    Lew, MS; Seve, N; Djeraba, C; Jain, R
  • Harvesting social images for bi-concept search
    Li, X; Snoek, CGM; Worring, M; Smeulders, AWM
  • Socializing the semantic gap: A comparative survey on image tag assignment, refinement, and retrieval
    Li, X; Uricchio, T; Ballan, L; Bertini, M; Snoek, CGM; Bimbo, AD
  • Proceedings of the 3rd ACM conference on international conference on multimedia retrieval, ICMR ’13
    Liu, X; Hue, B
  • Proceedings of the 17th international conference on knowledge discovery and data mining, KDD ’11
    Low, Y; Agarwal, D; Smola, AJ
  • Mowl: An ontology representation language for web-based multimedia applications
    Mallik, A; Ghosh, H; Chaudhury, S; Harit, G
  • Multimedia search reranking: A literature survey
    Mei, T; Rui, Y; Li, S; Tian, Q
  • Topic modelling of clickthrough data in image search
    Morrison, D; Tsikrika, T; Hollink, V; Vries, AP; Bruno, É; Marchand-Maillet, S
  • Social multimedia: highlighting opportunities for search and mining of multimedia data in social media applications
    Naaman, M
  • Proceedings of the 20th ACM international conference on multimedia, MM ’12
    Nie, L; Yan, S; Wang, M; Hong, R; Chua, TS
  • Proceedings of the 21st international conference on World Wide Web, WWW ’12
    Perelman, D; Bortnikov, E; Lempel, R; Sandler, R
  • Mediaeval 2014 workshop
    Petkos, G; Papadopoulos, S; Mezaris, V; Kompatsiaris, Y
  • Proceedings 19th ACM international conference on information and knowledge management (CIKM’10)
    Poblete, B; Bustos, B; Mendoza, M; Barrios, JM
  • Proceedings of the 1st ACM international conference on multimedia retrieval, ICMR ’11
    Popescu, A; Grefenstette, G
  • Proceedings of the 2015 workshop on community-organized multimodal mining: Opportunities for novel solutions, MMCommons ’15
    Popescu, A; Spyromitros-Xioufis, E; Papadopoulos, S; Le Borgne, H; Kompatsiaris, I
  • Proceedings of the 4th multimedia systems conference, MMSys ’13
    Schedl, M; Orio, N; Liem, CCS; Peeters, G
  • Proceedings of the 4th ACM multimedia systems conference, MMSys ’13
    Schmiedeke, S; Xu, P; Ferrané, I; Eskevich, M; Kofler, C; Larson, MA; Estève, Y; Lamel, L; Jones, GJF; Sikora, T
  • Proceedings of the 8th international conference on multimedia, MM ’00
    Shen, HT; Ooi, BC; Tan, KL
  • Content-based image retrieval at the end of the early years
    Smeulders, AWM; Worring, M; Santini, S; Gupta, A; Jain, R
  • Proceedings of the 1st international conference on multimedia retrieval, ICMR ’11
    Tan, HK; Ngo, CW
  • Proceedings of the 19th international conference on multimedia, MM ’11
    Tan, S; Ngo, CW; Tan, HK; Pang, L
  • Proceedings of the 6th international conference on music information retrieval, ISMIR 2005
    Typke, R; Wiering, F; Veltkamp, RC
  • CLEF (Online working notes/labs/workshop)
    Villegas, M; Paredes, R
  • Interactive image search by color map
    Wang, J; Hua, XS
  • Proceedings of the 36th international conference on research and development in information retrieval, SIGIR ’13
    Wang, D; Hoi, S; Wu, P; Zhu, J; He, Y; Miao, C
  • Content-based multimedia information access, RIAO ’00
    Westerveld, T
  • Proceedings 1st ACM workshop on large-scale multimedia retrieval and mining, LS-MMRM ’09
    Wu, L; Hoi, S; Yu, N
  • Proceedings of the 20th international conference on information and knowledge management, CIKM ’11
    Xu, S; Jiang, H; Lau, FCM
  • Special interest tracks and posters of the 14th international conference on World Wide Web, WWW ’05
    Yang, CC; Chan, KY
  • Learning to rank using user clicks and visual features for image retrieval
    Yu, J; Tao, D; Wang, M; Rui, Y
  • Narrowing the semantic gap—improved text-based web document retrieval using visual features
    Zhao, R; Grosky, WI

You’re reading a free preview. Subscribe to read the entire article.

Try 2 weeks free now

DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

or browse the journals available

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create folders to
organize your research

Export folders, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

Sign up
for free
Start 14 day
Free Trial