The VLDB Journal (2001) 9: 312–326 / Digital Object Identiﬁer (DOI) 10.1007/s007780100040
Supporting efﬁcient multimedia database exploration
Wen-Syan Li, K. Sel¸cuk Candan, Kyoji Hirata, Yoshinori Hara
C&C Research Laboratories, NEC USA, 110 Rio Robles, M/S SJ100, San Jose, CA 95134, USA;
Edited by S. Christodoulakis. Received: 9 June 1998/ Accepted: 21 July 2000
Published online: 4 May 2001 –
Abstract. Due to the fuzziness of query speciﬁcation and me-
dia matching, multimedia retrieval is conducted by way of ex-
ploration. It is essential to provide feedback so that users can
visualize query reformulation alternatives and database con-
tent distribution. Since media matching is an expensive task,
another issue is how to efﬁciently support exploration so that
the systemis not overloaded byperpetual queryreformulation.
In this paper, we present a uniform framework to represent sta-
tistical information of both semantics and visual metadata for
images in the databases.We propose the concept of query veri-
ﬁcation, which evaluates queries using statistics, and provides
users with feedback, including the strictness and reformula-
tion alternatives of each query condition as well as estimated
numbers of matches. With query veriﬁcation, the system in-
creases the efﬁciency of the multimedia database exploration
for both users and the system. Such statistical information
is also utilized to support progressive query processing and
Key words: Multimedia database – Exploration – Query re-
laxation – Progressive processing – Selectivity statistics – Hu-
man computer interaction
Query processing in multimedia databases is different from
query processing in traditional database systems. Contents
stored in traditional database systems are generally precise
and, as a result, query processing answers are deterministic.
On the other hand, in both document retrieval and image re-
trieval, results are based on similarity calculations. In docu-
ment retrieval, documents are represented as keyword lists. To
retrieve a document, information systems compare keywords
speciﬁed by users with the documents’keyword lists. Images,
Correspondence to: K.S. Candan, Computer Science and Engineer-
ing Department, College of Engineering and Applied Sciences, Ari-
zona State University, Box 875406 Tempe, AZ 85287-5406, USA;
This work was performed when K.S. Candan visited NEC, CCRL.
in a similar manner, are usually represented as media features.
However, image matching is carried out through comparing
these feature vectors.Comparison of these three types of query
processing are illustrated at the top of Fig. 1. We see that mul-
timedia database query processing is truly an integration of
these three types of query processing.
An image consists of three types of information repre-
senting its contents: visual features, structural layout (spatial
relationships), and semantics of image objects, as shown at the
bottom of Fig.1. A multimedia database query requires simi-
larity measures in all these aspects. For example, a query may
be posed as “retrieve images in which there is a woman to the
right of an object and the object is visually similar to the pro-
vided image". This query results in a list of candidate images
ranked by their aggregated scores with degrees of uncertainty
based on all above three aspects. Thus, we view multime-
dia database query processing as a combination of: (1) infor-
mation retrieval notions described in  (exploratory query-
ing, inexact match, query reﬁnement); and (2) ORDBMS or
OODBMS database notions (recognition of speciﬁc concepts,
variety of data types).
In most multimedia applications, supporting partial match
capability – in contrast to supporting only exact match func-
tionalities in relational DBMSs – is essential and desirable.
There are two major reasons:
1. There may not be a reasonable number of images which
match with the user query. Figure 2 (Query) shows the
conceptual representation of a query. Figures 2a–d show
examples of candidate images that may match this query.
The numbers next to the objects in these candidate images
denote the similarity values for the object-level matching.
The candidate image in Fig.2a satisﬁes object matching
conditions but its layout does not match user speciﬁcation.
Figure 2b and d satisfy image layout conditions butobjects
do not perfectly match the speciﬁcation. Figure 2c has
structural and object matching with low scores.
Note that in Fig.2a, the spatial constraint, and in Fig.2c,
the image similarity constraint for the lake completely fail
(i.e., the match is 0.0). Such candidate images in gen-
eral would not be returned by SQL/DBMS-based image
retrieval systems. Query relaxation is needed to include
such types of partially matched images.