Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web: A Systematic Review

Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide... Abstract Context The quality of consumer health information on the World Wide Web is an important issue for medicine, but to date no systematic and comprehensive synthesis of the methods and evidence has been performed. Objectives To establish a methodological framework on how quality on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, and to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research. Data Sources We searched MEDLINE and PREMEDLINE (1966 through September 2001), Science Citation Index (1997 through September 2001), Social Sciences Citation Index (1997 through September 2001), Arts and Humanities Citation Index (1997 through September 2001), LISA (1969 through July 2001), CINAHL (1982 through July 2001), PsychINFO (1988 through September 2001), EMBASE (1988 through June 2001), and SIGLE (1980 through June 2001). We also conducted hand searches, general Internet searches, and a personal bibliographic database search. Study Selection We included published and unpublished empirical studies in any language in which investigators searched the Web systematically for specific health information, evaluated the quality of Web sites or pages, and reported quantitative results. We screened 7830 citations and retrieved 170 potentially eligible full articles. A total of 79 distinct studies met the inclusion criteria, evaluating 5941 health Web sites and 1329 Web pages, and reporting 408 evaluation results for 86 different quality criteria. Data Extraction Two reviewers independently extracted study characteristics, medical domains, search strategies used, methods and criteria of quality assessment, results (percentage of sites or pages rated as inadequate pertaining to a quality criterion), and quality and rigor of study methods and reporting. Data Synthesis Most frequently used quality criteria used include accuracy, completeness, readability, design, disclosures, and references provided. Fifty-five studies (70%) concluded that quality is a problem on the Web, 17 (22%) remained neutral, and 7 studies (9%) came to a positive conclusion. Positive studies scored significantly lower in search (P = .02) and evaluation (P = .04) methods. Conclusions Due to differences in study methods and rigor, quality criteria, study population, and topic chosen, study results and conclusions on health-related Web sites vary widely. Operational definitions of quality criteria are needed. The Internet has become an important mass medium for consumers seeking health information and health care services online.1 A recent concern and public health issue has been the quality of health information on the World Wide Web. However, the scale of the problem and the "epidemiology" (distribution and determinants) of poor health information on the Web are still unclear, as is their impact on public health and the question of whether poor health information on the Web is a problem at all.2 Many studies have been conducted to describe, critically appraise, and analyze consumer health information on the Web. These typically report proportions of inaccurate or imperfect information as estimates of the prevalence of flawed information or the risk of encountering misinformation on the Web. However, to date no systematic and comprehensive synthesis of the methodology and evidence has been attempted. Two previous systematic reviews focused on compiling quality criteria and rating instruments, but did not synthesize evaluation results. Jadad and Gagliari3 reviewed non–research-based rating systems (eg, criteria used for "awards," "seals of approval," or gateways) published on the Internet but did not include rating systems published in the peer-reviewed literature. Kim et al4 compiled instruments or criteria proposed to evaluate Web sites, but it is unclear to what extent and with what results these criteria can be and have been applied in practice. To our knowledge, this is the first systematic review conducted to compile criteria actually used and to synthesize evaluation results from studies containing quantitative data on structure and process measures of the quality of health information on the Web. The objectives of this study were to establish a methodological framework on how "quality" on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research. Methods Inclusion and Exclusion Criteria Inclusion and exclusion criteria were established in advance in a written protocol. Studies were included when their authors: (1) searched the World Wide Web systematically for health information (eg, to identify all consumer Web sites for a given topic) or clearly defined a set of specific health information services to be included; (2) evaluated information likely to be accessed by consumers; (3) evaluated the quality of health information or services against certain criteria—for example, by judging the authority of source; assessing the accuracy of information, readability, or comprehensiveness; or by comparing Internet information and services against those outside of the Internet; and (4) provided quantitative results, such as proportions of Web sites complying with the quality criteria, or the distribution of quality scores, or information on the readability level(s) in the sample. Studies were excluded if authors simply listed "quality" Web sites without indicating that they had performed a comprehensive, systematic search, if they did not list quality criteria, or if they did not quantitate how many Web sites were assessed in total or how many did not comply with quality criteria. We also excluded studies assessing tools that enable or facilitate access to health information, such as search engines, directories, health portals, rating systems, and review services, or studies comparing different methods of information retrieval. Theoretical papers that only described quality criteria alone were also excluded, as were qualitative descriptions of Internet contents. We did not include studies that dealt with content other than health information, or studies evaluating information not intended for consumers (eg, those providing continuing medical education), but we did include studies evaluating services and information likely to be used by both professionals and consumers (eg, those providing drug information). Studies evaluating a single site or application were excluded, as were studies focusing exclusively on privacy and security issues, or on interactions via newsgroups or e-mail. Search Strategy We sought all relevant studies and unpublished reports, regardless of language or peer-review/publication status. We searched MEDLINE and PREMEDLINE (1966-September 2001) by entering the following query into the PubMed interface on September 20, 2001 (quality OR reliability OR accuracy OR readability OR evaluation OR assessment) AND (information OR education OR advice) AND (internet OR web OR ehealth OR "e-health" OR cyber* OR www). We retrieved 1545 citations and their titles and abstracts were screened for potential relevance by 2 independent reviewers. In addition, 1 reviewer used analogous search terms to screen citations in LISA (Library and Information Science Abstracts Database, 1969-July 2001) (1269 hits), CINAHL Nursing and Allied Health (1982-July 2001) (2312 hits), PsychINFO (American Psychological Association, 1988-September 2001 [week 2]) (321 hits), EMBASE (Elsevier Science BV, Amsterdam/NL, 1988-June 2001) (647 hits), and SIGLE (Grey Literature in Europe database, 1980-June 2001) (83 hits). The Web of Science database (Institute for Scientific Information, Philadelphia, Pa), which includes the Science Citation Index (1997-September 2001), the Social Sciences Citation Index (1997-September 2001), and the Arts and Humanities Citation Index (1997-September 2001) was searched using both a traditional backward search strategy (1120 hits) as well as using a forward citation-search strategy, using 5 seminal papers in this field3-7 as "seed publications" to identify all publications that subsequently cited 1 of these papers (533 hits). We also checked the references of all identified studies, hand-searched the Journal of Medical Internet Research (because it contains pertinent studies and because at that time it was not yet indexed on MEDLINE), conducted general Internet searches using combinations of the search terms at the Google and Northern Light search engines, reviewed 3 bibliographies of consumer health information studies collated by academic institutions, and used private literature databases. In total, we screened 7830 citations and retrieved 165 potentially eligible full articles to determine whether they met the inclusion criteria. Articles in Hungarian, Japanese, French, Spanish, Dutch, and Italian were translated by professional translators into English or German. Data Extraction Two reviewers (G.E., J.P.) independently extracted study characteristics using electronic data extraction forms, maintaining all data in an Access database (Microsoft Corp, Redmond, Wash). Each extraction form contained 85 items, of which 43 were closed questions, mostly to be answered on a scale (yes/no/partially/not applicable/to be discussed). We extracted information on search strategies used, quality criteria applied, and methodology of quality assessment used by authors, as well as quality and rigor of study methodology and reporting (see below and Table 1). We rated the overall tone of the authors' conclusions as "negative" if authors were wary and pessimistic about the Web as a source for health information, "positive" if authors did not express concern or recommended the Web as a source for health information, or "neutral" if authors discussed both risks and benefits, or if the reviewers disagreed. To extract evaluation results of each study, one reviewer tabulated data on how many Web sites or Web pages were evaluated for each quality criterion and how many of those evaluated did not comply with the respective criterion. Statistical Analysis To determine interobserver reliability for all coded items, we calculated standard κ values for each extracted item reported on a binary scale (eg, include study: yes/no), or weighted κ values for ordinal outcomes (eg, study characteristics graded on a scale [yes/partially/no]). We decided in advance not to include items with κ values less than 0.6.8 For the final consensus coding used for analysis, reviewers examined all disagreements or items coded "to be discussed" and resolved them by extensive discussion. We used meta-analysis of proportions9 to test for homogeneity of results for quality criteria that have been used by 3 or more authors to assess Web sites. We pooled study results for a given quality criterion if the assumption of homogeneity could not be rejected on a significance level of less than .05. To test whether the different conclusions of studies could be explained by their different quality scores we compared study assessment scores of study groups using the Wilcoxon test and logistic regression. All calculations were performed using SAS versions 6.12 and 8.2 (SAS Institute Inc, Cary, NC). Quality Scores We summarized the quality of reporting and the rigor of the studies by checking whether authors reported certain items deemed important for a well-reported, systematic, and comprehensive Internet search strategy (ie, S-score for search quality; range, 0-8 [transformed to percentage score]), and items thought to indicate a rigorous evaluation (ie, E-score for evaluation quality; [range, 0-10 [transformed to percentage score]). To ascertain face validity, an initial list of candidate items was compiled by 2 reviewers (G.E., J.P.) independently and later combined by consensus. As studies systematically evaluating Web information resemble in some ways systematic research overviews, the candidate criteria for the S-score partly derived from corresponding checklists.10 To ascertain score reliability, we eliminated items for which κ was less than 0.6; the remaining items are shown in Table 1. One point was given per item reported in order to calculate a raw summary score for search quality and evaluation quality. The final reported S-score and E-score in Online Table A (available in PDF format) represent the percentages of the maximum score achievable for each study (because some items are not applicable for some studies). To test construct validity we compared the scores of 5 letters to the editor with the scores of the 74 full publications. As expected, the letters had, on average, lower S-scores (28.8 vs 44.2, P = .06), but similar E-scores (22.4 vs 33.5, P = .45). The 7 full articles published in peer-reviewed non–English-language journals had lower average E-scores than articles published in English-language journals (mean E-score: 18.6 vs 34.8, P = .09), while their S-scores were similar (mean S-score: 42.3 vs 44.5, P = .85). While these differences are not significant at a 5% level, perhaps due to the small study number, the trend goes in the expected direction. Results Included Studies Of 170 articles reviewed, 85 met the inclusion criteria11-5960-95 (Online Table A available in PDF format). These constitute 79 distinct studies, as 6 studies were published either in duplicate,26,27,50,51,92,93 in a fragmented manner,87,88 or were reported both in a peer-reviewed publication and in an unpublished form.14,15,68,69 In these cases we pooled the results from the different articles and referred to these as a single study. Table 1 summarizes and Online Table A (available in PDF format) lists, all included studies. Quality Criteria and Methods Used to Evaluate Web Sites Technical Criteria (T). "Technical" quality criteria were defined as general, domain-independent criteria, ie, criteria referring to the question of how the information was presented or what meta-information was provided. The 24 technical quality criteria most frequently used (Table 2) are variations of what could be called "transparency criteria" from the print world, mentioned by Silberg et al7: authorship, attribution, disclosure, and currency. While the latter is strictly speaking a dimension of accuracy, almost all studies sought for provision of a date of creation or last update (rather than actual currency of the content), which is a technical criterion. Design (D). Fifteen studies also evaluated subjective design features such as the visual aspect of the site or layout, but only a few studies reported results (Online Table B available in PDF format), presumably because of their subjectivity and low reliability. For example, Gillois et al39 used a visual analog scale to assess the quality of visual presentation and of the interface, but noted high interobserver variability. Stausberg et al81 report a κ of 0.08 when evaluating navigation, and 0.23 when evaluating layout. Speed, browser compatibility, and presence of a search engine were considered technical criteria. Readability (R). Eleven studies used readability formulas to establish the reading level of a document based on the complexity and length of words and sentences (Online Table B available in PDF format). Nine studies used the Flesch-Kincaid (FK) Grade Level Index. Other formulas used include the SMOG Readability Formula, the Fry Readability Graph, the Gunning-Fog formula, and the Lexile Framework. As different formulas yield different reading levels for the same document, and as authors use different cutoffs, the results cannot be pooled, but studies suggest that reading levels are frequently too high (Online Table B available in PDF format).65 Using reading formulas has limitations, as readability scores do not reflect other factors that affect comprehension such as frequency and explanation of medical jargon, writing style (use of active voice, nonpatronizing language, motivational messages, tone/mood, how it relates to the audience), or use of culturally specific information. Few studies analyzed these important but subjective aspects—Oermann and Wilson66 discussed some of these parameters, Fitzmaurice and Adams35 scored the writing style, and Wilson et al94 examined the cultural sensitivity of the documents to various ethnic groups using Bloch's Ethnic/Cultural Assessment Tool. None of the studies conducted comprehension tests with actual consumers or used judgments of literacy experts. Accuracy (A). Accuracy can be defined as the degree of concordance of the information provided with the best evidence or with generally accepted medical practice. Alternative terms used for the concept of "accuracy" include "reliability"51 or "conventionality of information."79 We coded 2 studies as assessing accuracy "partially," because authors claimed to have evaluated "value and amount of text" without further specifying how they define "value."36,90 Authors who appraised the accuracy and/or comprehensiveness could either define clear criteria beforehand (a priori) or extract information from the Web first and then check these claims against the literature (a posteriori). Thirty studies assessed accuracy a posteriori, and only 19 a priori (Online Table A available in PDF format). Studies defining elements a priori varied considerably in the granularity and specificity of the items defined in advance: for example, 1 study stated that "paracetamol should be given in a dose of 10 to 15 mg/kg every 4 hours,"51 while others just predefined broad keywords that should be mentioned, eg, "oestrogen."75 Accuracy ideally should be defined using the best available evidence.5 Fourteen studies used evidence-based guidelines or systematic reviews to define elements a priori and 1 study used a guideline as the criterion standard a posteriori. Eleven studies used primary literature for a posteriori comparison; however, with 1 exception89 these studies did not provide details on how and what literature was screened, leaving open the possibility that some authors did not actually search the literature, but just compared information against their own knowledge of the literature. Textbooks or expert consensus were used as the criterion standard in 3 a priori and in 3 a posteriori studies.29,54,55,64,70,84 The remaining studies used unclear sources or the personal opinion of the author as the a priori (2 studies) or a posteriori (15 studies) criterion standard. Completeness/Comprehensiveness/Coverage/Scope (C). Several methods were used to evaluate "completeness," "comprehensiveness," "coverage," or "scope." Most authors12-15,22,29,35,37,42,44-46,50,51,54-56,58,59,70,75,78,79,85,92,93,95 calculated a proportion of a priori–defined elements covered by a Web site or reported the proportion of Web sites that mentioned all key elements (eg, from a clinical guideline). Willems and Bouvy92,93 used a 5-point scale to evaluate completeness. One study11 evaluated "balance," eg, whether the adverse effects and contraindications as well as the advantages of a drug are presented, but detailed subcriteria were not made explicit. The Soot score13,46,58,79 addresses completeness by measuring coverage of topic areas defined a priori, weighting some areas as being more important than others. While 19 studies evaluated completeness as a distinct entity, 21 other studies evaluated completeness as an integral part of accuracy (marked "AC" in Online Table A available in PDF format). For example, Impicciatore et al51 reported that only 4 of 44 Web sites "adhered closely to the guidelines," implying that these sites were both accurate and complete, whereas Davison27 only assessed the accuracy of statements made on the site against the respective statement in the guideline without requiring that all recommendations from nutritional guidelines appear on the Web site. While it is possible to evaluate accuracy without demanding completeness, studies in which authors just prepared checklists for completeness sometimes raise the question as to how and to what extent accuracy has been evaluated.29,35,54,55,75 For example, the Soot79 score is a weighted checklist giving points for each broad topic covered by the site (such as "treatment options"), but does not specify whether topics were merely present or absent or if they were evaluated for accuracy. Of the 19 studies that evaluated completeness as a distinct entity, 5 studies used an external source a priori to define elements: 2 used evidence-based guidelines, 2 used primary literature, and 1 used a textbook. The remaining studies used unclear sources or the personal opinion of the author, either a priori (7 studies) or a posteriori (7 studies). Score Systems Twenty-two studies11,13-15,23,31,35,36,38,42-44,46,58,59,70,72,74,75,78,79,87,90 used a composite score system, most frequently the Soot score,13,46,58,79 for adherence to quality criteria. Two studies38,87 used DISCERN96 and another used Suitability Assessment of Materials (SAM),65 both described by their developers as "validated" instruments to assess printed patient education material; however, none of these have been validated in a way such that a higher score would predict better health outcomes or consumer satisfaction. Quality of Studies The quality scores for each study are shown in Online Table A (available in PDF format). Out of the 79 studies, 49 received an S-score and an E-score of 50 or less, indicating that both search strategy and evaluation were not well reported or rigorous. However, of the 31 studies using more than 1 rater, 20 made efforts to ascertain the within-study reliability of their instruments, which is in contrast to what has been criticized in a previous study about Web-based instruments.97 Study Conclusions and Study Quality Most studies (55 [70%]) concluded that quality is a problem on the Internet (κ for coding conclusions, 0.88). Seventeen (21.5%) were neutral. Only 7 (9%) came to a more positive conclusion, none of which used evidence-based guidelines as a criterion standard. The mean S-score of the positive studies was significantly lower than that of the negative studies (29.4 vs 45.0; P = .02), as was the E-score (15.6 vs 37.9; P = .04), indicating that the more enthusiastic studies used a less rigorous search and evaluation strategy than did negative studies. In a logistic regression model with both scores as predictive variables and the study conclusion "positive/negative" as the dependent variable, the odds ratio (OR) for S-score is 0.929 (95% confidence interval [CI], 0.871-0.991; P = .02), ie, for each additional S-score percentage point the odds to reach a positive conclusion decreases by 7.1%; the OR for E-score is 0.938 (95% CI, 0.884-0.996; P = .04), ie, for each additional E-score percentage point the odds for a positive conclusion decreases by 6.2%. When studies evaluated accuracy, the proportions of inaccurate Web sites depended on the level of evidence used as a criterion standard: studies that didn't report the criterion standard or that used personal opinion found an average of 15.4% of Web sites inaccurate; those using literature, textbooks, or expert consensus, 35.3%; and those using clinical guidelines, 38.3%. In an ordinal logistic regression model, the reported proportions of inaccurate Web sites were significantly associated with the 3 levels of evidence used by authors as criterion standard (P<.001). Even if authors had similar results, they sometimes interpreted these differently. For example, 3 studies found that about 5% of cancer Web sites provided inaccurate information, but these results were interpreted as being "of concern,"16 "encouraging,"76 or even "reassuring."46 In at least 1 case investigators reviewed the same topic area, but arrived at opposite conclusions.28,75 Comment We reviewed 79 studies in which authors evaluated a total of 5941 Web sites and 1329 Web pages, and reported 408 evaluation results for 86 distinct quality criteria (Online Table B available in PDF format). Content Quality In our review, most authors who evaluated content found significant problems, criticizing lack of completeness, difficulty in finding high-quality sites, and lack of accuracy, in particular if "accuracy" also implied "completeness." Five of eight studies reporting results on completeness found that around 90% of Web sites were "incomplete" (Online Table B available in PDF format). However, completeness as a requirement has questionable validity from the perspective of the user or the public health researcher. First, too much information may overburden users. Web sites may deliberately and with good reason focus on a single topic in-depth rather than aiming for comprehensiveness. Second, in contrast to printed educational material, a single Web page or Web site is part of a universe of information: a topic not covered by one Web page or site may be covered by another (perhaps linked) Web page. Consumers will usually search across different Web sites when looking for specific health information.98 Mechanical comparison of elements from a guideline with elements covered by a single Web site without taking into account the context and purpose of the site or exploring links to other sites is of limited use. Perhaps a better approach would be to evaluate whether materials cover the topics they claim to be discussing96,99 and if they are balanced. Comparisons Across Studies Prevalence figures of inaccurate Web sites differ across different domains, eg, diet and nutrition sites (45.5%26 and 88.9%64 inaccurate information) vs cancer sites (4% for prostate cancer,46 5.1% for breast cancer,76 9% for English-language and 4% for Spanish-language breast cancer documents,14 6% for testicular cancer,46 or 6.2% for Ewing Sarcoma16). While such prevalence figures may suggest that diet information on the Web is of poorer quality than cancer information, unadjusted comparisons across studies have to be made with care, as at least 3 potential confounders should be considered. First, results heavily depend on the rigor of the methodology used: studies that used personal opinion as a criterion standard found fewer inaccurate Web sites and more often came to a positive conclusion than studies using more rigorous criteria. Second, many studies use the terms "Web site" and "Web page" interchangeably, making comparisons difficult. Third, different and often poorly described sampling and selection strategies were used. While in theory a truly random sample of Web sites could be identified (by choosing random Internet protocol [IP] addresses100), this approach is not practical for identifying Web sites for a given topic. Thus, all studies used search engines, catalogs, or lists of popular Web sites. However, the choice of the search strategy may greatly confound the results. Most studies mimicked how consumers would search (although only 3 involved actual consumers), hand-selecting popular sites. As there is no consensus and little research on how a typical consumer searches,98 studies used various strategies, mostly picking the top-ranked results from a search engine. As many search engines can rank the better sites first, the search tool could influence the results. Moreover, even the same search engine may give a different result if different sampling strategies (ie, which sites are picked) are used. For example, Suarez-Almazor et al83 showed that, in the Webcrawler search engine, the first 20 ranked hits are more relevant, have less financial interests, contain less alternative therapies, and are more often nonprofit organizations, than sites ranked lower. Moreover, the selection of search terms may critically determine which Web sites are retrieved. For example, using the term "coronary heart disease," Eachus31 found only 2 sites (among 110) provided by lay people. His conclusion that "the concern that the Internet would be a major source of low-quality health information, particularly that provided by unqualified members of the lay public, is not supported by [our] findings" may be confounded by the choice of search terms, as lay people might not necessarily use terms such as "coronary heart disease." Who devises the search strategy and conducts the search can also affect the quality of the retrieved sites. For example, in a comparison of results of a search devised by doctors with a search devised by information experts, Groot et al43 observed differences in the credibility and accuracy scores of the retrieved samples. Comparison With Other Media The quality of Web sites should be interpreted in the larger context of information in other media to determine whether the Web is "the beginning of an epidemic of misinformation or nothing more than a variation of what is endemic."2 Many of the shortcomings detected likely are not specific to the Web and are also present in other media. For example, 2 of the 4 "erroneous information" elements found on 65 Web pages identified by Biermann et al16 were found in the online version of the Encyclopaedia Britannica, which probably has the same inaccuracies in the printed edition. This issue, and the relatively low prevalence (6.2%) of inaccurate information, was generally ignored when this study was widely quoted in the lay media as evidence that the "Internet can be a quick link to bad health information."101 Studies assessing information in traditional media also frequently report high prevalences of inaccurate or incomplete information. In an early study, authors found 70% of health information broadcast on television to be inaccurate, misleading, or both.102 In another study, authors rated as inaccurate 76% of the information about oral hygiene from television, 53% from magazines, and 12% from newspapers.103 Another study of the popular press found 20% of the information on oral cancer to be a "mix of accurate and inaccurate information."104 The proportion of inaccurate press reports on healthy eating was found to be 55% in free advertising newspapers, 28.9% in lifestyle magazines, 29.9% in general interest magazines, 17.5% in health magazines, and 14.1% in newspapers.105 In another study, 50% of the advice in newspaper advice columns was rated inappropriate, with critical issues only partially covered or not covered at all in 76% of the articles, and 58% were rated unsafe or potentially dangerous.106 Inquiries by telephone to libraries yielded a 3.6% rate of inaccurate information.107 The perceived quality problem on the Internet is not restricted to the health sector: a study investigating the quality of general scientific information found that 10% to 34% was inaccurate, 20% to 35% was misleading, and 48% to 90% was unreferenced.108 Very few studies in our sample directly compared Internet information with information found elsewhere. One study35 evaluated both printed and Web-based patient education materials, concluding that "there was no significant difference between the ranges of scores [incorporating content, writing style, design, and readability] for Internet and non-Internet leaflets"; however, subjectively authors felt that "the overall quality of the Internet leaflets was more variable and the information less comprehensive." Two studies compared the readability of Internet information with the readability of printed information. One study33 concluded that SMOG readability levels of Internet information were significantly higher compared with other (printed) patient information materials; the other study65 observed no differences in the proportion of patient information written above the 9th grade level, but 87.5% of Web information materials vs only 14.3% of printed patient information, were deemed unsuitable based on the Suitability Assessment of Materials score (however, as the 8 Web-based patient education materials came from only 2 different Web sites, this result may not be representative). We identified only 1 study that compared the accuracy of advice obtained on the Internet (in a pharmacy newsgroup, hence this study was excluded from this review) with advice obtained in the real world (from drug information centers).109 In summary, the prevalence figures of inaccurate or incomplete Web information reported by the studies in this review are difficult to interpret or compare, are unlikely to be representative or generalizable, and must also be considered against the background of imperfect consumer health information in other media. Presentation Quality The presentation criteria evaluated by most investigators are considered to be quality criteria because: (1) their presence is deemed ethical according to several codes of conduct for Web publishing110-114 (eg, transparency and accountability criteria, such as disclosure of authorship); (2) they help to create context and to avoid misunderstandings (eg, disclosure of sponsorship, purpose); (3) they empower users to select the information that is best for them in their individual situation (eg, disclosure of target audience); (4) they empower users to validate the information themselves (eg, references, contact addresses); or (5) they may influence the accessibility of information or effectiveness of communication (eg, search capabilities, speed, design). In addition, several authors have attempted to establish whether technical criteria or other site characteristics may be used as predictors for content quality,11,42,47,58,87,91 but the results have been conflicting and inconclusive. A number of studies found, perhaps unsurprisingly, that the length of a document is correlated with better content scores.42,58,87 A few studies suggest that the source may be a predictor for content quality, with commercial sites often scoring lower than academic sites,59,74,75 but the way in which content is evaluated may also influence this relationship. Other technical criteria influenced by the source include references, which are more often found on Web sites targeting medical professionals,23 on academic46,79 or educational89 sites, or on sites owned by organizations.42 Individual authors often are not disclosed on government sites14 or sites of organizations and drug companies.42 Given these complex relationships, it seems unlikely that a simple scoring system could be developed that predicts content quality across domains and site populations. As with content criteria, the ability to compare or pool results across studies is impeded by wide variations in sampling strategies, methods, and operational definitions even for the same quality criteria (eg, what constitutes "attribution"). Moreover, there is no consensus on the unit of evaluation (eg, should the update cycle be published on each Web page, or is a site-specific disclosure sufficient) or granularity with which information should be provided (eg, is it sufficient to post the year of last update, or should the exact date be given). Despite this lack of consensus, some quality criteria are consistently given (eg, "ownership disclosure," with 99% of sites providing information) or not given (eg, "credentials of physicians not disclosed," with 97.5% not complying) (Table 2). As with content criteria, presentation criteria should be put into context by comparison with non–Web-based information. While in our study 64% of Web sites failed to provide a date of update, investigators evaluating printed patient leaflets have found that 53%,35 or, in another study, one-third,115 did not include publication dates. Omissions Since most authors used criteria derived from the print world,7 we noted important omissions concerning Web-specific criteria. For example, very few studies evaluated the privacy policy or the possibility to encrypt confidential information, and few studies checked whether the target audience or the target country (which is important in a global medium116) were clearly disclosed. None of the studies tested usability117 or accessibility (eg, compliance with guidelines of the Web Accessibility Initiative, ascertaining that the site is available to people with disabilities or with low-end technology), and only 1 study checked whether metadata were provided,76 which could greatly enhance the consumer's ability to select and filter information.6 Measuring Progress of Health Communication on the Web Among the public health objectives of the US Department of Health and Human Services is to "increase the proportion of health-related World Wide Web sites that disclose information that can be used to assess the quality of the site."118 How progress in this area might be achieved or evaluated is an open question. A meta-analysis of cross-sectional studies is one possible way, but one must take into account the different methods used (or the methods must be standardized) before meaningful conclusions can be drawn. Longitudinal studies could be used to assess changes over time, using a consistent methodology either to identify and assess Web sites for a given topic (with the caveat that observed changes may be changes in the ability of users to find better sites, ie, improvements in search engine technology to rank better sites first) or to follow up a cohort of Web sites. Two studies have attempted a longitudinal approach, with conflicting results; one observed an improvement, and the other a deterioration, in quality.36,57,90 The third possibility is to promote among site developers the use of machine-processable disclosure statements as Web site labels (meta-data),6,119 which would allow automatic tracking and analysis of the proportion and characteristics of Web sites making such disclosure statements. This would also facilitate the development of intelligent systems able to guide users to trustworthy Web sites.120 Conclusion The epidemiology of consumer health information on the Web is an emerging research discipline at the intersection of medical informatics and public health. Many descriptive, cross-sectional studies have attempted to draw attention to perceived "outbreaks" of misinformation on the Web by estimating the proportion or prevalence of inadequate health information. However, the individual's risk (R) of encountering an inadequate site on the Web is a function of both the proportion of inadequate information on the Web (P) and the inability (I) of the individual (or his tools) to filter the inadequate sites. Since studies usually report R, but not I, we cannot infer P, or adjust study results to make them comparable across domains or time. Even if we could know P, we would still not know how this measurement of "true" misinformation on the Web translates into health outcomes or critical incidents in a population. On an individual level, R can be reduced by improving the ability of the user to locate trustworthy sites or to filter the inadequate ones. Public e-health interventions such as MedCERTAIN121 therefore strive to reduce P and I by increasing, for example, the proportion of health information providers making disclosure statements119 and by empowering consumers to identify trusted sites through educational and technological innovations, including the possibilities of the semantic Web.120 Given the difficulty in interpreting descriptive studies without control or comparison groups, future studies should use analytic rather than descriptive approaches to investigate the relationship between quality markers and other variables, including outcomes, or to compare different Web site populations or media. Such studies are urgently needed to help in the ongoing process to develop methods and instruments to guide consumers to quality information and to identify factors that can be assessed to predict favorable patient outcomes. Finally, studies evaluating content should also harness the potential of the Web as a source for qualitative data122: rather than getting bogged down by the question of how much information is inaccurate, one could analyze where and why gaps exist between evidence-based medicine and health information on the Internet, which may elicit a wealth of valuable data that may inform priorities for research, health communication, and education. References 1. Eysenbach G. Consumer health informatics. BMJ.2000;320:1713-1716.Google Scholar 2. Coiera E. Information epidemics, economics, and immunity on the Internet. BMJ.1998;317:1469-1470.Google Scholar 3. Jadad AR, Gagliardi A. Rating health information on the Internet. JAMA.1998;279:611-614.Google Scholar 4. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ.1999;318:647-649.Google Scholar 5. Wyatt JC. Commentary: measuring quality and impact of the World Wide Web. BMJ.1997;314:1879-1881.Google Scholar 6. Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet. BMJ.1998;317:1496-1500.Google Scholar 7. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. JAMA.1997;277:1244-1245.Google Scholar 8. Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics.1977;33:159-174.Google Scholar 9. Bohning D, Dammann UP. The application of methods of meta-analysis for heterogeneity modeling in quality control and assurance. In: Schulze R, Holling H, Bohning D, eds. Meta-analysis: New Developments and Applications in Medical and Social Sciences. Seattle, Wash: Hogrefe & Huber. In press. 10. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol.1991;44:1271-1278.Google Scholar 11. Abbott VP. Web page quality: can we measure it and what do we find? J Public Health Med.2000;22:191-197.Google Scholar 12. Armstrong K, Schwartz JS, Asch DA. Direct sale of sildenafil (Viagra) to consumers over the Internet. N Engl J Med.1999;341:1389-1392.Google Scholar 13. Beredjiklian PK, Bozentka DJ, Steinberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet. J Bone Joint Surg Am.2000;82-A:1540-1543.Google Scholar 14. Berland GK, Elliott MN, Morales LS. et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA.2001;285:2612-2621.Google Scholar 15. Berland GK, Morales LS, Elliott MN. et al. Evaluation of English and Spanish Health Information on the Internet. Santa Monica: RAND/California HealthCare Foundation; 2001. Available at: http://www.rand.org/publications/documents/interneteval/. Accessibility verified April 25, 2002. 16. Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer.1999;86:381-390.Google Scholar 17. Bloom BS, Iannacone RC. Internet availability of prescription pharmaceuticals to the public. Ann Intern Med.1999;131:830-833.Google Scholar 18. Bogenschutz MP. Drug information libraries on the Internet. J Psychoactive Drugs.2000;32:249-258.Google Scholar 19. Boyer EW, Shannon M, Hibberd PL. Web sites with misinformation about illicit drugs. N Engl J Med.2001;345:469-471.Google Scholar 20. Breul H, Boue L, Martin A. Panorama des ressources documentaires hospitaliéres françaises sur Internet [Overview of Internet information resources of interest to French hospitals]. Semaine des Hopitaux.1999;75:257-262.Google Scholar 21. Butzke I, Kramer KL. Orthopadie im World Wide Web: Universitats-Homepages der G7-Staaten. Z Orthop Ihre Grenzgeb.2000;138:413-418.Google Scholar 22. Bykowski JL, Alora MB, Dover JS, Arndt KA. Accessibility and reliability of cutaneous laser surgery information on the World Wide Web. J Am Acad Dermatol.2000;42:784-786.Google Scholar 23. Chen LE, Minkes RK, Langer JC. Pediatric surgery on the Internet. J Pediatr Surg.2000;35:1179-1182.Google Scholar 24. Corpron CA, Lelli Jr JL. Evaluation of pediatric surgery information on the Internet. J Pediatr Surg.2001;36:1187-1189.Google Scholar 25. D'Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med.2001;155:807-812.Google Scholar 26. Davison K. The quality of dietary information on the World Wide Web. J Can Dietetic Assoc.1996;57:137-141.Google Scholar 27. Davison K. The quality of dietary information on the World Wide Web. Clin Perform Qual Health Care.1997;5:64-66.Google Scholar 28. Diering CL, Palmer MH. Professional information about urinary incontinence on the World Wide Web. J Wound Ostomy Continence Nurs.2001;28:55-62.Google Scholar 29. Doupi P, van der Lei J. Rx medication information for the public and the WWW: quality issues. Med Inform Internet Med.1999;24:171-179.Google Scholar 30. Dracos A, Seta M. Egeszsegugyi informaciok az Interneten laikusoknak: olaszorszagi tapasztalatok. [Health information for the lay person on the Internet: a survey of the Italian experience]. Tudomanyos es Muszaki Tajekoztatas.1998;45:118-120.Google Scholar 31. Eachus P. Health information on the Internet. Int J Health Prom Educ.1999;37:30-33.Google Scholar 32. Ellamushi H, Narenthiran G, Kitchen ND. Is current information available useful for patients and their families? Ann R Coll Surg Engl.2001;83:292-294.Google Scholar 33. Estrada CA, Hryniewicz MM, Higgs VB, Collins C, Byrd JC. Anticoagulant patient information material is written at high readability levels. Stroke.2000;31:2966-2970.Google Scholar 34. Eysenbach G. Online prescribing of sildanefil (Viagra) on the World Wide Web. J Med Internet Res.1999;1:e10. Available at: http://www.jmir.org/1999/2/index.htm. Accessibility verified March 25, 2002.Google Scholar 35. Fitzmaurice DA, Adams JL. A systematic review of patient information leaflets for hypertension. J Hum Hypertens.2000;14:259-262.Google Scholar 36. Frasca D, Malezieux R, Mertens P, Neidhardt JPH, Voiglio EJ. Review and evaluation of anatomy sites on the Internet (updated 1999). Surg Radiol Anat.2000;22:107-110.Google Scholar 37. Galimberti A, Jain S. Gynaecology on the Net. J Obstet Gynaecol.2000;20:297-299.Google Scholar 38. Gillies M. Access to the information should be made easier. BMJ.2000;321:47-48.Google Scholar 39. Gillois P, Colombet I, Dreau H, Degoulet P, Chatellier G. A critical appraisal of the use of Internet for calculating cardiovascular risk. Proc AMIA Symp.1999;:775-779.Google Scholar 40. Gordon JB, Barot LR, Fahey AL, Matthews MS. The Internet as a source of information on breast augmentation. Plast Reconstr Surg.2001;107:171-176.Google Scholar 41. Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract.1999;48:58-61.Google Scholar 42. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ.2000;321:1511-1515.Google Scholar 43. Groot D, Riet G, Khan KS, Misso K. Comparison of search strategies and quality of medical information of the Internet: a study relating to ankle sprain. Injury.2001;32:473-476.Google Scholar 44. Harmon D, Duggan M, Flynn N. Anaesthesia on the World Wide Web. Anaesthesia.2000;55:728-729.Google Scholar 45. Hatfield CL, May SK, Markoff JS. Quality of consumer drug information provided by four Web sites. Am J Health Syst Pharm.1999;56:2308-2311.Google Scholar 46. Hellawell GO, Turner KJ, Le Monnier KJ, Brewster SF. Urology and the Internet: an evaluation of Internet use by urology patients and of information available on urological topics. BJU Int.2000;86:191-194.Google Scholar 47. Hernández-Borges AA, Macías-Cervi P, Gaspar-Guardado MA, Torres-Álvarez de Arcaya ML, Ruiz-Rabaza A, Jiménez-Sosa A. Can examination of WWW usage statistics and other indirect quality indicators help to distinguish the relative quality of medical websites? J Med Internet Res.1999;1:e1. Available at: http://www.jmir.org/1999/1/e1/index.htm. Accessibility verified March 25, 2002.Google Scholar 48. Hersh WR, Gorman PN, Sacherek LS. Applicability and quality of information for answering clinical questions on the Web. JAMA.1998;280:1307-1308.Google Scholar 49. Hoffman-Goetz L, Clarke JN. Quality of breast cancer sites on the World Wide Web. Can J Public Health.2000;91:281-284.Google Scholar 50. Impicciatore P, Pandolfini C, Casella N, Bonati M. Possiamo fidarci delle informazioni sanitarie al pubblico che viaggiano su Internet? l'esempio della gestione domiciliare della febbre nel bambino. [Reliability of public-oriented health care resources in the World Wide Web: the case of the home management of fever in children]. Ricerca e Pratica.1997;13:67-75.Google Scholar 51. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web. BMJ.1997;314:1875-1879.Google Scholar 52. Jiang YL. Quality evaluation of orthodontic information on the World Wide Web. Am J Orthod Dentofacial Orthop.2000;118:4-9.Google Scholar 53. Kihlstrom LC. Evaluating pharmacy benefit management information on the Internet. Manag Care Interface.2001;14:64-68.Google Scholar 54. Latthe PM, Latthe M, Khan KS. Quality of medical information about menorrhagia on the worldwide web. BJOG.2000;107:39-43.Google Scholar 55. Latthe M, Latthe PM, Charlton R. Quality of information on emergency contraception on the internet. Br J Fam Plann.2000;26:39-43.Google Scholar 56. Latthe PM, Latthe M, Khan KS. Quality of information on female sterilisation on the Internet. J Obstet Gynaecol.2000;20:167-170.Google Scholar 57. Li L, Irvin E, Guzman J, Bombardier A. Surfing for back pain patients. Spine.2001;26:545-557.Google Scholar 58. Libertiny G, Perkins JM, Magee TR, Galland RB. Varicose veins on the internet. Eur J Vasc Endovasc Surg.2000;20:386-389.Google Scholar 59. Lissman TL, Boehnlein JK. A critical review of internet information about depression. Psychiatr Serv.2001;52:1046-1050.Google Scholar 60. Mallory C. What's on the Internet? services for women affected by HIV and AIDS. Health Care Women Int.1997;18:315-322.Google Scholar 61. Martinez-Lopez JF, Ruiz-Crespo EJ. Internet: calidad de la informacion sobre cirugia [Internet: quality of information available on orthopaedic surgery and traumatology]. Revista de Ortopedia y Traumatologia.1998;42:469-473.Google Scholar 62. Maugans TA, McComb JG, Levy ML. The internet as a pediatric neurosurgery information resource. Pediatr Neurosurg.1998;28:186-190.Google Scholar 63. McClung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics.1998;101:e2.Google Scholar 64. Miles J, Petrie C, Steel M. Slimming on the Internet. J R Soc Med.2000;93:254-257.Google Scholar 65. Murphy PW, Chesson AL, Berman SA, Arnold CL, Galloway G. Neurology patient education materials. J Neurosci Nurs.2001;33:99-104.Google Scholar 66. Oermann MH, Wilson FL. Quality of care information for consumers on the Internet. J Nurs Care Qual.2000;14:45-54.Google Scholar 67. O'Mahony B. Irish health care Web sites: a review. Ir Med J.1999;92:334-337.Google Scholar 68. Ogushi Y, Tatsumi H. Research on the Analysis of the Current State of the Provision and Use of Health Information Provided Through a New Technology Medium [in Japanese]. Tokyo, Japan: Health & Welfare Ministry Research Group; 2000. 69. Tatsumi H, Mitani H, Haruki Y, Ogushi Y. Internet medical usage in Japan. J Med Internet Res.2001;3:e12. Available at: http://www.jmir.org/2001/1/e12. Accessibility verified April 24, 2002.Google Scholar 70. Pandolfini C, Impicciatore P, Bonati M. Parents on the web. Pediatrics.2000;105:e1.Google Scholar 71. Payne C, Miller K. The quality of footcare information on the Internet. Australas J Podiatr Med.2000;34:63.Google Scholar 72. Peroutka SJ. Analysis of Internet sites for headache. Cephalalgia.2001;21:20-24.Google Scholar 73. Roberts JR, Spooner SA. Pediatric Internet resources. Arch Pediatr Adolesc Med.1997;151:592-597.Google Scholar 74. Sacchetti P, Zvara P, Plante MK. The Internet and patient education—resources and their reliability. Urology.1999;53:1117-1120.Google Scholar 75. Sandvik H. Health information and interaction on the Internet. BMJ.1999;319:29-32.Google Scholar 76. Shon J, Musen MA. The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web. Proc AMIA Symp.1999;:945-949.Google Scholar 77. Sing A, Salzman J, Sing D. Problems and risks of unsolicited e-mails in patient-physician encounters in travel medicine settings. J Trav Med.2001;8:109-112.Google Scholar 78. Smith M, Gertz E, Alvarez S, Lurie P. The content and accessibility of sex education information on the Internet. Health Educ Behav.2000;27:684-694.Google Scholar 79. Soot LC, Moneta GL, Edwards JM. Vascular surgery and the Internet. J Vasc Surg.1999;30:84-91.Google Scholar 80. Stausberg J, Fuchs J. Die chirurgische Fachabteilung im World Wide Web: Tribut an den Zeitgeist oder Informationsdrehscheibe? [Surgical specialty department in the World Wide Web: tribute to contemporary life style or information network?]. Chirurg.2000;71:472-477.Google Scholar 81. Stausberg J, Fuchs J, Husing J, Hirche H. Health care providers on the World Wide Web. Med Inform Internet Med.2001;26:17-24.Google Scholar 82. Stone TW, Jumper JM. Information about age-related macular degeneration on the Internet. South Med J.2001;94:22-25.Google Scholar 83. Suarez-Almazor ME, Kendall CJ, Dorgan M. Surfing the Net—information on the World Wide Web for persons with arthritis. J Rheumatol.2001;28:185-191.Google Scholar 84. Tamm EP, Raval BK, Huynh PT. Evaluation of the quality of self-education mammography material available for patients on the Internet. Acad Radiol.2000;7:137-141.Google Scholar 85. Thompson JM, Howard JE. HMO Web sites. Manag Care Q.2000;8:33-41.Google Scholar 86. Tu F, Zimmerman NP. It is not just a matter of ethics: a survey of the provision of health disclaimers, caveats, and other health-related alerts in consumer health information on eating disorders on the Internet. Int Inf Libr Rev.2001;32:325-339.Google Scholar 87. Türp JC, Gerds T, Neugebauer S. Myoarthorpathien des Kausystems: Beurteilung der Qualität von Patienteninformationen im Weltweiten Netz. Z Arztl Fortbild Qualitatssich.2001;95:539-547.Google Scholar 88. Neugebauer S, Türp JC. Vorsicht Internet! Webseiten uber myoarthropathien des kausystems [Attention Internet! Web sites on myoarthropathies of the mastication system]. Schweiz Monatsschr Zahnmed.2001;111:298-311.Google Scholar 89. Veronin MA, Ramirez G. The validity of health claims on the World Wide Web. Am J Health Promot.2000;15:21-28.Google Scholar 90. Voiglio E, Frasca D, Malezieux R, Moreau S, Rodier MN, Neidhardt JPH. Prospecting and evaluation of the anatomy sites on the internet. Surg Radiol Anat.1999;21:65-68.Google Scholar 91. von Danwitz F, Baehring T, Scherbaum W. Verbreitung, Gestaltung und Qualität von Deutschsprachigen World Wide Web Seiten zum Diabetes Mellitus. Düsseldorf, Germany: Deutsches Diabetes Forschungsinstitut; 1999. 92. Willems M, Bouvy ML. Gezondheidsinformatie op Internet: ze adviseren gewoon de spullen die op voorraad zijn. Pharmaceutisch Weekblad.2001;136:607-609.Google Scholar 93. Willems M, Bouvy ML. Gezondheidsinformatie op het internet : de kwaliteit van websites vergeleken. Medisch Contact.2001;56:673-676.Google Scholar 94. Wilson FL, Baker LM, Brown-Syed C, Gollop C. An analysis of the readability and cultural sensitivity of information on the National Cancer Institute's Web site: CancerNet. Oncol Nurs Forum.2000;27:1403-1409.Google Scholar 95. Wright B, Williams C, Partridge I. Management advice for children with chronic fatigue syndrome: a systematic study of information from the Internet. Ir J Psychol Med.1999;16:67-71.Google Scholar 96. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health.1999;53:105-111.Google Scholar 97. Delamothe T. Quality of websites: kitemarking the west wind. BMJ.2000;321:843-844.Google Scholar 98. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the World-Wide-Web? BMJ.2002;324:573-577.Google Scholar 99. Dalton L, Gartenfeld E. Evaluating printed health information for consumers. Bull Med Libr Assoc.1981;69:322-324.Google Scholar 100. Lawrence S, Giles CL. Accessibility of information on the web. Nature.1999;400:107-109.Google Scholar 101. Internet can be quick link to bad health information. CNN.July 29, 1999. Available at: http://www.cnn.com/HEALTH/9907/29/internet.health. Accessibility verified April 25, 2002.Google Scholar 102. Smith FA, Trivax G, Zuehlke DA, Lowinger P, Nghiem TL. Health information during a week of television. N Engl J Med.1972;286:516-520.Google Scholar 103. Frazier PJ, Jenny J, Ostman R, Frenick C. Quality of information in mass media: a barrier to the dental health education of the public. J Public Health Dent.1974;34:244-257.Google Scholar 104. Canto MT, Kawaguchi Y, Horowitz AM. Coverage and quality of oral cancer information in the popular press: 1987-98. J Public Health Dent.1998;58:241-247.Google Scholar 105. Institut für Ernährungswissenschaften der Universität Wien. Österreichischer Ernährungsbericht.Vienna, Austria: Bundesministerium für Arbeit, Gesundheit und Soziales und der Bundesministerin für Frauenangelegenheiten und Verbraucherschutz; 1998.Google Scholar 106. Molnar FJ, Man-Son-Hing M, Dalziel WB. et al. Assessing the quality of newspaper medical advice columns for elderly readers. CMAJ.1999;161:393-395.Google Scholar 107. Paskoff BM. Accuracy of telephone reference service in health sciences libraries. Bull Med Libr Assoc.1991;79:182-188.Google Scholar 108. Allen ES, Burke JM, Welch ME, Rieseberg LH. How reliable is science information on the web? Nature.1999;402:722.Google Scholar 109. Seaboldt JA, Kuiper R. Comparison of information obtained from a Usenet newsgroup and from drug information centers. Am J Health Syst Pharm.1997;54:1732-1735.Google Scholar 110. Risk A, Dzenowagis J. Review of Internet health information quality initiatives. J Med Internet Res.2001;3:e28. Available at: http://www.jmir.org/2001/4/e28.Google Scholar 111. Boyer C, Selby M, Scherrer JR, Appel RD. The Health on the Net Code of Conduct for medical and health Websites. Comput Biol Med.1998;28:603-610.Google Scholar 112. Hi-Ethics Group. Health Internet Ethics: Ethical Principles For Offering Internet Health Services to Consumers. 2000. Available at: http://www.hiethics.org/Principles/. Accessibility verified March 25, 2002. 113. e-Health Ethics Initiative. e-Health Code of Ethics. J Med Internet Res.2000;2:e9. Available at: http://www.jmir.org/2000/2/e9/. Accessibility verified March 25, 2002.Google Scholar 114. Winker MA, Flanagin A, Chi-Lum B. et al. Guidelines for medical and health information sites on the Internet. JAMA.2000;283:1600-1606.Google Scholar 115. Coulter A, Entwistle V, Gilbert D. Sharing decisions with patients: is the information good enough? BMJ.1999;318:318-322.Google Scholar 116. Eysenbach G, Diepgen TL. Labeling and filtering of medical information on the Internet. Methods Inf Med.1999;38:80-88.Google Scholar 117. Nielsen J. Usability Engineering. San Diego, Calif: Morgan Kaufmann; 1993. 118. US Department of Health and Human Services. Healthy People 2010: Understanding and Improving Health. 2nd ed. Washington, DC: Office of Disease Prevention and Health Promotion, US Dept of Health and Human Services; 2000. Publication 017-001-00547-9. Available at: http://www.health.gov/healthypeople/document/. Accessibility verified March 25, 2002. 119. Eysenbach G, Köhler C, Yihune G, Lampe K, Cross P, Brickley D. A metadata vocabulary for self- and third-party labeling of health web-sites. Proc AMIA Annu Fall Symp.2001:169-173.Google Scholar 120. Eysenbach G. An ontology of quality initiatives and a model for decentralized, collaborative quality management on the (semantic) World-Wide-Web. J Med Internet Res.2001;3:e34. Available at: http://www.jmir.org/2001/4/e34. Accessibility verified April 24, 2002.Google Scholar 121. Eysenbach G, Köhler C, Yihune G, Lampe K, Cross P, Brickley D. A framework for improving the quality of health information on the world-wide-web and bettering public (e-)health. Medinfo2001, Proceedings of the Tenth World Congress on Medical Informatics, London, England, September 2001 . Amsterdam, Holland: IOS Press; 2001:1450-1454. 122. Eysenbach G, Till JE. Ethical issues in qualitative research on Internet communities. BMJ.2001;323:1103-1105.Google Scholar http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png JAMA American Medical Association

Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web: A Systematic Review

Eysenbach, Gunther; Powell, John; Kuss, Oliver; Sa, Eun-Ryoung
JAMA , Volume 287 (20) – May 22, 2002
Download PDF
36 pages

Loading next page...
 
/lp/american-medical-association/empirical-studies-assessing-the-quality-of-health-information-for-RTXedMJsEN

References (137)

Publisher
American Medical Association
Copyright
Copyright © 2002 American Medical Association. All Rights Reserved.
ISSN
0098-7484
eISSN
1538-3598
DOI
10.1001/jama.287.20.2691
Publisher site
See Article on Publisher Site

Abstract

Abstract Context The quality of consumer health information on the World Wide Web is an important issue for medicine, but to date no systematic and comprehensive synthesis of the methods and evidence has been performed. Objectives To establish a methodological framework on how quality on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, and to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research. Data Sources We searched MEDLINE and PREMEDLINE (1966 through September 2001), Science Citation Index (1997 through September 2001), Social Sciences Citation Index (1997 through September 2001), Arts and Humanities Citation Index (1997 through September 2001), LISA (1969 through July 2001), CINAHL (1982 through July 2001), PsychINFO (1988 through September 2001), EMBASE (1988 through June 2001), and SIGLE (1980 through June 2001). We also conducted hand searches, general Internet searches, and a personal bibliographic database search. Study Selection We included published and unpublished empirical studies in any language in which investigators searched the Web systematically for specific health information, evaluated the quality of Web sites or pages, and reported quantitative results. We screened 7830 citations and retrieved 170 potentially eligible full articles. A total of 79 distinct studies met the inclusion criteria, evaluating 5941 health Web sites and 1329 Web pages, and reporting 408 evaluation results for 86 different quality criteria. Data Extraction Two reviewers independently extracted study characteristics, medical domains, search strategies used, methods and criteria of quality assessment, results (percentage of sites or pages rated as inadequate pertaining to a quality criterion), and quality and rigor of study methods and reporting. Data Synthesis Most frequently used quality criteria used include accuracy, completeness, readability, design, disclosures, and references provided. Fifty-five studies (70%) concluded that quality is a problem on the Web, 17 (22%) remained neutral, and 7 studies (9%) came to a positive conclusion. Positive studies scored significantly lower in search (P = .02) and evaluation (P = .04) methods. Conclusions Due to differences in study methods and rigor, quality criteria, study population, and topic chosen, study results and conclusions on health-related Web sites vary widely. Operational definitions of quality criteria are needed. The Internet has become an important mass medium for consumers seeking health information and health care services online.1 A recent concern and public health issue has been the quality of health information on the World Wide Web. However, the scale of the problem and the "epidemiology" (distribution and determinants) of poor health information on the Web are still unclear, as is their impact on public health and the question of whether poor health information on the Web is a problem at all.2 Many studies have been conducted to describe, critically appraise, and analyze consumer health information on the Web. These typically report proportions of inaccurate or imperfect information as estimates of the prevalence of flawed information or the risk of encountering misinformation on the Web. However, to date no systematic and comprehensive synthesis of the methodology and evidence has been attempted. Two previous systematic reviews focused on compiling quality criteria and rating instruments, but did not synthesize evaluation results. Jadad and Gagliari3 reviewed non–research-based rating systems (eg, criteria used for "awards," "seals of approval," or gateways) published on the Internet but did not include rating systems published in the peer-reviewed literature. Kim et al4 compiled instruments or criteria proposed to evaluate Web sites, but it is unclear to what extent and with what results these criteria can be and have been applied in practice. To our knowledge, this is the first systematic review conducted to compile criteria actually used and to synthesize evaluation results from studies containing quantitative data on structure and process measures of the quality of health information on the Web. The objectives of this study were to establish a methodological framework on how "quality" on the Web is evaluated in practice, to determine the heterogeneity of the results and conclusions, to compare the methodological rigor of these studies, to determine to what extent the conclusions depend on the methodology used, and to suggest future directions for research. Methods Inclusion and Exclusion Criteria Inclusion and exclusion criteria were established in advance in a written protocol. Studies were included when their authors: (1) searched the World Wide Web systematically for health information (eg, to identify all consumer Web sites for a given topic) or clearly defined a set of specific health information services to be included; (2) evaluated information likely to be accessed by consumers; (3) evaluated the quality of health information or services against certain criteria—for example, by judging the authority of source; assessing the accuracy of information, readability, or comprehensiveness; or by comparing Internet information and services against those outside of the Internet; and (4) provided quantitative results, such as proportions of Web sites complying with the quality criteria, or the distribution of quality scores, or information on the readability level(s) in the sample. Studies were excluded if authors simply listed "quality" Web sites without indicating that they had performed a comprehensive, systematic search, if they did not list quality criteria, or if they did not quantitate how many Web sites were assessed in total or how many did not comply with quality criteria. We also excluded studies assessing tools that enable or facilitate access to health information, such as search engines, directories, health portals, rating systems, and review services, or studies comparing different methods of information retrieval. Theoretical papers that only described quality criteria alone were also excluded, as were qualitative descriptions of Internet contents. We did not include studies that dealt with content other than health information, or studies evaluating information not intended for consumers (eg, those providing continuing medical education), but we did include studies evaluating services and information likely to be used by both professionals and consumers (eg, those providing drug information). Studies evaluating a single site or application were excluded, as were studies focusing exclusively on privacy and security issues, or on interactions via newsgroups or e-mail. Search Strategy We sought all relevant studies and unpublished reports, regardless of language or peer-review/publication status. We searched MEDLINE and PREMEDLINE (1966-September 2001) by entering the following query into the PubMed interface on September 20, 2001 (quality OR reliability OR accuracy OR readability OR evaluation OR assessment) AND (information OR education OR advice) AND (internet OR web OR ehealth OR "e-health" OR cyber* OR www). We retrieved 1545 citations and their titles and abstracts were screened for potential relevance by 2 independent reviewers. In addition, 1 reviewer used analogous search terms to screen citations in LISA (Library and Information Science Abstracts Database, 1969-July 2001) (1269 hits), CINAHL Nursing and Allied Health (1982-July 2001) (2312 hits), PsychINFO (American Psychological Association, 1988-September 2001 [week 2]) (321 hits), EMBASE (Elsevier Science BV, Amsterdam/NL, 1988-June 2001) (647 hits), and SIGLE (Grey Literature in Europe database, 1980-June 2001) (83 hits). The Web of Science database (Institute for Scientific Information, Philadelphia, Pa), which includes the Science Citation Index (1997-September 2001), the Social Sciences Citation Index (1997-September 2001), and the Arts and Humanities Citation Index (1997-September 2001) was searched using both a traditional backward search strategy (1120 hits) as well as using a forward citation-search strategy, using 5 seminal papers in this field3-7 as "seed publications" to identify all publications that subsequently cited 1 of these papers (533 hits). We also checked the references of all identified studies, hand-searched the Journal of Medical Internet Research (because it contains pertinent studies and because at that time it was not yet indexed on MEDLINE), conducted general Internet searches using combinations of the search terms at the Google and Northern Light search engines, reviewed 3 bibliographies of consumer health information studies collated by academic institutions, and used private literature databases. In total, we screened 7830 citations and retrieved 165 potentially eligible full articles to determine whether they met the inclusion criteria. Articles in Hungarian, Japanese, French, Spanish, Dutch, and Italian were translated by professional translators into English or German. Data Extraction Two reviewers (G.E., J.P.) independently extracted study characteristics using electronic data extraction forms, maintaining all data in an Access database (Microsoft Corp, Redmond, Wash). Each extraction form contained 85 items, of which 43 were closed questions, mostly to be answered on a scale (yes/no/partially/not applicable/to be discussed). We extracted information on search strategies used, quality criteria applied, and methodology of quality assessment used by authors, as well as quality and rigor of study methodology and reporting (see below and Table 1). We rated the overall tone of the authors' conclusions as "negative" if authors were wary and pessimistic about the Web as a source for health information, "positive" if authors did not express concern or recommended the Web as a source for health information, or "neutral" if authors discussed both risks and benefits, or if the reviewers disagreed. To extract evaluation results of each study, one reviewer tabulated data on how many Web sites or Web pages were evaluated for each quality criterion and how many of those evaluated did not comply with the respective criterion. Statistical Analysis To determine interobserver reliability for all coded items, we calculated standard κ values for each extracted item reported on a binary scale (eg, include study: yes/no), or weighted κ values for ordinal outcomes (eg, study characteristics graded on a scale [yes/partially/no]). We decided in advance not to include items with κ values less than 0.6.8 For the final consensus coding used for analysis, reviewers examined all disagreements or items coded "to be discussed" and resolved them by extensive discussion. We used meta-analysis of proportions9 to test for homogeneity of results for quality criteria that have been used by 3 or more authors to assess Web sites. We pooled study results for a given quality criterion if the assumption of homogeneity could not be rejected on a significance level of less than .05. To test whether the different conclusions of studies could be explained by their different quality scores we compared study assessment scores of study groups using the Wilcoxon test and logistic regression. All calculations were performed using SAS versions 6.12 and 8.2 (SAS Institute Inc, Cary, NC). Quality Scores We summarized the quality of reporting and the rigor of the studies by checking whether authors reported certain items deemed important for a well-reported, systematic, and comprehensive Internet search strategy (ie, S-score for search quality; range, 0-8 [transformed to percentage score]), and items thought to indicate a rigorous evaluation (ie, E-score for evaluation quality; [range, 0-10 [transformed to percentage score]). To ascertain face validity, an initial list of candidate items was compiled by 2 reviewers (G.E., J.P.) independently and later combined by consensus. As studies systematically evaluating Web information resemble in some ways systematic research overviews, the candidate criteria for the S-score partly derived from corresponding checklists.10 To ascertain score reliability, we eliminated items for which κ was less than 0.6; the remaining items are shown in Table 1. One point was given per item reported in order to calculate a raw summary score for search quality and evaluation quality. The final reported S-score and E-score in Online Table A (available in PDF format) represent the percentages of the maximum score achievable for each study (because some items are not applicable for some studies). To test construct validity we compared the scores of 5 letters to the editor with the scores of the 74 full publications. As expected, the letters had, on average, lower S-scores (28.8 vs 44.2, P = .06), but similar E-scores (22.4 vs 33.5, P = .45). The 7 full articles published in peer-reviewed non–English-language journals had lower average E-scores than articles published in English-language journals (mean E-score: 18.6 vs 34.8, P = .09), while their S-scores were similar (mean S-score: 42.3 vs 44.5, P = .85). While these differences are not significant at a 5% level, perhaps due to the small study number, the trend goes in the expected direction. Results Included Studies Of 170 articles reviewed, 85 met the inclusion criteria11-5960-95 (Online Table A available in PDF format). These constitute 79 distinct studies, as 6 studies were published either in duplicate,26,27,50,51,92,93 in a fragmented manner,87,88 or were reported both in a peer-reviewed publication and in an unpublished form.14,15,68,69 In these cases we pooled the results from the different articles and referred to these as a single study. Table 1 summarizes and Online Table A (available in PDF format) lists, all included studies. Quality Criteria and Methods Used to Evaluate Web Sites Technical Criteria (T). "Technical" quality criteria were defined as general, domain-independent criteria, ie, criteria referring to the question of how the information was presented or what meta-information was provided. The 24 technical quality criteria most frequently used (Table 2) are variations of what could be called "transparency criteria" from the print world, mentioned by Silberg et al7: authorship, attribution, disclosure, and currency. While the latter is strictly speaking a dimension of accuracy, almost all studies sought for provision of a date of creation or last update (rather than actual currency of the content), which is a technical criterion. Design (D). Fifteen studies also evaluated subjective design features such as the visual aspect of the site or layout, but only a few studies reported results (Online Table B available in PDF format), presumably because of their subjectivity and low reliability. For example, Gillois et al39 used a visual analog scale to assess the quality of visual presentation and of the interface, but noted high interobserver variability. Stausberg et al81 report a κ of 0.08 when evaluating navigation, and 0.23 when evaluating layout. Speed, browser compatibility, and presence of a search engine were considered technical criteria. Readability (R). Eleven studies used readability formulas to establish the reading level of a document based on the complexity and length of words and sentences (Online Table B available in PDF format). Nine studies used the Flesch-Kincaid (FK) Grade Level Index. Other formulas used include the SMOG Readability Formula, the Fry Readability Graph, the Gunning-Fog formula, and the Lexile Framework. As different formulas yield different reading levels for the same document, and as authors use different cutoffs, the results cannot be pooled, but studies suggest that reading levels are frequently too high (Online Table B available in PDF format).65 Using reading formulas has limitations, as readability scores do not reflect other factors that affect comprehension such as frequency and explanation of medical jargon, writing style (use of active voice, nonpatronizing language, motivational messages, tone/mood, how it relates to the audience), or use of culturally specific information. Few studies analyzed these important but subjective aspects—Oermann and Wilson66 discussed some of these parameters, Fitzmaurice and Adams35 scored the writing style, and Wilson et al94 examined the cultural sensitivity of the documents to various ethnic groups using Bloch's Ethnic/Cultural Assessment Tool. None of the studies conducted comprehension tests with actual consumers or used judgments of literacy experts. Accuracy (A). Accuracy can be defined as the degree of concordance of the information provided with the best evidence or with generally accepted medical practice. Alternative terms used for the concept of "accuracy" include "reliability"51 or "conventionality of information."79 We coded 2 studies as assessing accuracy "partially," because authors claimed to have evaluated "value and amount of text" without further specifying how they define "value."36,90 Authors who appraised the accuracy and/or comprehensiveness could either define clear criteria beforehand (a priori) or extract information from the Web first and then check these claims against the literature (a posteriori). Thirty studies assessed accuracy a posteriori, and only 19 a priori (Online Table A available in PDF format). Studies defining elements a priori varied considerably in the granularity and specificity of the items defined in advance: for example, 1 study stated that "paracetamol should be given in a dose of 10 to 15 mg/kg every 4 hours,"51 while others just predefined broad keywords that should be mentioned, eg, "oestrogen."75 Accuracy ideally should be defined using the best available evidence.5 Fourteen studies used evidence-based guidelines or systematic reviews to define elements a priori and 1 study used a guideline as the criterion standard a posteriori. Eleven studies used primary literature for a posteriori comparison; however, with 1 exception89 these studies did not provide details on how and what literature was screened, leaving open the possibility that some authors did not actually search the literature, but just compared information against their own knowledge of the literature. Textbooks or expert consensus were used as the criterion standard in 3 a priori and in 3 a posteriori studies.29,54,55,64,70,84 The remaining studies used unclear sources or the personal opinion of the author as the a priori (2 studies) or a posteriori (15 studies) criterion standard. Completeness/Comprehensiveness/Coverage/Scope (C). Several methods were used to evaluate "completeness," "comprehensiveness," "coverage," or "scope." Most authors12-15,22,29,35,37,42,44-46,50,51,54-56,58,59,70,75,78,79,85,92,93,95 calculated a proportion of a priori–defined elements covered by a Web site or reported the proportion of Web sites that mentioned all key elements (eg, from a clinical guideline). Willems and Bouvy92,93 used a 5-point scale to evaluate completeness. One study11 evaluated "balance," eg, whether the adverse effects and contraindications as well as the advantages of a drug are presented, but detailed subcriteria were not made explicit. The Soot score13,46,58,79 addresses completeness by measuring coverage of topic areas defined a priori, weighting some areas as being more important than others. While 19 studies evaluated completeness as a distinct entity, 21 other studies evaluated completeness as an integral part of accuracy (marked "AC" in Online Table A available in PDF format). For example, Impicciatore et al51 reported that only 4 of 44 Web sites "adhered closely to the guidelines," implying that these sites were both accurate and complete, whereas Davison27 only assessed the accuracy of statements made on the site against the respective statement in the guideline without requiring that all recommendations from nutritional guidelines appear on the Web site. While it is possible to evaluate accuracy without demanding completeness, studies in which authors just prepared checklists for completeness sometimes raise the question as to how and to what extent accuracy has been evaluated.29,35,54,55,75 For example, the Soot79 score is a weighted checklist giving points for each broad topic covered by the site (such as "treatment options"), but does not specify whether topics were merely present or absent or if they were evaluated for accuracy. Of the 19 studies that evaluated completeness as a distinct entity, 5 studies used an external source a priori to define elements: 2 used evidence-based guidelines, 2 used primary literature, and 1 used a textbook. The remaining studies used unclear sources or the personal opinion of the author, either a priori (7 studies) or a posteriori (7 studies). Score Systems Twenty-two studies11,13-15,23,31,35,36,38,42-44,46,58,59,70,72,74,75,78,79,87,90 used a composite score system, most frequently the Soot score,13,46,58,79 for adherence to quality criteria. Two studies38,87 used DISCERN96 and another used Suitability Assessment of Materials (SAM),65 both described by their developers as "validated" instruments to assess printed patient education material; however, none of these have been validated in a way such that a higher score would predict better health outcomes or consumer satisfaction. Quality of Studies The quality scores for each study are shown in Online Table A (available in PDF format). Out of the 79 studies, 49 received an S-score and an E-score of 50 or less, indicating that both search strategy and evaluation were not well reported or rigorous. However, of the 31 studies using more than 1 rater, 20 made efforts to ascertain the within-study reliability of their instruments, which is in contrast to what has been criticized in a previous study about Web-based instruments.97 Study Conclusions and Study Quality Most studies (55 [70%]) concluded that quality is a problem on the Internet (κ for coding conclusions, 0.88). Seventeen (21.5%) were neutral. Only 7 (9%) came to a more positive conclusion, none of which used evidence-based guidelines as a criterion standard. The mean S-score of the positive studies was significantly lower than that of the negative studies (29.4 vs 45.0; P = .02), as was the E-score (15.6 vs 37.9; P = .04), indicating that the more enthusiastic studies used a less rigorous search and evaluation strategy than did negative studies. In a logistic regression model with both scores as predictive variables and the study conclusion "positive/negative" as the dependent variable, the odds ratio (OR) for S-score is 0.929 (95% confidence interval [CI], 0.871-0.991; P = .02), ie, for each additional S-score percentage point the odds to reach a positive conclusion decreases by 7.1%; the OR for E-score is 0.938 (95% CI, 0.884-0.996; P = .04), ie, for each additional E-score percentage point the odds for a positive conclusion decreases by 6.2%. When studies evaluated accuracy, the proportions of inaccurate Web sites depended on the level of evidence used as a criterion standard: studies that didn't report the criterion standard or that used personal opinion found an average of 15.4% of Web sites inaccurate; those using literature, textbooks, or expert consensus, 35.3%; and those using clinical guidelines, 38.3%. In an ordinal logistic regression model, the reported proportions of inaccurate Web sites were significantly associated with the 3 levels of evidence used by authors as criterion standard (P<.001). Even if authors had similar results, they sometimes interpreted these differently. For example, 3 studies found that about 5% of cancer Web sites provided inaccurate information, but these results were interpreted as being "of concern,"16 "encouraging,"76 or even "reassuring."46 In at least 1 case investigators reviewed the same topic area, but arrived at opposite conclusions.28,75 Comment We reviewed 79 studies in which authors evaluated a total of 5941 Web sites and 1329 Web pages, and reported 408 evaluation results for 86 distinct quality criteria (Online Table B available in PDF format). Content Quality In our review, most authors who evaluated content found significant problems, criticizing lack of completeness, difficulty in finding high-quality sites, and lack of accuracy, in particular if "accuracy" also implied "completeness." Five of eight studies reporting results on completeness found that around 90% of Web sites were "incomplete" (Online Table B available in PDF format). However, completeness as a requirement has questionable validity from the perspective of the user or the public health researcher. First, too much information may overburden users. Web sites may deliberately and with good reason focus on a single topic in-depth rather than aiming for comprehensiveness. Second, in contrast to printed educational material, a single Web page or Web site is part of a universe of information: a topic not covered by one Web page or site may be covered by another (perhaps linked) Web page. Consumers will usually search across different Web sites when looking for specific health information.98 Mechanical comparison of elements from a guideline with elements covered by a single Web site without taking into account the context and purpose of the site or exploring links to other sites is of limited use. Perhaps a better approach would be to evaluate whether materials cover the topics they claim to be discussing96,99 and if they are balanced. Comparisons Across Studies Prevalence figures of inaccurate Web sites differ across different domains, eg, diet and nutrition sites (45.5%26 and 88.9%64 inaccurate information) vs cancer sites (4% for prostate cancer,46 5.1% for breast cancer,76 9% for English-language and 4% for Spanish-language breast cancer documents,14 6% for testicular cancer,46 or 6.2% for Ewing Sarcoma16). While such prevalence figures may suggest that diet information on the Web is of poorer quality than cancer information, unadjusted comparisons across studies have to be made with care, as at least 3 potential confounders should be considered. First, results heavily depend on the rigor of the methodology used: studies that used personal opinion as a criterion standard found fewer inaccurate Web sites and more often came to a positive conclusion than studies using more rigorous criteria. Second, many studies use the terms "Web site" and "Web page" interchangeably, making comparisons difficult. Third, different and often poorly described sampling and selection strategies were used. While in theory a truly random sample of Web sites could be identified (by choosing random Internet protocol [IP] addresses100), this approach is not practical for identifying Web sites for a given topic. Thus, all studies used search engines, catalogs, or lists of popular Web sites. However, the choice of the search strategy may greatly confound the results. Most studies mimicked how consumers would search (although only 3 involved actual consumers), hand-selecting popular sites. As there is no consensus and little research on how a typical consumer searches,98 studies used various strategies, mostly picking the top-ranked results from a search engine. As many search engines can rank the better sites first, the search tool could influence the results. Moreover, even the same search engine may give a different result if different sampling strategies (ie, which sites are picked) are used. For example, Suarez-Almazor et al83 showed that, in the Webcrawler search engine, the first 20 ranked hits are more relevant, have less financial interests, contain less alternative therapies, and are more often nonprofit organizations, than sites ranked lower. Moreover, the selection of search terms may critically determine which Web sites are retrieved. For example, using the term "coronary heart disease," Eachus31 found only 2 sites (among 110) provided by lay people. His conclusion that "the concern that the Internet would be a major source of low-quality health information, particularly that provided by unqualified members of the lay public, is not supported by [our] findings" may be confounded by the choice of search terms, as lay people might not necessarily use terms such as "coronary heart disease." Who devises the search strategy and conducts the search can also affect the quality of the retrieved sites. For example, in a comparison of results of a search devised by doctors with a search devised by information experts, Groot et al43 observed differences in the credibility and accuracy scores of the retrieved samples. Comparison With Other Media The quality of Web sites should be interpreted in the larger context of information in other media to determine whether the Web is "the beginning of an epidemic of misinformation or nothing more than a variation of what is endemic."2 Many of the shortcomings detected likely are not specific to the Web and are also present in other media. For example, 2 of the 4 "erroneous information" elements found on 65 Web pages identified by Biermann et al16 were found in the online version of the Encyclopaedia Britannica, which probably has the same inaccuracies in the printed edition. This issue, and the relatively low prevalence (6.2%) of inaccurate information, was generally ignored when this study was widely quoted in the lay media as evidence that the "Internet can be a quick link to bad health information."101 Studies assessing information in traditional media also frequently report high prevalences of inaccurate or incomplete information. In an early study, authors found 70% of health information broadcast on television to be inaccurate, misleading, or both.102 In another study, authors rated as inaccurate 76% of the information about oral hygiene from television, 53% from magazines, and 12% from newspapers.103 Another study of the popular press found 20% of the information on oral cancer to be a "mix of accurate and inaccurate information."104 The proportion of inaccurate press reports on healthy eating was found to be 55% in free advertising newspapers, 28.9% in lifestyle magazines, 29.9% in general interest magazines, 17.5% in health magazines, and 14.1% in newspapers.105 In another study, 50% of the advice in newspaper advice columns was rated inappropriate, with critical issues only partially covered or not covered at all in 76% of the articles, and 58% were rated unsafe or potentially dangerous.106 Inquiries by telephone to libraries yielded a 3.6% rate of inaccurate information.107 The perceived quality problem on the Internet is not restricted to the health sector: a study investigating the quality of general scientific information found that 10% to 34% was inaccurate, 20% to 35% was misleading, and 48% to 90% was unreferenced.108 Very few studies in our sample directly compared Internet information with information found elsewhere. One study35 evaluated both printed and Web-based patient education materials, concluding that "there was no significant difference between the ranges of scores [incorporating content, writing style, design, and readability] for Internet and non-Internet leaflets"; however, subjectively authors felt that "the overall quality of the Internet leaflets was more variable and the information less comprehensive." Two studies compared the readability of Internet information with the readability of printed information. One study33 concluded that SMOG readability levels of Internet information were significantly higher compared with other (printed) patient information materials; the other study65 observed no differences in the proportion of patient information written above the 9th grade level, but 87.5% of Web information materials vs only 14.3% of printed patient information, were deemed unsuitable based on the Suitability Assessment of Materials score (however, as the 8 Web-based patient education materials came from only 2 different Web sites, this result may not be representative). We identified only 1 study that compared the accuracy of advice obtained on the Internet (in a pharmacy newsgroup, hence this study was excluded from this review) with advice obtained in the real world (from drug information centers).109 In summary, the prevalence figures of inaccurate or incomplete Web information reported by the studies in this review are difficult to interpret or compare, are unlikely to be representative or generalizable, and must also be considered against the background of imperfect consumer health information in other media. Presentation Quality The presentation criteria evaluated by most investigators are considered to be quality criteria because: (1) their presence is deemed ethical according to several codes of conduct for Web publishing110-114 (eg, transparency and accountability criteria, such as disclosure of authorship); (2) they help to create context and to avoid misunderstandings (eg, disclosure of sponsorship, purpose); (3) they empower users to select the information that is best for them in their individual situation (eg, disclosure of target audience); (4) they empower users to validate the information themselves (eg, references, contact addresses); or (5) they may influence the accessibility of information or effectiveness of communication (eg, search capabilities, speed, design). In addition, several authors have attempted to establish whether technical criteria or other site characteristics may be used as predictors for content quality,11,42,47,58,87,91 but the results have been conflicting and inconclusive. A number of studies found, perhaps unsurprisingly, that the length of a document is correlated with better content scores.42,58,87 A few studies suggest that the source may be a predictor for content quality, with commercial sites often scoring lower than academic sites,59,74,75 but the way in which content is evaluated may also influence this relationship. Other technical criteria influenced by the source include references, which are more often found on Web sites targeting medical professionals,23 on academic46,79 or educational89 sites, or on sites owned by organizations.42 Individual authors often are not disclosed on government sites14 or sites of organizations and drug companies.42 Given these complex relationships, it seems unlikely that a simple scoring system could be developed that predicts content quality across domains and site populations. As with content criteria, the ability to compare or pool results across studies is impeded by wide variations in sampling strategies, methods, and operational definitions even for the same quality criteria (eg, what constitutes "attribution"). Moreover, there is no consensus on the unit of evaluation (eg, should the update cycle be published on each Web page, or is a site-specific disclosure sufficient) or granularity with which information should be provided (eg, is it sufficient to post the year of last update, or should the exact date be given). Despite this lack of consensus, some quality criteria are consistently given (eg, "ownership disclosure," with 99% of sites providing information) or not given (eg, "credentials of physicians not disclosed," with 97.5% not complying) (Table 2). As with content criteria, presentation criteria should be put into context by comparison with non–Web-based information. While in our study 64% of Web sites failed to provide a date of update, investigators evaluating printed patient leaflets have found that 53%,35 or, in another study, one-third,115 did not include publication dates. Omissions Since most authors used criteria derived from the print world,7 we noted important omissions concerning Web-specific criteria. For example, very few studies evaluated the privacy policy or the possibility to encrypt confidential information, and few studies checked whether the target audience or the target country (which is important in a global medium116) were clearly disclosed. None of the studies tested usability117 or accessibility (eg, compliance with guidelines of the Web Accessibility Initiative, ascertaining that the site is available to people with disabilities or with low-end technology), and only 1 study checked whether metadata were provided,76 which could greatly enhance the consumer's ability to select and filter information.6 Measuring Progress of Health Communication on the Web Among the public health objectives of the US Department of Health and Human Services is to "increase the proportion of health-related World Wide Web sites that disclose information that can be used to assess the quality of the site."118 How progress in this area might be achieved or evaluated is an open question. A meta-analysis of cross-sectional studies is one possible way, but one must take into account the different methods used (or the methods must be standardized) before meaningful conclusions can be drawn. Longitudinal studies could be used to assess changes over time, using a consistent methodology either to identify and assess Web sites for a given topic (with the caveat that observed changes may be changes in the ability of users to find better sites, ie, improvements in search engine technology to rank better sites first) or to follow up a cohort of Web sites. Two studies have attempted a longitudinal approach, with conflicting results; one observed an improvement, and the other a deterioration, in quality.36,57,90 The third possibility is to promote among site developers the use of machine-processable disclosure statements as Web site labels (meta-data),6,119 which would allow automatic tracking and analysis of the proportion and characteristics of Web sites making such disclosure statements. This would also facilitate the development of intelligent systems able to guide users to trustworthy Web sites.120 Conclusion The epidemiology of consumer health information on the Web is an emerging research discipline at the intersection of medical informatics and public health. Many descriptive, cross-sectional studies have attempted to draw attention to perceived "outbreaks" of misinformation on the Web by estimating the proportion or prevalence of inadequate health information. However, the individual's risk (R) of encountering an inadequate site on the Web is a function of both the proportion of inadequate information on the Web (P) and the inability (I) of the individual (or his tools) to filter the inadequate sites. Since studies usually report R, but not I, we cannot infer P, or adjust study results to make them comparable across domains or time. Even if we could know P, we would still not know how this measurement of "true" misinformation on the Web translates into health outcomes or critical incidents in a population. On an individual level, R can be reduced by improving the ability of the user to locate trustworthy sites or to filter the inadequate ones. Public e-health interventions such as MedCERTAIN121 therefore strive to reduce P and I by increasing, for example, the proportion of health information providers making disclosure statements119 and by empowering consumers to identify trusted sites through educational and technological innovations, including the possibilities of the semantic Web.120 Given the difficulty in interpreting descriptive studies without control or comparison groups, future studies should use analytic rather than descriptive approaches to investigate the relationship between quality markers and other variables, including outcomes, or to compare different Web site populations or media. Such studies are urgently needed to help in the ongoing process to develop methods and instruments to guide consumers to quality information and to identify factors that can be assessed to predict favorable patient outcomes. Finally, studies evaluating content should also harness the potential of the Web as a source for qualitative data122: rather than getting bogged down by the question of how much information is inaccurate, one could analyze where and why gaps exist between evidence-based medicine and health information on the Internet, which may elicit a wealth of valuable data that may inform priorities for research, health communication, and education. References 1. Eysenbach G. Consumer health informatics. BMJ.2000;320:1713-1716.Google Scholar 2. Coiera E. Information epidemics, economics, and immunity on the Internet. BMJ.1998;317:1469-1470.Google Scholar 3. Jadad AR, Gagliardi A. Rating health information on the Internet. JAMA.1998;279:611-614.Google Scholar 4. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ.1999;318:647-649.Google Scholar 5. Wyatt JC. Commentary: measuring quality and impact of the World Wide Web. BMJ.1997;314:1879-1881.Google Scholar 6. Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet. BMJ.1998;317:1496-1500.Google Scholar 7. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet. JAMA.1997;277:1244-1245.Google Scholar 8. Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics.1977;33:159-174.Google Scholar 9. Bohning D, Dammann UP. The application of methods of meta-analysis for heterogeneity modeling in quality control and assurance. In: Schulze R, Holling H, Bohning D, eds. Meta-analysis: New Developments and Applications in Medical and Social Sciences. Seattle, Wash: Hogrefe & Huber. In press. 10. Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol.1991;44:1271-1278.Google Scholar 11. Abbott VP. Web page quality: can we measure it and what do we find? J Public Health Med.2000;22:191-197.Google Scholar 12. Armstrong K, Schwartz JS, Asch DA. Direct sale of sildenafil (Viagra) to consumers over the Internet. N Engl J Med.1999;341:1389-1392.Google Scholar 13. Beredjiklian PK, Bozentka DJ, Steinberg DR, Bernstein J. Evaluating the source and content of orthopaedic information on the Internet. J Bone Joint Surg Am.2000;82-A:1540-1543.Google Scholar 14. Berland GK, Elliott MN, Morales LS. et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA.2001;285:2612-2621.Google Scholar 15. Berland GK, Morales LS, Elliott MN. et al. Evaluation of English and Spanish Health Information on the Internet. Santa Monica: RAND/California HealthCare Foundation; 2001. Available at: http://www.rand.org/publications/documents/interneteval/. Accessibility verified April 25, 2002. 16. Biermann JS, Golladay GJ, Greenfield ML, Baker LH. Evaluation of cancer information on the Internet. Cancer.1999;86:381-390.Google Scholar 17. Bloom BS, Iannacone RC. Internet availability of prescription pharmaceuticals to the public. Ann Intern Med.1999;131:830-833.Google Scholar 18. Bogenschutz MP. Drug information libraries on the Internet. J Psychoactive Drugs.2000;32:249-258.Google Scholar 19. Boyer EW, Shannon M, Hibberd PL. Web sites with misinformation about illicit drugs. N Engl J Med.2001;345:469-471.Google Scholar 20. Breul H, Boue L, Martin A. Panorama des ressources documentaires hospitaliéres françaises sur Internet [Overview of Internet information resources of interest to French hospitals]. Semaine des Hopitaux.1999;75:257-262.Google Scholar 21. Butzke I, Kramer KL. Orthopadie im World Wide Web: Universitats-Homepages der G7-Staaten. Z Orthop Ihre Grenzgeb.2000;138:413-418.Google Scholar 22. Bykowski JL, Alora MB, Dover JS, Arndt KA. Accessibility and reliability of cutaneous laser surgery information on the World Wide Web. J Am Acad Dermatol.2000;42:784-786.Google Scholar 23. Chen LE, Minkes RK, Langer JC. Pediatric surgery on the Internet. J Pediatr Surg.2000;35:1179-1182.Google Scholar 24. Corpron CA, Lelli Jr JL. Evaluation of pediatric surgery information on the Internet. J Pediatr Surg.2001;36:1187-1189.Google Scholar 25. D'Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med.2001;155:807-812.Google Scholar 26. Davison K. The quality of dietary information on the World Wide Web. J Can Dietetic Assoc.1996;57:137-141.Google Scholar 27. Davison K. The quality of dietary information on the World Wide Web. Clin Perform Qual Health Care.1997;5:64-66.Google Scholar 28. Diering CL, Palmer MH. Professional information about urinary incontinence on the World Wide Web. J Wound Ostomy Continence Nurs.2001;28:55-62.Google Scholar 29. Doupi P, van der Lei J. Rx medication information for the public and the WWW: quality issues. Med Inform Internet Med.1999;24:171-179.Google Scholar 30. Dracos A, Seta M. Egeszsegugyi informaciok az Interneten laikusoknak: olaszorszagi tapasztalatok. [Health information for the lay person on the Internet: a survey of the Italian experience]. Tudomanyos es Muszaki Tajekoztatas.1998;45:118-120.Google Scholar 31. Eachus P. Health information on the Internet. Int J Health Prom Educ.1999;37:30-33.Google Scholar 32. Ellamushi H, Narenthiran G, Kitchen ND. Is current information available useful for patients and their families? Ann R Coll Surg Engl.2001;83:292-294.Google Scholar 33. Estrada CA, Hryniewicz MM, Higgs VB, Collins C, Byrd JC. Anticoagulant patient information material is written at high readability levels. Stroke.2000;31:2966-2970.Google Scholar 34. Eysenbach G. Online prescribing of sildanefil (Viagra) on the World Wide Web. J Med Internet Res.1999;1:e10. Available at: http://www.jmir.org/1999/2/index.htm. Accessibility verified March 25, 2002.Google Scholar 35. Fitzmaurice DA, Adams JL. A systematic review of patient information leaflets for hypertension. J Hum Hypertens.2000;14:259-262.Google Scholar 36. Frasca D, Malezieux R, Mertens P, Neidhardt JPH, Voiglio EJ. Review and evaluation of anatomy sites on the Internet (updated 1999). Surg Radiol Anat.2000;22:107-110.Google Scholar 37. Galimberti A, Jain S. Gynaecology on the Net. J Obstet Gynaecol.2000;20:297-299.Google Scholar 38. Gillies M. Access to the information should be made easier. BMJ.2000;321:47-48.Google Scholar 39. Gillois P, Colombet I, Dreau H, Degoulet P, Chatellier G. A critical appraisal of the use of Internet for calculating cardiovascular risk. Proc AMIA Symp.1999;:775-779.Google Scholar 40. Gordon JB, Barot LR, Fahey AL, Matthews MS. The Internet as a source of information on breast augmentation. Plast Reconstr Surg.2001;107:171-176.Google Scholar 41. Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract.1999;48:58-61.Google Scholar 42. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ.2000;321:1511-1515.Google Scholar 43. Groot D, Riet G, Khan KS, Misso K. Comparison of search strategies and quality of medical information of the Internet: a study relating to ankle sprain. Injury.2001;32:473-476.Google Scholar 44. Harmon D, Duggan M, Flynn N. Anaesthesia on the World Wide Web. Anaesthesia.2000;55:728-729.Google Scholar 45. Hatfield CL, May SK, Markoff JS. Quality of consumer drug information provided by four Web sites. Am J Health Syst Pharm.1999;56:2308-2311.Google Scholar 46. Hellawell GO, Turner KJ, Le Monnier KJ, Brewster SF. Urology and the Internet: an evaluation of Internet use by urology patients and of information available on urological topics. BJU Int.2000;86:191-194.Google Scholar 47. Hernández-Borges AA, Macías-Cervi P, Gaspar-Guardado MA, Torres-Álvarez de Arcaya ML, Ruiz-Rabaza A, Jiménez-Sosa A. Can examination of WWW usage statistics and other indirect quality indicators help to distinguish the relative quality of medical websites? J Med Internet Res.1999;1:e1. Available at: http://www.jmir.org/1999/1/e1/index.htm. Accessibility verified March 25, 2002.Google Scholar 48. Hersh WR, Gorman PN, Sacherek LS. Applicability and quality of information for answering clinical questions on the Web. JAMA.1998;280:1307-1308.Google Scholar 49. Hoffman-Goetz L, Clarke JN. Quality of breast cancer sites on the World Wide Web. Can J Public Health.2000;91:281-284.Google Scholar 50. Impicciatore P, Pandolfini C, Casella N, Bonati M. Possiamo fidarci delle informazioni sanitarie al pubblico che viaggiano su Internet? l'esempio della gestione domiciliare della febbre nel bambino. [Reliability of public-oriented health care resources in the World Wide Web: the case of the home management of fever in children]. Ricerca e Pratica.1997;13:67-75.Google Scholar 51. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web. BMJ.1997;314:1875-1879.Google Scholar 52. Jiang YL. Quality evaluation of orthodontic information on the World Wide Web. Am J Orthod Dentofacial Orthop.2000;118:4-9.Google Scholar 53. Kihlstrom LC. Evaluating pharmacy benefit management information on the Internet. Manag Care Interface.2001;14:64-68.Google Scholar 54. Latthe PM, Latthe M, Khan KS. Quality of medical information about menorrhagia on the worldwide web. BJOG.2000;107:39-43.Google Scholar 55. Latthe M, Latthe PM, Charlton R. Quality of information on emergency contraception on the internet. Br J Fam Plann.2000;26:39-43.Google Scholar 56. Latthe PM, Latthe M, Khan KS. Quality of information on female sterilisation on the Internet. J Obstet Gynaecol.2000;20:167-170.Google Scholar 57. Li L, Irvin E, Guzman J, Bombardier A. Surfing for back pain patients. Spine.2001;26:545-557.Google Scholar 58. Libertiny G, Perkins JM, Magee TR, Galland RB. Varicose veins on the internet. Eur J Vasc Endovasc Surg.2000;20:386-389.Google Scholar 59. Lissman TL, Boehnlein JK. A critical review of internet information about depression. Psychiatr Serv.2001;52:1046-1050.Google Scholar 60. Mallory C. What's on the Internet? services for women affected by HIV and AIDS. Health Care Women Int.1997;18:315-322.Google Scholar 61. Martinez-Lopez JF, Ruiz-Crespo EJ. Internet: calidad de la informacion sobre cirugia [Internet: quality of information available on orthopaedic surgery and traumatology]. Revista de Ortopedia y Traumatologia.1998;42:469-473.Google Scholar 62. Maugans TA, McComb JG, Levy ML. The internet as a pediatric neurosurgery information resource. Pediatr Neurosurg.1998;28:186-190.Google Scholar 63. McClung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics.1998;101:e2.Google Scholar 64. Miles J, Petrie C, Steel M. Slimming on the Internet. J R Soc Med.2000;93:254-257.Google Scholar 65. Murphy PW, Chesson AL, Berman SA, Arnold CL, Galloway G. Neurology patient education materials. J Neurosci Nurs.2001;33:99-104.Google Scholar 66. Oermann MH, Wilson FL. Quality of care information for consumers on the Internet. J Nurs Care Qual.2000;14:45-54.Google Scholar 67. O'Mahony B. Irish health care Web sites: a review. Ir Med J.1999;92:334-337.Google Scholar 68. Ogushi Y, Tatsumi H. Research on the Analysis of the Current State of the Provision and Use of Health Information Provided Through a New Technology Medium [in Japanese]. Tokyo, Japan: Health & Welfare Ministry Research Group; 2000. 69. Tatsumi H, Mitani H, Haruki Y, Ogushi Y. Internet medical usage in Japan. J Med Internet Res.2001;3:e12. Available at: http://www.jmir.org/2001/1/e12. Accessibility verified April 24, 2002.Google Scholar 70. Pandolfini C, Impicciatore P, Bonati M. Parents on the web. Pediatrics.2000;105:e1.Google Scholar 71. Payne C, Miller K. The quality of footcare information on the Internet. Australas J Podiatr Med.2000;34:63.Google Scholar 72. Peroutka SJ. Analysis of Internet sites for headache. Cephalalgia.2001;21:20-24.Google Scholar 73. Roberts JR, Spooner SA. Pediatric Internet resources. Arch Pediatr Adolesc Med.1997;151:592-597.Google Scholar 74. Sacchetti P, Zvara P, Plante MK. The Internet and patient education—resources and their reliability. Urology.1999;53:1117-1120.Google Scholar 75. Sandvik H. Health information and interaction on the Internet. BMJ.1999;319:29-32.Google Scholar 76. Shon J, Musen MA. The low availability of metadata elements for evaluating the quality of medical information on the World Wide Web. Proc AMIA Symp.1999;:945-949.Google Scholar 77. Sing A, Salzman J, Sing D. Problems and risks of unsolicited e-mails in patient-physician encounters in travel medicine settings. J Trav Med.2001;8:109-112.Google Scholar 78. Smith M, Gertz E, Alvarez S, Lurie P. The content and accessibility of sex education information on the Internet. Health Educ Behav.2000;27:684-694.Google Scholar 79. Soot LC, Moneta GL, Edwards JM. Vascular surgery and the Internet. J Vasc Surg.1999;30:84-91.Google Scholar 80. Stausberg J, Fuchs J. Die chirurgische Fachabteilung im World Wide Web: Tribut an den Zeitgeist oder Informationsdrehscheibe? [Surgical specialty department in the World Wide Web: tribute to contemporary life style or information network?]. Chirurg.2000;71:472-477.Google Scholar 81. Stausberg J, Fuchs J, Husing J, Hirche H. Health care providers on the World Wide Web. Med Inform Internet Med.2001;26:17-24.Google Scholar 82. Stone TW, Jumper JM. Information about age-related macular degeneration on the Internet. South Med J.2001;94:22-25.Google Scholar 83. Suarez-Almazor ME, Kendall CJ, Dorgan M. Surfing the Net—information on the World Wide Web for persons with arthritis. J Rheumatol.2001;28:185-191.Google Scholar 84. Tamm EP, Raval BK, Huynh PT. Evaluation of the quality of self-education mammography material available for patients on the Internet. Acad Radiol.2000;7:137-141.Google Scholar 85. Thompson JM, Howard JE. HMO Web sites. Manag Care Q.2000;8:33-41.Google Scholar 86. Tu F, Zimmerman NP. It is not just a matter of ethics: a survey of the provision of health disclaimers, caveats, and other health-related alerts in consumer health information on eating disorders on the Internet. Int Inf Libr Rev.2001;32:325-339.Google Scholar 87. Türp JC, Gerds T, Neugebauer S. Myoarthorpathien des Kausystems: Beurteilung der Qualität von Patienteninformationen im Weltweiten Netz. Z Arztl Fortbild Qualitatssich.2001;95:539-547.Google Scholar 88. Neugebauer S, Türp JC. Vorsicht Internet! Webseiten uber myoarthropathien des kausystems [Attention Internet! Web sites on myoarthropathies of the mastication system]. Schweiz Monatsschr Zahnmed.2001;111:298-311.Google Scholar 89. Veronin MA, Ramirez G. The validity of health claims on the World Wide Web. Am J Health Promot.2000;15:21-28.Google Scholar 90. Voiglio E, Frasca D, Malezieux R, Moreau S, Rodier MN, Neidhardt JPH. Prospecting and evaluation of the anatomy sites on the internet. Surg Radiol Anat.1999;21:65-68.Google Scholar 91. von Danwitz F, Baehring T, Scherbaum W. Verbreitung, Gestaltung und Qualität von Deutschsprachigen World Wide Web Seiten zum Diabetes Mellitus. Düsseldorf, Germany: Deutsches Diabetes Forschungsinstitut; 1999. 92. Willems M, Bouvy ML. Gezondheidsinformatie op Internet: ze adviseren gewoon de spullen die op voorraad zijn. Pharmaceutisch Weekblad.2001;136:607-609.Google Scholar 93. Willems M, Bouvy ML. Gezondheidsinformatie op het internet : de kwaliteit van websites vergeleken. Medisch Contact.2001;56:673-676.Google Scholar 94. Wilson FL, Baker LM, Brown-Syed C, Gollop C. An analysis of the readability and cultural sensitivity of information on the National Cancer Institute's Web site: CancerNet. Oncol Nurs Forum.2000;27:1403-1409.Google Scholar 95. Wright B, Williams C, Partridge I. Management advice for children with chronic fatigue syndrome: a systematic study of information from the Internet. Ir J Psychol Med.1999;16:67-71.Google Scholar 96. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health.1999;53:105-111.Google Scholar 97. Delamothe T. Quality of websites: kitemarking the west wind. BMJ.2000;321:843-844.Google Scholar 98. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the World-Wide-Web? BMJ.2002;324:573-577.Google Scholar 99. Dalton L, Gartenfeld E. Evaluating printed health information for consumers. Bull Med Libr Assoc.1981;69:322-324.Google Scholar 100. Lawrence S, Giles CL. Accessibility of information on the web. Nature.1999;400:107-109.Google Scholar 101. Internet can be quick link to bad health information. CNN.July 29, 1999. Available at: http://www.cnn.com/HEALTH/9907/29/internet.health. Accessibility verified April 25, 2002.Google Scholar 102. Smith FA, Trivax G, Zuehlke DA, Lowinger P, Nghiem TL. Health information during a week of television. N Engl J Med.1972;286:516-520.Google Scholar 103. Frazier PJ, Jenny J, Ostman R, Frenick C. Quality of information in mass media: a barrier to the dental health education of the public. J Public Health Dent.1974;34:244-257.Google Scholar 104. Canto MT, Kawaguchi Y, Horowitz AM. Coverage and quality of oral cancer information in the popular press: 1987-98. J Public Health Dent.1998;58:241-247.Google Scholar 105. Institut für Ernährungswissenschaften der Universität Wien. Österreichischer Ernährungsbericht.Vienna, Austria: Bundesministerium für Arbeit, Gesundheit und Soziales und der Bundesministerin für Frauenangelegenheiten und Verbraucherschutz; 1998.Google Scholar 106. Molnar FJ, Man-Son-Hing M, Dalziel WB. et al. Assessing the quality of newspaper medical advice columns for elderly readers. CMAJ.1999;161:393-395.Google Scholar 107. Paskoff BM. Accuracy of telephone reference service in health sciences libraries. Bull Med Libr Assoc.1991;79:182-188.Google Scholar 108. Allen ES, Burke JM, Welch ME, Rieseberg LH. How reliable is science information on the web? Nature.1999;402:722.Google Scholar 109. Seaboldt JA, Kuiper R. Comparison of information obtained from a Usenet newsgroup and from drug information centers. Am J Health Syst Pharm.1997;54:1732-1735.Google Scholar 110. Risk A, Dzenowagis J. Review of Internet health information quality initiatives. J Med Internet Res.2001;3:e28. Available at: http://www.jmir.org/2001/4/e28.Google Scholar 111. Boyer C, Selby M, Scherrer JR, Appel RD. The Health on the Net Code of Conduct for medical and health Websites. Comput Biol Med.1998;28:603-610.Google Scholar 112. Hi-Ethics Group. Health Internet Ethics: Ethical Principles For Offering Internet Health Services to Consumers. 2000. Available at: http://www.hiethics.org/Principles/. Accessibility verified March 25, 2002. 113. e-Health Ethics Initiative. e-Health Code of Ethics. J Med Internet Res.2000;2:e9. Available at: http://www.jmir.org/2000/2/e9/. Accessibility verified March 25, 2002.Google Scholar 114. Winker MA, Flanagin A, Chi-Lum B. et al. Guidelines for medical and health information sites on the Internet. JAMA.2000;283:1600-1606.Google Scholar 115. Coulter A, Entwistle V, Gilbert D. Sharing decisions with patients: is the information good enough? BMJ.1999;318:318-322.Google Scholar 116. Eysenbach G, Diepgen TL. Labeling and filtering of medical information on the Internet. Methods Inf Med.1999;38:80-88.Google Scholar 117. Nielsen J. Usability Engineering. San Diego, Calif: Morgan Kaufmann; 1993. 118. US Department of Health and Human Services. Healthy People 2010: Understanding and Improving Health. 2nd ed. Washington, DC: Office of Disease Prevention and Health Promotion, US Dept of Health and Human Services; 2000. Publication 017-001-00547-9. Available at: http://www.health.gov/healthypeople/document/. Accessibility verified March 25, 2002. 119. Eysenbach G, Köhler C, Yihune G, Lampe K, Cross P, Brickley D. A metadata vocabulary for self- and third-party labeling of health web-sites. Proc AMIA Annu Fall Symp.2001:169-173.Google Scholar 120. Eysenbach G. An ontology of quality initiatives and a model for decentralized, collaborative quality management on the (semantic) World-Wide-Web. J Med Internet Res.2001;3:e34. Available at: http://www.jmir.org/2001/4/e34. Accessibility verified April 24, 2002.Google Scholar 121. Eysenbach G, Köhler C, Yihune G, Lampe K, Cross P, Brickley D. A framework for improving the quality of health information on the world-wide-web and bettering public (e-)health. Medinfo2001, Proceedings of the Tenth World Congress on Medical Informatics, London, England, September 2001 . Amsterdam, Holland: IOS Press; 2001:1450-1454. 122. Eysenbach G, Till JE. Ethical issues in qualitative research on Internet communities. BMJ.2001;323:1103-1105.Google Scholar

Journal

JAMA – American Medical Association

Published: May 22, 2002

Keywords: internet

There are no references for this article.