Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews

The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews Abstract Background: Traditional and largely qualitative reviews of evidence are now giving way to much more structured systematic overviews that use a quantitative method to calculate the overall effect of treatment. The latter approach is dependent on the quality of primary studies, which may introduce bias if they are of poor methodologic quality. Objective: To test the hypothesis that the inclusion of poor-quality trials in meta-analyses would bias the conclusions and produce incorrect estimates of treatment effect. Methods: An overview of randomized trials of antiestrogen therapy in subfertile men with oligospermia was performed to test the hypothesis. Data sources included online searching of MEDLINE and Science Citation Index databases between 1966 and 1994, scanning the bibliography of known primary studies and review articles, and contacting experts in the field. After independent, blind assessment, nine of 149 originally identified studies met the inclusion criteria and were selected. We assessed study quality independently. Outcome data from each study were pooled and statistically summarized. Results: There was a marginal improvement in pregnancy rate with antiestrogen treatment (odds ratio, 1.6; 95% confidence interval, 0.9 to 2.6). Sensitivity analyses on the basis of methodologic quality demonstrated that poor-quality studies produced a positive effect with treatment, whereas no benefit was observed with high-quality studies. Conclusions: The results of a meta-analysis are influenced by the quality of the primary studies included. Methodologically, poor studies tend to exaggerate the overall estimate of treatment effect and may lead to incorrect inferences.(Arch Intern Med. 1996;156:661-666) References 1. Eddy DM. Principles for making difficult decisions in difficult times . JAMA. 1994;271:1792-1798.Crossref 2. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine . JAMA. 1992;268:2420-2425.Crossref 3. Sackett DL. The Cochrane Collaboration. ACP J Club. 1994;120:A11. 4. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodologic quality associated with estimates of treatment effect in controlled clinical trials . JAMA. 1995;273:408-412.Crossref 5. Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials . N Engl J Med. 1983;309:1358-1361.Crossref 6. Olkin I. A critical look at some popular meta-analytic methods . Am J Epidemiol. 1994;140:297-299. 7. Greenland S. Quality scores are useless and potentially misleading . Am J Epidemiol. 1994;140:300-301. 8. Greenland S. A critical look at some popular meta-analytic methods . Am J Epidemiol. 1994;140:290-298. 9. Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues and future directions. Int J Technol Assess Health Care. In press. 10. Vandekerckhove P, O'Donovan PA, Lilford RJ, Harada TW. Infertility treatment: from cookery to science: the epidemiology of randomized controlled trials . Br J Obstet Gynaecol. 1993;100:1005-1036.Crossref 11. Gilbaugh JH, Lipshultz LI. Nonsurgical treatment of male infertility . Urol Clin North Am. 1994;21:531-548. 12. O'Donovan P, Vandekerckhove P, Lilford R, Hughes E. Treatment of male infertility: is it effective? review and meta-analysis of published randomized controlled trials . Hum Reprod. 1993;8:1209-1222. 13. Oxman AD, Guyatt GH. Guidelines for reading literature reviews . Can Med Assoc J. 1988;138:697-703. 14. Jadad AR. Meta-analysis of Randomised Clinical Trials in Pain Relief. Oxford, England: University of Oxford; 1994. Thesis. 15. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit . Psychol Bull. 1968;70:213-220.Crossref 16. Sokol RZ, Petersen G, Steiner BS, Swerdloff RS, Bustillo M. A controlled comparison of the efficacy of clomiphene in male infertility . Fertil Steril. 1988;49:865-870. 17. Ronnberg L. The effect of clomiphene citrate on different sperm parameters and serum hormone levels in preselected infertile men: a controlled double-blind cross-over study . Int J Androl. 1980;3:479-486.Crossref 18. Micic S, Doltic R. Evaluation of sperm parameters in clinical trial with clomiphene citrate of oligospermic men . J Urol. 1985;133:221-222. 19. Wang C, Chan CW, Wong KK, Yeung KK. Comparison of the effectiveness of placebo, clomiphene citrate, mesterolone, pentoxifylline, and testosterone rebound therapy for the treatment of idiopathic oligospermia . Fertil Steril. 1983; 40:358-365. 20. Abel BJ, Carswell G, Elton R, et al. Randomised trial of clomiphene citrate treatment and vitamin C for male infertility . Br J Urol. 1982;54:780-784.Crossref 21. AinMelk Y, Belisle S, Carmel M, Jean-Pierre T. Tamoxifen citrate in male infertility . Fertil Steril. 1987;48:113-117. 22. Torok L. Treatment of oligospermia with tamoxifen . Andrologia. 1985;17:497-501.Crossref 23. Krause W, Holland-Moritz H, Schramm P. Treatment of idiopathic oligozoospermia with tamoxifen: a randomized controlled study . Int J Androl. 1992; 15:14-18.Crossref 24. World Health Organization. A double-blind trial of clomiphene citrate for the treatment of idiopathic male infertility . Int J Androl. 1992;15:299-307.Crossref 25. Kotoulas IG, Cardamakis E, Michopoulos J, Mitropoulos D, Dounis A. Tamoxifen treatment in male infertility, I: effect on spermatozoa . Fertil Steril. 1994; 61:911-914. 26. Willis KJ, London DR, Bevis MA, Butt WR, Lynch SS, Holder G. Hormonal effects of tamoxifen in oligospermic men . J Endocrinol. 1977;73:171-178.Crossref 27. Paulson DF. Cortisone acetate versus clomiphene citrate in pre-germinal idiopathic oligospermia . J Urol. 1979;121:432-434. 28. Floss GL, Tindall VR, Birkett JP. The treatment of subfertile men with clomiphene citrate . J Reprod Fertil. 1973;32:167-170.Crossref 29. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease . J Natl Cancer Inst 1959;22:719-748. 30. Breslow NE, Day NE. Statistical Methods in Cancer Research: The Analysis of Case Control Studies . Lyons, France: LARC Scientific Publications; 1980;1. 31. Imperiale TF, McCullough AJ. Do corticosteroids reduce mortality from alcoholic hepatitis? a meta-analysis of randomised trials . Ann Intern Med. 1990; 113:299-307.Crossref 32. Brown SA. Meta-analysis of diabetes patient education research: variations in intervention effects across studies . Res Nurs Health. 1992;15:409-419.Crossref 33. Nurmohamed MT, Rosendaal FR, Buller HR, et al. Low-molecular-weight heparin versus standard heparin in general and orthopedic surgery: a meta-analysis . Lancet. 1992;340:152-156.Crossref 34. Gotzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis . Control Clin Trials. 1989;10:31-56.Crossref 35. Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM. Evidence favoring the use of anticoagulants in hospital phase of acute myocardial infarction . N Engl J Med. 1977;297:1091-1096.Crossref 36. Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparison of therapy, I: medical . Stat Med. 1989;8:441-454.Crossref 37. Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparison of therapy, II: surgical . Stat Med. 1989;8:455-466.Crossref 38. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer . J Clin Oncol. 1986;4:942-951. 39. Orwin RG. A fail-safe N for effect size in meta-analysis . J Educ Stat. 1983;8:157-159.Crossref 40. Rosenthal R. The file drawer problem and tolerance for null results . Psychol Bull. 1979;86:638-641.Crossref http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Archives of Internal Medicine American Medical Association

The Importance of Quality of Primary Studies in Producing Unbiased Systematic Reviews

Loading next page...
 
/lp/american-medical-association/the-importance-of-quality-of-primary-studies-in-producing-unbiased-bLjy1QeGMG
Publisher
American Medical Association
Copyright
Copyright © 1996 American Medical Association. All Rights Reserved.
ISSN
0003-9926
eISSN
1538-3679
DOI
10.1001/archinte.1996.00440060089011
Publisher site
See Article on Publisher Site

Abstract

Abstract Background: Traditional and largely qualitative reviews of evidence are now giving way to much more structured systematic overviews that use a quantitative method to calculate the overall effect of treatment. The latter approach is dependent on the quality of primary studies, which may introduce bias if they are of poor methodologic quality. Objective: To test the hypothesis that the inclusion of poor-quality trials in meta-analyses would bias the conclusions and produce incorrect estimates of treatment effect. Methods: An overview of randomized trials of antiestrogen therapy in subfertile men with oligospermia was performed to test the hypothesis. Data sources included online searching of MEDLINE and Science Citation Index databases between 1966 and 1994, scanning the bibliography of known primary studies and review articles, and contacting experts in the field. After independent, blind assessment, nine of 149 originally identified studies met the inclusion criteria and were selected. We assessed study quality independently. Outcome data from each study were pooled and statistically summarized. Results: There was a marginal improvement in pregnancy rate with antiestrogen treatment (odds ratio, 1.6; 95% confidence interval, 0.9 to 2.6). Sensitivity analyses on the basis of methodologic quality demonstrated that poor-quality studies produced a positive effect with treatment, whereas no benefit was observed with high-quality studies. Conclusions: The results of a meta-analysis are influenced by the quality of the primary studies included. Methodologically, poor studies tend to exaggerate the overall estimate of treatment effect and may lead to incorrect inferences.(Arch Intern Med. 1996;156:661-666) References 1. Eddy DM. Principles for making difficult decisions in difficult times . JAMA. 1994;271:1792-1798.Crossref 2. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine . JAMA. 1992;268:2420-2425.Crossref 3. Sackett DL. The Cochrane Collaboration. ACP J Club. 1994;120:A11. 4. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methodologic quality associated with estimates of treatment effect in controlled clinical trials . JAMA. 1995;273:408-412.Crossref 5. Chalmers TC, Celano P, Sacks HS, Smith H Jr. Bias in treatment assignment in controlled clinical trials . N Engl J Med. 1983;309:1358-1361.Crossref 6. Olkin I. A critical look at some popular meta-analytic methods . Am J Epidemiol. 1994;140:297-299. 7. Greenland S. Quality scores are useless and potentially misleading . Am J Epidemiol. 1994;140:300-301. 8. Greenland S. A critical look at some popular meta-analytic methods . Am J Epidemiol. 1994;140:290-298. 9. Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials: current issues and future directions. Int J Technol Assess Health Care. In press. 10. Vandekerckhove P, O'Donovan PA, Lilford RJ, Harada TW. Infertility treatment: from cookery to science: the epidemiology of randomized controlled trials . Br J Obstet Gynaecol. 1993;100:1005-1036.Crossref 11. Gilbaugh JH, Lipshultz LI. Nonsurgical treatment of male infertility . Urol Clin North Am. 1994;21:531-548. 12. O'Donovan P, Vandekerckhove P, Lilford R, Hughes E. Treatment of male infertility: is it effective? review and meta-analysis of published randomized controlled trials . Hum Reprod. 1993;8:1209-1222. 13. Oxman AD, Guyatt GH. Guidelines for reading literature reviews . Can Med Assoc J. 1988;138:697-703. 14. Jadad AR. Meta-analysis of Randomised Clinical Trials in Pain Relief. Oxford, England: University of Oxford; 1994. Thesis. 15. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit . Psychol Bull. 1968;70:213-220.Crossref 16. Sokol RZ, Petersen G, Steiner BS, Swerdloff RS, Bustillo M. A controlled comparison of the efficacy of clomiphene in male infertility . Fertil Steril. 1988;49:865-870. 17. Ronnberg L. The effect of clomiphene citrate on different sperm parameters and serum hormone levels in preselected infertile men: a controlled double-blind cross-over study . Int J Androl. 1980;3:479-486.Crossref 18. Micic S, Doltic R. Evaluation of sperm parameters in clinical trial with clomiphene citrate of oligospermic men . J Urol. 1985;133:221-222. 19. Wang C, Chan CW, Wong KK, Yeung KK. Comparison of the effectiveness of placebo, clomiphene citrate, mesterolone, pentoxifylline, and testosterone rebound therapy for the treatment of idiopathic oligospermia . Fertil Steril. 1983; 40:358-365. 20. Abel BJ, Carswell G, Elton R, et al. Randomised trial of clomiphene citrate treatment and vitamin C for male infertility . Br J Urol. 1982;54:780-784.Crossref 21. AinMelk Y, Belisle S, Carmel M, Jean-Pierre T. Tamoxifen citrate in male infertility . Fertil Steril. 1987;48:113-117. 22. Torok L. Treatment of oligospermia with tamoxifen . Andrologia. 1985;17:497-501.Crossref 23. Krause W, Holland-Moritz H, Schramm P. Treatment of idiopathic oligozoospermia with tamoxifen: a randomized controlled study . Int J Androl. 1992; 15:14-18.Crossref 24. World Health Organization. A double-blind trial of clomiphene citrate for the treatment of idiopathic male infertility . Int J Androl. 1992;15:299-307.Crossref 25. Kotoulas IG, Cardamakis E, Michopoulos J, Mitropoulos D, Dounis A. Tamoxifen treatment in male infertility, I: effect on spermatozoa . Fertil Steril. 1994; 61:911-914. 26. Willis KJ, London DR, Bevis MA, Butt WR, Lynch SS, Holder G. Hormonal effects of tamoxifen in oligospermic men . J Endocrinol. 1977;73:171-178.Crossref 27. Paulson DF. Cortisone acetate versus clomiphene citrate in pre-germinal idiopathic oligospermia . J Urol. 1979;121:432-434. 28. Floss GL, Tindall VR, Birkett JP. The treatment of subfertile men with clomiphene citrate . J Reprod Fertil. 1973;32:167-170.Crossref 29. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease . J Natl Cancer Inst 1959;22:719-748. 30. Breslow NE, Day NE. Statistical Methods in Cancer Research: The Analysis of Case Control Studies . Lyons, France: LARC Scientific Publications; 1980;1. 31. Imperiale TF, McCullough AJ. Do corticosteroids reduce mortality from alcoholic hepatitis? a meta-analysis of randomised trials . Ann Intern Med. 1990; 113:299-307.Crossref 32. Brown SA. Meta-analysis of diabetes patient education research: variations in intervention effects across studies . Res Nurs Health. 1992;15:409-419.Crossref 33. Nurmohamed MT, Rosendaal FR, Buller HR, et al. Low-molecular-weight heparin versus standard heparin in general and orthopedic surgery: a meta-analysis . Lancet. 1992;340:152-156.Crossref 34. Gotzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis . Control Clin Trials. 1989;10:31-56.Crossref 35. Chalmers TC, Matta RJ, Smith H Jr, Kunzler AM. Evidence favoring the use of anticoagulants in hospital phase of acute myocardial infarction . N Engl J Med. 1977;297:1091-1096.Crossref 36. Colditz GA, Miller JN, Mosteller F. How study design affects outcomes in comparison of therapy, I: medical . Stat Med. 1989;8:441-454.Crossref 37. Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparison of therapy, II: surgical . Stat Med. 1989;8:455-466.Crossref 38. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer . J Clin Oncol. 1986;4:942-951. 39. Orwin RG. A fail-safe N for effect size in meta-analysis . J Educ Stat. 1983;8:157-159.Crossref 40. Rosenthal R. The file drawer problem and tolerance for null results . Psychol Bull. 1979;86:638-641.Crossref

Journal

Archives of Internal MedicineAmerican Medical Association

Published: Mar 25, 1996

References