Background: QUALICOPC is an international survey of primary care performance. QUALICOPC data have been used in several studies, yet the representativeness of the Canadian QUALICOPC survey is unknown, potentially limiting the generalizability of findings. This study examined the representativeness of QUALICOPC physician and patient respondents in Ontario using health administrative data. Methods: This representativeness study linked QUALICOPC physician and patient respondents in Ontario to health administrative databases at the Institute for Clinical Evaluative Sciences. Physician respondents were compared to other physicians in their practice group and all Ontario primary care physicians on demographic and practice characteristics. Patient respondents were compared to other patients rostered to their primary care physicians, patients rostered to their physicians’ practice groups, and a random sample of Ontario residents on sociodemographic characteristics, morbidity, and health care utilization. Standardized differences were calculated to compare the distribution of characteristics across cohorts. Results: QUALICOPC physician respondents included a higher proportion of younger, female physicians and Canadian medical graduates compared to other Ontario primary care physicians. A higher proportion of physician respondents practiced in Family Health Team models, compared to the provincial proportion for primary care physicians. QUALICOPC patient respondents were more likely to be older and female, with significantly higher levels of morbidity and health care utilization, compared with the other patient groups examined. However, when looking at the QUALICOPC physicians’ whole rosters, rather than just the patient survey respondents, the practice profiles were similar to those of the other physicians in their practice groups and Ontario patients in general. Conclusions: Comparisons revealed some differences in responding physicians’ demographic and practice characteristics, as well as differences in responding patients’ characteristics compared to the other patient groups tested, which may have resulted from the visit-based sampling strategy. Ontario QUALICOPC physicians had similar practice profiles as compared to non-participating physicians, providing some evidence that the participating practices are representative of other non-participating practices, and patients selected by visit-based sampling may also be representative of visiting patients in other practices. Those using QUALICOPC data should understand this limited representativeness when generalizing results, and consider the potential for bias in their analyses. Keywords: Primary care, Survey bias, Canada, Representativeness * Correspondence: firstname.lastname@example.org Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada Department of Family & Community Medicine, University of Toronto, Toronto, Canada Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Li et al. BMC Family Practice (2018) 19:77 Page 2 of 10 Background distributed by practice staff to consecutive consenting Ongoing primary care reform in Canada and around the adult patients visiting the practice that day. world has spurred a need for comprehensive and mean- While QUALICOPC represents the largest study on ingful measurement of primary care performance . This quality, organization, and patient values and experiences is the case for primary care in the Canadian province of in primary care in Canada, limited resources were avail- Ontario, where, despite being publicly funded and central able for provider recruitment and response rates for to the health care system, there is a paucity of high quality physicians across the country were generally low, ran- data on primary care performance . ging from 2% in British Columbia to 21% in Nova Scotia Surveys are an important source of information in . In Ontario, Canada’s most populous province, the health services research, policy, and planning. However, response rate was 3% . With low physician response physician surveys often have low response rates, which rates from a self-selected sample, and the corresponding may introduce concern about their validity and repre- patient sample consisting of consecutive visit-based sam- sentativeness . While response rate is sometimes used pling, it is unknown to what extent the QUALICOPC as a marker of survey quality, Halbesleben and Whitman physician and patient respondents can be generalized to advocate for looking beyond response rates when assessing the province. If the respondents are not representative the quality of survey data . They recommend examining of the province, then are they representative of meaning- nonresponse bias, which occurs when there is a systematic ful subgroups, such as the other physicians and patients difference between those who do and do not respond to a in the same practice? These comparisons can help deter- survey . One common method of assessing nonresponse mine to what extent physician, patient, or practice-level bias in physician surveys is to compare respondents and inferences are valid. nonrespondents, on the basis of demographic and practice The objective of the current study was to examine the characteristics [5–13]. These comparisons have identified representativeness of QUALICOPC physician and pa- differences in responding physicians compared to nonre- tient respondents in Ontario by answering the following spondents, including differences in age, gender, and years questions: of schooling [7, 8]. There is limited research exploring nonresponse bias 1) To what extent are the QUALICOPC physician in primary care patient surveys. Some studies have found respondents representative of i) the physicians in that patient surveys have potential for biased results due their practice group, and ii) other primary care to method of survey administration [14–17]. One study physicians in Ontario? identified differences in gender, income, and age among 2) To what extent are the QUALICOPC patient patients who responded to a survey in the waiting room respondents representative of i) the patients in their compared to those who responded by e-mail . An- physicians’ rosters, ii) the patients in their other found telephone survey response rates differed by physicians’ groups’ rosters, and iii) the general patients’ age, tobacco use, and comorbidity scores . Ontario population? The Quality and Costs of Primary Care study (QUALICOPC) is a multi-national study on primary care Methods performance investigating quality, equity, and costs of pri- Measures and data sources mary care in 34 countries worldwide [2, 19]. Details of the This representativeness study linked a database of design and administration of the Canadian QUALICOPC QUALICOPC survey respondents to health administrative can be found elsewhere [2, 20]. Briefly, research teams databases held at the Institute for Clinical Evaluative from each province in Canada collected data for the Sciences (ICES). Data holdings at ICES include a number QUALICOPC study in 2013 and early 2014. The of databases with information on providers and patients, QUALICOPC study included practice, physician, and pa- such as physician billings, hospital inpatient and emer- tient surveys. In Ontario, a random sample of physicians gency room care, and census data [17, 21]. Universal was not possible as researchers did not have access to a health insurance in Canada means that the databases list of eligible physicians. Instead, emails were sent by capture the whole population. Physician databases and the Ontario College of Family Physicians inviting eligible public data have been used to examine survey represen- physicians working in general/family practice to register tativeness [7, 22], while census data has been used to for the study. The family physician survey was completed examine representativeness of an EMR database . by the participating physician and the practice survey Administrative data were captured from January 1, was completed by either the participating physician or 2013 to December 31, 2013, corresponding to the period of administrative staff. For the patient surveys, physicians data collection in Ontario. For the physician cohorts, were instructed to choose a day they felt represented 185 physicians completed QUALICOPC surveys in Ontario, their regular practice population, and surveys were and we were successful in linking 175 (95%) of these Li et al. BMC Family Practice (2018) 19:77 Page 3 of 10 respondents with administrative data using unique Demographic information for patients, including age, encrypted physician billing numbers. Using group number, sex, and postal code, was derived from the Registered a variable identifying groups of physicians practicing to- Person’s Database (RPDB). Health care utilization was gether, we then identified the other primary care physicians derived from the National Ambulatory Care Reporting belonging to the same practice groups as the survey re- System (NACRS), the Canadian Institute of Health Infor- spondents. We also identified the remaining primary care mation Discharge Abstract Database (CIHI-DAD), and the physicians in Ontario. Ontario Health Insurance Plan (OHIP). Of the 1698 patients who completed the QUALICOPC Rurality of the patients was measured using the Rurality patient experience survey, 1225 (72%) consented to link- Index of Ontario (RIO), a scale which assigns a number age to health administrative databases using their health score between 0 and 100 using postal codes and an algo- card numbers. We then identified the other patients in rithm which takes into account population density and their physicians’ rosters, as well as the other patients in travel times to referral centres. RIO scores of 0–9were their physicians’ practice groups’ rosters. We included considered urban, 10–39 as suburban, and 40 or greater patients who were formally or virtually rostered to the as rural . primary care physicians; formally rostered patients had Material deprivation of the patients was measured signed an enrollment form while virtually rostered pa- using the Canadian Marginalization Index, which is de- tients were those who saw a particular physician for the rived geographically from census data and includes mea- majority of their visits over the previous year. Finally, in sures such as proportion of the population without a order to construct a provincially representative sample high school diploma, proportion of households living in of patients for comparison, we also determined a 10% dwellings that are in need of major repair, and propor- simple random sample of all patients in Ontario aged 18 tion of the population above the age of 15 who are un- and older with a valid health card number. employed . The physician cohorts were compared on demographic To account for the morbidity burden of the patients, characteristics, including sex, age, years since graduation, resource utilization bands (RUBs) were used. RUBs are and whether they were Canadian medical graduates. part of the Johns Hopkins Adjusted Clinical Groups They were also compared on type of primary care prac- Casemix system and are derived from hospitalization and tice model they were practicing in, and roster size. The primary care visit records. RUBs range from 0 (non-users patient cohorts were compared on sociodemographic of the health care system) to 5 (very high users) . The characteristics, including sex, age, material deprivation, and prevalence of five specific chronic diseases was deter- rurality, as well as morbidity and health care utilization, in- mined using validated cohort databases at ICES: asthma, cluding primary care visits, emergency department visits, chronic obstructive pulmonary disease (COPD), congest- and acute care hospitalizations. These variables are com- ive heart failure (CHF), hypertension, and diabetes. monly found to vary among respondents and nonrespon- dents in other studies. Furthermore, these variables may be Analysis related to primary care performance and patient experi- The standardized difference, also known as effect size, ence, and are thus important to examine in the context of was calculated to compare the means and proportions of a primary care performance survey [16, 18]. variables across the physician and patient comparison Demographic and practice information for physicians groups. The standardized difference was selected as it is was derived from the ICES Physician Database (IPDB) not as sensitive to large sample sizes, such as those in and Client Agency Program Enrolment (CAPE) tables. our study, as traditional significance tests and it also Primary care models were also derived from CAPE, and provides information about the relative magnitude of classified according to type of practice (solo vs. group) differences between groups . Consistent with Cohen and remuneration: solo physicians (including enhanced (1988, as described in ), we considered a standard- fee for service and fee for service), group enhanced fee ized difference of 0.2 to indicate a small, but meaningful for service (i.e. Family Health Group), group capitated difference between groups. All analyses were conducted (i.e. Family Health Organization), and group capitated in SAS version 9.4. with an allied health team (i.e. Family Health Team). Family Health Network and Other group models were Results not included in the analysis as they each had fewer Physician respondents than 6 physician respondents in the QUALICOPC. Data from 175 physician QUALICOPC respondents See Additional file 1 for a summary of primary care were compared to 2507 physicians in the same practice models in Ontario. Since solo physicians, by definition, do groups, and 9758 Ontario primary care physicians not belong to a practice group, they were only compared (Table 1). Physician respondents were, on average, younger, to the other Ontario primary care physicians. had fewer years of experience, and consisted of a higher Li et al. BMC Family Practice (2018) 19:77 Page 4 of 10 Table 1 QUALICOPC physician respondents compared with physicians in their practice groups and Ontario primary care physicians Group 1: QUALICOPC Group 2: QUALICOPC physicians’ Group 3: Ontario primary Standardized difference physician respondents practice groups care physicians Group 2 vs. 1 Group 3 vs. 1 N = 175 N = 2507 N = 9758 Sex, N (%) Female 98 (56.0) 1177 (47.0) 4110 (42.1) 0.18 0.28 Male 77 (44.0) 1330 (53.0) 5642 (57.8) 0.18 0.28 Age, mean (SD) 49 (10) 51 (11) 51 (12) 0.19 0.20 Years in practice, mean (SD) 23 (11) 25 (12) 25 (13) 0.20 0.21 Canadian medical graduate, N (%) Yes 141 (80.6) 1878 (74.9) 7054 (72.3) 0.14 0.20 No 34 (19.4) 629 (25.1) 2698 (27.7) 0.14 0.20 Roster size, mean (SD) 1257 (582) 1126 (786) 1120 (1045) 0.19 0.16 Primary care model , N(%) Solo physicians 12 (6.9) 0 3711 (38.0) - 0.81 FHG 44 (25.1) 1117 (44.6) 2415 (24.8) 0.42 0.01 FHN < 6 27 (1.1) 202 (2.1) - - FHO 38 (21.7) 401 (16.0) 1765 (18.1) 0.15 0.09 FHT 73 (41.7) 923 (36.8) 1594 (16.3) 0.10 0.58 Other group < 6 39 (1.6) 71 (0.7) - - SD standard deviation, FHG Family Health Group, FHN Family Health Network, FHO Family Health Organization, FHT Family Health Team Clarify that standardized differences >=0.2 are considered a meaningful difference and are highlighted in italics Primary care models are classified according to type of practice model and remuneration: solo physicians (including enhanced fee for service and fee for service), group enhanced fee for service (i.e. Family Health Group), group capitated (i.e. Family Health Organization), and group capitated with an allied health team (i.e. Family Health Team). Family Health Network and Other group models were not included in the analysis as they each had fewer than 6 physician respondents in the QUALICOPC proportion of female physicians compared to the other patient populations in terms of material deprivation, with physicians in their practice groups, though these stan- 17% of respondents living in areas with high deprivation dardized differences were mostly below 0.2, with larger compared to 17% for the physicians’ rosters and 19% for differences when comparing respondents to the Ontario the province, and all standardized differences less than 0.2. primary care physicians. Survey respondents included a QUALICOPC survey respondents had more comorbidi- smaller proportion of physicians who attended medical ties as measured by RUBs than any of the other patient school abroad, with 19.4% international medical graduates populations. Survey respondents had a lower proportion of compared to 27.7% in Ontario. While roster sizes were “low morbidity,” and higher proportions of “high morbidity” comparable, survey respondents consisted of fewer solo and “very high morbidity” patients than comparator groups, physicians and more who practiced in Family Health with survey respondents including 24% “high morbidity,” Teams as compared to the Ontario average. compared to 15% in their physicians’ and practice groups’ rosters. Survey respondents had some differ- Patient respondents ences in terms of specific chronic conditions, demonstrat- In total, 1225 patient respondents to the QUALICOPC ing higher prevalence of asthma and hypertension study were compared to 158,537 patients within partici- compared to the province. However, there were not mean- pating physicians’ rosters, 2,270,380 patients rostered to ingful differences in COPD, CHF, or diabetes across the the participating physicians’ practice groups, and 831,056 comparison groups. patients representing a 10% simple random sample of Survey respondents were also more frequent users of Ontarians aged 18 years and older (Table 2). the health care system, with an average of 5.83 primary Patient survey respondents consisted of a greater pro- care visits per year, compared to an average of 3.46 visits portion of female patients compared to the population for the other patients in their physicians’ rosters, 3.69 in of Ontario. Survey respondents also included a lower the practice groups’ rosters, and the provincial average proportion of patients between the ages of 18 and 44, of 3.33. Emergency department visits and number of compared to their physicians’ rosters and the population hospitalizations demonstrated a similar trend, but stan- of Ontario. Patients surveyed did not differ from the other dardized differences were less than 0.2. Li et al. BMC Family Practice (2018) 19:77 Page 5 of 10 Table 2 QUALICOPC patient respondents compared with patients in their physicians’ rosters, practice groups’ rosters, and Ontario Group 1: QUALICOPC Group 2: Group 3: QUALICOPC Group 4: Ontario population, Standardized difference patient respondents QUALICOPC physicians’ practice 10% simple random sample Group 2 vs. 1 Group 3 vs. 1 Group 4 vs. 1 Group 3 vs. 2 Group 4 vs. 2 N = 1225 physicians’ groups’ rosters N = 831,056 rosters N = 2,270,380 N = 158,537 Sex, N (%) Female 782 (63.8) 88,682 (55.9) 1244,3224 (54.8) 420,085 (50.5) 0.16 0.18 0.27 0.02 0.11 Male 443 (36.2) 69,855 (44.1) 1,026,056 (45.2) 410,971 (49.5) 0.16 0.18 0.27 0.02 0.11 Age, N (%) 18–44 423 (34.5) 69,786 (44.0) 987,543 (43.5) 385,125 (46.3) 0.20 0.18 0.24 0.01 0.05 45–64 492 (40.2) 57,281 (36.1) 829,440 (36.5) 291,929 (35.1) 0.08 0.07 0.10 0.01 0.02 65+ 310 (25.3) 31,470 (19.9) 453,397 (20.0) 154,002 (18.5) 0.13 0.13 0.16 0.00 0.03 Material deprivation quintile, N (%) 1 (least deprived) 303 (25.1) 39,917 (25.7) 561,391 (25.1) 187,235 (22.9) 0.01 0.00 0.05 0.01 0.06 2 266 (22.0) 33,362 (21.4) 471,281 (21.1) 162,865 (20.0) 0.01 0.02 0.05 0.01 0.04 3 208 (17.2) 28,781 (18.5) 421,191 (18.9) 157,343 (19.3) 0.03 0.04 0.05 0.01 0.02 4 219 (18.1) 27,392 (17.6) 405,012 (18.1) 149,790 (18.4) 0.01 0.00 0.01 0.01 0.02 5 (most deprived) 211 (17.5) 26,084 (16.8) 373,716 (16.7) 158,614 (19.4) 0.02 0.02 0.05 0.00 0.07 Rurality Index of Ontario, N (%) < 10 (least rural) 795 (64.9) 104,620 (66.0) 1,688,247 (74.4) 608,395 (73.2) 0.02 0.21 0.18 0.18 0.16 10–40 316 (25.8) 39,638 (25.0) 463,532 (20.4) 155,884 (18.8) 0.02 0.13 0.17 0.11 0.15 40+ (most rural) 114 (9.3) 14,279 (9.0) 118,601 (5.2) 66,777 (8.0) 0.01 0.16 0.05 0.15 0.03 Resource utilization bands, N (%) 0 (non-user) 22 (1.8) 8856 (5.6) 133,031 (5.9) 92,008 (11.1) 0.20 0.21 0.38 0.01 0.20 1 (healthy user) 35 (2.9) 8746 (5.5) 120,487 (5.3) 49,519 (6) 0.13 0.12 0.15 0.01 0.02 2 (low morbidity) 87 (7.1) 26,594 (16.8) 376,574 (16.6) 141,200 (17) 0.30 0.30 0.31 0.01 0.01 3 (moderate morbidity) 662 (54.0) 81,632 (51.5) 1,185,228 (52.2) 397,248 (47.8) 0.05 0.04 0.13 0.01 0.07 4 (high morbidity) 292 (23.8) 24,256 (15.3) 339,211 (14.9) 111,158 (13.4) 0.22 0.23 0.27 0.01 0.05 5 (very high morbidity) 127 (10.4) 8453 (5.3) 115,849 (5.1) 39,923 (4.8) 0.19 0.20 0.21 0.01 0.02 Chronic disease, N (%) Asthma 255 (20.8) 22,792 (14.4) 329,256 (14.5) 112,173 (13.5) 0.17 0.17 0.20 0.00 0.03 COPD 56 (4.6) 4285 (2.7) 61,555 (2.7) 21,446 (2.6) 0.10 0.10 0.11 0.00 0.01 CHF 44 (3.6) 3523 (2.2) 51,127 (2.3) 18,228 (2.2) 0.08 0.08 0.08 0.00 0.00 Hypertension 446 (36.4) 43,269 (27.3) 640,891 (28.2) 213,398 (25.7) 0.20 0.18 0.23 0.02 0.04 Diabetes 206 (16.8) 18,653 (11.8) 277,755 (12.2) 93,806 (11.3) 0.14 0.13 0.16 0.01 0.01 Li et al. BMC Family Practice (2018) 19:77 Page 6 of 10 Table 2 QUALICOPC patient respondents compared with patients in their physicians’ rosters, practice groups’ rosters, and Ontario (Continued) Group 1: QUALICOPC Group 2: Group 3: QUALICOPC Group 4: Ontario population, Standardized difference patient respondents QUALICOPC physicians’ practice 10% simple random sample Group 2 vs. 1 Group 3 vs. 1 Group 4 vs. 1 Group 3 vs. 2 Group 4 vs. 2 N = 1225 physicians’ groups’ rosters N = 831,056 rosters N = 2,270,380 N = 158,537 Healthcare visits in the last year, mean (SD) Primary care 5.83 (6.24) 3.46 (4.08) 3.69 (4.32) 3.33 (4.38) 0.45 0.40 0.46 0.05 0.03 Emergency department 0.58 (1.23) 0.44 (1.23) 0.40 (1.15) 0.42 (1.25) 0.12 0.16 0.13 0.03 0.01 Acute care 0.12 (0.47) 0.08 (0.36) 0.08 (0.36) 0.07 (0.36) 0.10 0.11 0.12 0.01 0.02 SD standard deviation, COPD chronic obstructive pulmonary disease, CHF congestive heart failure Clarify that standardized differences >=0.2 are considered a meaningful difference and are highlighted in italics Li et al. BMC Family Practice (2018) 19:77 Page 7 of 10 Notably, when we looked at the QUALICOPC physicians’ consistent with literature on this topic. It has been sug- whole rosters, rather than just the patient survey respon- gested that one of the reasons for this is that physicians dents, the patient characteristics were very similar to those working in groups have more time to devote to of the other physicians in their practice groups and Ontario non-patient care, and may be more likely to complete a patients in general (group 2 vs. group 3 and group 4 in survey [5, 7, 11, 12]. The opportunity cost of answering Table 2). The only meaningful difference, according to our a survey would be higher for physicians paid by fee for threshold, was when looking at morbidity using RUBs, service compared to those paid by capitation, such as where there was a higher proportion of “non-users” in the those in a Family Health Team. Our finding that a higher province compared to the QUALICOPC physicians’ proportion of physician respondents were local rather rosters. than international medical graduates is also consistent with the literature . Discussion The patient respondents were recruited by consecutive This is thefirst studytoexamine therepresentativenessof visit-based sampling in primary care, which means they the QUALICOPC study within Canada. While other studies were patients with access to primary care who are more have explored the representativeness of QUALICOPC likely to need or use these services. Consistent with our physician respondents internationally, this is also the first findings regarding the characteristics of patient respon- study to assess representativeness of both the physician and dents, another Ontario primary care study also found patient respondents using comprehensive administrative that patients recruited by consecutive sampling in the databases. In one QUALICOPC study from Switzerland, waiting room sampled a population that was older, primary care physicians were randomly selected from a sicker, and more likely to be female compared to the rest database by mail to participate in the survey, with another of the practice population . Similarly, an American set of randomly selected physicians as the comparison. study of visit-based sampling in Veterans Affairs primary Physician survey respondents were found to be similar to care firms found that patients sampled were older, had their comparators in terms of age, sex and practice loca- more visits, and were in poorer health compared to the tion . Another QUALICOPC study from Australia general patient population . assessed nonresponse bias by contacting nonresponders In addition, the sampling method used in the Ontario by telephone; researchers concluded that the gender split QUALICOPC study involved first recruiting one physician of physicians was similar, but younger primary care from each practice, followed by recruiting the patients of physicians were underrepresented in the survey sample the responding physicians, meaning that patient respon- . These differences are likely due to the variability in dents were dependent on which physicians responded to sampling and recruitment used in the different itera- the survey. Another study that explored this recruitment tions of the QUALICOPC study internationally. The strategy found no difference between patients whose findings in our study may relate to the fact that in physicians participated and those whose physicians did Ontario the physician respondents were self-selected (i.e. not participate . In our study, we also found that invitations were sent to all physicians), whereas in the the respondent physicians’ patient rosters were similar Swiss and Australian contexts they were recruited by ran- to those of their practice groups and the Ontario popu- dom sampling. lation, although there were differences between partici- Physician survey respondents were younger on average pating and non-participating physicians’ demographic than nonrespondent physicians, which is consistent with and practice characteristics. Thus differences observed literature exploring nonresponse bias in primary care between the patient respondents and the general popu- surveys for physicians [5–8, 13]. A minority of studies lation are more likely due to the visit-based patient have concluded the opposite; however, these studies sampling methods than differences in the patient rosters used a random sampling strategy and were conducted of responding physicians. within different geographical contexts [10, 11]. It has Given that the profile of QUALICOPC physicians’ been suggested that differences in how physicians are whole rosters were similar to their practice groups’ and trained may help to explain why age is associated with the province, there is some evidence that the participating survey responses. If this is indeed the case, it may be practices are representative of other non-participating that Ontario physicians that have graduated more re- practices. While the QUALICOPC patient respondents cently have more interest in participating in research are not representative of their physicians’ rosters or all and primary care performance measurement than their Ontarians, their responses may still be representative of more experienced counterparts. other patients with a similar health profile and possibly of We identified that physician survey respondents in- patients who tend to visit their physicians. cluded more physicians who worked in Family Health With continued interest in primary care reform in Teams, rather than solo practice, a conclusion that is Canada and throughout the world, the QUALICOPC Li et al. BMC Family Practice (2018) 19:77 Page 8 of 10 study provides important data for further research. Sev- sample, in which the patient respondents of the Ontario eral Canadian and international studies have already QUALICOPC tended to be older, sicker, and more likely been published using the Canadian QUALICOPC data to be female than the other patient groups. However, des- [32–36]. Our study describes the extent to which the pite these differences, Ontario QUALICOPC physician re- Ontario QUALICOPC physician and patient respondents spondents had similar rosters overall compared to their are representative of their practices and the general practice groups and the general population. population, which is important for appropriately interpret- These results will have implications for studies relying ing results of studies relying on Canadian QUALICOPC on QUALICOPC data as well as other primary care sur- data. This study also highlights the importance of assessing veys. Those using QUALICOPC data should understand nonresponse bias to appropriately generalize the results of the limited representativeness of the respondents, and surveys to certain populations. Knowledge of the character- consider the potential for bias in their analyses. While istics of physicians and patients that are underrepresented physician and patient-level results are not representative in research may be helpful in considering survey recruit- of the entire Ontario population, the participating prac- ment and sampling strategies for future research, in order tices may be representative of other non-participating to maximize the representativeness of the sample . practices, and the patients selected by visit-based sam- pling may also be representative of visiting patients in Limitations other practices. Future primary care surveys are encour- This representativeness study has some limitations. The aged to consider consistent recruitment and sampling use of administrative databases allowed us to compare strategies across jurisdictions if possible, and to consider survey respondents with large cohorts of nonresponding integrating measurement of nonresponse bias into sur- physicians and patients. However, some characteristics vey protocols. We have demonstrated one method of relevant to this study were not available in health admin- assessing sample representativeness using administrative istrative databases, such as whether physicians are affili- data, which could be used regardless of the sampling ated with academic institutions, or how many hours per methodology selected. week they work in their respective clinics. These charac- teristics may have impacted the actual survey responses of Additional file respondents and nonrespondents thus potentially contrib- Additional file 1: Primary Care Models. Summary of primary care uting to nonresponse bias. We were also unable to identify models in Ontario including composition and characteristics, and exclude primary care physicians who predominantly physician compensation type, and whether patient enrolment is have a focused practice (e.g. sports medicine or travel required. (DOCX 14 kb) medicine) from the comparison groups, even though they were not eligible to participate in the QUALICOPC. Abbreviations CAPE: Client agency program enrolment tables; CHF : Congestive heart We only examined the representativeness of the failure; CIHI-DAD : Canadian Institute of health information discharge abstract Ontario subset of the Canadian QUALICOPC respon- database; COPD : Chronic Obstructive pulmonary disease; ICES : Institute for dents. Physician recruitment methods varied slightly be- clinical evaluative sciences; IPDB : ICES physician database; NACRS : National ambulatory care reporting system; OHIP : Ontario health insurance plan; tween provinces; therefore, the generalizability of the QUALICOPC : Quality and costs of primary care study; RIO : Rurality index of physician component of this study is certain only in the ontario; RPDB : Registered person’s database; RUB : Resource utilization band province of Ontario . The patients were recruited Acknowledgements by similar consecutive visit-based sampling across the This study was supported by the Health System Performance Research Network provinces. However, the generalizability of the patient (HSPRN grant #06034) and the Institute for Clinical Evaluative Sciences (ICES), results to the national sample depends on the extent which are funded by the Ontario Ministry of Health and Long-Term Care. The opinions, results and conclusions reported in this paper are those of the authors to which the differences in physician sampling across and are independent from the funding sources. No endorsement by HSPRN, provinces selected physicians with different patient ICES or the MOHLTC is intended or should be inferred. populations. This study highlights a need to examine the Funding representativeness of the QUALICOPC study in other This study was supported by the Health System Performance Research Network Canadian jurisdictions, to appropriately contextualize the (HSPRN grant #06034) and the Institute for Clinical Evaluative Sciences (ICES), results of studies relying on Canadian QUALICOPC data. which are funded by the Ontario Ministry of Health and Long-Term Care. The opinions, results and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by HSPRN, Conclusion ICES or the MOHLTC is intended or should be inferred. The physician respondents of the Ontario QUALICOPC differed slightly from their practice groups, and to a larger Availability of data and materials Ontario QUALICOPC data can be made available to researchers upon request. extent from other Ontario primary care physicians with The Institute for Clinical Evaluative Sciences (ICES) owns the comparator respect to most of the characteristics studied. Visit-based data underlying this study. ICES is a non-profit agency funded by the sampling may have led to a biased patient respondent Ontario government and a prescribed entity under the Ontario Personal Li et al. BMC Family Practice (2018) 19:77 Page 9 of 10 Health Information Protection Act. As such, ICES policies and procedures 9. Partin MR, Powell AA, Burgess DJ, Haggstrom DA, Gravely AA, Halek K, are approved by Ontario’s Information and Privacy Commissioner. These Bangerter A, Shaukat A, Nelson DB. Adding postal follow-up to a web-based policies require that access to data be limited to persons who require such access survey of primary care and gastroenterology clinic physician chiefs to perform their role on an approved ICES Project or Third-Party Project. Inquiries improved response rates but not response quality or representativeness. for data access can be sent to Data Services at the Institute for Clinical Evaluative Eval Health Prof. 2015;38(3):382–403. Sciences: (https://www.ices.on.ca/DAS/Submitting-your-request). 10. Parkinson A, Jorm L, Douglas KA, Gee A, Sargent GM, Lujic S, McRae IS. Recruiting general practitioners for surveys: reflections on the difficulties Authors’ contributions and some lessons learned. Aust J Prim Health. 2015;21:254–8. AL, SC, and WPW contributed to the conception and design of the study, 11. Bjertnaes OA, Garratt A, Botten G. Nonresponse bias and cost-effectiveness in a along with MA, SW, and WH. Analysis was performed by KW and YQB and Norwegian survey of family physicians. Eval Health Prof. 2008;31(1):65–80. data was analyzed and interpreted by AL, SC, KW, YQB, and WPW. The 12. Bjertnaes OA, Iversen HH, Bukholm G. International health policy survey in manuscript was drafted by AL, SC, and WPW, and revised critically by MA, 11 countries: assessment of non-response bias in the Norwegian sample. SW, and WH. All of the authors read and approved the final manuscript. BMC Health Serv Res. 2010;10:38. 13. Barclay S, Todd C, Finaly I, Grande G, Wyatt P. Not another questionnaire! Ethics approval and consent to participate Maximizing the response rate, predicting non-response and assessing non- The QUALICOPC survey received ethical approval from the University of response bias in postal questionnaire studies of GPs. Fam Pract. 2002;19(1):105–11. Toronto Research Ethics Board and this study also received approval from 14. Bowling A. Mode of questionnaire administration can have serious effects the Sunnybrook Hospital Research Ethics Board. Participants provided on data quality. J Public Health. 2005;27(3):281–91. consent to participate in the surveys. Patient respondents further consented 15. Gribble R, Haupt C. Quantitative and qualitative differences between handout to provide their OHIP numbers for the purposes of linking their de-identified and mailed patient satisfaction surveys. Med Care. 2005;27(3):276–81. survey information to health administrative databases at ICES. 16. Slater M, Kiran T. Measuring the patient experience in primary care. Can Fam Physician. 2016;62:e740–8. Competing interests 17. Green ME, Hogg W, Savage C, Johnston S, Russell G, Jaakkimainen L, Glazier The authors declare that they have no competing interests. R, Barnsley J, Birtwhistle R. Assessing methods for measurement of clinical outcomes and quality of care in primary care practices. BMC Health Serv Res. 2012;2:214. Publisher’sNote 18. Shortreed SM, Von Korff M, Thielke S, LeResche L, Saunders K, Rosenberg D, Springer Nature remains neutral with regard to jurisdictional claims in Turner JA. Electronic health records to evaluate and account for published maps and institutional affiliations. nonresponse bias: a survey of patients using chronic opioid therapy. Obs Stud. 2016;2:24–38. Author details 1 19. Schäfer WLA, Boerma WGW, Kringos DS, De Maeseneer J, Gress S, Institute of Health Policy, Management and Evaluation, University of 2 Heinemann S, Rotar-Pavlic D, Seghieri C, Svab I, Van den Berg MJ, Vainieri Toronto, Toronto, Canada. Department of Family & Community Medicine, 3 M, Westert GP, Willems S, Groenewegen PP. QUALICOPC, a multi-country University of Toronto, Toronto, Canada. Institute for Clinical Evaluative 4 study evaluating quality, costs and equity in primary care. BMC Fam Pract. Sciences, Toronto, Canada. School of Public Policy & Administration, 5 2011;12:115. Carleton University, Ottawa, Canada. Department of Family Medicine, 6 20. Laberge M, Pang J, Walker K, Wong S, Hogg W, Wodchis WP. QUALICOPC University of Ottawa, Ottawa, Canada. Bruyere Research Institute, Ottawa, 7 (Quality and Costs of Primary Care) Canada: a focus on the aspects of Canada. School of Nursing, University of British Columbia, Vancouver, 8 primary care most highly rated by current patients of primary care practices. Canada. Centre for Health Services and Policy Research, University of British 9 Ottawa: Canadian Foundation for Healthcare Improvement; 2014. Columbia, Vancouver, Canada. Toronto Rehabilitation Institute, Toronto, 21. Jaakkimainen L, Upshur R, Klein-Geltink J, Leong A, Maaten S, Schultz S, Canada. Wang L. Primary Care in Ontario: ICES Atlas. Toronto: Institute for Clinical Evaluative Sciences; 2006. Received: 29 August 2017 Accepted: 18 May 2018 22. Galliher JM, Bonham AJ, Dickinson LM, Staton EW, Pace WD. Representativeness of PBRN physician practice patterns and related beliefs: the case of the AAFP national research network. Ann Fam Med. References 2009;7:547–54. 1. Marchildon GP, Hutchison B. Primary Care in Ontario, Canada: new 23. Queenan JA, Williamson T, Khan S, Drummond N, Garies S, Morkem R, proposals after 15 years of reform. Health Policy. 2016;120:732–8. Birtwhistle R. Representativeness of patients and providers in the Canadian 2. Wong ST, Chau LW, Hogg W, Teare GF, Miedema B, Breton M, Aubrey- primary care sentinel surveillance network: a cross-sectional study. CMAJ Bassler K, Katz A, Burge F, Boivin A, Cooke T, Francoeur D, Wodchis WP. An Open. 2016;4(1):E28-32. international cross-sectional survey on the quality and costs of primary care 24. Kralj B. Measuring Rurality - RIO2008_Basic: methodology and results. (QUALICO-PC): recruitment and data collection of places delivering primary Toronto: OMA Department of Economics; 2009. care across Canada. BMC Fam Pract. 2015;16:20. 25. Matheson FI, Dunn JR, Smith KLW, Moineddin R, Glazier RH. Development 3. VanGeest JB, Johnson TP, Welch VL. Methodologies for improving response of the Canadian marginalization index: a new tool for the study of rates in surveys of physicians: a systematic review. Eval Health Prof. 2007; inequality. Can J Public Health. 2012;103(Suppl 2):12–6. 30(4):303–21. 26. The Johns Hopkins University, “The Johns Hopkins ACG System,” 18 11 4. Halbesleben JRB, Whitman MV. Evaluating survey quality in health services 2016.;[Online]. Available: https://www.hopkinsacg.org/. research: a decision framework for assessing nonresponse bias. Health Serv 27. Mamdami M, Sykora K, Li P, Norman SLT, Streiner DL, Austin PC, Rochon PA, Res. 2013;48(3):913–30. Anderson GM. Reader's guide to critical appraisal of cohort studies: 2. 5. Armstrong D, Ashworth M. When questionnaire response rates do matter: a Assessing potential for confounding. BMJ. 2005;330:960–2. survey of general practitioners and their views of NHS changes. Brit J Gen 28. Sawilowsky SS. New effect size rules of thumb. J Mod App Stat Meth. Pract. 2000;50:479–80. 2009;8(2):597–9. 6. Templeton L, Deehan A, Taylor C, Drummond C, Strang J. Surveying 29. Selby K, Cornuz J, Senn N. Establishment of a representative practice-based general practitioners: does a low response rate matter? Brit J Gen Pract. research network for the monitoring of primary care in Switzerland. JABFM. 1997;47:91–4. 2015;28(5):673–5. 7. Lippmann S, Frese T, Herrmann K, Scheller K, Sandholzer H. Primary care research - trade-off between representativeness and response rate of GP 30. Lee ML, Yano EM, Wang MM, Simon BF, Rubenstein LV. What patient teachers for undergraduates. Swiss Med Wkly. 2012;142:w13537. population does visit-based sampling in primary care settings represent? 8. Wetzel D, Himmel W, Heidenreich R, Hummers-Pradier E, Kochen MM, Med Care. 2002;40(9):761–70. Rogausch A, Sigle J, Boeckmann H, Kuehnel S, Niebling W, Scheidt-Nave C. 31. Fourrier-Reglat A, Droz-Perroteau C, Benichou J, Depont F, Amouretti M, Participation in a quality of care study and consequences for generalizability Begaud B, Moride Y, Blin P, Moore N. Impact of prescriber nonresponse on of general practice research. Fam Pract. 2005;22:458–64. patient representativeness. Epidemiology. 2008;19(2):186–90. Li et al. BMC Family Practice (2018) 19:77 Page 10 of 10 32. Thompson AE, Anisimowicz Y, Miedema B, Hogg W, Wodchis WP, Aubrey-Bassler K. The influence of gender and other patient characteristics on health care-seeking behaviour: a QUALICOPC study. BMC Fam Pract. 2016;17:38. 33. Miedema B, Easley J, Thompson AE, Boivin A, Aubrey-Bassler K, Katz A, Hogg WE, Breton M, Francoeur D, Wong ST, Wodchis WP. Do new and traditional models of primary care differ with regard to access? Canadian QUALICOPC study. Can Fam Physician. 2016;62:54–61. 34. Rumball-Smith J, Wodchis WP, Kone A, Kenealy T, Barnsley J, Ashton T. Under the same roof: co-location of practitioners within primary care is associated with specialized chronic care management. BMC Fam Pract. 2014;15:149. 35. Pavlic DR, Sever M, Klemenc-Ketis Z, Svab I. Process quality indicators in family medicine: results of an international comparison. BMC Fam Pract. 2015;16:172. 36. van Loenen T, van den Berg MJ, Faber MJ, Westert GP. Propensity to seek healthcare in different healthcare systems: analysis of patient data in 34 countries. BMC Health Serv Res. 2015;15:465.
BMC Family Practice – Springer Journals
Published: May 30, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera