TY - JOUR
AU - Nemes, Maria Ines Baptistella
AB - Introduction High-quality primary health care (PHC) can improve population and individual outcomes while reducing total health care costs [1], but quality in PHC has not been fully achieved in most countries [2–4]. Therefore, PHC quality assessment takes on special importance for prompting actions that improve health care team training, favoring change processes in the organization of care practices [5, 6]. Health facility assessment can be based on different methodologies and data collection approaches, all of which have advantages and drawbacks. To choose the best strategy, it is necessary to consider the purpose of the assessment, the themes that guide it, and its theoretical framework, as well as the analysis of its feasibility, reliability, utility, and access to its results by the facilities that took part in the assessment [7]. Assessments carried out through online surveys can be questioned in terms of confidentiality, confidentiality maintenance, response rate and sample representativeness [7–9]. On the other hand, they do offer several advantages. In addition to suitably fitting with pandemic contexts, such as the COVID-19 one, they show high feasibility as they allow large-scale surveys, can be conducted in a short period of time, offer a good cost-benefit ratio, and allow quick access to results by assessment researchers and users, thus constituting an important assessment survey tool [8, 9]. Data collection through in-person interviews in health facility assessment is a qualitative strategy that can be used in surveys. This type of survey is costlier and tends to use structured instruments, such as questionnaires or checklists. This more structured form of collection differs from approaches based on interviews that seek to expand explanatory power through the analysis of the facilities’ daily experience and the relationships established between teams and between teams and users [10]. In Brazil, PHC health facility assessment processes have been carried out using different instruments, with different methodologies, geographic coverage, and assessment focus [11–14]. In general, these are cross-sectional studies aimed at professionals and/or users based on in-person interviews using structured questionnaires in paper or in digital format, installed on tablets or smartphones that upload the data, or even in the form of online self-response. No studies have been conducted to compare information gathered through the different methods used, often similar in aim and data collection methods. The advance of information technology and the high cost of in-person interview surveys, especially in a continent-sized country like Brazil, has motivated the investigation into the correspondence between responses collected via web-based self-administered questionnaires and those collected through in-person structured interviews. The institutionalization of assessment processes in Brazilian PHC facilities was consolidated with the implementation of the National Program for Improvement of Access and Quality of Primary Care—PMAQ-AB, which completed its third assessment cycle in 2018. Based on voluntary participation, the program awarded financial incentives that varied according to facility performance. Three assessment cycles were carried out throughout the country, by means of integrated steps that used data from health information systems, self-assessment instruments, and external assessments made in loco by trained interviewers (AE-PMAQ-AB). The criteria and standards used were based on the guidelines and protocols of the National Primary Care Policy (PNAB) and on the ethical and political principles of the Brazilian Unified Health System (SUS) [15, 16]. In this context of encouragement of PHC facility assessment, the system for Assessment of Primary Health Care Services (QualiAB) was created in the state of São Paulo, Brazil. It is a computerized system using a web-based self-administered questionnaire, with assessment criteria and standards following the PNAB and SUS guidelines; adherence is voluntary and not linked to financial incentives. Eight QualiAB surveys have been conducted in different regions of the country. In the state of São Paulo, the last survey was conducted in 2017, preceding the AE-PMAQ-AB in the state by a few months [17]. By focusing on a particular fraction of the complex aspects involved in health facility assessment, this study investigates whether there is concordance between the responses to two assessment surveys that used structured questionnaires and adopted different data collection methodologies: the PMAQ-AB external assessment (AE-PMAQ-AB), conducted by in-person interviewers and considered here as the “gold standard,” and the QualiAB external assessment, conducted through a computerized web-based self-response system. Materials and methods This is a concordance analysis study comparing answers of 1,855 PHC facilities to two surveys conducted with the aid of structured instruments in the state of São Paulo, Brazil, in 2017–2018. The two surveys consist in external assessments that use different information collection methods: the AE-PMAQ-AB assessment is conducted by in-person interviewers, whereas the QualiAB assessment uses a web-based self-response questionnaire. The selection of assessment surveys was based on the comparability between answers, according to the equivalence of the following criteria: 1) focus on the organization of the work process; 2) use of the same technical and political references to define the criteria and standards of their indicators; 3) use of structure and process indicators; 4) occurrence in periods close to each other; and 5) assessment of PHC facilities in the same region. The assessments were carried out at PHC facilities in the state of São Paulo, which has a municipally managed network of public PHC facilities covering approximately 60% of the population during the period studied. The state has 45.5 million inhabitants (21.9% of the Brazilian population) and has the country’s second highest Human Development Index (HDI): 0.783. However, its 645 municipalities display great geographical, populational, and socioeconomic heterogeneity. Forty percent of the municipalities have fewer than 10,000 inhabitants, are geographically distributed across coastal and mountainous regions that are poorly accessible, and have municipal HDIs between 0.862 and 0.639, thus presenting inequalities similar to Brazil’s [18–20]. The AE-PMAQ-AB assessment under analysis was conducted in the state of São Paulo between May and August 2018, as part of the third PMAQ-AB cycle. It was the result of a partnership between the Ministry of Health and public higher education institutions responsible for selecting, training, and hiring the university-level professionals who conducted the AE-PMAQ-AB data collection in loco16. The interviewers formed regionalized teams supervised by a coordinator responsible for planning their travel itinerary and checking the recorded information. Data collection was carried out through structured questionnaires installed on tablets equipped with a computerized system that sends the data to the Ministry of Health at the end of each interview. In the state of São Paulo, 2,693 family health teams based in 564 municipalities were assessed in loco. The municipal participation rate was 87.4%. No data are available on the total number of PHC facilities in the municipalities participating in the PMAQ assessment. Participation was encouraged by the Ministry of Health and partner institutions through presentations of the project in regional meetings and financial incentives to participate [21]. The Ministry of Health was slow in disclosing the end results of the PMAQ assessment. The final score determined the ranking of each facility and the amount of financial incentive for performance. The score was composed of the results achieved in three stages: (1) implementation of self-assessment procedures, accounting for 10% of the score; (2) assessment of contractual indicators, corresponding to 30%; and (3) the AE-PMAQ-AB assessment, which was the last stage and accounted for 60% of the final score [21]. The partial results relative to the AE-PMAQ-AB stage were not disclosed to the participants. The QualiAB survey was conducted in the state of São Paulo between May and November 2017 as part of a research project resulting from the partnership of two public higher education institutions (UNESP and USP) and the São Paulo State Department of Health (SES SP). It was supported by the Council of Municipal Health Secretaries of the state of São Paulo (COSEMS SP) and encompassed 2,739 PHC facilities located in 514 municipalities in the state of São Paulo. A total of 79.7% of municipalities participated in the QualiAB survey, with a participation rate of 88.2% for the PHC facilities located in the participating municipalities. The QualiAB assessment project was presented by the SES SP to municipal health secretaries in regional meetings to encourage municipal participation. The other stages of the assessment were carried out via web: enrollment of the municipalities with password definition; enrollment of facilities by local managers with individualized access passwords; questionnaire response; and hierarchized access to results and standards according to the institutional rank of the participants. The computerized system provided participants with immediate access to their score and performance level, measured both globally and by indicator, as well as access to recommendations according to the criteria and standards used. There was no financial incentive to participate in the QualiAB assessment [22]. Both questionnaires were written in Portuguese, Brazil’s official language, and answered by two different professionals of each PHC facility. The content of the questionnaires used in both assessments was analyzed at two different times by two researchers, who identified 158 comparable questions. Even though these questions are not identical, they have equivalent phrasings, as they address the same aspects of work organization. The QualiAB answer alternatives tend to be more detailed, while in the AE-PMAQ-AB assessment, they are overall dichotomous (yes/no), as shown in Table 1. To allow comparison, the equivalence was established through the presence, or lack thereof, of paired items. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 1. An example of paired questions from the AE-PMAQ-AE and QualiAB data collection instruments. Brazil, 2017/2018. https://doi.org/10.1371/journal.pone.0281085.t001 The 158 comparable questions were grouped into nine domains relative to territory, structure, management, and attention to main demand and programs, as provided in the Brazilian National Primary Care Policy [23]. The domains comprehend different aspects of the organization of PHC facilities and make it possible to analyze concordance differences between facilities. The nine domains are: 1. Territory characteristics; 2. Local management and external support; 3. Structure; 4. Health promotion, disease prevention, and therapeutic procedures; 5. Attention to unscheduled patients; 6. Women’s health; 7. Children’s health; 8. Attention to chronic conditions; 9. Oral health. After the comparable questions were identified, the facilities that responded to the two external assessments in the state of São Paulo were paired, resulting in a total of 1,855 facilities located in 434 municipalities. The AE-PMAQ-AB assessment was used as the gold standard to validate the answers. This choice is justified because the AE-PMAQ-AB assessment was part of PMAQ, an official program of the Brazilian government for assessing PHC facilities during the period analyzed in this study, and because its criteria, standards, and indicators reflected the technical and political proposals for PHC facilities [16, 23]. That choice is also corroborated by the fact that PMAQ has nationwide reach and defines a "baseline" for PHC assessment in Brazil, which is confirmed by the multiple studies and assessments based on AE-PMAQ-AB data [14, 24–27]. The proportion of similar answers to the comparable questions from the two questionnaires was calculated, which made it possible to calculate sensitivity (TP/(TP+FN)), specificity (TN/(FP+TN)), and accuracy ((TP+TN)/N). TP stands for "true positive" (where both questionnaires have an affirmative answer for the same comparable question); FN stands for "false negative" (affirmative answer in QualiAB, negative in AE-PMAQ-AB); TN is "true negative" (negative answers in both questionnaires); FP is "false positive" (negative answer in QualiAB, affirmative in AE-PMAQ-AB); and N is the total of answers. The concordance between answers was analyzed with the Kappa coefficient test; all analyses were performed using the SPSS 21® statistical package. This study complied with the ethical principles laid out in Brazilian legislation, according to Resolution No. 446/2012 on the guidelines and regulations for medical research on human beings, and garnered approval from the ethics committee (Resolution No. 2.532.658, CAAE: 83479417.7.0000.5411, March 8, 2018). Results The pairing of questions from the two external assessments made possible the analysis of the concordance of the answers from 1,855 PHC facilities, geographically distributed across 67% of the municipalities of the state of São Paulo (Fig 1). Download: PPT PowerPoint slide PNG larger image TIFF original image Fig 1. Geographic distribution of the 434 municipalities participating in the QualiAB, 2017 and AE-PMAQ-AB, 2018 assessments. Source: authors; Datasus software. https://doi.org/10.1371/journal.pone.0281085.g001 The grouping of questions into nine domains allowed for the analysis of the main aspects involved in the organization of the PHC facilities. A high concordance level was found between the answers to both assessments: 81% (128) of the QualiAB answers showed a concordance level higher than 0.700 in relation to the AE-PMAQ-AB answers. Only in the question addressing treatment of people living with HIV/AIDS was the concordance level lower than 0.500 (Acc = 0,414) (Table 2). Download: PPT PowerPoint slide PNG larger image TIFF original image Table 2. Comparison of the answers to the QualiAB and the AE-PMAQ-AB questionnaires relative to territory characteristics, structure, and management and external support, according to accuracy, sensitivity, specificity, confidence interval, and Kappa coefficient. Brazil, 2017/2018. https://doi.org/10.1371/journal.pone.0281085.t002 Table 2 comprehends domains relative to territory, some of the available resources, and management characteristics, showing high accuracy for nearly all compared questions, and varying sensitivity and specificity. In Table 3, all the questions show accuracy higher than 0.900. While sensitivity is high for all three subdomains (health promotion, disease prevention, and therapeutic procedures), specificity is considerably lower for the subdomain Therapeutic Procedures. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 3. Comparison of the answers to the QualiAB and the AE-PMAQ-AB questionnaires relative to health promotion, disease prevention, and therapeutic procedures, according to accuracy, sensitivity, specificity, confidence interval, and Kappa coefficient. Brazil, 2017/2018. https://doi.org/10.1371/journal.pone.0281085.t003 Table 4 shows low accuracy for the question about risk assessment of unscheduled patients, as well as for the questions in the subdomain Attention to Chronic Communicable Conditions, except for the question about bacilloscopy for tuberculosis. In the subdomain of Chronic Non-communicable Conditions, the questions about ocular fundus examination and mental health care also show low accuracy. The item with the lowest accuracy, "Delivery of care to persons living with HIV/AIDS", contrasts with the high accuracy seen in consultations for persons with diabetes and hypertension, as well as in the delivery of care to women and children. Download: PPT PowerPoint slide PNG larger image TIFF original image Table 4. Comparison of the answers to the QualiAB and the AE-PMAQ-AB questionnaires relative to the delivery of care to unscheduled and scheduled patients (women, children, chronic conditions, oral health) according to accuracy, sensitivity, specificity, confidence interval, and Kappa coefficient. Brazil, 2017/2018. https://doi.org/10.1371/journal.pone.0281085.t004 The high accuracy combined with low, even negative, Kappa values is explained by the vulnerability of the Kappa coefficient test against marginal distributions and asymmetric joint distributions, since too high concordances, without a normal distribution, compromise Kappa values [27]. Discussion The results presented a high concordance level between answers to the paired questions of both assessments, showing that web-based questionnaires are a viable tool to assess work organization in PHC facilities when it comes to structure and processes. The highest accuracy level was found in the more traditional actions of the Brazilian PHC programs, especially in relation to questions of health promotion, disease prevention, therapeutic procedures, definition of coverage area, local management, and health care programs such as care delivery to women, children, persons with hypertension and type 2 diabetes, and more traditional oral health procedures. The lowest-accuracy questions may point to a lack of clarity in their formulation, respondents’ limited knowledge of the subject, or recent implementation of the service in question. For example, the item “Dedicated vaccine refrigerator” showed low accuracy level (0.533), which may be linked to the lack of clarity in the question’s phrasing. Neither the QualiAB nor the AE-PMAQ-AB questionnaire specifies the type of refrigerator (for home or commercial use), which may have led to diverging interpretations, as it is recommended to replace home-use refrigerators with ones that meet safety and quality standards [28]. The lowest accuracy level among the 158 paired questions was found in “Delivery of care to persons living with HIV/AIDS” (0.414), which has only recently been incorporated into the Brazilian PHC [29], followed by “Ulotomy/Ulectomy” (0.516), which are low-demand procedures unknown to many team members [30]. It is important to point out that the AE-PMAQ-AB questionnaire should be answered either by the facility manager or by the head of each department, or even by a doctor, whereas the QualiAB questionnaire should preferably be answered in a team meeting. It behooved the facility personnel to find the best way to answer the questionnaires without compromising service to patients. They were, therefore, answered by different team members of each facility, which may have influenced the answers regarding low-demand or recently implemented services. Other not fully implemented services, such as electronic medical records, risk assessment protocols for persons with diabetes and hypertension, and mental health care, also showed low accuracy levels, which may be related to the reorganization of the facilities during that period. In general, the questions showed lower specificity, i.e., higher discordance in relation to negative answers. In addition to the possible reasons mentioned above, this discrepancy may be due to the adjustment to expected standards as a result of the self-assessment process that preceded AE-PMAQ-AB. This process was a component of both the PMAQ and the QualiAB programs. PMAQ was the first institutional program of PHC facility assessment that covered the entire Brazilian territory, involving a large number of in loco AE-PMAQ-AB interviewers. In this process, transportation difficulties arose from the great distances between municipalities. Municipalities with large rural areas also proved difficult to reach. Thus, journeys frequently took hours or days, by different routes—air, river, or land. Roads were often precarious, and weather incidents, such as rainy seasons, blocked roads or isolated PHC facilities in the state of Amazonas [31–33]. Brazil’s great territorial extension and geographic diversity make in loco data collection surveys difficult and costly in many respects. Additionally, the high cost of the whole process of organization, selection, training, and hiring of a large number of professionals must be taken into account when choosing the best form of data collection [7]. Information and Communications Technology (ICT) has long been gaining ground in the health sector, with the incorporation of telemedicine technologies that make it possible to expand patient service and health care professionals’ training and support [34], as well as improving information record. As of 2020, with the measures to prevent the spread of Covid-19, the use of ICTs and computer equipment has been amplified in Brazil [35], which increases its potential for use in health care services and assessment processes. Web-based structured assessments are limited by the number of questions that can be asked, which also limits the scope of the assessment and requires the selection of high-sensitivity and -specificity indicators. Additionally, this type of assessment requires great investment in establishing partnerships that will participate actively in and be committed to the assessment, thus yielding high response rates [7]. In-person assessments, on the other hand, even when based on structured questionnaires, make it possible not only to expand and diversify the subjects and interviewed professionals, but to observe the facilities directly and to use various instruments, such as semi-structured interviews with patients. Some limitations of this study worthy of pointing out are the difference in subjects addressed by the two instruments, the time gap between the surveys, and their different levels of institutionalization and ability to induce participation. The AE-PMAQ-AB assessment was part of a financial incentive-granting evaluation program of the Brazilian Ministry of Health to improve service quality, whereas the QualiAB program was rather a self-assessment opportunity for the participating facilities. On the other hand, web-based assessments, such as the one conducted through the QualiAB system, in addition to not posing the challenges of in loco data collection, can provide immediate access to results, reports and orientation to the system’s users. Quick access to results make them more likely to be used by the participants, particularly their direct users—the PHC teams–, thus allowing the assessment to complete all the stages of an assessment process. Timely disclosure of results and good strategies for communicating them are mentioned in the literature [36, 37] as factors that promote knowledge and use of the assessments to underpin political decision making, redesign measures, and allocate financial resources [38, 39]. They also increase acceptance of the assessment, according to Rissi, Sager (2013) [40]. Another advantage is that this format does not interfere in the routine health care service, as it makes it possible to save the information record in case it is necessary to interrupt the response process. We can also add the ease of answering the questionnaire in a partial and scheduled way, which favors the involvement of a larger number of professionals in the discussion of the answers [7, 17]. The high level of concordance found between both assessments points to advantages in the use of web-based assessment instruments. These advantages become even more pronounced in light of the need for investments to expand assessment surveys, amplify the assessment culture, navigate pandemic scenarios, and further computerize PHC services, thus highlighting the importance of investing in web-based assessments as one more tool to improve the quality of PHC services and facilities. Supporting information S1 Data. https://doi.org/10.1371/journal.pone.0281085.s001 (XLSX) Acknowledgments Our acknowledgments to Alessandro Barriviera for his careful revision of the English version of the manuscript and his support during the submission process.
TI - Performance assessment of primary health care facilities in Brazil: Concordance between web-based questionnaire and in-person interviews with health personnel
JF - PLoS ONE
DO - 10.1371/journal.pone.0281085
DA - 2023-02-02
UR - https://www.deepdyve.com/lp/public-library-of-science-plos-journal/performance-assessment-of-primary-health-care-facilities-in-brazil-IteSWqbH7c
SP - e0281085
VL - 18
IS - 2
DP - DeepDyve
ER -