Background: Involving groups of community stakeholders (e.g., steering committees) to lead community-wide health interventions appears to support multiple outcomes ranging from policy and systems change to individual biology. While numerous tools are available to measure stakeholder characteristics, many lack detail on reliability and validity, are not context specific, and may not be sensitive enough to capture change over time. This study describes the development and reliability of a novel survey to measure Stakeholder-driven Community Diffusion via assessment of stakeholders’ social networks, knowledge, and engagement about childhood obesity prevention. Methods: This study was completed in three phases. Phase 1 included conceptualization and online survey development through literature reviews and expert input. Phase 2 included a retrospective study with stakeholders from two completed whole-of-community interventions. Between May–October 2015, 21 stakeholders from the Shape Up Somerville and Romp & Chomp interventions recalled their social networks, knowledge, and engagement pre-post intervention. We also assessed one-week test-retest reliability of knowledge and engagement survey modules among ShapeUp Somervillerespondents.Phase3includedsurveymodifications and a second prospective reliability assessment. Test-retest reliability was assessed in May 2016 among 13 stakeholders involved in ongoing interventions in Victoria, Australia. Results: In Phase 1, we developed a survey with 7, 20 and 50 items for the social networks, knowledge, and engagement survey modules, respectively. In the Phase 2 retrospective study, Shape Up Somerville and Romp & Chomp networks included 99 and 54 individuals. Pre-post Shape Up Somerville and Romp & Chomp mean knowledge scores increased by 3.5 points (95% CI: 0.35–6.72) and (− 0.42–7.42). Engagement scores did not change significantly (Shape Up Somerville: 1.1 points (− 0.55–2.73); Romp & Chomp: 0.7 points (− 0.43–1.73)). Intraclass correlation coefficients (ICCs) for knowledge and engagement were 0.88 (0.67–0.97) and 0.97 (0.89–0.99). In Phase 3, the modified knowledge and engagement survey modules included 18 and 25 items, respectively. Knowledge and engagement ICCs were 0.84 (0.62–0.95) and 0.58 (0.23–0.86). Conclusions: The survey measures upstream stakeholder properties—social networks, knowledge, and engagement—with good test-retest reliability. Future research related to Stakeholder-driven Community Diffusionshould focusonprospective change and survey validation for intervention effectiveness. Keywords: Community-based interventions, Community engagement, Childhood obesity prevention, Survey development * Correspondence: email@example.com Friedman School of Nutrition Science and Policy, Tufts University, 150 Harrison Ave., Boston, MA 02111, USA Full list of author information is available at the end of the article © The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Korn et al. BMC Public Health (2018) 18:681 Page 2 of 11 Background Methods and results Community-based interventions have demonstrated ef- This study was completed in three phases (Fig. 1). fective childhood obesity prevention at the population Methods and results are reported below by study level [1–3]. In particular, “whole-of-community” inter- phase. Phase 1 included conceptualization and survey ventions are recommended in which entire communities development assessed for content validity. Phase 2 in- are exposed to programs, policies, and environments cluded a pre-post assessment and reliability testing intended to reduce obesity risk [4–9]. Successful (test-retest) with stakeholders from two completed whole-of-community interventions necessitates the rec- whole-of-community interventions using retrospective ognition of complex organizational and community dy- reporting . Phase 3 included survey modifications namics and the influence of community leaders and and a second prospective reliability assessment. stakeholders (hereafter referred to as stakeholders) from various sectors to build capacity, enhance community Phase 1: Survey development well-being, and promote systems change [2, 10–16]. The COMPACT Stakeholder-driven Community Dif- Understanding the upstream processes by which in- fusion Survey included three modules to (a) assess vestigators and stakeholder groups (e.g., coalitions, the network structure of stakeholders’ professional re- steering committees) conceive, design, implement, and lationships related to childhood obesity prevention ef- adapt whole-of-community interventions is a critical forts, (b) knowledge about childhood obesity step to inform prevention efforts and impact research prevention, and (c) engagement with the issue. outcomes [17, 18]. Extant tools to measure stake- holder characteristics, such as empowerment , Part A: Social networks collaboration , and readiness for change  have In Stakeholder-driven Community Diffusion, social notable weaknesses that limit utility, such as lack of networks represent pathways for knowledge and en- detail on reliability and validity and/or are not con- gagement diffusion. The survey was designed to allow text specific. Sensitive, reliable, and valid tools to respondentstonameupto20people withwhom measure longitudinal information on context, includ- they had “discussed issues related to childhood obes- ing differences in stakeholder social networks and dif- ity” during an intervention . Due to the retro- fusion of information, are needed to shift how spective nature of the initial survey and to diminish investigators approach, understand, and work with likelihood of inaccurate recall, we used two name community partners. This may contribute to the generation methods (free recall and a roster of stake- widespread adoption and scaling of the whole-of-com- holderswho hadalreadyprovidedinformedconsent munity model to improve population health . for the survey) in a three-stage procedure (free recall, The Childhood Obesity Modeling for Prevention And roster identification, final free recall) . This ap- Community Transformation (COMPACT) study funded proach was used to capture the complete network of by the National Institutes of Health (R01HL115485; bounded stakeholder groups (e.g., steering commit- 2013–2018) seeks to apply systems methods to better tees) and stakeholders’ broader networks when ex- understand stakeholders’ leadership roles in whole-of- ploring community-wide connections . community interventions . We hypothesize that stakeholder groups may be a driving factor in the suc- Part B: Knowledge cess of interventions through a process of “Stakeholder-- We conceptualized knowledge as stakeholders’ under- driven Community Diffusion”. As an initial test of this standing of community-wide efforts to prevent childhood theory, an agent-based model has been designed to obesity. We identified five domains from completed inter- understand how stakeholders (the agents) involved in vention trials that reduced unnecessary weight gain completed and ongoing whole-of-community interven- among children [6, 7, 26]: tions in the US, Australia, and New Zealand use their social networks to diffuse their knowledge about and en- 1. The problem of childhood obesity (“Problem”) gagement with childhood obesity prevention efforts. This 2. Modifiable determinants of childhood obesity and work, however, is also dependent upon reliable and valid level of social ecology to address them, e.g., measurement of stakeholder characteristics. Therefore, individual behavior change versus environment and this paper describes the development and reliability test- policy change (“Intervention factors”) ing of the COMPACT Stakeholder-driven Community 3. Stakeholders’ roles in the whole intervention, what Diffusion Survey, a unique multi-method survey that al- others are doing, and knowledge of multi-setting lows quantification of changes in social networks, know- components (“Roles”) ledge, and engagement properties of stakeholders 4. How to intervene to achieve sustainability involved in whole-of-community interventions. (“Sustainability”) Korn et al. BMC Public Health (2018) 18:681 Page 3 of 11 Fig. 1 Overview of the development and reliability testing of the COMPACT Stakeholder-driven Community Diffusion Survey 5. Available resources (“Resources”) Stakeholder-driven Community Diffusion theory suggests that engagement motivates stakeholders to share their We conducted comprehensive literature reviews knowledge with others, and represents stakeholders’ de- (peer-reviewed and grey) to source relevant instruments sires and ability to translate their knowledge into effective and survey items measuring aspects of community readi- action for whole-of-community interventions. ness, group dynamics, coalitions, and community-based We used the CBPR Model to identify domains de- participatory research (CBPR) to adapt and apply to the scribing stakeholder engagement [12, 28]: identified domains. For “resources”, we adapted four items from the Community Capacity Index  and the Com- 1. Exchange of skills and understanding (“Dialogue munity Readiness Handbook . For the remaining do- and mutual learning”) mains, we identified eight items from the CBPR 2. Willingness to compromise and adapt (“Flexibility”) Conceptual Model matrix of variables and instruments 3. Ability or capacity to have an effect on course of [12, 17, 28] and the coalition literature. Five research team events, others’ thinking, and behavior (“Influence members with experience in community-based interven- and power”) tions scored items to assess content validity. Scoring re- 4. Action of directing and being responsible for a sulted in disagreement on items to include. Through group of people or course of events (“Leadership iterative critique and feedback, the team developed new and stewardship”) fact-based, multiple-choice items for domains 1–4(four 5. Belief and confidence in others (“Trust”) items per domain). The knowledge domain included 20 total items. We used 46 items from existing instruments cited in the CBPR Model to construct an engagement scale . Part C: Engagement We also conducted a secondary search in Scopus, We conceptualized engagement as a latent construct PubMed, and the National Cancer Institute’s Team Sci- representing stakeholders’ enthusiasm and agency for ence Toolkit  for community and group partnership preventing childhood obesity in their community. Our tools, yielding 104 total items from 20 instruments. Korn et al. BMC Public Health (2018) 18:681 Page 4 of 11 Six research team members evaluated the 104 items question that asked participants to “write any names, for content validity. They scored items from 0 to 2 phrases, or keywords that describe what was going on points (0 = no; 1 = maybe; 2 = yes), with a maximum in your life” during the intervention period. We in- per-item score of 12 points. Item scores ranged from 3 formed participants that this response would not be to 11 points (mean = 7.2; SD = 1.8). We eliminated items retained and that the purpose was to help them pro- with low scores (≤ 6 points; n = 37) and/or if an item vide more accurate recalled responses . scored lower than a similar item. We retained 50 items We then asked participants to identify social relation- from 17 instruments: dialogue and mutual learning (11), ships and to report their own levels of knowledge and flexibility (8), influence and power (4), leadership and engagement related to childhood obesity prevention at stewardship (22), and trust (5) (Additional file 1: Table the start (T1) and end (T2) of their involvement in SUS S1A). We set response options to a 5-point agree/dis- or R&C. Time was based on intervention involvement agree Likert scale and adapted wording to fit the context due to varying participation and attrition in stakeholder of whole-of-community childhood obesity prevention meetings. Participants reported their gender, current age, interventions. education, and affiliated organizational sector (e.g., school administration) at the start of the intervention. Phase 2: Retrospective study To assess the test-retest reliability of the knowledge Methods and engagement survey components, we asked partici- pants to complete a second web-based survey, one week Participants Respondents were members of stakeholder after the first survey. groups involved in two completed whole-of-community In the SUS study, we offered participants up to $49 childhood obesity interventions: Shape Up Somerville (electronic Amazon gift card) for completing both (SUS)  and Romp & Chomp (R&C) . Both inter- test-retest surveys. Consistent with usual practices in ventions demonstrated measured reductions in child- Australian studies of this type, no monetary incentive hood obesity prevalence. SUS was a community-based was offered to R&C participants. Procedures for individ- environmental change intervention from 2003 to 2005 uals participating in research were approved by the Tufts targeting early elementary school children in Somerville, University Institutional Review Board and the Deakin Massachusetts, USA. The SUS Community Advisory University Human Ethics Advisory Group for the SUS Council included stakeholders from academia, public and R&C studies, respectively. schools, foodservice, local health department, community-based organizations, and met every 2– Data analysis 4 months throughout the intervention. R&C was a Demographics capacity-building and environmental intervention from We calculated frequencies for categorical variables (gen- 2004 to 2008 targeting children from birth to five years der, education, organizational sector affiliation) and in Geelong, Victoria, Australia. The R&C Management means and standard deviations (SD) for participant age. Committee  included stakeholders from academia, local health department, government, department of hu- man services, and the local kindergarten association, and met every 1–2 months. Social networks We extracted online data from the three-stage name Procedures generator of childhood obesity ‘discussion’ networks We identified potential participants’ names from his- andimportedtothe [sna], [igraph],and[network] torical SUS and R&C records and meeting minutes, packages in the R programming language to conduct and then acquired current contact information (email descriptive analyses and produce sociograms [31–33]. and/or telephone) via records, existing contacts, and In the sociograms, participants are represented as the internet. We first contacted participants for in- nodes and are connected by a directed tie to represent a formed consent. Upon providing consent, we invited discussion relationship. Visualizations demonstrate struc- participants to complete the web-based (Qualtrics) tural attributes of networks and are useful in generating survey in May–June 2015 for SUS and August–Octo- hypotheses about pathways for knowledge and engage- ber 2015 for R&C. ment diffusion. Calculated descriptive connectivity statis- To aid participants’ memories in what life was like tics included number of nodes and ties, density (the during the interventions, the surveys started by listing proportion of ties to the number of possible ties between historical milestones at the local, state, and national level node pairs), and in-degree centralization (an indicator of (e.g., elected government officials, major sports victor- node connectivity, or the extent to which one or few ies). This was followed by an optional, open-ended nodes in the network receive a high number of ties). Korn et al. BMC Public Health (2018) 18:681 Page 5 of 11 Knowledge and engagement 4; 30.8%), school administration (n = 1; 7.7%), afterschool We calculated composite and domain-specific scores programs (n = 2; 15.4%), and local health department (n = (mean, SD) at both time points. Knowledge domains 1–4 2; 15.4%). R&C respondents represented university/aca- each had four multiple-choice questions with a maximum demia (n = 5; 62.5%), community-based organizations (n = score of four points per domain (− 1 = incorrect response; 1; 12.5%), and local government (n = 2; 25.0%). 0 = not sure; 1 = correct response). Knowledge domain 5 had four 4-point agree/disagree Likert-scale items (− 1= Social networks strongly disagree; − 0.5 = disagree; 0.5 = agree; 1 = strongly The SUS and R&C stakeholder networks are shown agree) with a maximum score of 4 points. The maximum in Fig. 2. The SUS network had 99 nodes (individ- composite score was 20 points. There were 50 5-point uals), 218 ties (relationships between individuals), agree/disagree Likert-scale engagement items. We density of 0.02 (proportion of ties to the total number weighted scores based on number of items per domain to of possible ties between node pairs), and in-degree ease domain-domain comparisons, with a maximum com- centralization of 0.09 (the extent to which one or few posite score of 25 points (1 = strongly disagree to 5 = nodes receive a high number of ties). The R&C net- strongly agree). We used paired t-tests and corresponding work had 54 nodes, 126 ties, a density of 0.04, and 95% CIs to assess change in mean knowledge and engage- in-degree centralization of 0.07. Readers are referred ment scores from T1 to T2 within interventions (test sur- to McGlashan et al. for further description of SUS vey data used). and R&C stakeholder networks . Knowledge and engagement reliability Knowledge and engagement scores We analyzed reliability data from T1. We assessed Mean composite and domain-specific scores are re- item-specific test-retest reliability using Cohen’s ported in Table 1. Of 20-points maximum, the mean weighted Kappa statistic (κ ). We calculated intra- SUS composite knowledge score increased from 10.4 class correlation coefficients (ICCs) and within-subject points (SD = 5.2) at T1 to 13.9 points (SD = 3.8) at T2 coefficients of variation (WSCV), each with 95% confi- (3.5 points; 95% CI: 0.35–6.72). The mean composite dence intervals (CIs), to inform composite and knowledge R&C score increased from 10.1 points (SD = domain-specific reliability. We used Cronbach’s alpha (α) 6.3) at T1 to 13.6 points (SD = 2.7) at T2 (3.5 points; to assess composite and domain-specific engagement in- 95% CI: -0.42-7.42). Mean engagement scores were simi- ternal scale consistency. We did not calculate scale lar among SUS and R&C respondents at T1 and T2. consistency for the retrospective knowledge measure, as items in domains 1–4 assessed fact-based knowledge Knowledge and engagement reliability and were not expected to relate. Data were analyzed SUS test-retest reliability data are presented, but not using SAS 9.3 (Cary, NC) and StataSE 14 (College Sta- from R&C due to low retest sample size (n = 6). Eleven tion, TX). of 13 SUS respondents completed the one-week retest survey (84.6%). Composite and domain-specific results Results are shown in Table 2, while per-item results are available Sample characteristics in Additional file 2: Table S2B1-B2. The ICC and WSCV From historical records, we identified 25 SUS stake- for composite knowledge were 0.88 (95% CI: 0.67–0.97) holders and acquired contact information for 23, of and 0.06 (95% CI: 0.04–0.10), respectively. The ICC and which 15 provided consent (65.2%). Consenting par- WSCV for composite engagement were 0.97 (95% CI: ticipants’ names were included in the network roster. 0.89–0.99) and 0.04 (95% CI: 0.03–0.07). Across Thirteen participants completed the first reliability test-retest surveys, the average Cronbach’s α for compos- survey (56.5%). For R&C, we identified 21 stake- ite engagement scale consistency was 0.99. holders and acquired contact information for 12. Eleven provided consent (91.7%) and were included in Phase 3: Prospective study the network roster. Eight participants completed the Methods first survey (66.7%). Most SUS and R&C respondents were female (n =11; Survey modifications We modified the retrospective sur- 84.6% and n = 5; 62.5%). At T1, mean ages were 40.9 (SD = vey to evaluate whole-of-community childhood obesity 9.7) and 41.4 (SD = 10.8) years for SUS and R&C, respect- prevention interventions prospectively (Additional file 3: ively. The majority of SUS and R&C respondents had a Appendix). The social network name generator was lim- Bachelor’sdegreeorhigher(n = 13; 100% and n = 7; 87.5%). ited to free recall in anticipation of prospectively asses- SUS respondents reported affiliations with university/aca- sing new stakeholder networks, in which names were demia (n = 4; 30.8%), community-based organizations (n = not yet known to populate a roster. Items were changed Korn et al. BMC Public Health (2018) 18:681 Page 6 of 11 Fig. 2 Phase 2 stakeholder networks from the Shape Up Somerville and Romp & Chomp interventions. Data are from the three-stage name generator of childhood obesity discussion networks. Visualizations are for illustrative purposes only and were not used to interpret results or draw conclusions from past to present tense. Multiple-choice and true/ We used the same procedures as Phase 2, but with false knowledge items with “correct” answers (domains two-week test-retest reliability. No incentive was offered 1–4) were adapted to fit a 5-point Likert agree/disagree to participants. Procedures for individuals participating scale. This change was implemented to capture greater in research were approved by the Deakin University Hu- variability in scaled responses over time (for future sur- man Ethics Advisory Group. vey applications) and to include a neutral response op- tion, consistent with engagement items. Domain 5 Results knowledge items were changed from a 4- to 5-point Sample characteristics Likert scale for consistency with other items. The en- We identified 13 coalition members from the SEA gagement scale was reduced from 50 to 25 items based Change and GenR8 Change initiatives and acquired con- on Phase 2 κ values (0.3 cut-off) and expert input tact information for all members. All members agreed to (Additional file 2: Table S2B2). participate in the reliability study, with 13 paired re- sponses. The majority of the sample was female (n =11; Prospective reliability study 84.6%) and had a Bachelor’sdegree orhigher (n =11; We tested revised knowledge and engagement scales for 84.6%). The mean sample age was 41.8 years (SD = 12.0). reliability (test-retest and internal scale consistency) in May 2016 among a convenience sample of stakeholders Knowledge and engagement scores from the ongoing SEA (Sustainable Eating and Activity) Mean composite and domain-specific scores are re- Change and GenR8 Change community-based childhood ported in Table 3. On average, respondents agreed or obesity prevention initiatives in Victoria, Australia . strongly agreed with knowledge and engagement items. Korn et al. BMC Public Health (2018) 18:681 Page 7 of 11 Table 1 Phase 2 knowledge and engagement scores at the start and end of stakeholders’ intervention involvement Constructs and # items Max. Shape Up Somerville (n = 13) Romp & Chomp (n =8) domains score Mean score (SD) T2-T1 difference Mean score (SD) T2-T1 difference a a (95% CI) (95% CI) T1 T2 T1 T2 Knowledge Composite 20 20 10.38 (5.16) 13.92 (3.78) 3.54 (0.35–6.72)* 10.13 (6.28) 13.63 (2.68) 3.50 (−0.42–7.42) Domain-specific 1. Problem 4 4 3.23 (1.24) 3.77 (0.60) 0.54 (−0.23–1.30) 3.13 (1.36) 4.00 (0.00) 0.88 (−0.26–2.01) 2. Intervention factors 4 4 0.92 (2.06) 2.54 (1.45) 1.62 (0.50–2.73)* 1.50 (2.00) 1.88 (1.36) 0.38 (−0.96–1.71) 3. Roles 4 4 2.31 (1.93) 3.38 (1.26) 1.08 (−0.07–2.22) 1.63 (2.00) 3.38 (0.92) 1.75 (−0.13–3.63) 4. Sustainability 4 4 2.46 (1.39) 2.46 (1.20) 0.00 (−0.55–0.55) 2.88 (1.25) 2.75 (1.39) −0.13 (−1.07–0.82) d d 5. Resources 4 4 1.46 (1.74) 2.09 (1.83) 0.73 (− 0.64–2.09) 1.00 (1.25) 1.63 (0.79) 0.63 (0.00–1.25)* Engagement Composite 50 25 17.89 (3.28) 18.98 (3.43) 1.09 (−0.55–2.73) 19.02 (2.11) 19.67 (1.52) 0.65 (−0.43–1.73) Domain-specific 1. Dialogue & mutual 11 5 3.99 (0.75) 4.29 (0.54) 0.29 (−0.04–0.62) 3.93 (0.49) 3.98 (0.40) 0.05 (−0.13–0.22) learning 2. Flexibility 8 5 3.68 (0.71) 3.66 (1.16) −0.02 (− 0.70–0.66) 3.89 (0.28) 3.94 (0.27) 0.05 (− 0.05–0.14) 3. Influence & power 4 5 3.12 (0.81) 3.42 (0.88) 0.31 (0.02–0.59)* 3.47 (0.67) 3.66 (0.50) 0.19 (−0.12–0.50) 4. Leadership & 22 5 3.60 (0.71) 3.84 (0.69) 0.23 (−0.12–0.58) 3.78 (0.44) 3.88 (0.36) 0.10 (−0.19–0.38) stewardship e e e 5. Trust 5 5 3.78 (0.78) 4.08 (0.81) 0.30 (− 0.07–0.67) 3.95 (0.67) 4.23 (0.46) 0.28 (−0.05–0.60) Notes: T1 and T2 are the start and end, respectively, of stakeholders’ intervention involvement. CI = confidence interval. *p < 0.05 Paired t-test Knowledge items for domains 1–4 were multiple choice or true/false with the following scoring: − 1 = incorrect; 0 = not sure; 1 = correct. Items for domain 5 were on a 4-point agree/disagree Likert scale with the following scoring (to remain consistent with domains 1–4 scores): − 1 = strongly disagree; − 0.5 = disagree; 0.5 = agree; 1 = strongly agree Engagement items were on a 5-point agree/disagree Likert scale. Data were weighted to reflect the number of items per domain to ease domain-to-domain comparisons. Composite scores are a mean of the total, not a sum of means; therefore, domain scores may not add up to composite score n = 11; difference and 95% CI calculated from paired respondents n =12 Knowledge and engagement reliability understanding of modifiable factors to intervene on Composite and domain-specific results are shown in (SUS only) and available resources (R&C only). Stake- Table 3, while per-item results are available in Add- holders’ mean composite engagement scores did not itional file 4: Tables S3C1-C2. The ICCs for composite change during the interventions but remained high. knowledge and engagement were 0.84 (95% CI: 0.62– Domain-level change in engagement was only ob- 0.95) and 0.58 (95% CI: 0.23–0.86), respectively. Corre- served in SUS with a pre-post increase in stake- sponding WSCVs were 0.02 (95% CI: 0.01–0.03) and holders’ recalled influence and power. In this paper, 0.05 (95% CI: 0.03–0.08). Across test-retest surveys, the we demonstrate the type of network data collected by average Cronbach’s α for composite knowledge and en- the survey. Future prospective survey applications gagement internal scale consistencies were 0.81 and with further social network analysis will allow us to 0.91, respectively. determine if patterns of connectivity among stake- holder groups exist, if there are changes and stability Discussion in group structures, and to identify opportunities to We developed and pilot-tested a novel survey that create cohesive relationships among stakeholders. quantifies three potentially key properties of stake- Phase 2 retrospective ICC values suggest excellent holders involved in whole-of-community childhood test-retest reliability for knowledge (ICC = 0.88) and obesity prevention interventions: social networks, engagement (ICC = 0.97) survey components . knowledge, and engagement. In the retrospective We assessed reliability using T1 data because we as- study with stakeholders from SUS and R&C, we ob- sumed that participants could better recall their pos- served pre-post increases in recalled total mean ition at the start versus the end of intervention knowledge scores, partly driven by increased involvement. Korn et al. BMC Public Health (2018) 18:681 Page 8 of 11 Table 2 Phase 2 reliability results (n = 11 paired responses; Shape Up Somerville Community Advisory Council members) a b Constructs and # items One-week test-retest reliability Internal scale consistency (Cronbach’s α) domains ICC (95% CI) WSCV (95% CI) Test Retest Average Knowledge –– – Composite 20 0.88 (0.67–0.97) 0.06 (0.04–0.10) Domain-specific 1. Problem 4 0.08 (0.00–1.00) 0.14 (0.09–0.23) –– – 2. Intervention factors 4 0.83 (0.55–0.95) 0.19 (0.11–0.33) –– – 3. Roles 4 0.76 (0.44–0.93) 0.15 (0.09–0.25) –– – 4. Sustainability 4 0.70 (0.34–0.91) 0.13 (0.08–0.21) –– – 5. Resources 4 0.59 (0.21–0.88) 0.23 (0.13–0.40) –– – Engagement Composite 50 0.97 (0.89–0.99) 0.04 (0.03–0.07) 0.98 0.99 0.99 Domain-specific 1. Dialogue & mutual 11 0.96 (0.86–0.99) 0.05 (0.03–0.08) 0.95 0.97 0.96 learning 2. Flexibility 8 0.86 (0.61–0.96) 0.10 (0.06–0.16) 0.92 0.95 0.94 3. Influence & power 4 0.88 (0.67–0.97) 0.15 (0.09–0.26) 0.91 0.95 0.93 4. Leadership & 22 0.97 (0.90–0.99) 0.04 (0.03–0.07) 0.97 0.98 0.98 stewardship 5. Trust 5 0.93 (0.78–0.98) 0.07 (0.04–0.11) 0.94 0.97 0.96 ICC intraclass correlation coefficient, WSCV within-subject coefficient of variation, CI confidence interval Reliability results from T1, i.e., the start of Community Advisory Council members’ involvement in the Shape Up Somerville intervention Internal scale consistency was not calculated for the retrospective knowledge survey component. Items were fact-based (multiple choice or true/false), and therefore not expected to relate to each other The Phase 3 revised prospective survey represents To our knowledge, the COMPACT Stakeholder-driven how we intend to use the survey with ongoing interven- Community Diffusion Survey is the first survey developed tions. The ICCs for composite knowledge and engage- that aims to examine change in social network, know- ment scores were 0.84 and 0.58, which suggest excellent ledge, and engagement properties of stakeholders involved and fair-to-good test-retest reliability, respectively . in designing and implementing whole-of-community pre- The decrease in engagement ICC from Phase 2 to Phase vention interventions. Also using the CBPR Model as a 3 may attribute to assessing test-retest reliability at one guiding framework [12, 17], Zoellner and colleagues re- versus two weeks. The Phase 3 two-week assessment cently developed an instrument to assess community cap- reflected concern of participant burden in repeating acity of advisory board members planning a childhood measurements in short turnaround times. We also obesity treatment program. While social networks were present an alternative measure of test-retest reliability: not assessed, the instrument included dimensions related within-subject coefficient of variation (WSCV). Both to our knowledge (group roles, resources, sustainability) knowledge and engagement WSCVs were low (0.02 and and engagement (communication, trust, participation and 0.05, respectively), which indicates 2 and 5% variation in influence, leadership) domains . The authors did not scores among test-retest participants. These findings in- assess test-retest reliability but report good internal scale crease our confidence in the survey’s test-retest reliabil- consistency for most dimensions (α > 0.7), similar to our ity; however, further testing is needed to better Phase 3 scale reliability for knowledge (α = 0.8) and en- understand construct dynamics over time. gagement (α =0.9). Phase 3 knowledge and engagement scores were high Our study strengths include an initial pilot test of on average, indicating that respondents agreed or the survey’s sensitivity among stakeholders involved in strongly agreed with most survey items. Among this two whole-of-community interventions, SUS and cross-sectional sample, respondents may have strongly R&C: studies that occurred nearly concurrently but understood and were invested in their communities’ far apart and with no communication between their childhood obesity prevention efforts. It is also possible stakeholders. The similar results across studies in- that the survey needs further adaptations for local con- creases our confidence in applying the survey to di- text and to capture greater variability in responses. verse whole-of-community interventions in multiple Korn et al. BMC Public Health (2018) 18:681 Page 9 of 11 Table 3 Phase 3 reliability results (n = 13 paired responses; SEA Change and GenR8 Change coalition members) Construct and # items Max. Mean Two-week test-retest reliability Internal scale consistency domains score score (SD) (Cronbach’s α) ICC (95% CI) WSCV (95% CI) Test Retest Average Knowledge Composite 18 25 22.24 (1.28) 0.84 (0.62–0.95) 0.02 (0.01–0.03) 0.83 0.81 0.81 Domain-specific 1. Problem 3 5 4.74 (0.43) 0.43 (0.11–0.82) 0.06 (0.04–0.09) 0.95 0.23 0.68 2. Intervention factors 6 5 4.60 (0.30) 0.67 (0.35–0.89) 0.03 (0.02–0.05) 0.72 0.44 0.58 3. Roles 3 5 4.64 (0.35) 0.82 (0.57–0.94) 0.03 (0.02–0.05) 0.14 0.58 0.35 4. Sustainability 3 5 4.00 (0.51) 0.59 (0.25–0.86) 0.08 (0.05–0.11) 0.76 0.89 0.82 5. Resources 3 5 4.26 (0.58) 0.78 (0.51–0.92) 0.06 (0.04–0.09) 0.78 0.68 0.74 Engagement Composite 25 25 21.34 (1.77) 0.58 (0.23–0.86) 0.05 (0.03–0.08) 0.88 0.94 0.91 Domain-specific 1. Dialogue & mutual 7 5 4.73 (0.32) 0.54 (0.20–0.85) 0.05 (0.04–0.08) 0.79 0.93 0.88 learning 2. Flexibility 3 5 4.28 (0.56) 0.40 (0.09–0.82) 0.09 (0.06–0.14) 0.85 0.77 0.82 3. Influence & power 2 5 3.81 (0.83) 0.55 (0.21–0.85) 0.13 (0.09–0.21) 0.93 0.66 0.84 4. Leadership & 10 5 4.32 (0.43) 0.53 (0.19–0.84) 0.06 (0.04–0.09) 0.81 0.80 0.81 stewardship 5. Trust 3 5 4.21 (0.32) 0.25 (0.02–0.84) 0.10 (0.07–0.15) 0.56 0.84 0.78 ICC intraclass correlation coefficient, WSCV within-subject coefficient of variation, CI confidence interval Scores calculated from test data. All items were on a 5-point agree/disagree Likert scale. Data were weighted to reflect the number of items per domain to ease domain-to-domain comparisons. Composite scores are a mean of the total, not a sum of means; therefore, domain scores may not add up to composite score One item was dropped in the analysis due to zero variance (“Preventing obesity early in life is important”) geographies. Further, we included two rounds of reli- Additionally, we are currently using the COMPACT ability testing, which helped us modify the retrospect- Stakeholder-driven Community Diffusion Survey pro- ivesurveyfor prospectiveuse. spectively to evaluate early childhood obesity prevention Study limitations include small sample sizes and in- studies in Somerville, Massachusetts, USA and Auck- complete representation from SUS and R&C stake- land, New Zealand. By having data from multiple inter- holders due to nonresponse and inability to acquire ventions in communities across the world, we aim to contact information. Phase 2 data were collected retro- iteratively develop and rigorously test an agent-based spectively and responses may be inaccurate due to recall model with wide applicability. Social network, know- and memory issues. We did not assess the reliability of ledge, and engagement data from the survey may also be the social network survey module; however, our type of used to inform community intervention efforts in name generator questions have been extensively used real-time (e.g., to convene stakeholders with high con- across varied survey research settings , and we nectivity to others; to implement stakeholder leadership followed best practices in guarding against recall biases trainings; to develop community-wide channels for dis- in formulating our research design and question wording seminating information and available resources related [38, 39]. to obesity prevention). We expect to further adapt the To increase our understanding of how Stakeholder- survey based on longitudinal study insights and partici- driven Community Diffusion operates within whole-of- pant feedback. Future research is needed to identify po- community interventions, future work will use insights tential sources of response error and to assess the from system science [40–42]. One approach is reliability and validity of revised surveys among larger agent-based modeling, which simulates individuals inter- samples, including predictive validity for implementation acting with one another and their environment with spe- outcomes. cified behavioral rules . We will use SUS and R&C data from this study to parameterize, calibrate, and test Conclusion models that demonstrate knowledge and engagement Whole-of-community interventions may be a major po- diffusion throughout social networks. tential response to curbing the childhood obesity Korn et al. BMC Public Health (2018) 18:681 Page 10 of 11 epidemic. Tailoring precise prevention interventions to Ethics approval and consent to participate All human subjects’ procedures were performed in accordance with the community characteristics and contexts, for example Declaration of Helsinki. The Tufts University Institutional Review Board and stakeholders’ social network structures, knowledge, and the Deakin University Human Ethics Advisory Group approved all study engagement, may lead to sustained success . If that procedures and participants gave written informed consent. is true, then the novel survey developed and evaluated Competing interests for this paper could be a key piece of that tailoring. The authors declare that they have no competing interests. Additional files Publisher’sNote Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Additional file 1: Table S1A. Engagement literature review and scale development. (DOCX 54 kb) Author details Additional file 2: Table S2B1-B2. Phase 2 retrospective Shape Up Friedman School of Nutrition Science and Policy, Tufts University, 150 Somerville per-item knowledge and engagement reliability results (n =11 Harrison Ave., Boston, MA 02111, USA. The Brookings Institution, 1775 paired responses). Data from test-retest surveys administered online one Massachusetts Ave., NW, Washington, DC 20036, USA. Global Obesity Centre week apart in May and June 2015: members of the 2003–2005 Shape Up (GLOBE), Deakin University, 1 Gheringhap St, Geelong, VIC 3220, Australia. Somerville Community Advisory Council. (DOCX 33 kb) Division of Chronic Disease Research Across the Lifecourse, Department of Additional file 3: Appendix. Example COMPACT Stakeholder-driven Population Medicine, Harvard Medical School and Harvard Pilgrim Health Community Diffusion Survey. (DOCX 39 kb) Care Institute, Landmark Center, 401 Park Drive, Suite 401 East, Boston, MA 02215, USA. Department of Sociology, University of Massachusetts Amherst, Additional file 4: Table S3C1-C2. Phase 3 prospective per-item know- 200 Hicks Way, Thompson Hall 532, Amherst, MA 01003, USA. School of ledge and engagement reliability results (n = 13 paired responses). Data Population Health, University of Auckland, Private Bag 92019, Auckland 1142, from test-retest surveys administered online two weeks apart in May New Zealand. Department of Nutrition and Food Sciences, University of 2016: members of the SEA Change and GenR8 Change coalitions in Rhode Island, 125 Fogarty Hall, Kingston, RI 02881, USA. Victoria, Australia. (DOCX 28 kb) Received: 17 November 2017 Accepted: 23 May 2018 Abbreviations CBPR: Community-based participatory research; CI: Confidence interval; COMPACT: Childhood Obesity Modeling for Prevention And Community References Transformation; ICC: Intraclass correlation coefficient; R&C: Romp & Chomp; 1. World Health Organization. Population-based approaches to childhood SD: Standard deviation; SUS: Shape Up Somerville; WSCV: Within-subject obesity Prevention. Geneva: WHO Press; 2012. coefficient of variation 2. IOM (Institute of Medicine). Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation. Washington, DC: The National Academies Acknowledgements Press; 2012. The authors wish to acknowledge Peter Bakun for his assistance with data 3. Bleich SN, Segal J, Wu Y, Wilson R, Wang Y. Systematic review of management and analysis. The authors are grateful to the study participants community-based childhood obesity prevention studies. Pediatrics. 2013; who generously contributed their time and insights for this research. 132(1):e201–10. 4. Wolfenden L, Wyse R, Nichols M, Allender S, Millar L, McElduff P. A Funding systematic review and meta-analysis of whole of community interventions This work was supported by the National Institutes of Health (NHLBI and to prevent excessive population weight gain. Prev Med. 2014;62:193–200. OBSSR, R01HL115485) and the Brookings Institution. The funders were not 5. Boelsen-Robinson T, Peeters A, Beauchamp A, Chung A, Gearon E, involved in the study design, data collection, analysis, or manuscript writing Backholer K. A systematic review of the effectiveness of whole-of- nor in the decision to submit the manuscript for publication. The views community interventions by socioeconomic position. Obes Rev. 2015; expressed in this article do not necessary represent the views of the US 16(9):806–16. Government, the Department of Health and Human Services, or the National 6. Economos CD, Hyatt RR, Goldberg JP, Must A, Naumova EN, Collins JJ, Institutes of Health. Nelson ME. A community intervention reduces BMI z-score in children: shape up Somerville first year results. Obesity (Silver Spring). 2007;15(5): Availability of data and materials 1325–36. The datasets supporting the conclusions of this article are available upon 7. de Silva-Sanigorski AM, Bell AC, Kremer P, Nichols M, Crellin M, Smith M, request to the corresponding author. Sharp S, de Groot F, Carpenter L, Boak R, et al. Reducing obesity in early childhood: results from Romp & Chomp, an Australian community-wide Authors’ contributions intervention program. Am J Clin Nutr. 2010;91(4):831–40. ARK assisted with survey design, collected data, interpreted data, and drafted 8. Millar L, Robertson N, Allender S, Nichols M, Bennett C, Swinburn B. the manuscript. EH, RAH, SA, MWG, MK, BS, and CDE conceptualized the Increasing community capacity and decreasing prevalence of overweight study and study design, assisted with survey design, and interpreted data. and obesity in a community based intervention among Australian JM analyzed and interpreted the data. LM and BO assisted with survey adolescents. Prev Med. 2013;56(6):379–84. design, collected data, and analyzed and interpreted data. MCP assisted with 9. Sanigorski AM, Bell AC, Kremer PJ, Cuttler R, Swinburn BA. Reducing survey design, and analyzed and interpreted data. AT assisted with survey unhealthy weight gain in children through community capacity-building: design and interpreted data. All authors provided critical feedback on the results of a quasi-experimental intervention program, be active eat well. Int draft manuscript. All authors read and approved the final manuscript. J Obes. 2008;32(7):1060–7. 10. Economos C, Blondin S. Obesity interventions in the community: engaged Authors’ information and participatory approaches. Curr Obes Rep. 2014;3(2):199–205. MWG is now Director of the Environmental Influences on Child Health 11. Economos CD, Hammond RA. Designing effective and sustainable Outcomes (ECHO) Program, Office of the Director, National Institutes of multifaceted interventions for obesity prevention and healthy communities. Health, 9000 Rockville Pike, Bethesda, Maryland 20,892, USA. LM is now at Obesity (Silver Spring). 2017;25(7):1155–6. the Australian Health Policy Collaboration, Victoria University, 300 Queen 12. Wallerstein N, Oetzel J, Duran B, Tafoya G, Belone L, Rae R. What predicts Street, Melbourne, Victoria 3000, Australia and the Australian Institute for outcomes in CBPR? In: Meredith Minkler NW, editor. Community-based Musculoskeletal Science (AIMSS), The University of Melbourne and Western participatory research for health: from process to outcomes. 2nd ed. San Health, 176 Furlong Road, St Albans, Victoria 3021, Australia. Francisco: John Wiley & Sons; 2008. Korn et al. BMC Public Health (2018) 18:681 Page 11 of 11 13. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. community-academic advisory board addressing childhood obesity. Health Developing and evaluating complex interventions: the new Medical Promot Pract. 2017;18(6):833–53. Research Council guidance. BMJ. 2008;337:a1655. 38. Brewer DD. Forgetting in the recall-based elicitation of personal and social 14. Ewart-Pierce E, Mejia Ruiz MJ, Gittelsohn J. "Whole-of-community" obesity networks. Soc Networks. 2000;22(1):29–43. prevention: a review of challenges and opportunities in multilevel, 39. Bell DC, Belli-Mcqueen B, Haider A. Partner naming and forgetting: recall of multicomponent interventions. Curr Obes Rep. 2016;5(3):361–74. network members. Soc Networks. 2007;29(2):279–99. 15. Tanner-Smith EE, Grant S. Meta-analysis of complex interventions. Annu Rev 40. Luke DA, Stamatakis KA. Systems science methods in public health: Public Health. 2018;39(1):135-151. dynamics, networks, and agents. Annu Rev Public Health. 2012;33:357–76. 41. Hammond RA. Complex systems modeling for obesity research. Prev 16. Singer HH, Kegler MC. Assessing interorganizational networks as a dimension Chronic Dis. 2009;6(3):A97. of community capacity: illustrations from a community intervention to prevent 42. Frerichs L, Lich KH, Dave G, Corbie-Smith G. Integrating systems science and lead poisoning. Health Educ Behav. 2004;31(6):808–21. community-based participatory research to achieve health equity. Am J 17. Sandoval JA, Lucero J, Oetzel J, Avila M, Belone L, Mau M, Pearson C, Tafoya Public Health. 2016;106(2):215–22. G, Duran B, Iglesias Rios L, et al. Process and outcome constructs for 43. Hennessy E, Ornstein JT, Economos CD, Herzog JB, Lynskey V, Coffield E, evaluating community-based participatory research projects: a matrix of Hammond RA. Designing an agent-based model for childhood obesity existing measures. Health Educ Res. 2012;27(4):680–90. interventions: a case study of ChildObesity180. Prev Chronic Dis. 2016; 18. Gillman MW, Hammond RA. Precision Treatment and Precision prevention: 13:E04. integrating "below and above the skin". JAMA Pediatr. 2016;170(1):9–10. 19. Israel BA, Checkoway B, Schulz A, Zimmerman M. Health education and community empowerment: conceptualizing and measuring perceptions of individual, organizational, and community control. Health Educ Q. 1994; 21(2):149–70. 20. Mattessich P, Murray-Close M, Monsey B. Wilder Collaboration Factors Inventory. St. Paul: Amherst H. Wilder Foundation; 2001. 21. Plested BA, Edwards RW, Jumper-Thurman P. Community Readiness: A Handbook for Successful Change. Fort Collins: Tri-Ethnic Center for Prevention Research; 2006. 22. COMPACT Study. http://www.compactstudy.org/ Accessed 25 May 2018. 23. Osypuk TL, Kehm R, Misra DP. Where we used to live: validating retrospective measures of childhood neighborhood context for life course epidemiologic studies. PLoS One. 2015;10(4):e0124635. 24. McGlashan J, Nichols M, Korn A, Millar L, Marks J, Sanigorski A, Pachucki M, Swinburn B, Allender S, Economos C. Social network analysis of stakeholder networks from two community-based obesity prevention interventions. PLoS One. 2018;13(4):e0196211. 25. Marsden PV. Survey methods for network data. In: Scott J, Carrington PJ, eds. The SAGE Handbook of Social Network Analysis. London: SAGE Publications Ltd; 2011. p. 370–88. 26. Committee on Prevention of Obesity in Children and Youth; Food and Nutrition Board; Board on Health Promotion and Disease Prevention: Preventing childhood obesity: health in the balance. Washington, D.C.: National Academies Press; 2005. 27. Bush R, Dower J, Mutch A. Community capacity index version 2. Centre for Primary Health Care: Brisbane; 2002. 28. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100(Suppl 1): S40–6. 29. National Cancer Institute Science of Team Science (SciTS). Team Science Toolkit. https://www.teamsciencetoolkit.cancer.gov/public/Home.aspx . Accessed 25 May 2018. 30. de Groot FP, Robertson NM, Swinburn BA, de Silva-Sanigorski AM. Increasing community capacity to prevent childhood obesity: challenges, lessons learned and results from the Romp & Chomp intervention. BMC Public Health. 2010;10:522. 31. Butts CT. "Sna: tools for social network analysis" R package version 2.4. 2016. 32. Butts CT. Network: a package for managing relational data in R. J Stat Softw. 2008;24(2). 33. Csardi G, Nepusz T. The igraph software package for complex network research. InterJ Complex Syst. 2006;1695. https://cran.r-project.org/web/ packages/igraph/citation.html. Accessed 25 May 2018. 34. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–20. 35. Allender S, Millar L, Hovmand P, Bell C, Moodie M, Carter R, Swinburn B, Strugnell C, Lowe J, de la Haye K, et al. Whole of systems trial of prevention strategies for childhood obesity: WHO STOPS childhood obesity. Int J Environ Res Public Health. 2016;13(11):1143. 36. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–90. 37. Zoellner J, Hill JL, Brock D, Barlow ML, Alexander R, Brito F, Price B, Jones CL, Marshall R, Estabrooks PA. One-year mixed-methods case study of a
BMC Public Health – Springer Journals
Published: May 31, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera