Question-Order Effects in the Evaluation of Political Institutions in Decentralized Polities

Question-Order Effects in the Evaluation of Political Institutions in Decentralized Polities Abstract This research studies question-order effects in the evaluation of political institutions in socially heterogeneous regions within decentralized countries. Split-ballot experiments were embedded in 3 representative surveys fielded in the Spanish region of Catalonia. Significant consistency effects were spotted in all samples. Respondents who first evaluated the relatively less valued institutions of one governmental level assessed less favorably the relatively better rated institutions of the other level. Clear evidence of the reverse effect was present only in 1 of the experiments. In addition, heterogeneous question-order effects emerged among the 2 distinct national communities coexisting in Catalonia. The article suggests a mechanism beneath this sort of question-order effects and further proposes and empirically tests a remedial solution. Introduction Survey questions are not asked in a vacuum but embedded in the conversational flow of an interview, which involves several additional surrounding items. The order in which questions are asked in surveys can have important effects on the results. In survey methodology, “order effects” are produced by the order in which questions (or response options) are presented to the respondents (Schuman, 1992; Schuman & Presser, 1996; Schwarz, 1999). Questions that are asked first provide a particular context that can alter the way in which subsequent questions are responded. This phenomenon has the potential to threat the substantive interpretation of survey results. Either if the interest is in studying a single moment in time or changes over time, inferences from survey data would be biased were they subject to unexpected order effects owing to how questions are placed on the questionnaire. Question-order effects tend to arise because questions similar in content influence one another (Schuman & Presser, 1996). This research addresses question-order effects that emerge in the evaluation of political institutions in decentralized polities. In particular, it deals with the contamination that can appear when respondents in those contexts have to consecutively assess governmental institutions that have a national and a regional autonomous manifestation. First, it studies the specific case of trust in politicians. It analyzes two similar questions: trust in Spanish and Catalan politicians, using an experiment embedded in two different surveys held in the Spanish region of Catalonia. These experiments test whether the order in which politicians of each of these two political levels of government are presented to respondents has an impact on the response obtained. Is the trust in the relatively more trusted politicians belonging to one governmental level undermined by previously asking about the trust in the relatively less trusted leaders of the other governmental level? Order effects could eventually work in the reverse direction: When relatively more trusted politicians of one level are rated first, comparatively less trusted ones of the other level might see their evaluations improve. Furthermore, in Catalonia, people can identify (to a different degree) with two distinct national communities of reference—the Catalan or the Spanish one—that in turn correspond with the two governmental levels. In such a context, citizens can feel closer to one specific governmental level above the other, according to their subjective national identification. Therefore, question-order effects might vary depending on the national identity of the respondent. Second, this research expands the analysis of this sort of question-order effects by investigating a different political institution: the government, in its national and its regional separate counterpart, exploring the evaluation of government performance. Results of this second experiment confirm what was seen in the former and allow speaking of a general case for question-order effects in the evaluation of political institutions in decentralized polities. Question-Order Effects Researchers identified order effects in early survey research studies (Cantril, 1944; Hyman & Sheatsley, 1950). Despite the important risks order effects can pose to generalization from survey results, not enough substantial literature has dealt with this phenomenon (Schuman & Presser, 1996, p. 24). Order effects can either refer to the order of questions (question-order effects) or to the order of response options within questions (response-order effects1). There is specific literature related to these two avenues of research. The focus of this investigation, however, is on a particular type of question-order effects. Question-order effects usually involve questions about similar issues in which a “transfer of meaning” between them takes place. These situations are usually named after the label of context effects (Schuman, 1992). Question-order effects can be further classified according to the types of relations between questions, yielding to two main types: the part–whole and the part–part combinations (Schuman & Presser, 1996). Part–whole combinations involve two or more questions where one question is more general and comprises the other one. Part–part combinations include questions that are at the same level of specificity. Question-order effects are also classified by the types of effects they produce. The two main types are consistency effects, also known as assimilation effects, and contrast effects. Consistency effects happen when answers to the later question are more similar to the earlier one than would have been if the order of questions would have been different. The rationale for this situation is the respondent's need to appear consistent when answering. Conversely, contrast effects produce the opposite situation: greater differences between questions as a result of the ordering. Both part–whole and part–part combinations can be subject to either consistency or contrast effects (for a detailed list of examples, see Schuman & Presser, 1996). Strack (1992) digs into the psychological mechanism of question-order effects that lead to either consistency or contrast effects. The influence of the preceding question can be conceptualized as a prime that has the functions of activation and information. The function of activation refers to the fact that the prime automatically activates certain concepts that can be brought to mind more easily later on. If the respondent is not aware of the prime, it would mechanically lead to a consistency effect. Conversely, if the respondent becomes aware of the prime, and perceives the relationship between the two questions, they can use this information to intentionally act on it. The preceding question then would have an additional function of information in the sense that it provides information on the intended meaning of the question (Strack, 1992). The results could then be either consistency or contrast depending on whether respondents perceive the two questions as meant to belong together or not. For instance, it would lead to contrast if the respondent recognizes a part–whole combination and deliberately subtracts the part when evaluating the whole. Moore (2002) refers to a complementary sort of question-order effects. Although consistency and contrast are part of the item dimension, there is an additional dimension of question-order effects, potentially independent of the former, named the framework dimension. It relates to the fact that each question is presented to the respondent as either belonging to a comparative or a noncomparative context. In split-ballot experiments, the first question is always the noncomparative, because respondents are still unaware of the second question. However, the second question is the comparative one, because it could be influenced by the former. Framework effects occur when the noncomparative questions obtain different scores than the comparative ones. If the scores of the noncomparative questions are higher than the comparative, Moore speaks of subtractive effects, whereas if the scores are lower, the effects are called additive. All in all, perceiving questions as part of a comparative context can affect results, irrespective of the item dimension.2 Decentralized Political Systems This research investigates question-order effects that take place when respondents evaluate political institutions in socio-politically heterogeneous regions within countries that have a decentralized multilevel structure of government. It studies the particular case of Catalonia within Spain, a territory with autonomous political institutions within the quasi-federal Spanish political system, in which citizens have divergent national allegiances to either Catalonia or Spain. Within Spain itself, other similar cases are those of the Basque country, Galicia, or the Valencian community. There are also analogous situations over the world where regional heterogeneity is based on diverging national allegiances, such as Veneto or South Tirol in Italy, Scotland in the U.K., the Flemish community in Belgium, or Quebec in Canada. However, if regional heterogeneity is more broadly defined in terms of ethnic, racial, religious, or even ideological diversity, we could include many more cases.3 In such decentralized and internally diverse polities, order effects may arise when respondents have to evaluate consecutively two political objects that belong to the same general category (e.g., politicians) but refer to each specific polity or layer of government (e.g., regional- or national-level politicians). Furthermore, I will show that the type of question-order effect that occurs is consistency and that it follows the basic logic described by Strack (1992). These consistency effects are related to the prime of the preceding question referred to one governmental level and follow an automatic sort of mechanism that instills respondents with a drive to compensate or balance the ratings of the posterior question referred to the other governmental level. In so doing, it produces a contamination across levels. Decentralized polities might create informational effects for respondents who have to sequentially evaluate institutions of each level of government, making the order of questions consequential. This type of regional information cues that may affect perceptions of governance factors has not yet been investigated, to the best of the author’s knowledge. Question-order effects of this sort might, for instance, have the capacity to artificially downgrade the general evaluation of the performance of the regional-level government as to obtain a falling grade if asked after that of national-level government, or vice versa. Furthermore, this type of effects could be quite usual and might have gone unnoticed until now. Both the political context in which the effects take place, decentralized political systems, and the survey questions affected, evaluation of political institutions of different governmental levels, are common. Most countries in the world have decentralized constitutional arrangements, and many decentralized regions are dissimilar or have distinctive identity or characteristics. Besides, the survey questions affected are not artificial or rare. The evaluation of political institutions of each governmental level is part of the usual repertoire of questions of most political surveys in those contexts. Survey practitioners in decentralized polities have to deal with the possibility of this type of question-order effects occurring on a regular basis. To mention just the Spanish case, any political survey that aims at measuring political evaluations of national and regional autonomous institutions in the same questionnaire (the majority of political surveys in this decentralized country) should keep in mind the analysis and recommendations presented here. Previous Research The existence of question-order effects in the evaluation of political institutions in the context of decentralized polities has not received scholarly attention yet, as far as the author knows. This is why this article can only relate to previous investigations on question-order effects about similar survey questions but not about similar contexts. Since the context -decentralized polities- is a key factor involved in the emergence of these effects, it is precisely the lack of research antecedents that makes the present contribution most valuable. With regard to studies on similar survey questions, not much research has specifically dealt with question-order effects in trust in politicians, and even less with government performance evaluation. A previous investigation explicitly analyzing trust issues was the experiment by Schwarz and Bless (1992a) priming scandals before general and specific questions on trust in politicians. Making respondents think about a politician who was involved in a scandal decreased the trust of politicians in general through a consistency effect but increased the trust in other individual politicians by means of a contrast effect. In a similar vein, Erikson, Luttberg, and Tedin (1988) showed how American citizens distrust Congress but trusted their own representative in this institution. Schwarz (1999) offers a possible reason for this effect. The media presents extreme cases of untrustworthiness and corruption to the public, which become highly accessible to memory. These extreme examples can strongly influence the representation of the general trust, but they can be used as a standard of comparison and contrast in evaluating individual instances. Moore (2002) also found question-order effects when asking about the honesty and trustworthiness of pairs of U.S. political leaders in Gallup polls. Asking first about Al Gore benefited the ratings of Bill Clinton and asking first about Clinton undermined those of Gore. Overall, both ratings became more similar to what they would have been had they been asked independently, in tune with consistency effects. In the case of Gingrich and Dole, Moore spotted contrast effects: Respondents tended to emphasize the differences between the two leaders rather than the similarities as a result of the order of questions. Stapel and Schwartz (1998) investigated carryover effects between parties (the Republican), candidates (Dole), and party members (Powell) in the evaluation of these objects. They found that initial judgments of the party affected subsequent evaluations about candidates, and vice versa in an additive manner. Their results were consistent with the Schwarz and Bless (1992b) inclusion/exclusion model of social judgment. Other researchers of question-order effects have dealt with relatively related topics such as presidential popularity (Darcy & Schramm, 1979; Sigelman, 1981) and candidate preferences for governor and senator (Crespi & Morris, 1984). Crespi and Morris (1984) studied question-order effects on preferences for candidates to two different U.S. races, the senatorial and the gubernatorial, using a split-ballot experiment. Preferences for candidates to the Senate became affected by the order in which preferences for governor were asked. They concluded that asking first about the race in which a party’s candidate is stronger has a coattail effect4 among the party’s followers when preference in the other race is measured first. Another important implication from this research was that order effects were not homogeneous across the sample but associated with specific political attitudes (such as party identification, candidate preference, or education).5 In any case, to the best of my knowledge, no research has specifically dealt with the evaluation of political institutions in heterogeneous regional contexts within decentralized countries. In such settings, the two different levels of government (regional and national) also represent two potentially adversarial communities of reference, as citizens have a repertoire of identities to resort to. When evaluating a political object that has a separated manifestation at the national and at the regional levels, informational cues might arise and foster potential question-order effects. In Catalonia, these adversarial regional identities take the form of opposed national allegiances. Catalonia is divided in terms of national identity, with a majority defined as mainly or exclusively Catalan (50% according to Centre d’Estudis d’Opinio [CEO] 816, 2016) and a minority defined as Spanish (11%), and a more heterogeneous group of dual identifiers (34%). In recent years, the Catalan independence debate has made identity issues more salient to the public (Tormos, Muñoz, & Hierro, 2014), and identities turned more conflictual (Hierro & Gallego, 2016). This conflict of identities might have a role in loading the survey questions referred to each level of government in the experiments presented here. In this respect, Wilson (2010) investigated the impact of group identity and identity activation on question-order effects. He found that question order interacted with the racial group of the respondent to influence beliefs about the amount of interracial prejudice between Blacks and Whites in the United States. Wilson showed how in-group members tended to view out-group members as having more dislike toward their in-group only when the in-group was asked about first, a contrast effect. When in-group members were evaluated after the out-group, they viewed their in-group’s dislike as similar to that of the out-group’s, an assimilation. This research will test whether indeed identity plays an active a role as a mechanism guiding question-order effects (identity activation) in tune with Wilson’s findings, or just a passive one as a source of heterogeneous effects that arise from the particular aggregate social configuration of the region. Trust in Politicians in a Multilevel Setting In multilevel political systems, such as the Spanish one, pollsters in charge of political surveys usually ask citizens’ perception of political institutions of different governmental levels in the same questionnaire. Informational cues from institutions of the level of government assessed first can influence the response to the corresponding political institution at the other level evaluated later. I test if asking on the first place for the trust in Spanish politicians influences the trust expressed in Catalan politicians asked later on, and vice versa. I first use the survey on the political situation from 20156 (CEO 806, 2015) from the CEO.7 The questions on trust in Spanish and Catalan politicians were placed together at about the middle of the questionnaire. They had the following format: P20a. All in all, please rate the degree of trust you have in the Catalan politicians in a scale from 0 to 10, where 0 is no trust at all and 10 is a lot of trust. P20b. All in all, please rate the degree of trust you have in the Spanish politicians in a scale from 0 to 10, where 0 is no trust at all and 10 is a lot of trust. In the administration of the questionnaire, the order of appearance of these two questions was randomized in the context of a split-ballot experiment. Half of the sample was exposed to a questionnaire in which trust in Catalan politicians was asked first and trust in Spanish politicians just after. The other half of the sample was first faced to the rating of Spanish politicians and right after they had to rate Catalan politicians. The treatment of the experiment was, therefore, the order in which both questions on trust in Spanish and Catalan politicians were formulated. This single split-ballot experiment with two groups could be essentially understood as containing two pairs of treatment and control conditions (Figure 1). One pair (Structure 1) tackles whether the question on trust in Spanish politicians, Q(Spa), has an effect on trust in Catalan politicians, Q(Cat). The other one (Structure 2) seeks whether the question on trust in Catalan politicians, Q(Cat), has an impact on trust in Spanish politicians, Q(Spa). The experiment can also be conceptualized according to Moore’s framework dimension (2002). Then we would have a pair of comparative and a pair of noncomparative questions. The first questions in any of the experimental groups would be considered the noncomparative questions, whereas the second questions would be the comparative ones. Figure 1 View largeDownload slide Experimental conditions to test question-order effects Figure 1 View largeDownload slide Experimental conditions to test question-order effects The prime implied by the preceding question can be qualified in terms of the direction of its valence, and then speak of a positive or a negative valence of the prime (Strack, 1992). We know from previous surveys (e.g., CEO 804, 2015) that Spanish politicians in Catalonia are overall less trusted than Catalan politicians. Therefore, it can be anticipated that the dominant valence associated to Spanish politicians would be negative. In fact, this negative appreciation of national-level politicians in comparison with regional-level ones extends to other political institutions such as the government or the parliament. In Catalonia, Catalan institutions are viewed under a more positive light than Spanish ones (Tormos, 2017). In this region, the image of the Spanish politicians might be associated with issues repeatedly appearing in the media such as corruption scandals, inability to deal with the crisis, problems of representation, and their responsibility on the current bad state of relations between Spain and Catalonia. The valence associated with Catalan politicians is less negative—still not good though. Their image might share some negative elements with the Spanish one, but it could also be associated with positive issues, at least for some, such as a certain sense of differentness with respect to politicians of the rest of Spain, as well as the projected hope and expectations for an eventual secession and the building of a new state. In view of these informational cues that may emerge when evaluating institutions of each governmental level, we can derive that in Catalonia, trust in Spanish politicians as a preceding question might play the role of a negative prime, whereas trust in Catalan politicians may be a positive prime in relative terms, or at least a less negative one. A relevant element when considering the psychological mechanisms leading to question-order effects is whether the respondent is aware of the priming episode. Previous experiments (Lombardi, Higgins, & Bargh, 1987) indicate that subjects able to recall the prime were more likely to show contrast effects, whereas subjects unable to recall it were prone to consistency effects. The influence of the prime of the preceding question may only be counterbalanced in the form of contrast if subjects are conscious of it and react upon this information. If they are not aware, a mechanical process of assimilation (consistency) would likely apply. In this experiment, my assumption is that the prime is so subtle and apparently inadvertent that it would be improbable for respondents to be aware of it or infer intentionality. As a result, the most likely sort of effects that may appear would be consistency effects. However, if contrast effects shall be spotted instead, it would imply that respondents were aware of the prime, and other mechanisms such as identity activation might be operating. Another consideration is whether the experiment is dealing with a part–whole combination of questions or with a part–part. It is not clear which of these two combinations is perceived by the respondent. In principle, it may seem a part–whole combination, given that formally Catalan politicians belong to the larger set of Spanish politicians. However, in practice, respondents may not relate to this scheme depending on their national identification. To people who feel predominantly Catalan (the largest share), the question pair might appear more like a part–part than a part–whole combination. If the part–whole logic shall play a relevant role in this experiment, we might be able to witness contrast effects when respondents subtract the part when evaluating the whole as in the inclusion/exclusion framework. All these considerations help in the elaboration of the hypotheses. I sustain that the type of question-order effects that will appear in the context of this decentralized polity as a result of the experiment would be consistency effects. However, consistency effects can potentially operate in two directions: positive or negative, depending on the regional cues that influence the valence of the prime (the preceding question). On the one hand, it is expected that asking first for the governmental level with the less trusted politicians (negative prime) would undermine the ratings of the relatively more trusted ones at the other level, and therefore produce a negative consistency effect. On the other hand, asking first for the government level with the more trusted politicians (positive prime) could better the ratings of the governmental level with the less trusted ones, generating positive consistency effects. This would translate into the following general hypotheses: Hypothesis 1: Negative consistency effects: When the Spanish institution is asked first, it will negatively influence the ratings of the Catalan institution that would be asked second. Hypothesis 2: Positive consistency effects: When the Catalan institution is asked first, it will positively affect the ratings of the Spanish institution asked right after. If contrast effects shall appear instead of consistency, it could be a signal that either identity activation (Wilson, 2010) or inclusion/exclusion processes (Schwarz & Bless, 1992b) may be taking place. Besides testing for the presence of consistency or contrast effects in the item dimension, this research also explores the potential existence of framework effects (Moore, 2002), comparing the comparative and the noncomparative pair of questions on trust in Catalan and Spanish politicians to look for either subtractive or additive effects. Question-order effects are not necessarily an across-the-board phenomenon. As in the Crespi and Morris (1984) experiment, effects might differ quite markedly across subgroups of the sample. I can anticipate a clear source of heterogeneity related to the national identification of the respondent. In Catalonia, the subjective national identification of individuals is a powerful filter through which the sociopolitical reality is evaluated (Guinjoan & Rodon, 2016). In fact, the national identification of the respondent is the most powerful explanatory factor of the differences in trust between Spanish and Catalan politicians among a set of usual structural predictors of Catalan political behavior (see the regression analysis in Table A1 in the appendix). With some confidence, I can forestall that feeling Catalan vis-à-vis Spanish will influence the trust the respondent has in either Catalan or Spanish politicians.8 The third hypothesis takes into consideration this heterogeneity among the Catalan public, and the consequences it can have for the experiment. The positive and negative valences of the primes would be different according to the national identification of respondents. However, it will still be valid that a negative prime would potentially produce negative consistency effects across governmental levels, and a positive prime would generate positive consistency effects. More specifically, this translates into the following general propositions: Hypothesis 3: Heterogeneous effects by subjective national identification: For those who feel predominantly Catalan, (H3.1) being first exposed to the evaluation of a Spanish institution would negatively affect their ratings of the Catalan institution. And (H3.2) being first faced with the evaluation of a Catalan institution would positively affect their ratings of the Spanish institution. For those who feel predominantly Spanish, however, (H3.3) being first exposed to the evaluation of a Catalan institution would negatively affect their evaluation of the Spanish institution. And subsequently (H3.4) being first exposed to the evaluation of a Spanish institution would positively affect their rating of the Catalan institution. An alternative explanation in competition with the third hypothesis refers to the activation of group identities. As noted by Wilson (2010), question order and group membership can interact to strengthen in-group and out-group categorization, leading to a contrast effect by which one’s in-group is favored and differentiated over the out-group. The first question can heighten the salience of group membership and promote group comparison together with the drive to view one’s in-group more positively and out-group more negatively. As Wilson observed, groups will differentiate their in-group with an out-group when their own group is considered first but will assimilate their group to the out-group in a comparative context. Here, if the premise about identity activation is correct, those who feel Catalan when asked about a Catalan institution first would tend to differentiate the ratings of the Spanish institution asked about second in tune with a contrast effect, and vice versa for those with a mainly Spanish identity. In this situation, respondents would be aware of the comparative context; their identity would become activated and would consciously respond differentiating the ratings of the in-group institution asked on the first place from those of the out-group asked second. Results If we take the whole sample estimates, without distinguishing among experimental groups, Catalan politicians are more trusted than Spanish leaders (see the upper part of Table 1). In both cases, the average is below the middle point, but Catalan politicians obtain a mean of 3.77, whereas Spanish politicians get a 2.65. This difference of 1.12 is statistically significant (p < .001). Is trust in Catalan political leaders downgraded when trust in Spanish ones is asked on the first place? The middle part of Table 1 presents the average trust in Catalan and Spanish politicians across the two experimental groups. The group that assesses their trust in Catalan politicians first gives them an average of 4, higher than the mean obtained when the question came after the rating of Spanish politicians (3.55). It is a statistically significant difference of 0.45 among the two groups (p < .001). Hypothesis 1 becomes then confirmed: A negative consistency effect takes place affecting the trust in Catalan politicians as a result of the impact of the negative prime established by the preceding question on trust in Spanish politicians. Changing the order, however, does not affect the evaluation of the Spanish politicians. Both the group in which the question on Catalan politicians is asked first and the group in which it is asked second assess Spanish politicians in the same way (an average of 2.63 and 2.66, respectively). Therefore, Hypothesis 2 stating a potential positive consistency effect is not confirmed. The potentially positive prime of the preceding question—trust in Catalan politicians—does not have the capacity to influence the assessment of Spanish politicians. It could either be because the prime is not really powerful or positive enough (Catalan politicians are only slightly more trusted than Spanish ones) or because the actual assessment of Spanish politicians is so negative that it is difficult to change it by any means. As a result of the experiment, the average difference among the Catalan and Spanish leaders is reduced from 1.37, in the group in which Catalan leaders are asked first, to 0.89, in which they are asked second. The distance between these two differences is 0.48, and it is statistically significant (p < .001). This is in fact another way of assessing the consistency effect: by means of observing a higher resemblance between the two groups of politicians after the treatment. No signs of contrast effects are spotted; therefore, the inclusion/exclusion frame seems not to be operating. In addition, the current experiment seems not to have effects on the framework dimension. The average trust in Spanish and Catalan politicians is very similar in the comparative and noncomparative frameworks. Therefore, the fact that this pair of questions is presented together does not per se generate an additive or a subtractive effect. In the upper part of Table 2, respondents are divided into three groups according to their national identification in response to the Linz–Moreno question9: (1) those who feel predominantly Catalan (“only Catalan” and “more Catalan than Spanish”), (2) the dual identifiers (“as Catalan as Spanish”), and (3) the predominantly Spanish (“only Spanish” and “more Spanish than Catalan”). Those who feel predominantly Catalan evaluate positively the Catalan leaders (5.06) and negatively the Spanish leaders (2.07). Conversely, respondents who feel mainly Spanish value better the Spanish politicians (3.92) than the Catalan politicians (1.95), although not really well. Dual identifiers trust Spanish (3.22) more than Catalan (2.44) politicians, presenting a pattern more similar to those who feel Spanish though more attenuated. On the whole, the national identification of the respondent clearly influences trust in both groups of politicians. Respondents tend to look favorably to the political leaders of their own national community of reference, and with a side-glance the leaders of the other national community. This evidence points to a potential heterogeneous effect of the treatment conditional on the national identification of the respondent. According to the main hypothesis defended here, if respondents who feel mainly Spanish are first exposed to the question on trust in Catalan politicians, they would rate Spanish politicians worst; the reverse to what would happen in the group of those who feel predominantly Catalan. The reason for it would be that the more trusted politicians for those who feel Spanish are Spanish politicians; the group of politicians that is their positive reference point. In contrast, Catalan politicians would be their negative reference point. However, if group identities should become activated (the alternative explanation), consistency will only happen when the out-group institution is asked first, as a means of compensation. Yet, when the out-group institution is asked second, contrast effects would tend to emerge, as respondents will try to differentiate the institution belonging to the in-group from that of the out-group. Table 2 also presents the results of the experiment across national identity groups. The experiment only has statistically significant effects in the group that feels predominantly Catalan (the one with a larger subsample). In this segment, when trust in Catalan politicians is asked first, the resulting level is as high as 5.28, well above the midpoint of the scale. Whereas, when this group of respondents is first exposed to the rating of Spanish politicians, their trust in Catalan politicians falls to 4.82. This is a 0.46 statistically significant difference (p < .018). In the other groups (duals and Spanish), the effects of the experiment do not reach the threshold of statistical significance. However, as it was expected, question-order effects have the inverse impact. In the segment of Spanish identifiers, asking first for their trust in Catalan politicians reduces their trust in Spanish politicians, though the difference is not statistically significant, given the small size of the subsample. In this same group, asking first for the Spanish politicians increases their trust in the Catalan ones: a positive consistency effect. However, the differences neither reach the level of statistical significance. The type of question-order effect spotted here refutes the alternative explanation based on identity activation. There are no signs of contrast effects among identity groups. Therefore, in tune with Strack (1992), the mechanism seems much more automatic and related to the respondent’s reflex reaction to the positive or negative prime of the preceding question, which is informed by regional cues. Although identity seems not to play a direct role in the causal mechanism, it has an indirect one as source of heterogeneous effects across groups and connected with the informational cues about governmental levels. Given the aggregate distribution of national identifications in the population, the order of questions has the opposite effect in Catalan identifiers as compared with Spanish identifiers, although both groups experience the same kind of mental process. The case of a region with autonomous institutions within a decentralized country provides the context for these effects to happen. Decentralized contexts of this sort facilitate the presence of information cues affecting each level of government. In this case, depending on the national identification of the respondent, the cues have different effects, although the mechanism at stake is the same. To further illustrate the heterogeneous effects of the treatment by national identity, Figure 2 shows the coefficients of the interaction of both variables (from a regression further including the treatment and national identity as predictors). Although they are not statistically significant, we can clearly observe that treatment effects are opposed, according to the respondent’s national identity. Asking first for the Spanish politicians to those who have a Catalan identity makes them assess Catalan politicians worse, whereas for those with a Spanish identity, asking first for the Spanish politicians makes them rate Catalan leaders better. Figure 3 shows the interaction of the treatment with national identification on trust in Spanish politicians. Again, the effects are not statistically significant; however, it is possible to see the inverse pattern of effects. Asking first for the trust in Catalan politicians tends to reduce the trust in Spanish politicians in the group of those who feel predominantly Spanish. In contrast, asking first for the trust in Catalan leaders tends to improve the trust in Spanish leaders in the group of those who consider themselves mainly Catalans. Figure 2 View largeDownload slide Linear regression to explain trust in Catalan politicians. Question-order effects by national identity (regr. coeff.) Figure 2 View largeDownload slide Linear regression to explain trust in Catalan politicians. Question-order effects by national identity (regr. coeff.) Figure 3 View largeDownload slide Linear regression to explain trust in Spanish politicians. Question-order effects by national identity (regr. coeff.) Figure 3 View largeDownload slide Linear regression to explain trust in Spanish politicians. Question-order effects by national identity (regr. coeff.) Apart from subjective national identification, no other variable usually mentioned in the literature of question-order effects (such as education, age, sex, or interest in politics) interacts with this particular treatment. Verification A usual practice when performing experiments is to check for imbalances among experimental groups after random assignment. As noted by Mutz and Pemantle (2015), randomization checks should be used with the purpose of testing whether the randomization mechanism has worked properly. An imbalance in the covariates after random assignment is not a problem per se if the randomization mechanism performed well. Usual statistical tests such as analysis of variance (ANOVA) are designed to account for this sort of imbalance. There would only be a problem if the source of the imbalance is to be found on a defective random assignment process. To explore possible imbalances I perform a set of randomization checks. Figure 4 shows a test of the equivalence of the two groups by means of a logistic regression in which the dependent variable is the experimental group and the independent variables are the relevant characteristics that might distinguish them. Figure 4 View largeDownload slide Randomization test. Logistic regression to explain experimental group membership Figure 4 View largeDownload slide Randomization test. Logistic regression to explain experimental group membership There are statistically significant differences between the two groups by subjective national identification and interest in politics. The group that rates Spanish politicians first contains more dual identifiers and less who feel only Catalan, as well as respondents with a little less interest in politics. These differences are unlikely attributable to an effect of the experiment. In the sequence of the questionnaire, the question on subjective national identification is asked much later than the treatment, near the end of the interview. Besides that, the treatment itself, a question-order change, is so mild and subtle that it is highly unlikely it would have had the power to influence the national identification of the respondent. From this it follows that the difference between the two groups might be owing to mere chance, an “unlucky draw”, and not to an unexpected consequence of the experiment. I performed additional randomization checks to rule out the possibility that an interviewer effect or an effect of the way in which the fieldwork was carried out could eventually be the reason of the differences in interest in politics and subjective national identification across the two experimental groups. I explore differences between the two groups by the profile of the interviewer who performed the fieldwork. There is a theoretical possibility that specific interviewers were assigned to one group and not to the other. If this hypothetical uneven assignment of interviewers would have been combined with the fact that some of them were more prone to introduce an involuntary bias in the process of interview, respondents of one of the groups could have expressed more interest in politics or a particular national identity owing to an interviewer effect. However, Table 3 indicates that there are no differences with respect to the interviewers assigned across the two groups. There is only a difference due to the sex of the interviewer, but this trait has no effect on the experiment (this is tested through a regression analysis not presented for simplicity). There are no differences in the length of the interviews, the day in which they were performed, and the language used. These additional tests reinforce the idea that differences between the two groups might be owing to mere chance, and not to a fieldwork problem that might have biased the random assignment process. Table 3 Bivariate Randomization Tests Across Experimental Groups ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 Table 3 Bivariate Randomization Tests Across Experimental Groups ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 Replication To be able to safely generalize the presence of question-order effects, it is convenient to repeat the current experiment on different samples of the same population. Descriptive inference as well as causal inference relate to the idea of repeated samples and experiments. Replication reduces variability in experimental results and increases the confidence on the effects of the treatment. If a treatment has a truly causal impact, the average effect of different replications would show it. A replication of the first experiment was embedded on a similar survey performed by the same institution just 4 months after the first one (CEO 816, 2016). On this occasion, it was a CAPI survey representative of the same population and with a larger sample size (N = 1,500). Table 4 presents the main effects of the experiment. Table 4 Main Effects of the Second Experiment Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 **Sig. < 0.01; *Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2+ x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Table 4 Main Effects of the Second Experiment Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 **Sig. < 0.01; *Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2+ x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Again, those who assess the Spanish politicians first give a worst rating to the Catalan politicians afterwards. The magnitude of this negative consistency effect is similar to that of the first experiment, though slightly smaller. In addition, in this new experiment, positive consistency effects are also spotted, unlike in the first probe. When respondents have to rate Catalan politicians initially, they assess Spanish politicians better later on. Once more, this second experiment does not portray any framework effects. The average trusts in Spanish and Catalan politicians are almost the same in the comparative and in the noncomparative frameworks. Heterogeneous question-order effects do not reach the level of statistical significance on this one occasion (not presented for the sake of simplicity). In this case, both experimental groups were balanced in the key covariates after random assignment.10 Evaluation of Government Performance An additional experiment was introduced in another survey (CEO 835, 2016) to test question-order effects in the evaluation of a different political institution, the government, using a measure of respondents’ assessment of government performance. Respondents had to evaluate on a scale from 0 to 10 both the performance of the national institution and its regional autonomous counterpart. Results from this experiment are presented in Table 5. They follow the same pattern as in the case of trust in politicians. When respondents evaluate the Catalan Government first, they give it a better rating (4.53) than when they do it after evaluating the Spanish Government (4.00), in accordance with a negative consistency effect (p < .001). In this case, positive consistency effects do not reach the level of statistical significance but can be spotted nonetheless. When question-order effects are decomposed by the national identification of respondents, results are in a similar vein to previous experiments on trust in politicians. Table 5 Main Effects of the Third Experiment and Results by Subjective National Identification How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 **Sig. < 0.01; *Sig. < 0.05. Table 5 Main Effects of the Third Experiment and Results by Subjective National Identification How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 **Sig. < 0.01; *Sig. < 0.05. These results further speak of the robustness of the findings and allow generalizing beyond a single specific political institution. Decentralized countries with socially heterogeneous regions can provide the context for such question-order effects to happen when citizens of a region have to sequentially evaluate political institutions that have a national and a regional autonomous manifestation. Conclusions This research has analyzed question-order effects in the evaluation of political institutions using split-ballot experiments embedded in representative surveys fielded in the Catalan region of Spain, a decentralized country. Results indicate that when respondents of this region who have to evaluate political institutions with a national and a regional independent expression are first exposed to the assessment of the less appreciated institution belonging to one of the two levels of government, the ratings of the relatively more appreciated institution at the other governmental level become affected by means of a negative consistency effect. In Catalonia, across three different samples, Catalan institutions are, on average, more valued than Spanish ones. The three experiments performed show how the ratings of Catalan political institutions diminish when the evaluation of their Spanish counterparts is requested first. Clear evidence of the reverse, a positive consistency effect from the ratings of the Catalan institutions to those of the Spanish ones, is only statistically significant on the main effects of one of the experiments. Apart from the item dimension, no additive or subtractive effects in the framework dimension of question-order effects were found in any of the experiments. The subjective national identification of the respondent plays a role as a source of heterogeneous effects across groups, but not in the very mechanism of question-order effects. This is assessed by the absence of contrast effects in the experiments, which allows inferring that no process of identity activation might have been at play. Catalan citizens can either identify with Spain or Catalonia in national terms (or have a dual identification). Respondents who feel Catalan rate Catalan institutions better than Spanish ones, whereas interviewees who feel Spanish do the reverse. Therefore, respondents with a Catalan national identification end up rating Catalan institutions worst if they are first exposed to the evaluation of a Spanish institution. Likewise, respondents with a Spanish national identity express less appreciation of Spanish institutions when they have to rate a Catalan institution first (although results are not statistically significant owing to the small subsample of Spanish identifiers). Results also point to positive consistency effects across subgroups on national identification. However, the reduced sample sizes of those subgroups do not usually allow for a safe generalization of the relationship. To summarize, evidence from the main effects of these experiments confirm the hypothesis on the negative consistency effects (Hypothesis 1) but provide only partial support to the positive consistency effects one (Hypothesis 2). Results from the interaction of the experiment with subjective national identification speak favorably of each statement of the third hypothesis (Hypothesis 3) on the heterogeneous effects of the treatment had we used larger subsamples. The alternative explanations for the psychological mechanisms based on group identity or on a part–whole subtraction become discarded because contrast effects are not spotted in any case. The necessary context for these question-order effects to happen is that of a decentralized country with a multilevel structure of governance, a constitutional arrangement frequent across the world. These politically decentralized contexts enable the existence of information cues that affect the perception of the different governmental levels and allow for eventual contaminations across levels. The implications of these results are twofold. On the one hand, they add to the literature on question-order effects the case of the evaluation of political institutions in heterogeneous regions within decentralized countries, and on the other hand, they have practical consequences for survey design. In regions within decentralized countries where two or more group identities coexist, the order in which questions about institutions of either of those communities are presented to respondents might not be neutral. The risk of bias is likely worsened when some kind of clash between the region and the nation takes place, such as in the current case of Catalonia in Spain. A number of comparable cases in the world might fulfill this condition, as decentralized countries and regional diversity are abundant. The evidence presented here calls for a close monitoring of the order of questions in such contexts.11 Apart from substantive implications, a clear practical consequence for survey practice can be derived from this research. The mechanism involved in the sort of question-order effects identified here is related to the prime of the preceding question referred to a given governmental level, which is loaded with either negative or positive valences. Therefore, an advice for future surveys would be to separate the items related to one governmental level from those of the other level over the questionnaire, at least using some “buffer” questions (Lasorsa, 2003). If they are put together, randomization of the order of appearance—the usual cautionary procedure in survey practice (Oldendick, 2008)—would only reduce the bias by half (as can be seen in the experiments presented here), but it will continue to exist. However, separating the blocks of questions related to each governmental level can deactivate the priming effect that otherwise takes place when they are next to each other in the questionnaire. This remedy has been tested in a recent survey (CEO 850, 2017). Question blocks on Spanish and Catalan institutions were separated by seven buffer questions, and the administration of the survey followed a split-ballot design changing the order of blocks. The results obtained confirm that separating the items allows circumventing order effects and obtaining unbiased estimates (see Table A2 in appendix for more information). Supplementary Data Supplementary data are available at IJPOR online. Acknowledgements The author would like to thank Jordi Argelaguet, Malcolm Fairbrother, Willem Saris, Jordi Muñoz, and the two anonymous reviewers for their valuable comments and suggestions. Conflicts of interest: None declared. Raül Tormos (PhD) is a survey scientist at the Centre d’Estudis d’Opinió (CEO) of the Government of Catalonia in Spain. He is also a lecturer at the University of Barcelona, Department of Sociology. Footnotes 1Response-order effects usually arise from the difficulty the respondent has in keeping in mind all the alternatives presented, yielding to primacy or recency effects. 2Schuman and Presser (1996) also mention additional types of question-order effects such as salience, rapport, fatigue, and initial frame of reference effects. Potentially relevant to this research are the initial frame of reference effects. This type of effects occurs when respondents are requested to rate a series of items on numerical scales. In such situations, a problem establishing an initial reference point arises. This happens because respondents when asked to form a judgment about a particular matter might need a standard of comparison (Sudman, Bradburn, & Schwarz, 1996). The first question is then used as this sort of standard. 3Just to mention some concrete ones: the Muslim-majority states in the Indian federation, the North–South divide in the multiethnic Nigeria, or the ethnically diverse federation of South Africa. 4A coattail effect is the propensity for a popular leader of a political party to attract votes for other candidates of that particular party in an election. 5Lacy (2001) studied question-order effects related to individual choices applying the theory of nonseparable preferences. He explored question-order effects when preferences were separable and when they were not, finding that people with nonseparable preferences had greater response instability across question orders. However, Lacy’s object of study, choices or preferences on issues, differs from that of the current investigation: the ratings of institutions. Furthermore, if the theory of nonseparable preferences would have been applicable to the case studied here, contrast effects would have been more likely to appear than consistency. For instance, if preferences for Catalan and Spanish institutions were nonseparable, when a person who identifies very much with Catalonia (and very little with Spain) is asked about the Spanish institution first, she could use this information to differentiate the assessment of the Catalan institution asked on the second place. This is not what finally happens in the case studied here. 6This is a CATI survey with a stratified proportional sample of 1,050 individuals, representative of the population above 17 years of age living in Catalonia and with Spanish citizenship (±3.02% margin of error for P=Q=50). The questionnaire lasted for an average of 15 min and revolved around sociopolitical topics. 7The Centre d’Estudis d’Opinió (Center for Opinion Studies) is the official institute for public opinion studies of the regional government of Catalonia in Spain. 8Additional heterogeneous effects mentioned in the literature of question-order effects will be explored, especially those linked to education (Crespi & Morris, 1984; Narayan & Krosnick, 1996). Table 1 Main Effects of the First Experiment Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 **Sig. < 0.01; * Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2 + x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Table 1 Main Effects of the First Experiment Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 **Sig. < 0.01; * Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2 + x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. 9See Guinjoan and Rodon (2016) for more information about the Linz–Moreno question. Table 2 Results of the First Experiment by Subjective National Identification Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 **Sig. < 0.01; *Sig. < 0.05. Table 2 Results of the First Experiment by Subjective National Identification Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 **Sig. < 0.01; *Sig. < 0.05. 10Tests are available upon request. 11Beyond the national versus regional logic, this type of cross-level question-order effects could eventually operate in cases in which supranational institutions, such as those of the European Union, and national institutions of a given country, like Great Britain, were included together in the same questionnaire. References Cantril H. (Ed.) ( 1944 ). Gauging public opinion . Princeton NJ : Princeton University Press . Google Scholar CrossRef Search ADS CEO (Centre d’Estudis d’Opinio) . Surveys no. REO 804 and 806 (2015), 816 and 835 (2016), and 850 (2017). Crespi I. , Morris D. ( 1984 ). Question order effect and the measurement of candidate preference in the 1982 Connecticut elections . Public Opinion Quarterly , 48 , 578 – 591 . Google Scholar CrossRef Search ADS Darcy R. , Schramm S. S. ( 1979 ). Comment on Kernell . American Political Science Review , 73 , 543 – 545 . Google Scholar CrossRef Search ADS Erikson R. S. , Luttberg N. R. , Tedin K. T. ( 1988 ). American public opinion . New York, NY : MacMillan . Guinjoan M. , Rodon T. ( 2016 ). A scrutiny of the Linz-Moreno question . Publius: The Journal of Federalism , 46 , 128 – 142 . Google Scholar CrossRef Search ADS Hierro M. J. , Gallego A. ( 2016 ). Identities in between: Political conflict and ethnonational identities in multicultural states . Journal of Conflict Resolution . First published December 30, 2016. doi: 10.1177/0022002716682593 Hyman H. H. , Sheatsley P. B. ( 1950 ). The current status of American public opinion. In Payne J. C. (Ed.), The teaching of contemporary affairs (pp. 11 – 34 ). Washington: Twenty-first Yearbook of the National Council of Social Studies . Lacy D. ( 2001 ). A theory of nonseparable preferences in survey responses . American Journal of Political Science , 45 , 239 – 258 . Google Scholar CrossRef Search ADS Lasorsa D. L. ( 2003 ). Question-order effects in surveys: The case of political interest, news attention, and knowledge . Journalism and Mass Communication Quarterly , 80 , 499 – 512 . Google Scholar CrossRef Search ADS Lombardi W. J. , Higgins H. E. , Bargh J. A. ( 1987 ). The role of consciousness in priming effects on categorization: assimilation versus contrast as a function of awareness of the priming task . Personality and Social Psychology Bulletin , 13 , 411 – 429 . Google Scholar CrossRef Search ADS Moore D. M. ( 2002 ). Measuring new types of question-order effects. Additive and subtractive . Public Opinion Quarterly , 66 , 80 – 91 . Google Scholar CrossRef Search ADS Mutz D. C. , Pemantle R. ( 2015 ). Standards for experimental research: encouraging a better understanding of experimental methods . Journal of Experimental Political Science , 2 , 192 – 215 . Google Scholar CrossRef Search ADS Narayan S. , Krosnick J. ( 1996 ). Education moderates some response effects in attitude measurement . Public Opinion Quarterly , 60 , 58 – 88 . Google Scholar CrossRef Search ADS Oldendick R. ( 2008 ). Question order effects. In Lavrakas P. (Ed.), Encyclopedia of survey research methods . Thousand Oaks, CA : Sage . Schuman H. ( 1992 ). Context effects: State of the past/state of the art. In Schwarz N. , Sudman S. (Eds.), Context effects in social and psychological research . New York, NY : Springer-Verlag . Schuman H. , Presser S. ( 1996 ). Questions and answers in attitude surveys . Thousand Oaks, CA : Sage . Schwarz N. ( 1999 ). Self-reports. How the questions shape the answers . American Psychologist , 54 , 93 – 105 . Google Scholar CrossRef Search ADS Schwarz N. , Bless H. ( 1992a ). Scandals and the public’s trust in politicians: Assimilation and contrast effects . Personality and Social Psychology Bulletin , 18 , 574 – 579 . Google Scholar CrossRef Search ADS Schwarz N. , Bless H. ( 1992b ). Constructing reality and its alternatives: An inclusion/exclusion model of assimilation and contrast effects in social judgment. In Martin L. L. , Tesser A. (Eds.), The construction of social judgements . Hillsdale, NJ : Lawrence Erlbaum Associates, Publishers . Sigelman L. ( 1981 ). Question-order effects on presidential popularity . Public Opinion Quarterly , 2 , 199 – 207 . Google Scholar CrossRef Search ADS Stapel D. A. , Schwartz N. ( 1998 ): The republican who did not want to become president: Colin Powell’s impact on evaluations of the Republican Party and Bob Dole . Personality and Social Psychology Bulletin , 24 , 690 – 698 . Google Scholar CrossRef Search ADS Strack F. ( 1992 ). ‘Order effects’ in survey research: Activation and information functions of preceding questions. In Schwarz N. , Sudman S. (Eds.), Context effects in social and psychological research . New York, NY : Springer-Verlag . Sudman S. , Bradburn N. M. , Schwarz N. ( 1996 ). Thinking about answers: The application of cognitive processes to survey methodology . San Francisco : Jossey-Bass . Tormos R. ( 2017 ). L’efecte de la situació econòmica personal en la confiança en les institucions en època de crisi. Barcelona: Papers de treball del CEO. Tormos R. , Muñoz J. , Hierro M. J. ( 2014 ). Endogenous identities? How the independence debate is reshaping Catalans’ identity. Paper presented at the IBEI Workshop The Politics of Identity Adoption, Barcelona. Available at: http://www.cuimpb.cat/files/TormosMu%C3%B1ozHierro_EndogenousIdentities.pdf Wilson D. C. ( 2010 ). Perceptions about the amount of interracial prejudice depend on racial group membership and question order . Public Opinion Quarterly , 74 , 344 – 356 . Google Scholar CrossRef Search ADS © The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png International Journal of Public Opinion Research Oxford University Press

Question-Order Effects in the Evaluation of Political Institutions in Decentralized Polities

Loading next page...
 
/lp/ou_press/question-order-effects-in-the-evaluation-of-political-institutions-in-zAZOnRWHBj
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved.
ISSN
0954-2892
eISSN
1471-6909
D.O.I.
10.1093/ijpor/edy013
Publisher site
See Article on Publisher Site

Abstract

Abstract This research studies question-order effects in the evaluation of political institutions in socially heterogeneous regions within decentralized countries. Split-ballot experiments were embedded in 3 representative surveys fielded in the Spanish region of Catalonia. Significant consistency effects were spotted in all samples. Respondents who first evaluated the relatively less valued institutions of one governmental level assessed less favorably the relatively better rated institutions of the other level. Clear evidence of the reverse effect was present only in 1 of the experiments. In addition, heterogeneous question-order effects emerged among the 2 distinct national communities coexisting in Catalonia. The article suggests a mechanism beneath this sort of question-order effects and further proposes and empirically tests a remedial solution. Introduction Survey questions are not asked in a vacuum but embedded in the conversational flow of an interview, which involves several additional surrounding items. The order in which questions are asked in surveys can have important effects on the results. In survey methodology, “order effects” are produced by the order in which questions (or response options) are presented to the respondents (Schuman, 1992; Schuman & Presser, 1996; Schwarz, 1999). Questions that are asked first provide a particular context that can alter the way in which subsequent questions are responded. This phenomenon has the potential to threat the substantive interpretation of survey results. Either if the interest is in studying a single moment in time or changes over time, inferences from survey data would be biased were they subject to unexpected order effects owing to how questions are placed on the questionnaire. Question-order effects tend to arise because questions similar in content influence one another (Schuman & Presser, 1996). This research addresses question-order effects that emerge in the evaluation of political institutions in decentralized polities. In particular, it deals with the contamination that can appear when respondents in those contexts have to consecutively assess governmental institutions that have a national and a regional autonomous manifestation. First, it studies the specific case of trust in politicians. It analyzes two similar questions: trust in Spanish and Catalan politicians, using an experiment embedded in two different surveys held in the Spanish region of Catalonia. These experiments test whether the order in which politicians of each of these two political levels of government are presented to respondents has an impact on the response obtained. Is the trust in the relatively more trusted politicians belonging to one governmental level undermined by previously asking about the trust in the relatively less trusted leaders of the other governmental level? Order effects could eventually work in the reverse direction: When relatively more trusted politicians of one level are rated first, comparatively less trusted ones of the other level might see their evaluations improve. Furthermore, in Catalonia, people can identify (to a different degree) with two distinct national communities of reference—the Catalan or the Spanish one—that in turn correspond with the two governmental levels. In such a context, citizens can feel closer to one specific governmental level above the other, according to their subjective national identification. Therefore, question-order effects might vary depending on the national identity of the respondent. Second, this research expands the analysis of this sort of question-order effects by investigating a different political institution: the government, in its national and its regional separate counterpart, exploring the evaluation of government performance. Results of this second experiment confirm what was seen in the former and allow speaking of a general case for question-order effects in the evaluation of political institutions in decentralized polities. Question-Order Effects Researchers identified order effects in early survey research studies (Cantril, 1944; Hyman & Sheatsley, 1950). Despite the important risks order effects can pose to generalization from survey results, not enough substantial literature has dealt with this phenomenon (Schuman & Presser, 1996, p. 24). Order effects can either refer to the order of questions (question-order effects) or to the order of response options within questions (response-order effects1). There is specific literature related to these two avenues of research. The focus of this investigation, however, is on a particular type of question-order effects. Question-order effects usually involve questions about similar issues in which a “transfer of meaning” between them takes place. These situations are usually named after the label of context effects (Schuman, 1992). Question-order effects can be further classified according to the types of relations between questions, yielding to two main types: the part–whole and the part–part combinations (Schuman & Presser, 1996). Part–whole combinations involve two or more questions where one question is more general and comprises the other one. Part–part combinations include questions that are at the same level of specificity. Question-order effects are also classified by the types of effects they produce. The two main types are consistency effects, also known as assimilation effects, and contrast effects. Consistency effects happen when answers to the later question are more similar to the earlier one than would have been if the order of questions would have been different. The rationale for this situation is the respondent's need to appear consistent when answering. Conversely, contrast effects produce the opposite situation: greater differences between questions as a result of the ordering. Both part–whole and part–part combinations can be subject to either consistency or contrast effects (for a detailed list of examples, see Schuman & Presser, 1996). Strack (1992) digs into the psychological mechanism of question-order effects that lead to either consistency or contrast effects. The influence of the preceding question can be conceptualized as a prime that has the functions of activation and information. The function of activation refers to the fact that the prime automatically activates certain concepts that can be brought to mind more easily later on. If the respondent is not aware of the prime, it would mechanically lead to a consistency effect. Conversely, if the respondent becomes aware of the prime, and perceives the relationship between the two questions, they can use this information to intentionally act on it. The preceding question then would have an additional function of information in the sense that it provides information on the intended meaning of the question (Strack, 1992). The results could then be either consistency or contrast depending on whether respondents perceive the two questions as meant to belong together or not. For instance, it would lead to contrast if the respondent recognizes a part–whole combination and deliberately subtracts the part when evaluating the whole. Moore (2002) refers to a complementary sort of question-order effects. Although consistency and contrast are part of the item dimension, there is an additional dimension of question-order effects, potentially independent of the former, named the framework dimension. It relates to the fact that each question is presented to the respondent as either belonging to a comparative or a noncomparative context. In split-ballot experiments, the first question is always the noncomparative, because respondents are still unaware of the second question. However, the second question is the comparative one, because it could be influenced by the former. Framework effects occur when the noncomparative questions obtain different scores than the comparative ones. If the scores of the noncomparative questions are higher than the comparative, Moore speaks of subtractive effects, whereas if the scores are lower, the effects are called additive. All in all, perceiving questions as part of a comparative context can affect results, irrespective of the item dimension.2 Decentralized Political Systems This research investigates question-order effects that take place when respondents evaluate political institutions in socio-politically heterogeneous regions within countries that have a decentralized multilevel structure of government. It studies the particular case of Catalonia within Spain, a territory with autonomous political institutions within the quasi-federal Spanish political system, in which citizens have divergent national allegiances to either Catalonia or Spain. Within Spain itself, other similar cases are those of the Basque country, Galicia, or the Valencian community. There are also analogous situations over the world where regional heterogeneity is based on diverging national allegiances, such as Veneto or South Tirol in Italy, Scotland in the U.K., the Flemish community in Belgium, or Quebec in Canada. However, if regional heterogeneity is more broadly defined in terms of ethnic, racial, religious, or even ideological diversity, we could include many more cases.3 In such decentralized and internally diverse polities, order effects may arise when respondents have to evaluate consecutively two political objects that belong to the same general category (e.g., politicians) but refer to each specific polity or layer of government (e.g., regional- or national-level politicians). Furthermore, I will show that the type of question-order effect that occurs is consistency and that it follows the basic logic described by Strack (1992). These consistency effects are related to the prime of the preceding question referred to one governmental level and follow an automatic sort of mechanism that instills respondents with a drive to compensate or balance the ratings of the posterior question referred to the other governmental level. In so doing, it produces a contamination across levels. Decentralized polities might create informational effects for respondents who have to sequentially evaluate institutions of each level of government, making the order of questions consequential. This type of regional information cues that may affect perceptions of governance factors has not yet been investigated, to the best of the author’s knowledge. Question-order effects of this sort might, for instance, have the capacity to artificially downgrade the general evaluation of the performance of the regional-level government as to obtain a falling grade if asked after that of national-level government, or vice versa. Furthermore, this type of effects could be quite usual and might have gone unnoticed until now. Both the political context in which the effects take place, decentralized political systems, and the survey questions affected, evaluation of political institutions of different governmental levels, are common. Most countries in the world have decentralized constitutional arrangements, and many decentralized regions are dissimilar or have distinctive identity or characteristics. Besides, the survey questions affected are not artificial or rare. The evaluation of political institutions of each governmental level is part of the usual repertoire of questions of most political surveys in those contexts. Survey practitioners in decentralized polities have to deal with the possibility of this type of question-order effects occurring on a regular basis. To mention just the Spanish case, any political survey that aims at measuring political evaluations of national and regional autonomous institutions in the same questionnaire (the majority of political surveys in this decentralized country) should keep in mind the analysis and recommendations presented here. Previous Research The existence of question-order effects in the evaluation of political institutions in the context of decentralized polities has not received scholarly attention yet, as far as the author knows. This is why this article can only relate to previous investigations on question-order effects about similar survey questions but not about similar contexts. Since the context -decentralized polities- is a key factor involved in the emergence of these effects, it is precisely the lack of research antecedents that makes the present contribution most valuable. With regard to studies on similar survey questions, not much research has specifically dealt with question-order effects in trust in politicians, and even less with government performance evaluation. A previous investigation explicitly analyzing trust issues was the experiment by Schwarz and Bless (1992a) priming scandals before general and specific questions on trust in politicians. Making respondents think about a politician who was involved in a scandal decreased the trust of politicians in general through a consistency effect but increased the trust in other individual politicians by means of a contrast effect. In a similar vein, Erikson, Luttberg, and Tedin (1988) showed how American citizens distrust Congress but trusted their own representative in this institution. Schwarz (1999) offers a possible reason for this effect. The media presents extreme cases of untrustworthiness and corruption to the public, which become highly accessible to memory. These extreme examples can strongly influence the representation of the general trust, but they can be used as a standard of comparison and contrast in evaluating individual instances. Moore (2002) also found question-order effects when asking about the honesty and trustworthiness of pairs of U.S. political leaders in Gallup polls. Asking first about Al Gore benefited the ratings of Bill Clinton and asking first about Clinton undermined those of Gore. Overall, both ratings became more similar to what they would have been had they been asked independently, in tune with consistency effects. In the case of Gingrich and Dole, Moore spotted contrast effects: Respondents tended to emphasize the differences between the two leaders rather than the similarities as a result of the order of questions. Stapel and Schwartz (1998) investigated carryover effects between parties (the Republican), candidates (Dole), and party members (Powell) in the evaluation of these objects. They found that initial judgments of the party affected subsequent evaluations about candidates, and vice versa in an additive manner. Their results were consistent with the Schwarz and Bless (1992b) inclusion/exclusion model of social judgment. Other researchers of question-order effects have dealt with relatively related topics such as presidential popularity (Darcy & Schramm, 1979; Sigelman, 1981) and candidate preferences for governor and senator (Crespi & Morris, 1984). Crespi and Morris (1984) studied question-order effects on preferences for candidates to two different U.S. races, the senatorial and the gubernatorial, using a split-ballot experiment. Preferences for candidates to the Senate became affected by the order in which preferences for governor were asked. They concluded that asking first about the race in which a party’s candidate is stronger has a coattail effect4 among the party’s followers when preference in the other race is measured first. Another important implication from this research was that order effects were not homogeneous across the sample but associated with specific political attitudes (such as party identification, candidate preference, or education).5 In any case, to the best of my knowledge, no research has specifically dealt with the evaluation of political institutions in heterogeneous regional contexts within decentralized countries. In such settings, the two different levels of government (regional and national) also represent two potentially adversarial communities of reference, as citizens have a repertoire of identities to resort to. When evaluating a political object that has a separated manifestation at the national and at the regional levels, informational cues might arise and foster potential question-order effects. In Catalonia, these adversarial regional identities take the form of opposed national allegiances. Catalonia is divided in terms of national identity, with a majority defined as mainly or exclusively Catalan (50% according to Centre d’Estudis d’Opinio [CEO] 816, 2016) and a minority defined as Spanish (11%), and a more heterogeneous group of dual identifiers (34%). In recent years, the Catalan independence debate has made identity issues more salient to the public (Tormos, Muñoz, & Hierro, 2014), and identities turned more conflictual (Hierro & Gallego, 2016). This conflict of identities might have a role in loading the survey questions referred to each level of government in the experiments presented here. In this respect, Wilson (2010) investigated the impact of group identity and identity activation on question-order effects. He found that question order interacted with the racial group of the respondent to influence beliefs about the amount of interracial prejudice between Blacks and Whites in the United States. Wilson showed how in-group members tended to view out-group members as having more dislike toward their in-group only when the in-group was asked about first, a contrast effect. When in-group members were evaluated after the out-group, they viewed their in-group’s dislike as similar to that of the out-group’s, an assimilation. This research will test whether indeed identity plays an active a role as a mechanism guiding question-order effects (identity activation) in tune with Wilson’s findings, or just a passive one as a source of heterogeneous effects that arise from the particular aggregate social configuration of the region. Trust in Politicians in a Multilevel Setting In multilevel political systems, such as the Spanish one, pollsters in charge of political surveys usually ask citizens’ perception of political institutions of different governmental levels in the same questionnaire. Informational cues from institutions of the level of government assessed first can influence the response to the corresponding political institution at the other level evaluated later. I test if asking on the first place for the trust in Spanish politicians influences the trust expressed in Catalan politicians asked later on, and vice versa. I first use the survey on the political situation from 20156 (CEO 806, 2015) from the CEO.7 The questions on trust in Spanish and Catalan politicians were placed together at about the middle of the questionnaire. They had the following format: P20a. All in all, please rate the degree of trust you have in the Catalan politicians in a scale from 0 to 10, where 0 is no trust at all and 10 is a lot of trust. P20b. All in all, please rate the degree of trust you have in the Spanish politicians in a scale from 0 to 10, where 0 is no trust at all and 10 is a lot of trust. In the administration of the questionnaire, the order of appearance of these two questions was randomized in the context of a split-ballot experiment. Half of the sample was exposed to a questionnaire in which trust in Catalan politicians was asked first and trust in Spanish politicians just after. The other half of the sample was first faced to the rating of Spanish politicians and right after they had to rate Catalan politicians. The treatment of the experiment was, therefore, the order in which both questions on trust in Spanish and Catalan politicians were formulated. This single split-ballot experiment with two groups could be essentially understood as containing two pairs of treatment and control conditions (Figure 1). One pair (Structure 1) tackles whether the question on trust in Spanish politicians, Q(Spa), has an effect on trust in Catalan politicians, Q(Cat). The other one (Structure 2) seeks whether the question on trust in Catalan politicians, Q(Cat), has an impact on trust in Spanish politicians, Q(Spa). The experiment can also be conceptualized according to Moore’s framework dimension (2002). Then we would have a pair of comparative and a pair of noncomparative questions. The first questions in any of the experimental groups would be considered the noncomparative questions, whereas the second questions would be the comparative ones. Figure 1 View largeDownload slide Experimental conditions to test question-order effects Figure 1 View largeDownload slide Experimental conditions to test question-order effects The prime implied by the preceding question can be qualified in terms of the direction of its valence, and then speak of a positive or a negative valence of the prime (Strack, 1992). We know from previous surveys (e.g., CEO 804, 2015) that Spanish politicians in Catalonia are overall less trusted than Catalan politicians. Therefore, it can be anticipated that the dominant valence associated to Spanish politicians would be negative. In fact, this negative appreciation of national-level politicians in comparison with regional-level ones extends to other political institutions such as the government or the parliament. In Catalonia, Catalan institutions are viewed under a more positive light than Spanish ones (Tormos, 2017). In this region, the image of the Spanish politicians might be associated with issues repeatedly appearing in the media such as corruption scandals, inability to deal with the crisis, problems of representation, and their responsibility on the current bad state of relations between Spain and Catalonia. The valence associated with Catalan politicians is less negative—still not good though. Their image might share some negative elements with the Spanish one, but it could also be associated with positive issues, at least for some, such as a certain sense of differentness with respect to politicians of the rest of Spain, as well as the projected hope and expectations for an eventual secession and the building of a new state. In view of these informational cues that may emerge when evaluating institutions of each governmental level, we can derive that in Catalonia, trust in Spanish politicians as a preceding question might play the role of a negative prime, whereas trust in Catalan politicians may be a positive prime in relative terms, or at least a less negative one. A relevant element when considering the psychological mechanisms leading to question-order effects is whether the respondent is aware of the priming episode. Previous experiments (Lombardi, Higgins, & Bargh, 1987) indicate that subjects able to recall the prime were more likely to show contrast effects, whereas subjects unable to recall it were prone to consistency effects. The influence of the prime of the preceding question may only be counterbalanced in the form of contrast if subjects are conscious of it and react upon this information. If they are not aware, a mechanical process of assimilation (consistency) would likely apply. In this experiment, my assumption is that the prime is so subtle and apparently inadvertent that it would be improbable for respondents to be aware of it or infer intentionality. As a result, the most likely sort of effects that may appear would be consistency effects. However, if contrast effects shall be spotted instead, it would imply that respondents were aware of the prime, and other mechanisms such as identity activation might be operating. Another consideration is whether the experiment is dealing with a part–whole combination of questions or with a part–part. It is not clear which of these two combinations is perceived by the respondent. In principle, it may seem a part–whole combination, given that formally Catalan politicians belong to the larger set of Spanish politicians. However, in practice, respondents may not relate to this scheme depending on their national identification. To people who feel predominantly Catalan (the largest share), the question pair might appear more like a part–part than a part–whole combination. If the part–whole logic shall play a relevant role in this experiment, we might be able to witness contrast effects when respondents subtract the part when evaluating the whole as in the inclusion/exclusion framework. All these considerations help in the elaboration of the hypotheses. I sustain that the type of question-order effects that will appear in the context of this decentralized polity as a result of the experiment would be consistency effects. However, consistency effects can potentially operate in two directions: positive or negative, depending on the regional cues that influence the valence of the prime (the preceding question). On the one hand, it is expected that asking first for the governmental level with the less trusted politicians (negative prime) would undermine the ratings of the relatively more trusted ones at the other level, and therefore produce a negative consistency effect. On the other hand, asking first for the government level with the more trusted politicians (positive prime) could better the ratings of the governmental level with the less trusted ones, generating positive consistency effects. This would translate into the following general hypotheses: Hypothesis 1: Negative consistency effects: When the Spanish institution is asked first, it will negatively influence the ratings of the Catalan institution that would be asked second. Hypothesis 2: Positive consistency effects: When the Catalan institution is asked first, it will positively affect the ratings of the Spanish institution asked right after. If contrast effects shall appear instead of consistency, it could be a signal that either identity activation (Wilson, 2010) or inclusion/exclusion processes (Schwarz & Bless, 1992b) may be taking place. Besides testing for the presence of consistency or contrast effects in the item dimension, this research also explores the potential existence of framework effects (Moore, 2002), comparing the comparative and the noncomparative pair of questions on trust in Catalan and Spanish politicians to look for either subtractive or additive effects. Question-order effects are not necessarily an across-the-board phenomenon. As in the Crespi and Morris (1984) experiment, effects might differ quite markedly across subgroups of the sample. I can anticipate a clear source of heterogeneity related to the national identification of the respondent. In Catalonia, the subjective national identification of individuals is a powerful filter through which the sociopolitical reality is evaluated (Guinjoan & Rodon, 2016). In fact, the national identification of the respondent is the most powerful explanatory factor of the differences in trust between Spanish and Catalan politicians among a set of usual structural predictors of Catalan political behavior (see the regression analysis in Table A1 in the appendix). With some confidence, I can forestall that feeling Catalan vis-à-vis Spanish will influence the trust the respondent has in either Catalan or Spanish politicians.8 The third hypothesis takes into consideration this heterogeneity among the Catalan public, and the consequences it can have for the experiment. The positive and negative valences of the primes would be different according to the national identification of respondents. However, it will still be valid that a negative prime would potentially produce negative consistency effects across governmental levels, and a positive prime would generate positive consistency effects. More specifically, this translates into the following general propositions: Hypothesis 3: Heterogeneous effects by subjective national identification: For those who feel predominantly Catalan, (H3.1) being first exposed to the evaluation of a Spanish institution would negatively affect their ratings of the Catalan institution. And (H3.2) being first faced with the evaluation of a Catalan institution would positively affect their ratings of the Spanish institution. For those who feel predominantly Spanish, however, (H3.3) being first exposed to the evaluation of a Catalan institution would negatively affect their evaluation of the Spanish institution. And subsequently (H3.4) being first exposed to the evaluation of a Spanish institution would positively affect their rating of the Catalan institution. An alternative explanation in competition with the third hypothesis refers to the activation of group identities. As noted by Wilson (2010), question order and group membership can interact to strengthen in-group and out-group categorization, leading to a contrast effect by which one’s in-group is favored and differentiated over the out-group. The first question can heighten the salience of group membership and promote group comparison together with the drive to view one’s in-group more positively and out-group more negatively. As Wilson observed, groups will differentiate their in-group with an out-group when their own group is considered first but will assimilate their group to the out-group in a comparative context. Here, if the premise about identity activation is correct, those who feel Catalan when asked about a Catalan institution first would tend to differentiate the ratings of the Spanish institution asked about second in tune with a contrast effect, and vice versa for those with a mainly Spanish identity. In this situation, respondents would be aware of the comparative context; their identity would become activated and would consciously respond differentiating the ratings of the in-group institution asked on the first place from those of the out-group asked second. Results If we take the whole sample estimates, without distinguishing among experimental groups, Catalan politicians are more trusted than Spanish leaders (see the upper part of Table 1). In both cases, the average is below the middle point, but Catalan politicians obtain a mean of 3.77, whereas Spanish politicians get a 2.65. This difference of 1.12 is statistically significant (p < .001). Is trust in Catalan political leaders downgraded when trust in Spanish ones is asked on the first place? The middle part of Table 1 presents the average trust in Catalan and Spanish politicians across the two experimental groups. The group that assesses their trust in Catalan politicians first gives them an average of 4, higher than the mean obtained when the question came after the rating of Spanish politicians (3.55). It is a statistically significant difference of 0.45 among the two groups (p < .001). Hypothesis 1 becomes then confirmed: A negative consistency effect takes place affecting the trust in Catalan politicians as a result of the impact of the negative prime established by the preceding question on trust in Spanish politicians. Changing the order, however, does not affect the evaluation of the Spanish politicians. Both the group in which the question on Catalan politicians is asked first and the group in which it is asked second assess Spanish politicians in the same way (an average of 2.63 and 2.66, respectively). Therefore, Hypothesis 2 stating a potential positive consistency effect is not confirmed. The potentially positive prime of the preceding question—trust in Catalan politicians—does not have the capacity to influence the assessment of Spanish politicians. It could either be because the prime is not really powerful or positive enough (Catalan politicians are only slightly more trusted than Spanish ones) or because the actual assessment of Spanish politicians is so negative that it is difficult to change it by any means. As a result of the experiment, the average difference among the Catalan and Spanish leaders is reduced from 1.37, in the group in which Catalan leaders are asked first, to 0.89, in which they are asked second. The distance between these two differences is 0.48, and it is statistically significant (p < .001). This is in fact another way of assessing the consistency effect: by means of observing a higher resemblance between the two groups of politicians after the treatment. No signs of contrast effects are spotted; therefore, the inclusion/exclusion frame seems not to be operating. In addition, the current experiment seems not to have effects on the framework dimension. The average trust in Spanish and Catalan politicians is very similar in the comparative and noncomparative frameworks. Therefore, the fact that this pair of questions is presented together does not per se generate an additive or a subtractive effect. In the upper part of Table 2, respondents are divided into three groups according to their national identification in response to the Linz–Moreno question9: (1) those who feel predominantly Catalan (“only Catalan” and “more Catalan than Spanish”), (2) the dual identifiers (“as Catalan as Spanish”), and (3) the predominantly Spanish (“only Spanish” and “more Spanish than Catalan”). Those who feel predominantly Catalan evaluate positively the Catalan leaders (5.06) and negatively the Spanish leaders (2.07). Conversely, respondents who feel mainly Spanish value better the Spanish politicians (3.92) than the Catalan politicians (1.95), although not really well. Dual identifiers trust Spanish (3.22) more than Catalan (2.44) politicians, presenting a pattern more similar to those who feel Spanish though more attenuated. On the whole, the national identification of the respondent clearly influences trust in both groups of politicians. Respondents tend to look favorably to the political leaders of their own national community of reference, and with a side-glance the leaders of the other national community. This evidence points to a potential heterogeneous effect of the treatment conditional on the national identification of the respondent. According to the main hypothesis defended here, if respondents who feel mainly Spanish are first exposed to the question on trust in Catalan politicians, they would rate Spanish politicians worst; the reverse to what would happen in the group of those who feel predominantly Catalan. The reason for it would be that the more trusted politicians for those who feel Spanish are Spanish politicians; the group of politicians that is their positive reference point. In contrast, Catalan politicians would be their negative reference point. However, if group identities should become activated (the alternative explanation), consistency will only happen when the out-group institution is asked first, as a means of compensation. Yet, when the out-group institution is asked second, contrast effects would tend to emerge, as respondents will try to differentiate the institution belonging to the in-group from that of the out-group. Table 2 also presents the results of the experiment across national identity groups. The experiment only has statistically significant effects in the group that feels predominantly Catalan (the one with a larger subsample). In this segment, when trust in Catalan politicians is asked first, the resulting level is as high as 5.28, well above the midpoint of the scale. Whereas, when this group of respondents is first exposed to the rating of Spanish politicians, their trust in Catalan politicians falls to 4.82. This is a 0.46 statistically significant difference (p < .018). In the other groups (duals and Spanish), the effects of the experiment do not reach the threshold of statistical significance. However, as it was expected, question-order effects have the inverse impact. In the segment of Spanish identifiers, asking first for their trust in Catalan politicians reduces their trust in Spanish politicians, though the difference is not statistically significant, given the small size of the subsample. In this same group, asking first for the Spanish politicians increases their trust in the Catalan ones: a positive consistency effect. However, the differences neither reach the level of statistical significance. The type of question-order effect spotted here refutes the alternative explanation based on identity activation. There are no signs of contrast effects among identity groups. Therefore, in tune with Strack (1992), the mechanism seems much more automatic and related to the respondent’s reflex reaction to the positive or negative prime of the preceding question, which is informed by regional cues. Although identity seems not to play a direct role in the causal mechanism, it has an indirect one as source of heterogeneous effects across groups and connected with the informational cues about governmental levels. Given the aggregate distribution of national identifications in the population, the order of questions has the opposite effect in Catalan identifiers as compared with Spanish identifiers, although both groups experience the same kind of mental process. The case of a region with autonomous institutions within a decentralized country provides the context for these effects to happen. Decentralized contexts of this sort facilitate the presence of information cues affecting each level of government. In this case, depending on the national identification of the respondent, the cues have different effects, although the mechanism at stake is the same. To further illustrate the heterogeneous effects of the treatment by national identity, Figure 2 shows the coefficients of the interaction of both variables (from a regression further including the treatment and national identity as predictors). Although they are not statistically significant, we can clearly observe that treatment effects are opposed, according to the respondent’s national identity. Asking first for the Spanish politicians to those who have a Catalan identity makes them assess Catalan politicians worse, whereas for those with a Spanish identity, asking first for the Spanish politicians makes them rate Catalan leaders better. Figure 3 shows the interaction of the treatment with national identification on trust in Spanish politicians. Again, the effects are not statistically significant; however, it is possible to see the inverse pattern of effects. Asking first for the trust in Catalan politicians tends to reduce the trust in Spanish politicians in the group of those who feel predominantly Spanish. In contrast, asking first for the trust in Catalan leaders tends to improve the trust in Spanish leaders in the group of those who consider themselves mainly Catalans. Figure 2 View largeDownload slide Linear regression to explain trust in Catalan politicians. Question-order effects by national identity (regr. coeff.) Figure 2 View largeDownload slide Linear regression to explain trust in Catalan politicians. Question-order effects by national identity (regr. coeff.) Figure 3 View largeDownload slide Linear regression to explain trust in Spanish politicians. Question-order effects by national identity (regr. coeff.) Figure 3 View largeDownload slide Linear regression to explain trust in Spanish politicians. Question-order effects by national identity (regr. coeff.) Apart from subjective national identification, no other variable usually mentioned in the literature of question-order effects (such as education, age, sex, or interest in politics) interacts with this particular treatment. Verification A usual practice when performing experiments is to check for imbalances among experimental groups after random assignment. As noted by Mutz and Pemantle (2015), randomization checks should be used with the purpose of testing whether the randomization mechanism has worked properly. An imbalance in the covariates after random assignment is not a problem per se if the randomization mechanism performed well. Usual statistical tests such as analysis of variance (ANOVA) are designed to account for this sort of imbalance. There would only be a problem if the source of the imbalance is to be found on a defective random assignment process. To explore possible imbalances I perform a set of randomization checks. Figure 4 shows a test of the equivalence of the two groups by means of a logistic regression in which the dependent variable is the experimental group and the independent variables are the relevant characteristics that might distinguish them. Figure 4 View largeDownload slide Randomization test. Logistic regression to explain experimental group membership Figure 4 View largeDownload slide Randomization test. Logistic regression to explain experimental group membership There are statistically significant differences between the two groups by subjective national identification and interest in politics. The group that rates Spanish politicians first contains more dual identifiers and less who feel only Catalan, as well as respondents with a little less interest in politics. These differences are unlikely attributable to an effect of the experiment. In the sequence of the questionnaire, the question on subjective national identification is asked much later than the treatment, near the end of the interview. Besides that, the treatment itself, a question-order change, is so mild and subtle that it is highly unlikely it would have had the power to influence the national identification of the respondent. From this it follows that the difference between the two groups might be owing to mere chance, an “unlucky draw”, and not to an unexpected consequence of the experiment. I performed additional randomization checks to rule out the possibility that an interviewer effect or an effect of the way in which the fieldwork was carried out could eventually be the reason of the differences in interest in politics and subjective national identification across the two experimental groups. I explore differences between the two groups by the profile of the interviewer who performed the fieldwork. There is a theoretical possibility that specific interviewers were assigned to one group and not to the other. If this hypothetical uneven assignment of interviewers would have been combined with the fact that some of them were more prone to introduce an involuntary bias in the process of interview, respondents of one of the groups could have expressed more interest in politics or a particular national identity owing to an interviewer effect. However, Table 3 indicates that there are no differences with respect to the interviewers assigned across the two groups. There is only a difference due to the sex of the interviewer, but this trait has no effect on the experiment (this is tested through a regression analysis not presented for simplicity). There are no differences in the length of the interviews, the day in which they were performed, and the language used. These additional tests reinforce the idea that differences between the two groups might be owing to mere chance, and not to a fieldwork problem that might have biased the random assignment process. Table 3 Bivariate Randomization Tests Across Experimental Groups ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 Table 3 Bivariate Randomization Tests Across Experimental Groups ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 ANOVA F χ2 df Sig. Interviewer code – 27.13 28 0.511 Age of the interviewer 0.24 – 1049 0.625 Sex of the interviewer – 7.51 1 0.006 Education of the interviewer – 1.50 3 0.683 Length of the interview 1.34 – 1049 0.247 Day of the interview – 6.31 7 0.504 Language of the interview – 1.04 2 0.594 Replication To be able to safely generalize the presence of question-order effects, it is convenient to repeat the current experiment on different samples of the same population. Descriptive inference as well as causal inference relate to the idea of repeated samples and experiments. Replication reduces variability in experimental results and increases the confidence on the effects of the treatment. If a treatment has a truly causal impact, the average effect of different replications would show it. A replication of the first experiment was embedded on a similar survey performed by the same institution just 4 months after the first one (CEO 816, 2016). On this occasion, it was a CAPI survey representative of the same population and with a larger sample size (N = 1,500). Table 4 presents the main effects of the experiment. Table 4 Main Effects of the Second Experiment Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 **Sig. < 0.01; *Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2+ x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Table 4 Main Effects of the Second Experiment Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 Trust in …. politicians Diff. Catalan Spanish Cat. − Spa.c Experimental groups  a) Catalan politicians first x̅1 3.86 x̅3 2.44 x̅a= (x̅1− x̅3) 1.41** S1 2.57 S3 2.11 Sa 2.77 n1 753 n3 752 na 749  b) Spanish politicians first x̅2 3.54 x̅4 2.17 x̅b = (x̅2− x̅4) 1.39** S2 2.55 S4 2.11 Sb 2.65 n2 730 n4 735 nb 730 Item dimensiona     Difference among groups x̅1− x̅2 0.32** x̅3− x̅4 0.27** x̅a− x̅b 0.03 Framework dimensionb     Comparative frame x̅c 2.99     Noncomparative frame x̅nc 3.02     Difference among groups x̅c–x̅nc 0.02 **Sig. < 0.01; *Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2+ x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Again, those who assess the Spanish politicians first give a worst rating to the Catalan politicians afterwards. The magnitude of this negative consistency effect is similar to that of the first experiment, though slightly smaller. In addition, in this new experiment, positive consistency effects are also spotted, unlike in the first probe. When respondents have to rate Catalan politicians initially, they assess Spanish politicians better later on. Once more, this second experiment does not portray any framework effects. The average trusts in Spanish and Catalan politicians are almost the same in the comparative and in the noncomparative frameworks. Heterogeneous question-order effects do not reach the level of statistical significance on this one occasion (not presented for the sake of simplicity). In this case, both experimental groups were balanced in the key covariates after random assignment.10 Evaluation of Government Performance An additional experiment was introduced in another survey (CEO 835, 2016) to test question-order effects in the evaluation of a different political institution, the government, using a measure of respondents’ assessment of government performance. Respondents had to evaluate on a scale from 0 to 10 both the performance of the national institution and its regional autonomous counterpart. Results from this experiment are presented in Table 5. They follow the same pattern as in the case of trust in politicians. When respondents evaluate the Catalan Government first, they give it a better rating (4.53) than when they do it after evaluating the Spanish Government (4.00), in accordance with a negative consistency effect (p < .001). In this case, positive consistency effects do not reach the level of statistical significance but can be spotted nonetheless. When question-order effects are decomposed by the national identification of respondents, results are in a similar vein to previous experiments on trust in politicians. Table 5 Main Effects of the Third Experiment and Results by Subjective National Identification How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 **Sig. < 0.01; *Sig. < 0.05. Table 5 Main Effects of the Third Experiment and Results by Subjective National Identification How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 How would you rate the task of the… Subjective national identification Catalan identity Dual identity Spanish identity Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Cat. Gov. Spa. Gov. Total sample     x̅Tc 4.26 1.82 5.66 1.21 3.10 2.25 2.72 2.94     STc 2.58 2.14 2.17 1.71 2.20 2.22 2.45 2.63     nTc 1460 1473 691 689 568 575 152 156 Experimental groups     a) Catalan Gov. first     x̅1 4.53 1.92 5.85 1.44 3.28 2.20 3.23 3.12     S1 2.51 2.11 2.05 1.79 2.20 2.13 2.61 2.72     n1 731 744 354 357 283 287 74 78     b) Spanish Gov. first     x̅2 4.00 1.71 5.45 0.96 2.92 2.30 2.23 2.76     S2 2.62 2.16 2.28 1.59 2.19 2.30 2.19 2.54     n2 729 729 337 332 285 288 78 78     Difference     x̅1− x̅2 0.53** 0.21 0.40* 0.48** 0.36 −0.10 1.00* 0.36 **Sig. < 0.01; *Sig. < 0.05. These results further speak of the robustness of the findings and allow generalizing beyond a single specific political institution. Decentralized countries with socially heterogeneous regions can provide the context for such question-order effects to happen when citizens of a region have to sequentially evaluate political institutions that have a national and a regional autonomous manifestation. Conclusions This research has analyzed question-order effects in the evaluation of political institutions using split-ballot experiments embedded in representative surveys fielded in the Catalan region of Spain, a decentralized country. Results indicate that when respondents of this region who have to evaluate political institutions with a national and a regional independent expression are first exposed to the assessment of the less appreciated institution belonging to one of the two levels of government, the ratings of the relatively more appreciated institution at the other governmental level become affected by means of a negative consistency effect. In Catalonia, across three different samples, Catalan institutions are, on average, more valued than Spanish ones. The three experiments performed show how the ratings of Catalan political institutions diminish when the evaluation of their Spanish counterparts is requested first. Clear evidence of the reverse, a positive consistency effect from the ratings of the Catalan institutions to those of the Spanish ones, is only statistically significant on the main effects of one of the experiments. Apart from the item dimension, no additive or subtractive effects in the framework dimension of question-order effects were found in any of the experiments. The subjective national identification of the respondent plays a role as a source of heterogeneous effects across groups, but not in the very mechanism of question-order effects. This is assessed by the absence of contrast effects in the experiments, which allows inferring that no process of identity activation might have been at play. Catalan citizens can either identify with Spain or Catalonia in national terms (or have a dual identification). Respondents who feel Catalan rate Catalan institutions better than Spanish ones, whereas interviewees who feel Spanish do the reverse. Therefore, respondents with a Catalan national identification end up rating Catalan institutions worst if they are first exposed to the evaluation of a Spanish institution. Likewise, respondents with a Spanish national identity express less appreciation of Spanish institutions when they have to rate a Catalan institution first (although results are not statistically significant owing to the small subsample of Spanish identifiers). Results also point to positive consistency effects across subgroups on national identification. However, the reduced sample sizes of those subgroups do not usually allow for a safe generalization of the relationship. To summarize, evidence from the main effects of these experiments confirm the hypothesis on the negative consistency effects (Hypothesis 1) but provide only partial support to the positive consistency effects one (Hypothesis 2). Results from the interaction of the experiment with subjective national identification speak favorably of each statement of the third hypothesis (Hypothesis 3) on the heterogeneous effects of the treatment had we used larger subsamples. The alternative explanations for the psychological mechanisms based on group identity or on a part–whole subtraction become discarded because contrast effects are not spotted in any case. The necessary context for these question-order effects to happen is that of a decentralized country with a multilevel structure of governance, a constitutional arrangement frequent across the world. These politically decentralized contexts enable the existence of information cues that affect the perception of the different governmental levels and allow for eventual contaminations across levels. The implications of these results are twofold. On the one hand, they add to the literature on question-order effects the case of the evaluation of political institutions in heterogeneous regions within decentralized countries, and on the other hand, they have practical consequences for survey design. In regions within decentralized countries where two or more group identities coexist, the order in which questions about institutions of either of those communities are presented to respondents might not be neutral. The risk of bias is likely worsened when some kind of clash between the region and the nation takes place, such as in the current case of Catalonia in Spain. A number of comparable cases in the world might fulfill this condition, as decentralized countries and regional diversity are abundant. The evidence presented here calls for a close monitoring of the order of questions in such contexts.11 Apart from substantive implications, a clear practical consequence for survey practice can be derived from this research. The mechanism involved in the sort of question-order effects identified here is related to the prime of the preceding question referred to a given governmental level, which is loaded with either negative or positive valences. Therefore, an advice for future surveys would be to separate the items related to one governmental level from those of the other level over the questionnaire, at least using some “buffer” questions (Lasorsa, 2003). If they are put together, randomization of the order of appearance—the usual cautionary procedure in survey practice (Oldendick, 2008)—would only reduce the bias by half (as can be seen in the experiments presented here), but it will continue to exist. However, separating the blocks of questions related to each governmental level can deactivate the priming effect that otherwise takes place when they are next to each other in the questionnaire. This remedy has been tested in a recent survey (CEO 850, 2017). Question blocks on Spanish and Catalan institutions were separated by seven buffer questions, and the administration of the survey followed a split-ballot design changing the order of blocks. The results obtained confirm that separating the items allows circumventing order effects and obtaining unbiased estimates (see Table A2 in appendix for more information). Supplementary Data Supplementary data are available at IJPOR online. Acknowledgements The author would like to thank Jordi Argelaguet, Malcolm Fairbrother, Willem Saris, Jordi Muñoz, and the two anonymous reviewers for their valuable comments and suggestions. Conflicts of interest: None declared. Raül Tormos (PhD) is a survey scientist at the Centre d’Estudis d’Opinió (CEO) of the Government of Catalonia in Spain. He is also a lecturer at the University of Barcelona, Department of Sociology. Footnotes 1Response-order effects usually arise from the difficulty the respondent has in keeping in mind all the alternatives presented, yielding to primacy or recency effects. 2Schuman and Presser (1996) also mention additional types of question-order effects such as salience, rapport, fatigue, and initial frame of reference effects. Potentially relevant to this research are the initial frame of reference effects. This type of effects occurs when respondents are requested to rate a series of items on numerical scales. In such situations, a problem establishing an initial reference point arises. This happens because respondents when asked to form a judgment about a particular matter might need a standard of comparison (Sudman, Bradburn, & Schwarz, 1996). The first question is then used as this sort of standard. 3Just to mention some concrete ones: the Muslim-majority states in the Indian federation, the North–South divide in the multiethnic Nigeria, or the ethnically diverse federation of South Africa. 4A coattail effect is the propensity for a popular leader of a political party to attract votes for other candidates of that particular party in an election. 5Lacy (2001) studied question-order effects related to individual choices applying the theory of nonseparable preferences. He explored question-order effects when preferences were separable and when they were not, finding that people with nonseparable preferences had greater response instability across question orders. However, Lacy’s object of study, choices or preferences on issues, differs from that of the current investigation: the ratings of institutions. Furthermore, if the theory of nonseparable preferences would have been applicable to the case studied here, contrast effects would have been more likely to appear than consistency. For instance, if preferences for Catalan and Spanish institutions were nonseparable, when a person who identifies very much with Catalonia (and very little with Spain) is asked about the Spanish institution first, she could use this information to differentiate the assessment of the Catalan institution asked on the second place. This is not what finally happens in the case studied here. 6This is a CATI survey with a stratified proportional sample of 1,050 individuals, representative of the population above 17 years of age living in Catalonia and with Spanish citizenship (±3.02% margin of error for P=Q=50). The questionnaire lasted for an average of 15 min and revolved around sociopolitical topics. 7The Centre d’Estudis d’Opinió (Center for Opinion Studies) is the official institute for public opinion studies of the regional government of Catalonia in Spain. 8Additional heterogeneous effects mentioned in the literature of question-order effects will be explored, especially those linked to education (Crespi & Morris, 1984; Narayan & Krosnick, 1996). Table 1 Main Effects of the First Experiment Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 **Sig. < 0.01; * Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2 + x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. Table 1 Main Effects of the First Experiment Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 Trust in … politicians Diff. Catalan Spanish Cat. − Spa.c Total sample x̅Tc 3.77 x̅Ts 2.65 x̅Td = (x̅Tc− x̅Ts) 1.27** STc 2.62 STs 2.27 STd 3.24 nTc 1040 nTs 1044 nTd 1039 Experimental groups  a) Catalan politicians first x̅1 4.00 x̅3 2.63 x̅a = (x̅1− x̅3) 1.37** S1 2.62 S3 2.16 Sa 3.21 n1 517 n3 520 na 517  b) Spanish politicians first x̅2 3.55 x̅4 2.66 x̅b = (x̅2− x̅4) 0.89** S2 3.77 S4 2.38 Sb 3.26 n2 523 n4 524 nb 522 Item dimensiona     Difference among groups x̅1− x̅2 0.45** x̅3− x̅4 −0.03 x̅a− x̅b 0.48** Framework dimensionb     Comparative frame x̅c 3.33     Noncomparative frame x̅nc 3.09     Difference among groups x̅c− x̅nc 0.24 **Sig. < 0.01; * Sig. < 0.05. aT-tests for independent samples are applied to compare averages across experimental groups. bThe comparative frame average is the result of averaging the means of the second question of each experimental group ((x̅2 + x̅3)/2). The noncomparative frame average is the equivalent for the first question in each experimental group ((x̅1 + x̅4)/2). T-tests for independent samples are applied to compare averages across comparative and noncomparative frames. cT-tests for related samples are applied to compare averages within the same experimental groups. 9See Guinjoan and Rodon (2016) for more information about the Linz–Moreno question. Table 2 Results of the First Experiment by Subjective National Identification Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 **Sig. < 0.01; *Sig. < 0.05. Table 2 Results of the First Experiment by Subjective National Identification Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 Subjective national identification Catalan identity Dual identity Spanish identity Trust in … politicians Trust in … politicians Trust in … politicians Catalan Spanish Catalan Spanish Catalan Spanish Total sample     x̅ 5.06 2.07 2.44 3.22 1.95 3.92     S 2.27 1.93 2.16 2.37 2.10 2.68     n 544 547 367 368 92 92 Experimental groups     a) Catalan politicians first     x̅ 5.28 2.18 2.53 3.06 1.69 3.6     S 2.20 1.95 2.11 2.17 2.01 2.60     n 284 286 165 165 52 53     b) Spanish politicians first     x̅ 4.82 1.95 2.37 3.35 2.28 4.36     S 2.32 1.89 2.21 2.51 2.21 2.76     n 260 261 202 203 40 39     Difference among groups 0.46* 0.23 0.16 −0.29 −0.59 −0.76     Levene's test         Equality of variance         F 0.58 0.01 0.40 4.60* 0.26 0.09     T test         Equality of variance         t 2.38* 1.36 0.71 −1.32 −1.34         No equality of variance         t −1.20 **Sig. < 0.01; *Sig. < 0.05. 10Tests are available upon request. 11Beyond the national versus regional logic, this type of cross-level question-order effects could eventually operate in cases in which supranational institutions, such as those of the European Union, and national institutions of a given country, like Great Britain, were included together in the same questionnaire. References Cantril H. (Ed.) ( 1944 ). Gauging public opinion . Princeton NJ : Princeton University Press . Google Scholar CrossRef Search ADS CEO (Centre d’Estudis d’Opinio) . Surveys no. REO 804 and 806 (2015), 816 and 835 (2016), and 850 (2017). Crespi I. , Morris D. ( 1984 ). Question order effect and the measurement of candidate preference in the 1982 Connecticut elections . Public Opinion Quarterly , 48 , 578 – 591 . Google Scholar CrossRef Search ADS Darcy R. , Schramm S. S. ( 1979 ). Comment on Kernell . American Political Science Review , 73 , 543 – 545 . Google Scholar CrossRef Search ADS Erikson R. S. , Luttberg N. R. , Tedin K. T. ( 1988 ). American public opinion . New York, NY : MacMillan . Guinjoan M. , Rodon T. ( 2016 ). A scrutiny of the Linz-Moreno question . Publius: The Journal of Federalism , 46 , 128 – 142 . Google Scholar CrossRef Search ADS Hierro M. J. , Gallego A. ( 2016 ). Identities in between: Political conflict and ethnonational identities in multicultural states . Journal of Conflict Resolution . First published December 30, 2016. doi: 10.1177/0022002716682593 Hyman H. H. , Sheatsley P. B. ( 1950 ). The current status of American public opinion. In Payne J. C. (Ed.), The teaching of contemporary affairs (pp. 11 – 34 ). Washington: Twenty-first Yearbook of the National Council of Social Studies . Lacy D. ( 2001 ). A theory of nonseparable preferences in survey responses . American Journal of Political Science , 45 , 239 – 258 . Google Scholar CrossRef Search ADS Lasorsa D. L. ( 2003 ). Question-order effects in surveys: The case of political interest, news attention, and knowledge . Journalism and Mass Communication Quarterly , 80 , 499 – 512 . Google Scholar CrossRef Search ADS Lombardi W. J. , Higgins H. E. , Bargh J. A. ( 1987 ). The role of consciousness in priming effects on categorization: assimilation versus contrast as a function of awareness of the priming task . Personality and Social Psychology Bulletin , 13 , 411 – 429 . Google Scholar CrossRef Search ADS Moore D. M. ( 2002 ). Measuring new types of question-order effects. Additive and subtractive . Public Opinion Quarterly , 66 , 80 – 91 . Google Scholar CrossRef Search ADS Mutz D. C. , Pemantle R. ( 2015 ). Standards for experimental research: encouraging a better understanding of experimental methods . Journal of Experimental Political Science , 2 , 192 – 215 . Google Scholar CrossRef Search ADS Narayan S. , Krosnick J. ( 1996 ). Education moderates some response effects in attitude measurement . Public Opinion Quarterly , 60 , 58 – 88 . Google Scholar CrossRef Search ADS Oldendick R. ( 2008 ). Question order effects. In Lavrakas P. (Ed.), Encyclopedia of survey research methods . Thousand Oaks, CA : Sage . Schuman H. ( 1992 ). Context effects: State of the past/state of the art. In Schwarz N. , Sudman S. (Eds.), Context effects in social and psychological research . New York, NY : Springer-Verlag . Schuman H. , Presser S. ( 1996 ). Questions and answers in attitude surveys . Thousand Oaks, CA : Sage . Schwarz N. ( 1999 ). Self-reports. How the questions shape the answers . American Psychologist , 54 , 93 – 105 . Google Scholar CrossRef Search ADS Schwarz N. , Bless H. ( 1992a ). Scandals and the public’s trust in politicians: Assimilation and contrast effects . Personality and Social Psychology Bulletin , 18 , 574 – 579 . Google Scholar CrossRef Search ADS Schwarz N. , Bless H. ( 1992b ). Constructing reality and its alternatives: An inclusion/exclusion model of assimilation and contrast effects in social judgment. In Martin L. L. , Tesser A. (Eds.), The construction of social judgements . Hillsdale, NJ : Lawrence Erlbaum Associates, Publishers . Sigelman L. ( 1981 ). Question-order effects on presidential popularity . Public Opinion Quarterly , 2 , 199 – 207 . Google Scholar CrossRef Search ADS Stapel D. A. , Schwartz N. ( 1998 ): The republican who did not want to become president: Colin Powell’s impact on evaluations of the Republican Party and Bob Dole . Personality and Social Psychology Bulletin , 24 , 690 – 698 . Google Scholar CrossRef Search ADS Strack F. ( 1992 ). ‘Order effects’ in survey research: Activation and information functions of preceding questions. In Schwarz N. , Sudman S. (Eds.), Context effects in social and psychological research . New York, NY : Springer-Verlag . Sudman S. , Bradburn N. M. , Schwarz N. ( 1996 ). Thinking about answers: The application of cognitive processes to survey methodology . San Francisco : Jossey-Bass . Tormos R. ( 2017 ). L’efecte de la situació econòmica personal en la confiança en les institucions en època de crisi. Barcelona: Papers de treball del CEO. Tormos R. , Muñoz J. , Hierro M. J. ( 2014 ). Endogenous identities? How the independence debate is reshaping Catalans’ identity. Paper presented at the IBEI Workshop The Politics of Identity Adoption, Barcelona. Available at: http://www.cuimpb.cat/files/TormosMu%C3%B1ozHierro_EndogenousIdentities.pdf Wilson D. C. ( 2010 ). Perceptions about the amount of interracial prejudice depend on racial group membership and question order . Public Opinion Quarterly , 74 , 344 – 356 . Google Scholar CrossRef Search ADS © The Author(s) 2018. Published by Oxford University Press on behalf of The World Association for Public Opinion Research. All rights reserved. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

International Journal of Public Opinion ResearchOxford University Press

Published: May 30, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off