A simple diagnostic measure of inattention bias in discrete choice models

A simple diagnostic measure of inattention bias in discrete choice models Abstract This note introduces a simple, easy-to-understand measure of inattention bias in discrete choice models. The metric, ranging from 0 to 1, can be compared across studies and samples. Specifically, a latent class logit model is estimated with all parameters in one class restricted to zero. The estimated share of observations falling in the class with null parameters (representing random choices) is the diagnostic measure of interest – the random response share. We validate the metric with an empirical study that identifies inattentive respondents via a trap question. 1. Introduction Survey quality has been declining for some time, with a primary concern being measurement error (Meyer, Mok and Sullivan, 2015). Non-response rates are often used as a primary signal of survey quality (Johnson and Wislar, 2012), but there are concerns even about the quality of data attained from respondents. One concern is inattention, particularly when choices are hypothetical (Murphy et al., 2005; Carlsson, 2011). Survey methods can lead to ill-considered selections, as the participant might be concerned with the minimum amount of effort required to finish the survey to obtain their participation fee or to devote time to other activities. While comparing observed outcomes to random choice has been proposed to test the hypothesis of utility maximisation using revealed preference data (Bronars, 1987), there is, at present, no simple metric available to infer issues related to inattention bias in survey research. This problem is particularly acute for analysts relying on stated preference choice data because there are no universally agreed upon measures of model fit and performance, such as the R2 used in linear regression models. Although there are a variety of commonly reported performance metrics, such as the percent of correct predictions, the likelihood ratio index (also called McFadden’s R2 or the pseudo R2) and various indices suggested by authors such as Ben-Akiva and Lerman (1985), Cramer (1999) and Efron (1978), all have shortcomings as evidenced by the lack of consistency in terms of reporting across studies employing discrete choice models (that is, if any measure of model performance is reported at all). This paper proposes a simple, easy-to-use method that produces an intuitive fit metric that is comparable across samples and studies. In particular, we propose a metric we call the random response share (RRS) that ranges from 0 to 1, where a lower number indicates a more informative model. This note is most related to recent research that has used instructional manipulation checks, or ‘trap questions’ to check whether an individual is reading questions on the survey. Where trap questions create an opportunity for revealed participant inattention, the objective of this article is to describe a method for researchers to infer participant inattention. The RRS is calculated by estimating a latent class logit model where all parameters in one class are restricted to zero. This approach is similar to the latent class analysis conducted by Meade and Craig (2012), except that our approach is consistent with random utility theory (RUM). For the standard RUM approach, the utility of choice option j is given by Uij=Vij+εij, where Vj is the deterministic portion of utility and εij is a random term, which is typically assumed to be known to the individual but not the analyst. Analysis typically proceeds by assuming a functional form for Vij. Our conjecture here is that there may be some individuals for whom Vij=0, meaning their responses are determined completely by the random term – i.e. Uij=εij. Assuming that the εij are distributed iid type I extreme value, the probability of choosing option j is conditional on the latent class C: sj|C={exp(Vij)∑k=1Jexp(Vik)ηifC=1exp(0)JifC=0, (1) where C = 1 are choices described by a traditional multinomial logit model and C = 0 is the restricted class where systematic utility has been set to zero (i.e. choices are defined only by the random term). RRS is equal to the estimated probability that participant i is in the restricted class ( Qic=0) with utility equal to zero such that Prob(class=C)=Qic=exp(θczi}∑c=1Cexp(θczi), (2) where the probability is determined by individual choice characteristics zi and θc are parameter estimates for each latent class. RSS is equal to the probability that C = 0. Stated simply, the model determines the probability that choices are effectively random. An RRS of 1 implies all choices are best explained by a model of pure random choice; an RRS of 0 means all choices are best explained by the estimated utility function (or functions if there are more than two classes).1 RRS is meant to be a measure of inattention in discrete choice models and can be readily estimated using ‘canned’ procedures in software packages, such as NLOGIT. The remainder of this research note is organised into three parts. In the following section, we describe how the RRS fits into the data quality and inattention literature. We then validate the RRS concept using stated preference choice data where we can compare RRS across people who have and have not missed a trap question. The final section concludes. 2. Inattention bias The notion that research participants do not always attentively respond to surveys is not new. Decades ago, Nichols, Greene and Schmolck (1989) identified two types of inattention bias: content responsive faking and content nonresponsivity. The key difference is the influence of the survey content: content responsive faking assumes that inattention bias is not all encompassing, and instead, participants simply respond via a basic heuristic. The discrete choice literature has developed a rich literature in response to this concern via attribute non-attendance (ANA) modelling. ANA has been studied substantially in the health economics literature and other methods have been proposed to address ANA in discrete choice settings (Hole, 2011; Hole, Norman and Viney, 2016). In essence, ANA reveals that some people do not place any utility weight (or do not pay attention to) particular attributes (e.g. Campbell, Hensher and Scarpa, 2011; LaGarde, 2013; Scarpa et al., 2013). Stated another way, the ANA itself is a method whose intention is to impose an a priori behavioural restriction onto choice data. By contrast, the RRS is a simple measure whose intention is to provide quick, meaningful information regarding the issues surrounding measurement error due to inattention bias. Although methodologically related, our approach also differs from the research on ANA as this article focuses on the second type of inattention bias: content nonresponsivity. Rather than seeking to systemically identify heuristics in the survey data, this concern is driven by ‘careless responding’ (Meade and Craig, 2012) or, for our purposes, ‘random responses’. Research in the psychology literature has suggested multiple methods for identifying careless respondents, although none has proven to be a panacea (Curran, 2016). For example, although response time is related to respondent inattention, it is difficult to identify an ‘optimal’ response time (Börger, 2016). Recent research in the discrete choice literature has generally identified poor respondents via questions with obvious answers, or ‘trap questions’ (Gao, House, and Xie, 2016). Although these trap questions are unlikely to perfectly identify inattention bias, participants who incorrectly respond to the trap questions are more likely to provide questionable answers to choice questions (Jones, House and Gao, 2015). These inattentive responses have the potential to inflate policy recommendations and willingness-to-pay estimates (Malone and Lusk, 2018). As such, we compare the results from our RRS model with stated preference data where a trap question was used to identify inattention bias. 3. Validation with stated preference choice data and a trap question We now move to an empirical application where we explore whether RRS varies with an alternative measure of inattention produced via a trap question. Data for the choice experiment come from one of the monthly editions of the Food Demand Survey, an online survey where participants are asked to choose between nine randomly ordered options (two beef, two pork, two chicken, two non-meat alternatives and a ‘none’ option) for nine choices. The choice experiment is a simple ‘branded’ choice experiment where the only attribute varying across options is product price (Lusk, 2017). After completing the choice experiment, subjects were asked the trap question shown in Figure 1, which is similar to that used in the broader literature on trap questions.2 Participants were asked the trap question directly after the choice questions so as not to induce protest-like or other reactive behaviour when answering the choice questions (Oppenheimer, Meyvis and Davidenko, 2009). The actual instruction provided to the respondent was embedded in a paragraph-long survey question. Reading only the first and last few lines would direct participants to answer the question based on their mood, but fully reading the instructions in the question would notify attentive participants to select ‘none of the above’. Fig. 1. View largeDownload slide Trap question used to identify potentially inattentive respondents. Twenty different emotions are listed after the question, with the final option being ‘none of the above’. Fig. 1. View largeDownload slide Trap question used to identify potentially inattentive respondents. Twenty different emotions are listed after the question, with the final option being ‘none of the above’. Of the 1,017 people who successfully completed the survey, 378 (37.2 per cent) did not correctly answer the trap question. Table 1 provides a comparison of parameter estimates for the latent class logit models when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random. Likelihood ratio tests indicate that relative to the traditional multinomial models, the restricted latent class models improve model fit for all three specifications (χ2Correct = 1,449.6, χ2Incorrect = 489.3 and χ2Full data = 2,261.2). Responses from the participants who correctly answered the trap question were as good as random 33.5 per cent of the time. However, the probability a choice is randomly made nearly doubles for the participants who incorrectly responded to the trap question (62.5 per cent). That is, the RRS measure is nearly double for people who missed the trap question, suggesting that the proposed method is a valid measure of inattention.3 Table 1. Latent class logit model estimates when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 aAsterisk represents statistical significance at the 0.05 level. bNumbers in parentheses are standard errors. Table 1. Latent class logit model estimates when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 aAsterisk represents statistical significance at the 0.05 level. bNumbers in parentheses are standard errors. A key concern for choice experimenters is that participant inattention might inflate willingness-to-pay (WTP) estimates. From the full dataset, Figure 2 displays participant WTP for the traditional multinomial logit (MNL) model versus the models with an estimated RRS. Results suggest that random responses inflated WTP estimates by approximately 23 per cent. For example, the traditional model suggests that consumer WTP for steak was $4.14; when we control for random responses via the RRS, consumer WTP decreased to $2.93. The only exception was WTP for pasta, which interestingly increased by 28 per cent when we control for participant inattention. Fig. 2. View largeDownload slide Participant willingness-to-pay estimates based on choice data from the discrete choice experiment for meat (N = 1,017). Fig. 2. View largeDownload slide Participant willingness-to-pay estimates based on choice data from the discrete choice experiment for meat (N = 1,017). 4. Conclusion While there may be no clear threshold that might identify when measurement error due to participant inattention completely invalidates survey results, our results imply that a non-trivial fraction of survey takers is inattentive, with their choice behaviour being statistically indistinguishable from random answers. Inattentive survey takers can influence model estimates, biasing willingness-to-pay estimates and resulting policy recommendations. For example, the WTP estimates for correct respondents were not statistically different between the traditional MNL and the restricted latent class model. However, the restricted latent class model for incorrect participants indicates that severe participant inattention can skew WTP estimates. Although the metric proposed here represents a useful contribution for discrete choice modellers, the RRS is not a panacea against inattention bias. It focuses on the percentage of individuals who ignored all attributes and is not intended to provide a measure of ANA. It is also possible that the probability of randomness is related to the number of choices, meaning that cross-study interpretation of the RRS should be cognisant of differences in design of the choice experiment. This article is also not recommending that the RRS approach be applied instead of the ANA or mixed logit approaches, as those models may often provide a better fit. Regardless, this study highlights the problem of inattentiveness in stated preference data and provides a simple way for practitioners to determine the problem’s severity for their own research. Footnotes 1 We provide a ‘proof of concept’ by calculating the RRS with simulated choice data where the share of true random responses is known in the Appendix in supplementary data at ERAE online. 2 For a more thorough description of the survey methods, see Malone and Lusk (2018). 3 The RRS model with all participants suggests that 44.7% of responses were random, although only 37.2% of respondents incorrectly missed the trap question. If we were to completely trust the effectiveness of the trap question in identifying inattention, this would imply that the RRS is overstating participant inattention. However, research in psychology suggests that a single trap question is not a comprehensive method for identifying inattention (Curran, 2016). Because of our findings via the simulated data in the Appendix in supplementary data at ERAE online, we can still safely conclude that the RRS is fairly accurate at identifying randomness in a sample. References Ben-Akiva , M. E. and Lerman , S. R. ( 1985 ). Discrete Choice Analysis: Theory and Application to Travel Demand . Cambridge, MA : MIT Press . Bronars , S. G. ( 1987 ). The power of nonparametric tests of preference maximization . Econometrica 55 ( 3 ): 693 – 698 . Google Scholar CrossRef Search ADS Börger , T. ( 2016 ). Are fast responses more random? Testing the effect of response time on scale in an online choice experiment . Environmental and Resource Economics 65 ( 2 ): 389 – 413 . Google Scholar CrossRef Search ADS Campbell , D. , David , A. H. and Scarpa , R. ( 2011 ). Non-attendance to attributes in environmental choice analysis: a latent class specification . Journal of Environmental Planning and Management 54 ( 8 ): 1061 – 1076 . Google Scholar CrossRef Search ADS Carlsson , F. ( 2011 ). Non-market valuation: stated preference methods. In: J. L. Lusk , J. Roosen and J. E. Shogren (eds), The Oxford Handbook of the Economics of Food Consumption and Policy . New York : Oxford University Press , 181 – 214 . Cramer , J. S. ( 1999 ). Predictive performance of the binary logit model in unbalanced samples . Journal of the Royal Statistical Society 48 ( 1 ): 85 – 94 . Curran , P. G. ( 2016 ). Methods for the detection of carelessly invalid responses in survey data . Journal of Experimental Social Psychology 66 : 4 – 19 . Google Scholar CrossRef Search ADS Efron , B. ( 1978 ). Regression and ANOVA with zero-one data: measures of residual variation . Journal of the American Statistical Association 73 ( 361 ): 113 – 212 . Google Scholar CrossRef Search ADS Gao , Z. , Lisa , A. H. and Xie , J. ( 2016 ). Online survey data quality and its implication for willingness-to-pay: a cross-country comparison . Canadian Journal of Agricultural Economics 64 ( 2 ): 199 – 221 . Google Scholar CrossRef Search ADS Hole , A. R. ( 2011 ). A discrete choice model with endogenous attribute attendance . Economics Letters 110 ( 3 ): 203 – 205 . Google Scholar CrossRef Search ADS Hole , A. R. , Norman , R. and Viney , R. ( 2016 ). Response patterns in health state valuation using endogenous attribute attendance and latent class analysis . Health Economics 25 ( 2 ): 212 – 224 . Google Scholar CrossRef Search ADS PubMed Johnson , T. P. and Wislar , J. S. ( 2012 ). Response rates and nonresponse errors in surveys . Journal of the American Medical Association 307 ( 17 ): 1805 – 1806 . Google Scholar CrossRef Search ADS PubMed Jones , M. S. , Lisa , A. H. and Gao , Z. ( 2015 ). Participant screening and revealed preference axioms: testing quarantining methods for enhanced data quality in web panel surveys . Public Opinion Quarterly 79 ( 3 ): 687 – 709 . Google Scholar CrossRef Search ADS Lagarde , M. ( 2013 ). Investigating attribute non‐attendance and its consequences in choice experiments with latent class models . Health Economics 22 ( 5 ): 554 – 567 . Google Scholar CrossRef Search ADS PubMed Lusk , J. L. ( 2017 ). Consumer research with big data: applications from the Food Demand Survey (FooDS) . American Journal of Agricultural Economics 99 ( 2 ): 303 – 320 . Google Scholar CrossRef Search ADS Malone , T. and Lusk , J. L. ( 2018 ). Consequences of participant inattention with an application to carbon taxes for meat products . Ecological Economics 145 : 218 – 230 . Google Scholar CrossRef Search ADS Meade , A. W. and Craig , S. B. ( 2012 ). Identifying careless responses in survey data . Psychological Methods 17 ( 3 ): 437 – 455 . Google Scholar CrossRef Search ADS PubMed Meyer , B. D. , Wallace , K. C. M. and James , X. S. ( 2015 ). Household surveys in crisis . Journal of Economic Perspectives 29 ( 4 ): 1 – 29 . Google Scholar CrossRef Search ADS Murphy , J. J. , Allen , P. G. , Stevens , T. H. and Weatherhead. , D. ( 2005 ). A meta-analysis of hypothetical bias in stated preference valuation . Environmental and Resource Economics 30 ( 3 ): 313 – 325 . Google Scholar CrossRef Search ADS Nichols , D. S. , Roger , L. G. and Schmolck , P. ( 1989 ). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials . Journal of Clinical Psychology 45 ( 2 ): 239 – 250 . Google Scholar CrossRef Search ADS PubMed Oppenheimer , D. , Meyvis , T. and Davidenko , N. ( 2009 ). Instructional manipulation checks: detecting satisficing to increase statistical power . Journal of Experimental Social Psychology 45 ( 4 ): 867 – 872 . Google Scholar CrossRef Search ADS Scarpa , R. , Zanoli , R. , Bruschi , V. and Naspetti , S. ( 2013 ). Inferred and stated attribute non-attendance in food choice experiments . American Journal of Agricultural Economics 95 ( 1 ): 165 – 180 . Google Scholar CrossRef Search ADS Author notes Review coordinated by Iain Fraser © Oxford University Press and Foundation for the European Review of Agricultural Economics 2018; all rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png European Review of Agricultural Economics Oxford University Press

A simple diagnostic measure of inattention bias in discrete choice models

Loading next page...
 
/lp/ou_press/a-simple-diagnostic-measure-of-inattention-bias-in-discrete-choice-N9GSBQQZ0G
Publisher
Oxford University Press
Copyright
© Oxford University Press and Foundation for the European Review of Agricultural Economics 2018; all rights reserved. For permissions, please e-mail: journals.permissions@oup.com
ISSN
0165-1587
eISSN
1464-3618
D.O.I.
10.1093/erae/jby005
Publisher site
See Article on Publisher Site

Abstract

Abstract This note introduces a simple, easy-to-understand measure of inattention bias in discrete choice models. The metric, ranging from 0 to 1, can be compared across studies and samples. Specifically, a latent class logit model is estimated with all parameters in one class restricted to zero. The estimated share of observations falling in the class with null parameters (representing random choices) is the diagnostic measure of interest – the random response share. We validate the metric with an empirical study that identifies inattentive respondents via a trap question. 1. Introduction Survey quality has been declining for some time, with a primary concern being measurement error (Meyer, Mok and Sullivan, 2015). Non-response rates are often used as a primary signal of survey quality (Johnson and Wislar, 2012), but there are concerns even about the quality of data attained from respondents. One concern is inattention, particularly when choices are hypothetical (Murphy et al., 2005; Carlsson, 2011). Survey methods can lead to ill-considered selections, as the participant might be concerned with the minimum amount of effort required to finish the survey to obtain their participation fee or to devote time to other activities. While comparing observed outcomes to random choice has been proposed to test the hypothesis of utility maximisation using revealed preference data (Bronars, 1987), there is, at present, no simple metric available to infer issues related to inattention bias in survey research. This problem is particularly acute for analysts relying on stated preference choice data because there are no universally agreed upon measures of model fit and performance, such as the R2 used in linear regression models. Although there are a variety of commonly reported performance metrics, such as the percent of correct predictions, the likelihood ratio index (also called McFadden’s R2 or the pseudo R2) and various indices suggested by authors such as Ben-Akiva and Lerman (1985), Cramer (1999) and Efron (1978), all have shortcomings as evidenced by the lack of consistency in terms of reporting across studies employing discrete choice models (that is, if any measure of model performance is reported at all). This paper proposes a simple, easy-to-use method that produces an intuitive fit metric that is comparable across samples and studies. In particular, we propose a metric we call the random response share (RRS) that ranges from 0 to 1, where a lower number indicates a more informative model. This note is most related to recent research that has used instructional manipulation checks, or ‘trap questions’ to check whether an individual is reading questions on the survey. Where trap questions create an opportunity for revealed participant inattention, the objective of this article is to describe a method for researchers to infer participant inattention. The RRS is calculated by estimating a latent class logit model where all parameters in one class are restricted to zero. This approach is similar to the latent class analysis conducted by Meade and Craig (2012), except that our approach is consistent with random utility theory (RUM). For the standard RUM approach, the utility of choice option j is given by Uij=Vij+εij, where Vj is the deterministic portion of utility and εij is a random term, which is typically assumed to be known to the individual but not the analyst. Analysis typically proceeds by assuming a functional form for Vij. Our conjecture here is that there may be some individuals for whom Vij=0, meaning their responses are determined completely by the random term – i.e. Uij=εij. Assuming that the εij are distributed iid type I extreme value, the probability of choosing option j is conditional on the latent class C: sj|C={exp(Vij)∑k=1Jexp(Vik)ηifC=1exp(0)JifC=0, (1) where C = 1 are choices described by a traditional multinomial logit model and C = 0 is the restricted class where systematic utility has been set to zero (i.e. choices are defined only by the random term). RRS is equal to the estimated probability that participant i is in the restricted class ( Qic=0) with utility equal to zero such that Prob(class=C)=Qic=exp(θczi}∑c=1Cexp(θczi), (2) where the probability is determined by individual choice characteristics zi and θc are parameter estimates for each latent class. RSS is equal to the probability that C = 0. Stated simply, the model determines the probability that choices are effectively random. An RRS of 1 implies all choices are best explained by a model of pure random choice; an RRS of 0 means all choices are best explained by the estimated utility function (or functions if there are more than two classes).1 RRS is meant to be a measure of inattention in discrete choice models and can be readily estimated using ‘canned’ procedures in software packages, such as NLOGIT. The remainder of this research note is organised into three parts. In the following section, we describe how the RRS fits into the data quality and inattention literature. We then validate the RRS concept using stated preference choice data where we can compare RRS across people who have and have not missed a trap question. The final section concludes. 2. Inattention bias The notion that research participants do not always attentively respond to surveys is not new. Decades ago, Nichols, Greene and Schmolck (1989) identified two types of inattention bias: content responsive faking and content nonresponsivity. The key difference is the influence of the survey content: content responsive faking assumes that inattention bias is not all encompassing, and instead, participants simply respond via a basic heuristic. The discrete choice literature has developed a rich literature in response to this concern via attribute non-attendance (ANA) modelling. ANA has been studied substantially in the health economics literature and other methods have been proposed to address ANA in discrete choice settings (Hole, 2011; Hole, Norman and Viney, 2016). In essence, ANA reveals that some people do not place any utility weight (or do not pay attention to) particular attributes (e.g. Campbell, Hensher and Scarpa, 2011; LaGarde, 2013; Scarpa et al., 2013). Stated another way, the ANA itself is a method whose intention is to impose an a priori behavioural restriction onto choice data. By contrast, the RRS is a simple measure whose intention is to provide quick, meaningful information regarding the issues surrounding measurement error due to inattention bias. Although methodologically related, our approach also differs from the research on ANA as this article focuses on the second type of inattention bias: content nonresponsivity. Rather than seeking to systemically identify heuristics in the survey data, this concern is driven by ‘careless responding’ (Meade and Craig, 2012) or, for our purposes, ‘random responses’. Research in the psychology literature has suggested multiple methods for identifying careless respondents, although none has proven to be a panacea (Curran, 2016). For example, although response time is related to respondent inattention, it is difficult to identify an ‘optimal’ response time (Börger, 2016). Recent research in the discrete choice literature has generally identified poor respondents via questions with obvious answers, or ‘trap questions’ (Gao, House, and Xie, 2016). Although these trap questions are unlikely to perfectly identify inattention bias, participants who incorrectly respond to the trap questions are more likely to provide questionable answers to choice questions (Jones, House and Gao, 2015). These inattentive responses have the potential to inflate policy recommendations and willingness-to-pay estimates (Malone and Lusk, 2018). As such, we compare the results from our RRS model with stated preference data where a trap question was used to identify inattention bias. 3. Validation with stated preference choice data and a trap question We now move to an empirical application where we explore whether RRS varies with an alternative measure of inattention produced via a trap question. Data for the choice experiment come from one of the monthly editions of the Food Demand Survey, an online survey where participants are asked to choose between nine randomly ordered options (two beef, two pork, two chicken, two non-meat alternatives and a ‘none’ option) for nine choices. The choice experiment is a simple ‘branded’ choice experiment where the only attribute varying across options is product price (Lusk, 2017). After completing the choice experiment, subjects were asked the trap question shown in Figure 1, which is similar to that used in the broader literature on trap questions.2 Participants were asked the trap question directly after the choice questions so as not to induce protest-like or other reactive behaviour when answering the choice questions (Oppenheimer, Meyvis and Davidenko, 2009). The actual instruction provided to the respondent was embedded in a paragraph-long survey question. Reading only the first and last few lines would direct participants to answer the question based on their mood, but fully reading the instructions in the question would notify attentive participants to select ‘none of the above’. Fig. 1. View largeDownload slide Trap question used to identify potentially inattentive respondents. Twenty different emotions are listed after the question, with the final option being ‘none of the above’. Fig. 1. View largeDownload slide Trap question used to identify potentially inattentive respondents. Twenty different emotions are listed after the question, with the final option being ‘none of the above’. Of the 1,017 people who successfully completed the survey, 378 (37.2 per cent) did not correctly answer the trap question. Table 1 provides a comparison of parameter estimates for the latent class logit models when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random. Likelihood ratio tests indicate that relative to the traditional multinomial models, the restricted latent class models improve model fit for all three specifications (χ2Correct = 1,449.6, χ2Incorrect = 489.3 and χ2Full data = 2,261.2). Responses from the participants who correctly answered the trap question were as good as random 33.5 per cent of the time. However, the probability a choice is randomly made nearly doubles for the participants who incorrectly responded to the trap question (62.5 per cent). That is, the RRS measure is nearly double for people who missed the trap question, suggesting that the proposed method is a valid measure of inattention.3 Table 1. Latent class logit model estimates when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 aAsterisk represents statistical significance at the 0.05 level. bNumbers in parentheses are standard errors. Table 1. Latent class logit model estimates when one class estimates the deterministic portion of the indirect utility function and the other class is restricted to be random Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 Parameter Correctly answered the trap question Incorrectly answered the trap question All participants Price −0.906*a (0.023)b −0.732* (0.043) −0.873* (0.020) Hamburger 4.328* (0.117) 4.856* (0.303) 4.404* (0.107) Steak 2.359* (0.123) 3.540* (0.309) 2.555* (0.113) Pork chop 3.668* (0.121) 4.135* (0.302) 3.727* (0.110) Ham 2.490* (0.119) 3.009* (0.309) 2.555* (0.110) Chicken breast 4.563* (0.116) 5.042* (0.299) 4.627* (0.106) Chicken wing 2.288* (0.115) 2.796* (0.305) 2.333* (0.107) Beans and rice 1.891* (0.110) 2.203* (0.305) 1.895* (0.104) Pasta 2.995* (0.140) 3.006* (0.351) 2.967* (0.128) Probability of random responses 0.335* (0.020) 0.625* (0.028) 0.447* (0.017) Log likelihood function −10,460.0 −6,852.1 −17,374.1 AIC 20,939.9 13,724.3 34,768.2 Number of observations 5,751 3,402 9,153 Number of respondents 639 378 1,017 aAsterisk represents statistical significance at the 0.05 level. bNumbers in parentheses are standard errors. A key concern for choice experimenters is that participant inattention might inflate willingness-to-pay (WTP) estimates. From the full dataset, Figure 2 displays participant WTP for the traditional multinomial logit (MNL) model versus the models with an estimated RRS. Results suggest that random responses inflated WTP estimates by approximately 23 per cent. For example, the traditional model suggests that consumer WTP for steak was $4.14; when we control for random responses via the RRS, consumer WTP decreased to $2.93. The only exception was WTP for pasta, which interestingly increased by 28 per cent when we control for participant inattention. Fig. 2. View largeDownload slide Participant willingness-to-pay estimates based on choice data from the discrete choice experiment for meat (N = 1,017). Fig. 2. View largeDownload slide Participant willingness-to-pay estimates based on choice data from the discrete choice experiment for meat (N = 1,017). 4. Conclusion While there may be no clear threshold that might identify when measurement error due to participant inattention completely invalidates survey results, our results imply that a non-trivial fraction of survey takers is inattentive, with their choice behaviour being statistically indistinguishable from random answers. Inattentive survey takers can influence model estimates, biasing willingness-to-pay estimates and resulting policy recommendations. For example, the WTP estimates for correct respondents were not statistically different between the traditional MNL and the restricted latent class model. However, the restricted latent class model for incorrect participants indicates that severe participant inattention can skew WTP estimates. Although the metric proposed here represents a useful contribution for discrete choice modellers, the RRS is not a panacea against inattention bias. It focuses on the percentage of individuals who ignored all attributes and is not intended to provide a measure of ANA. It is also possible that the probability of randomness is related to the number of choices, meaning that cross-study interpretation of the RRS should be cognisant of differences in design of the choice experiment. This article is also not recommending that the RRS approach be applied instead of the ANA or mixed logit approaches, as those models may often provide a better fit. Regardless, this study highlights the problem of inattentiveness in stated preference data and provides a simple way for practitioners to determine the problem’s severity for their own research. Footnotes 1 We provide a ‘proof of concept’ by calculating the RRS with simulated choice data where the share of true random responses is known in the Appendix in supplementary data at ERAE online. 2 For a more thorough description of the survey methods, see Malone and Lusk (2018). 3 The RRS model with all participants suggests that 44.7% of responses were random, although only 37.2% of respondents incorrectly missed the trap question. If we were to completely trust the effectiveness of the trap question in identifying inattention, this would imply that the RRS is overstating participant inattention. However, research in psychology suggests that a single trap question is not a comprehensive method for identifying inattention (Curran, 2016). Because of our findings via the simulated data in the Appendix in supplementary data at ERAE online, we can still safely conclude that the RRS is fairly accurate at identifying randomness in a sample. References Ben-Akiva , M. E. and Lerman , S. R. ( 1985 ). Discrete Choice Analysis: Theory and Application to Travel Demand . Cambridge, MA : MIT Press . Bronars , S. G. ( 1987 ). The power of nonparametric tests of preference maximization . Econometrica 55 ( 3 ): 693 – 698 . Google Scholar CrossRef Search ADS Börger , T. ( 2016 ). Are fast responses more random? Testing the effect of response time on scale in an online choice experiment . Environmental and Resource Economics 65 ( 2 ): 389 – 413 . Google Scholar CrossRef Search ADS Campbell , D. , David , A. H. and Scarpa , R. ( 2011 ). Non-attendance to attributes in environmental choice analysis: a latent class specification . Journal of Environmental Planning and Management 54 ( 8 ): 1061 – 1076 . Google Scholar CrossRef Search ADS Carlsson , F. ( 2011 ). Non-market valuation: stated preference methods. In: J. L. Lusk , J. Roosen and J. E. Shogren (eds), The Oxford Handbook of the Economics of Food Consumption and Policy . New York : Oxford University Press , 181 – 214 . Cramer , J. S. ( 1999 ). Predictive performance of the binary logit model in unbalanced samples . Journal of the Royal Statistical Society 48 ( 1 ): 85 – 94 . Curran , P. G. ( 2016 ). Methods for the detection of carelessly invalid responses in survey data . Journal of Experimental Social Psychology 66 : 4 – 19 . Google Scholar CrossRef Search ADS Efron , B. ( 1978 ). Regression and ANOVA with zero-one data: measures of residual variation . Journal of the American Statistical Association 73 ( 361 ): 113 – 212 . Google Scholar CrossRef Search ADS Gao , Z. , Lisa , A. H. and Xie , J. ( 2016 ). Online survey data quality and its implication for willingness-to-pay: a cross-country comparison . Canadian Journal of Agricultural Economics 64 ( 2 ): 199 – 221 . Google Scholar CrossRef Search ADS Hole , A. R. ( 2011 ). A discrete choice model with endogenous attribute attendance . Economics Letters 110 ( 3 ): 203 – 205 . Google Scholar CrossRef Search ADS Hole , A. R. , Norman , R. and Viney , R. ( 2016 ). Response patterns in health state valuation using endogenous attribute attendance and latent class analysis . Health Economics 25 ( 2 ): 212 – 224 . Google Scholar CrossRef Search ADS PubMed Johnson , T. P. and Wislar , J. S. ( 2012 ). Response rates and nonresponse errors in surveys . Journal of the American Medical Association 307 ( 17 ): 1805 – 1806 . Google Scholar CrossRef Search ADS PubMed Jones , M. S. , Lisa , A. H. and Gao , Z. ( 2015 ). Participant screening and revealed preference axioms: testing quarantining methods for enhanced data quality in web panel surveys . Public Opinion Quarterly 79 ( 3 ): 687 – 709 . Google Scholar CrossRef Search ADS Lagarde , M. ( 2013 ). Investigating attribute non‐attendance and its consequences in choice experiments with latent class models . Health Economics 22 ( 5 ): 554 – 567 . Google Scholar CrossRef Search ADS PubMed Lusk , J. L. ( 2017 ). Consumer research with big data: applications from the Food Demand Survey (FooDS) . American Journal of Agricultural Economics 99 ( 2 ): 303 – 320 . Google Scholar CrossRef Search ADS Malone , T. and Lusk , J. L. ( 2018 ). Consequences of participant inattention with an application to carbon taxes for meat products . Ecological Economics 145 : 218 – 230 . Google Scholar CrossRef Search ADS Meade , A. W. and Craig , S. B. ( 2012 ). Identifying careless responses in survey data . Psychological Methods 17 ( 3 ): 437 – 455 . Google Scholar CrossRef Search ADS PubMed Meyer , B. D. , Wallace , K. C. M. and James , X. S. ( 2015 ). Household surveys in crisis . Journal of Economic Perspectives 29 ( 4 ): 1 – 29 . Google Scholar CrossRef Search ADS Murphy , J. J. , Allen , P. G. , Stevens , T. H. and Weatherhead. , D. ( 2005 ). A meta-analysis of hypothetical bias in stated preference valuation . Environmental and Resource Economics 30 ( 3 ): 313 – 325 . Google Scholar CrossRef Search ADS Nichols , D. S. , Roger , L. G. and Schmolck , P. ( 1989 ). Criteria for assessing inconsistent patterns of item endorsement on the MMPI: rationale, development, and empirical trials . Journal of Clinical Psychology 45 ( 2 ): 239 – 250 . Google Scholar CrossRef Search ADS PubMed Oppenheimer , D. , Meyvis , T. and Davidenko , N. ( 2009 ). Instructional manipulation checks: detecting satisficing to increase statistical power . Journal of Experimental Social Psychology 45 ( 4 ): 867 – 872 . Google Scholar CrossRef Search ADS Scarpa , R. , Zanoli , R. , Bruschi , V. and Naspetti , S. ( 2013 ). Inferred and stated attribute non-attendance in food choice experiments . American Journal of Agricultural Economics 95 ( 1 ): 165 – 180 . Google Scholar CrossRef Search ADS Author notes Review coordinated by Iain Fraser © Oxford University Press and Foundation for the European Review of Agricultural Economics 2018; all rights reserved. For permissions, please e-mail: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

European Review of Agricultural EconomicsOxford University Press

Published: Mar 24, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off