Expanding the Methodological Arsenal of Applied Linguistics with a Robust Statistical Procedure

Expanding the Methodological Arsenal of Applied Linguistics with a Robust Statistical Procedure There is a growing awareness in methodological literature that parametric statistical tests may not be appropriate in studies with a small sample size and non-normal data (Larson-Hall and Herrington 2010; Mizumoto and Plonsky 2016). In addition, the presence of outliers or cases widely different from the rest of the data set poses serious methodological challenges: even a single extreme case can invalidate the estimated statistics in parametric methods (Larson-Hall and Herrington 2010). As Larson-Hall (2010) noted, removing outliers from the data set is undesirable for several reasons. First, such decisions are based on the researcher’s subjective opinion. Secondly, when one outlier is removed another may emerge and the subsequent removal of all outlying cases could drastically reduce the number of observations. Thirdly, removing outliers may violate a data distribution assumption (Huber 2004). In the case of mixed-methods studies eliminating outliers from the quantitative data set could yield inaccurate findings and conclusions because, by definition, such studies must synthesize both the quantitative and qualitative data. To overcome these challenges, applied linguists need to employ robust statistical methods that are not unduly affected by outliers and which do not assume that the data follow normal distribution. The present study proposes such a method; it combines a robust statistical procedure called quantile regression (QR) with bootstrapping. Bootstrapping, which is an alternative technique to traditional parametric methods, is gaining popularity in applied linguistics research because it requires fewer assumptions to be met and permits the analysis of data that are non-normal or have skewed distributions (Efron 1979; Larson-Hall and Herrington 2010; Plonsky et al. 2015). In this computer-intensive technique, the observed data are repeatedly and randomly resampled to allow for inferences to be made about unknown population from the available sample. A drawback of bootstrapping is that the proportion of outliers in the bootstrapped data could be greater than that in the original data (Salibian-Barrera and Zamar 2002). To minimize the potential effects of the outliers, bootstrapping can be combined with a robust outlier-resistant statistical method (Dietz et al. 1987; Bollen and Stine 1990). One such method is QR (Koenker and Bassett 1978). In contrast to ordinary least squares (OLS) regression, which uses the mean value, QR can be based on the middle value of the sorted data or median. The median is more resilient against outliers, so that the outcome of the statistical analysis is less likely to be affected by the extreme values (Chen and Chalhoub-Deville 2014). The main aim of this article is to demonstrate step by step how combining bootstrapping with QR can help applied linguists to solve a number of methodological challenges. Data used in the current study were extracted from a larger mixed-methods investigation of the psychological factors involved in learning a foreign language. Specifically, we examined whether language learners’ attitudes towards a target language country were associated with their stereotypes about the country and their attitudes towards the people who speak the target language. PARTICIPANTS AND VARIABLES Twenty-eight learners of Russian language (N = 28) were asked to write down their images of Russia in a free-response format. The students were then instructed to evaluate each image in their lists on a scale from −2 (‘very negative’) to +2 (‘very positive’). These ratings formed the variable ‘stereotypes’. Two thermometer-type scales ranging from 0 to 100 °C were employed to measure the students’ general attitudes towards Russia (Scale 1) and Russians (Scale 2). These attitudes formed the variables ‘attitude to country’ and ‘attitude to people’. The null hypothesis was: H0: Attitudes towards the target language country are not influenced by the students’ stereotypes about this country, nor are they associated with their attitudes towards its people. THE PROPOSED STATISTICAL PROCEDURE AND ITS FINDINGS The proposed robust statistical procedure consisted of six steps. The analyses can be performed using various software packages, such as SPSS, EViews, Stata, MATLAB, and R. To assess the null hypothesis, calculation of the confidence intervals (CI) was done using an Excel spreadsheet. The null hypothesis is rejected when zero is not contained in the CI. First of all, to highlight how findings can be misleading if basic assumptions for a statistical method are not met, the null hypothesis was tested using a standard OLS regression. As Table 1 shows, zero was included in the CI at all levels of significance; therefore, the null hypothesis could not be rejected. Table 1: OLS analysis results (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Note: *Indicates significance at the 1 per cent level. Table 1: OLS analysis results (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Note: *Indicates significance at the 1 per cent level. This finding suggests that ‘stereotypes’ and ‘attitude to people’ were not statistically significant variables to influence the language learners’ attitudes towards the target language country. This result is not only counter-intuitive; it contradicts the basic tenets of the psychological theory which postulates that individually held attitudes are derived, even if in part, from the beliefs and stereotypes associated with the attitude object (Eagly and Chaiken 1998). As Larson-Hall (2010) noted, when faced with perplexing or counter-intuitive results researchers would usually offer plausible alternative theories when the reason is clearly inappropriate statistical methods. Therefore, in the second step of the analysis, the Jarque–Bera (JB) test was performed to determine whether an OLS assumption of the residual normality was met. The JB statistic was 19.2, indicating that the assumption had been violated (see Table 2 and Figure 1). Table 2: Residual normality analysis Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Note: *Indicates significance at the 1 per cent level. Table 2: Residual normality analysis Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Note: *Indicates significance at the 1 per cent level. Figure 1: View largeDownload slide Histogram with overlaid normal distribution Figure 1: View largeDownload slide Histogram with overlaid normal distribution To address this problem, the next step was to perform the bootstrapped OLS analysis. The findings indicated that all CI included zero (see Table 3); hence, the null hypothesis could not be rejected. Table 3: Findings from bootstrapped OLS (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Note: *Indicates significance at the 1 per cent level. Table 3: Findings from bootstrapped OLS (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Note: *Indicates significance at the 1 per cent level. Bearing in mind that the outlying cases might have affected the findings, our next task was to perform the hat matrix test. An advantage of this test is that it gives a visual display of the findings. The hat matrix test identified two prominent outliers—Cases #14 and #22 (see Figure 2). Figure 2: View largeDownload slide Hat matrix test results Figure 2: View largeDownload slide Hat matrix test results To lessen the potential effects of these outliers, we re-analysed the data using QR, which is a robust outlier-resistant test. This time, the null hypothesis could be rejected but only for the variable ‘attitude to people’ (at the 5 per cent level of significance). As shown in Table 4, the 90 and 95 per cent CI for this variable did not include zero. Table 4: Findings from QR (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Notes: *Indicates significance at the 5 per cent level. ** indicates significance at the 10 per cent level. Table 4: Findings from QR (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Notes: *Indicates significance at the 5 per cent level. ** indicates significance at the 10 per cent level. In the next step, to perform a more rigorous analysis that would be robust against both non-normality and outliers, we combined bootstrapping and QR. The findings in Table 5 show that the 90 per cent CI for the variables ‘attitude to people’ and ‘stereotypes’ did not include zero; hence, the null hypothesis could be rejected for both variables at the 10 per cent level of significance. As Larson-Hall (2010) noted, in social sciences and humanities research relationships with a 10 per cent level of statistical significance deserve due attention. Table 5: Bootstrapped QR findings (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Note: *Indicates significance at the 10 per cent level. Table 5: Bootstrapped QR findings (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Note: *Indicates significance at the 10 per cent level. In sum, statistically significant relationships between the study’s variables—the result in line with psychological theories—could be detected only when assumptions for all statistical tests were checked and appropriate robust statistical procedure was adopted. Table 6 outlines the sequence of steps in this study. Table 6: Tests assumptions and sequence Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Table 6: Tests assumptions and sequence Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR METHODOLOGICAL IMPLICATIONS AND CONCLUDING REMARKS The proposed robust statistical procedure and its high resilience against outliers have some notable methodological implications, especially for mixed-methods research. The current study, as a part of a larger mixed-methods inquiry into language learners’ country images and attitudes, can serve as the case in point. To be more specific, excluding the outlying Cases #14 and #22 from the statistical analysis would have entailed discarding the qualitative data provided by these respondents. This means that 13 of 209 images about Russia would have had to be removed. Furthermore, other outliers might have cropped up requiring even more data to be discarded! As a result, the structure of students’ mental representations of the target language country would have been compromised. More importantly, in mixed-methods research, eliminating qualitative data obtained from ‘statistical outliers’ is methodologically unjustifiable because the qualitative analysis of outlying cases can lead to a deeper understanding of the issue under study. The alternative, where the qualitative data are retained but the quantitative data are discarded, is not feasible either. This is because such an approach will undoubtedly impair the credibility of findings, which must be based on a synthesis of the two kinds of data. Although this study focused on psychological variables involved in language learning, the method described here can be adopted, with some modifications if needed, in studies on a wide variety of topics that require statistical analysis. Consequently, we hope that the proposed robust statistical procedure will be a useful addition to the methodological arsenal of applied linguistics research. References Bollen K. A. , Stine R. . 1990 . ‘ Direct and indirect effects: Classical and bootstrap estimates of variability ,’ Sociological Methodology 20 : 115 – 40 . Google Scholar CrossRef Search ADS Chen F. , Chalhoub-Deville M. . 2014 . ‘ Principles of quantile regression and an application ,’ Language Testing 31 : 63 – 87 . Google Scholar CrossRef Search ADS Dietz T. , Frey R. , Kalof L. . 1987 . ‘ Estimation with cross-national data: Robust and nonparametric methods ,’ American Sociological Review 52 : 380 – 90 . Google Scholar CrossRef Search ADS Eagly A. H. , Chaiken S. . 1998 . ‘Attitude structure and function’ in Gilbert D. T. , Fiske S. T. , Lindzey G. (eds): The Handbook of Social Psychology . McGraw- Hill , pp. 269 – 322 . Efron B. 1979 . ‘ Bootstrap methods: Another look at the Jackknife ,’ The Annals of Statistics 7 : 1 – 26 . Google Scholar CrossRef Search ADS Huber P. J. 2004 . Robust Statistics . John Wiley & Sons . Koenker R. , Bassett G. . 1978 . ‘ Regression quantiles ,’ Econometrica 46 : 33 – 50 . Google Scholar CrossRef Search ADS Larson-Hall J. 2010 . A Guide to Doing Statistics in Second Language Research Using SPSS . Routledge . Larson-Hall J. , Herrington R. . 2010 . ‘ Improving data analysis in second language acquisition by utilizing modern developments in applied statistics ,’ Applied Linguistics 31 : 368 – 90 . Google Scholar CrossRef Search ADS Mizumoto A. , Plonsky L. . 2016 . ‘ R as a lingua franca: Advantages of using R for quantitative research in applied linguistics ,’ Applied Linguistics 37 : 284 – 91 . Google Scholar CrossRef Search ADS Plonsky L. , Egbert J. , Laflair G. T. . 2015 . ‘ Bootstrapping in applied linguistics: Assessing its potential using shared data ,’ Applied Linguistics 36 : 591 – 610 . Salibian-Barrera M. , Zamar R. H. . 2002 . ‘ Bootstrapping robust estimates of regression ,’ Annals of Statistics 30 : 556 – 82 . Google Scholar CrossRef Search ADS © Oxford University Press 2017 This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Linguistics Oxford University Press

Expanding the Methodological Arsenal of Applied Linguistics with a Robust Statistical Procedure

Loading next page...
 
/lp/ou_press/expanding-the-methodological-arsenal-of-applied-linguistics-with-a-57xxNLaaRX
Publisher
Oxford University Press
Copyright
© Oxford University Press 2017
ISSN
0142-6001
eISSN
1477-450X
D.O.I.
10.1093/applin/amx026
Publisher site
See Article on Publisher Site

Abstract

There is a growing awareness in methodological literature that parametric statistical tests may not be appropriate in studies with a small sample size and non-normal data (Larson-Hall and Herrington 2010; Mizumoto and Plonsky 2016). In addition, the presence of outliers or cases widely different from the rest of the data set poses serious methodological challenges: even a single extreme case can invalidate the estimated statistics in parametric methods (Larson-Hall and Herrington 2010). As Larson-Hall (2010) noted, removing outliers from the data set is undesirable for several reasons. First, such decisions are based on the researcher’s subjective opinion. Secondly, when one outlier is removed another may emerge and the subsequent removal of all outlying cases could drastically reduce the number of observations. Thirdly, removing outliers may violate a data distribution assumption (Huber 2004). In the case of mixed-methods studies eliminating outliers from the quantitative data set could yield inaccurate findings and conclusions because, by definition, such studies must synthesize both the quantitative and qualitative data. To overcome these challenges, applied linguists need to employ robust statistical methods that are not unduly affected by outliers and which do not assume that the data follow normal distribution. The present study proposes such a method; it combines a robust statistical procedure called quantile regression (QR) with bootstrapping. Bootstrapping, which is an alternative technique to traditional parametric methods, is gaining popularity in applied linguistics research because it requires fewer assumptions to be met and permits the analysis of data that are non-normal or have skewed distributions (Efron 1979; Larson-Hall and Herrington 2010; Plonsky et al. 2015). In this computer-intensive technique, the observed data are repeatedly and randomly resampled to allow for inferences to be made about unknown population from the available sample. A drawback of bootstrapping is that the proportion of outliers in the bootstrapped data could be greater than that in the original data (Salibian-Barrera and Zamar 2002). To minimize the potential effects of the outliers, bootstrapping can be combined with a robust outlier-resistant statistical method (Dietz et al. 1987; Bollen and Stine 1990). One such method is QR (Koenker and Bassett 1978). In contrast to ordinary least squares (OLS) regression, which uses the mean value, QR can be based on the middle value of the sorted data or median. The median is more resilient against outliers, so that the outcome of the statistical analysis is less likely to be affected by the extreme values (Chen and Chalhoub-Deville 2014). The main aim of this article is to demonstrate step by step how combining bootstrapping with QR can help applied linguists to solve a number of methodological challenges. Data used in the current study were extracted from a larger mixed-methods investigation of the psychological factors involved in learning a foreign language. Specifically, we examined whether language learners’ attitudes towards a target language country were associated with their stereotypes about the country and their attitudes towards the people who speak the target language. PARTICIPANTS AND VARIABLES Twenty-eight learners of Russian language (N = 28) were asked to write down their images of Russia in a free-response format. The students were then instructed to evaluate each image in their lists on a scale from −2 (‘very negative’) to +2 (‘very positive’). These ratings formed the variable ‘stereotypes’. Two thermometer-type scales ranging from 0 to 100 °C were employed to measure the students’ general attitudes towards Russia (Scale 1) and Russians (Scale 2). These attitudes formed the variables ‘attitude to country’ and ‘attitude to people’. The null hypothesis was: H0: Attitudes towards the target language country are not influenced by the students’ stereotypes about this country, nor are they associated with their attitudes towards its people. THE PROPOSED STATISTICAL PROCEDURE AND ITS FINDINGS The proposed robust statistical procedure consisted of six steps. The analyses can be performed using various software packages, such as SPSS, EViews, Stata, MATLAB, and R. To assess the null hypothesis, calculation of the confidence intervals (CI) was done using an Excel spreadsheet. The null hypothesis is rejected when zero is not contained in the CI. First of all, to highlight how findings can be misleading if basic assumptions for a statistical method are not met, the null hypothesis was tested using a standard OLS regression. As Table 1 shows, zero was included in the CI at all levels of significance; therefore, the null hypothesis could not be rejected. Table 1: OLS analysis results (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Note: *Indicates significance at the 1 per cent level. Table 1: OLS analysis results (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.237 0.830 90 per cent [−0.208, 0.602] 95 per cent [−0.291, 0.686] 99 per cent [−0.464, 0.859] Stereotypes −1.512 6.284 −0.240 90 per cent [−12.246, 9.221] 95 per cent [−14.455, 11.430] 99 per cent [−19.029, 16.004] Constant 51.738* 15.577 3.321 90 per cent [25.130, 78.345] 95 per cent [19.654, 83.821] 99 per cent [8.317, 95.158] Note: *Indicates significance at the 1 per cent level. This finding suggests that ‘stereotypes’ and ‘attitude to people’ were not statistically significant variables to influence the language learners’ attitudes towards the target language country. This result is not only counter-intuitive; it contradicts the basic tenets of the psychological theory which postulates that individually held attitudes are derived, even if in part, from the beliefs and stereotypes associated with the attitude object (Eagly and Chaiken 1998). As Larson-Hall (2010) noted, when faced with perplexing or counter-intuitive results researchers would usually offer plausible alternative theories when the reason is clearly inappropriate statistical methods. Therefore, in the second step of the analysis, the Jarque–Bera (JB) test was performed to determine whether an OLS assumption of the residual normality was met. The JB statistic was 19.2, indicating that the assumption had been violated (see Table 2 and Figure 1). Table 2: Residual normality analysis Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Note: *Indicates significance at the 1 per cent level. Table 2: Residual normality analysis Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Mean −0.001 Skewness −1.341 Median 2.317 Kurtosis 6.049 Standard deviation 17.171 JB statistic 19.249* Note: *Indicates significance at the 1 per cent level. Figure 1: View largeDownload slide Histogram with overlaid normal distribution Figure 1: View largeDownload slide Histogram with overlaid normal distribution To address this problem, the next step was to perform the bootstrapped OLS analysis. The findings indicated that all CI included zero (see Table 3); hence, the null hypothesis could not be rejected. Table 3: Findings from bootstrapped OLS (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Note: *Indicates significance at the 1 per cent level. Table 3: Findings from bootstrapped OLS (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Variable Beta Standard error t-ratio CIs Attitude to people 0.197 0.302 0.651 90 per cent [−0.320, 0.714] 95 per cent [−0.426, 0.820] 99 per cent −0.646, 1.041] Stereotypes −1.512 7.697 −0.196 90 per cent [−14.660, 11.635] 95 per cent [−17.366, 14.341] 99 per cent [−22.969, 19.943] Constant 51.738* 14.884 3.476 90 per cent [26.314, 77.162] 95 per cent [21.082, 82.393] 99 per cent [10.249, 93.226] Note: *Indicates significance at the 1 per cent level. Bearing in mind that the outlying cases might have affected the findings, our next task was to perform the hat matrix test. An advantage of this test is that it gives a visual display of the findings. The hat matrix test identified two prominent outliers—Cases #14 and #22 (see Figure 2). Figure 2: View largeDownload slide Hat matrix test results Figure 2: View largeDownload slide Hat matrix test results To lessen the potential effects of these outliers, we re-analysed the data using QR, which is a robust outlier-resistant test. This time, the null hypothesis could be rejected but only for the variable ‘attitude to people’ (at the 5 per cent level of significance). As shown in Table 4, the 90 and 95 per cent CI for this variable did not include zero. Table 4: Findings from QR (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Notes: *Indicates significance at the 5 per cent level. ** indicates significance at the 10 per cent level. Table 4: Findings from QR (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.283 2.157 90 per cent [0.127, 1.094] 95 per cent [0.027, 1.193] 99 per cent −0.178, 1.399] Stereotypes −12.658 7.489 −1.690 90 per cent [−25.451, 0.135] 95 per cent [−28.084, 2.767] 99 per cent [−33.535, 8.218] Constant 33.132** 18.565 1.784 90 per cent [1.421, 64.844] 95 per cent [−5.104, 71.370] 99 per cent [−18.616, 84.882] Notes: *Indicates significance at the 5 per cent level. ** indicates significance at the 10 per cent level. In the next step, to perform a more rigorous analysis that would be robust against both non-normality and outliers, we combined bootstrapping and QR. The findings in Table 5 show that the 90 per cent CI for the variables ‘attitude to people’ and ‘stereotypes’ did not include zero; hence, the null hypothesis could be rejected for both variables at the 10 per cent level of significance. As Larson-Hall (2010) noted, in social sciences and humanities research relationships with a 10 per cent level of statistical significance deserve due attention. Table 5: Bootstrapped QR findings (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Note: *Indicates significance at the 10 per cent level. Table 5: Bootstrapped QR findings (dependent variable: attitude to country) Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Variable Beta Standard error t-ratio CIs Attitude to people 0.610* 0.329 1.853 90 per cent [0.048, 1.173] 95 per cent [−0.067, 1.289] 99 per cent [−0.307, 1.529] Stereotypes −12.658* 7.046 −1.796 90 per cent [−24.694, −0.622] 95 per cent [−27.170, 1.854] 99 per cent [−32.299, 6.982] Constant 33.132 20.838 1.590 90 per cent [−2.460, 68.726] 95 per cent [−9.785, 76.051] 99 per cent [−24.951, 91.217] Note: *Indicates significance at the 10 per cent level. In sum, statistically significant relationships between the study’s variables—the result in line with psychological theories—could be detected only when assumptions for all statistical tests were checked and appropriate robust statistical procedure was adopted. Table 6 outlines the sequence of steps in this study. Table 6: Tests assumptions and sequence Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Table 6: Tests assumptions and sequence Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR Data non-normality Outliers Appropriate method No No OLS Yes No Bootstrapped OLS No Yes QR Yes Yes Bootstrapped QR METHODOLOGICAL IMPLICATIONS AND CONCLUDING REMARKS The proposed robust statistical procedure and its high resilience against outliers have some notable methodological implications, especially for mixed-methods research. The current study, as a part of a larger mixed-methods inquiry into language learners’ country images and attitudes, can serve as the case in point. To be more specific, excluding the outlying Cases #14 and #22 from the statistical analysis would have entailed discarding the qualitative data provided by these respondents. This means that 13 of 209 images about Russia would have had to be removed. Furthermore, other outliers might have cropped up requiring even more data to be discarded! As a result, the structure of students’ mental representations of the target language country would have been compromised. More importantly, in mixed-methods research, eliminating qualitative data obtained from ‘statistical outliers’ is methodologically unjustifiable because the qualitative analysis of outlying cases can lead to a deeper understanding of the issue under study. The alternative, where the qualitative data are retained but the quantitative data are discarded, is not feasible either. This is because such an approach will undoubtedly impair the credibility of findings, which must be based on a synthesis of the two kinds of data. Although this study focused on psychological variables involved in language learning, the method described here can be adopted, with some modifications if needed, in studies on a wide variety of topics that require statistical analysis. Consequently, we hope that the proposed robust statistical procedure will be a useful addition to the methodological arsenal of applied linguistics research. References Bollen K. A. , Stine R. . 1990 . ‘ Direct and indirect effects: Classical and bootstrap estimates of variability ,’ Sociological Methodology 20 : 115 – 40 . Google Scholar CrossRef Search ADS Chen F. , Chalhoub-Deville M. . 2014 . ‘ Principles of quantile regression and an application ,’ Language Testing 31 : 63 – 87 . Google Scholar CrossRef Search ADS Dietz T. , Frey R. , Kalof L. . 1987 . ‘ Estimation with cross-national data: Robust and nonparametric methods ,’ American Sociological Review 52 : 380 – 90 . Google Scholar CrossRef Search ADS Eagly A. H. , Chaiken S. . 1998 . ‘Attitude structure and function’ in Gilbert D. T. , Fiske S. T. , Lindzey G. (eds): The Handbook of Social Psychology . McGraw- Hill , pp. 269 – 322 . Efron B. 1979 . ‘ Bootstrap methods: Another look at the Jackknife ,’ The Annals of Statistics 7 : 1 – 26 . Google Scholar CrossRef Search ADS Huber P. J. 2004 . Robust Statistics . John Wiley & Sons . Koenker R. , Bassett G. . 1978 . ‘ Regression quantiles ,’ Econometrica 46 : 33 – 50 . Google Scholar CrossRef Search ADS Larson-Hall J. 2010 . A Guide to Doing Statistics in Second Language Research Using SPSS . Routledge . Larson-Hall J. , Herrington R. . 2010 . ‘ Improving data analysis in second language acquisition by utilizing modern developments in applied statistics ,’ Applied Linguistics 31 : 368 – 90 . Google Scholar CrossRef Search ADS Mizumoto A. , Plonsky L. . 2016 . ‘ R as a lingua franca: Advantages of using R for quantitative research in applied linguistics ,’ Applied Linguistics 37 : 284 – 91 . Google Scholar CrossRef Search ADS Plonsky L. , Egbert J. , Laflair G. T. . 2015 . ‘ Bootstrapping in applied linguistics: Assessing its potential using shared data ,’ Applied Linguistics 36 : 591 – 610 . Salibian-Barrera M. , Zamar R. H. . 2002 . ‘ Bootstrapping robust estimates of regression ,’ Annals of Statistics 30 : 556 – 82 . Google Scholar CrossRef Search ADS © Oxford University Press 2017 This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Applied LinguisticsOxford University Press

Published: Sep 14, 2017

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off