# A quantitative, longitudinal analysis of the impact of mathematics support in an Irish university

A quantitative, longitudinal analysis of the impact of mathematics support in an Irish university Abstract Mathematics support has become embedded in a large number of higher education institutes in Ireland and UK in particular. Measuring the impact of such support is a challenging task, which can be attempted either qualitatively, through the use of surveys or focus groups, or quantitatively, looking at data such as number of visits and subsequent exam performance. Here, we consider a quantitative analysis of the impact of a mathematics support centre in an Irish university, based on data gathered over the past 12 years. A binary logistic regression was carried out which showed that, when prior mathematical achievement and module studied were kept constant, the odds of a student who attended mathematics support once passing their module were 1.63 times higher than for one who had never engaged with the service. The odds for those who attended 15 or more times were almost 14 times higher. This study also showed that there was a significant difference between those who never engaged with mathematics support and those who attended once, meaning that such cohorts should be considered differently, which has not traditionally been done in other studies conducted to this point. 1. Introduction Measuring the impact of mathematics support is a difficult process, given that it should, by its nature, support the traditional lecture-tutorial format for students who need additional help. Isolating the impact of this support from students’ learning experiences in lectures, tutorials, lab classes, online homework and so on is challenging. However, as mathematics support becomes embedded in higher education over a longer period of time, longitudinal data and larger sample sizes mean that a more reliable picture of this impact can be investigated from a quantitative point of view. The Mathematics Learning Centre (MLC) in Dublin City University (DCU) was set up in February 2004 with the aim of providing additional assistance to all undergraduate students undertaking any mathematics module as part of their degree. In 2006, some early indications of the effectiveness of the MLC were reported by Dowling & Nolan (2006). This work presented evidence that the MLC had a positive impact on student retention and that the pass rates of First Science students (who can go on to specialize in programmes such as Biotechnology, Environmental Science and Health, Analytical Science or Chemical and Pharmaceutical Sciences) and at-risk students were higher for those who had attended the MLC than those who had not. While the findings at the time were not conclusive, they were important in validating the role of and acquiring funding for the MLC, which is in operation to date. In fact, in 2014/2015, the MLC was relocated to the ground floor of the Library, to a location which is more accessible, more comfortable and has a higher seating capacity. Given this recent upgrade and the duration for which the MLC has been in operation, it is an appropriate time to conduct a longitudinal study of the impact of mathematics support within DCU to date. Therefore, in this paper, we will address the following research questions: (1) When controlling for factors such as prior mathematical background and module studied, what impact does usage of mathematics support services have upon student success in final examinations? (2) Given the varying levels of support that may be optimal for different students, is it appropriate to consider even a single visit as engaging with mathematics support? 2. Literature review A number of studies on the effectiveness of mathematics support have been conducted in Ireland and UK, with some relying on qualitative data such as surveys, focus groups and interviews and others taking a more quantitative approach, studying attendance rates and examination performance. A comprehensive review of all of these studies is outside the scope of this paper, but can instead be found in the study by Matthews et al. (2013). Here, we will focus in detail on some of the quantitative studies conducted to date. Patel & Little (2006) looked at pass rates of students who had availed of the MLC service versus those who had not and found that 92% of mathematics-related passes were associated with MLC use in comparison to 88% passes related to non-use of the MLC. Pell & Croft (2008) analysed data from five first-year modules in the Engineering department in Loughborough University. They found that 20% of students in all but one module attended the MLC between 2 and 9 times, whereas 3–8% of students used it 10 times or more. Surprisingly, however, 35% of students who had achieved an A* attended the support more than once and from this the authors concluded that the facility was being used more by students seeking excellence than by less-able students looking to avoid failure. Moreover, it is noteworthy that despite the success rate of students who did attend the MLC more than once, Pell and Croft reported that roughly 10% of students visited the MLC less frequently (once or never) and failed and some of these students, i.e. those who had failed with an E grade, could have passed with more regular visits. Mac An Bhaird et al. (2009) considered data for the MLC at Maynooth University and found significant differences in the pass rates between First Arts students taking mathematics who had attended the MLC more than once (89%) and those who had attended once or never (62%). To account for prior mathematical knowledge, the study grouped the students by the grade (A, B, C and D) achieved and level (Higher or Ordinary) undertaken for the Leaving Certificate (LC), the terminal examination taken at the end of secondary education in Ireland. When the authors considered the First Arts students, the mean mark in every subgroup was higher if they had attended the MLC more than once. For First Science students, the earlier results were not statistically significant but when diagnostic test scores were used to categorize at-risk students, there was a small but significant difference in pass rates as well as mean final mark between those who attended the MLC more than once and those who attended once or never. A subsequent study for the same MLC by Berry et al. (2015) further validated the advantage to at-risk students who used the MLC in two academic years 2011/2012 and 2012/2013. This report used multiple regression analysis to identify the predictors of final mark among First Science students using their LC points (note: a certain number of ‘points’ are awarded by the Higher Education Institutes (HEIs) for the grade achieved in a student’s best six subjects in the LC, with entry to higher education courses contingent on achieving a specified number of points), diagnostic test scores, number of visits to the MLC and the total duration spent in the MLC. The study found the LC points, diagnostic test scores and the total duration to be predictors for final mark in 2011/2012 and LC points, diagnostic test scores and total visits to be predictors in 2012/2013. To gain further insight, the paper also conducted a two-step cluster analysis from which it was found that the subgroup that performed least well in both years was the one which demonstrated low engagement with the MLC and had weak mathematical background. Lee et al. (2008) also considered linear regression for predicting exam performance among engineering students. The significant predictors were found to be diagnostic test scores, whether a student had visited the MLC or not and how many statistics modules the students had taken at A-level. They did argue, however, that not all diagnostic test scores may be good predictors, as these vary widely in design, and that the model would not be directly transferrable to other diagnostic tests, but that, nonetheless, their diagnostic test scores were an efficient and reliable way of identifying at-risk students. All of these papers recognize that accrediting student performance to engagement with mathematics support is non-trivial and other factors, which can be all but impossible to measure, are also predictors of student success. However, in a longitudinal study, with a large sample size, it is possible to investigate whether the contribution of mathematics support may be significant, as described in the next section. 3. Methodology To add to and enhance the existing body of work, data for first year students who enrolled in first year Mathematics service modules from 2004/2005 to 2015/2016, inclusive, were collected. This data set included over 10,000 students who undertook one of the following modules, which will be identified by the labels Modules A–E in this paper: IT Mathematics (Module A) Mathematics for Scientists (Module B) Calculus (Module C) Mathematics for Economics and Business (Module D) Accounting Mathematics 1 (Module E) In addition, information on academic performance in these modules i.e. students’ examination mark, final mark and examination result (pass, fail, etc.) and their prior mathematics background i.e. LC grade in Mathematics, the level at which they studied Mathematics (Higher or Ordinary) and their overall LC points were collected. The data also provided information on their course choice, department and faculty. From the MLC, data were collected on student attendance, the module for which they sought assistance, how many refresher and revision classes students attended (where relevant) and their diagnostic test scores (where available). These sources of information were collated to create the data set that has been used in this report. The data was inputted into Statistical Package for the Social Sciences (SPSS) and this was used for all the statistical tests undertaken. Initially, some generic analysis of how the student profile has changed was conducted. This included comparing the number of Higher Level (HL) with Ordinary Level (OL) students, mean final mark, attendance at the MLC, etc. Subsequently some initial correlation tests were conducted to assess if there was any relationship between attendance at the MLC and the final mark as a way to validate applying regression analysis, which was the last step. Binary logistic regression was applied to the data set to investigate if and how much the MLC has an impact on pass rates. The independent variables that were included in the regression were diagnostic score, LC points, total visits at the MLC, LC grade and final examination outcome. Diagnostic test scores were categorized in bands of 10 percentage points, but the first four categories were merged into two due to the low number of students in these bands, meaning that they were grouped 0–19, 20–39, 40–49, 50–59 and so on. The total number of visits to the MLC were grouped as 0, 1, 2–5, 6–10, 11–14 and 15 or greater, for reasons that will be explored in the next section. For the LC grade, students were categorized into Higher (H) or Ordinary (O) level, and then each level was split into A, B, C or D, representing the grade category achieved by the student. (In fact, each grade is subdivided, with A split into A1 and A2 and B, C and D split in three e.g. B1, B2, B3 and so on, but these grades were merged to avoid an excessive number of categories with a low number of entries.) In addition, HA and HB were combined, as were OC and OD, as the numbers in the extreme categories were too low for accurate analysis otherwise. Those not fitting into these categories were deemed to be missing data points. While over 10,000 students enrolled in at least one of the above modules, there are data missing for various reasons. For example, every year a certain portion of students do not take the diagnostic test as it is not mandatory. From the central data set, there were some students who did not have a LC grade, LC points or level at which Mathematics was taken at LC (e.g. international students or mature students) and these will be treated as missing data and left out of the analysis where necessary. Every effort has been made to ensure that the maximum number of data points is used in the following analysis. 4. Results and discussion 4.1 Student background 2004/2005–2015/2016 The data included information on 10,504 students over 12 years, from the academic years 2004/2005 to 2015/2016. Descriptive statistics on student intake is presented below. Figure 1 shows the total number of students enrolled in a first-year service mathematics module in DCU each year. Fig. 1. View largeDownload slide Number of students registered in first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Fig. 1. View largeDownload slide Number of students registered in first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). This was further categorized into the individual modules involved, which is presented in Fig. 2. The fluctuation in module numbers from year to year is dependent upon the intake into certain programmes as well as the movement of programmes from one module to another if some redesign of the programme had taken place. Fig. 2. View largeDownload slide Number of students registered in each of the first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Fig. 2. View largeDownload slide Number of students registered in each of the first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Categorizing students by the grade they achieved in LC Mathematics and the level at which they took the examination, there are clear changes in profile. For example, the number of students achieving an HD and HC has increased since the academic year 2012/2013 (labelled 2013 in Fig. 3). This is a response that has been observed nationwide and is likely to be due to an initiative whereby 25 bonus points have been awarded by all HEIs to all students who pass HL Mathematics since 2013. Equivalently, the number of OA and OB students dropped as these are the cohort of students most likely to be incentivized by the bonus points to take the HL paper instead of OL. Fig. 3. View largeDownload slide Profile of LC Mathematics levels and grades for incoming service mathematics students from 2004/2005–2015/2016, where H represents HL and O represents OL. Fig. 3. View largeDownload slide Profile of LC Mathematics levels and grades for incoming service mathematics students from 2004/2005–2015/2016, where H represents HL and O represents OL. In line with this, there have been changes in the absolute number of students enrolling in DCU first-year service mathematics modules who have studied mathematics at Higher or Ordinary levels, as shown in Fig. 4. Before 2013/2014, OL students outnumbered HL students but in the last three years, this trend has reversed. There was also a large increase in the HL students in 2012/2013 which marked the first year of the bonus points. This profile is again representative of the national picture. Fig. 4. View largeDownload slide Number of incoming service mathematics students who took HL or OL mathematics for LC from 2004/2005–2015/2016. Fig. 4. View largeDownload slide Number of incoming service mathematics students who took HL or OL mathematics for LC from 2004/2005–2015/2016. All first-year service mathematics students in DCU are also given a mathematics diagnostic test during Orientation Week, the results of which are shown in Fig. 5. Fig. 5. View largeDownload slide Percentage of incoming service mathematics students who achieved certain diagnostic test marks during Orientation Week from 2004/2005–2015/2016. Note that no data is available for 2013/2014, as the diagnostic test did not take place that year. Fig. 5. View largeDownload slide Percentage of incoming service mathematics students who achieved certain diagnostic test marks during Orientation Week from 2004/2005–2015/2016. Note that no data is available for 2013/2014, as the diagnostic test did not take place that year. It should be noted that there was no diagnostic test in 2013/2014, and also that the marking scheme changed in 2007/2008 to introduce negative marking. This was done with the aim of providing a more accurate assessment of student knowledge, as the multiple-choice test without penalties which was used prior to 2007/2008 may have artificially inflated some students’ marks, which is backed up by the fact that the number of students in the 80–100% band dramatically reduced once negative marking was brought in. 4.2 Student examination performance 2004/2005–2015/2016 To also gain a better understanding of student examination performance, the changes in pass rates over this period were also considered, as shown in Fig. 6. This figure excludes students who did not sit their final examinations due to illness or other reasons, roughly 2% on average. Fig. 6. View largeDownload slide Pass and fail rates for first-year service mathematics modules from 2004/2005–2015-2016. Fig. 6. View largeDownload slide Pass and fail rates for first-year service mathematics modules from 2004/2005–2015-2016. Fig. 7. View largeDownload slide Pass rates by first-year service mathematics module from 2004/2005–2015-2016. Fig. 7. View largeDownload slide Pass rates by first-year service mathematics module from 2004/2005–2015-2016. If these pass rates are broken down by module, as shown in Fig. 7, a variable picture emerges, depending on the module in question, with Module C appearing particularly changeable, but it must be borne in mind that in many years, the student numbers in this module were almost 10 times smaller than in the largest module considered (Module D), so such fluctuations are to be expected with smaller sample sizes. 4.3 Student engagement with mathematics support 2004/2005–2015/2016 Attendance at the MLC from first-year service mathematics students fluctuated vastly during the 12 years analysed here, with numbers ranging from around 500–1,500 visits in a year. These variations were due to a number of factors, ranging from raw student numbers in any given year to the types and number of assessments in certain modules (as peaks in attendance are usually noticed immediately prior to in-class tests, particularly in larger modules). In order to analyse the potential impact of mathematics support upon individual students, it was necessary to categorize students by the number of visits they made to mathematics support. As mentioned in the previous section, in order to address our second research question as to whether or not a single visit to mathematics support can have an impact upon student examination performance, students who did not engage with mathematics support in any way were categorized differently from those who visited once, as shown in Fig. 8. Fig. 8. View largeDownload slide Percentage of first-year service mathematics students who made 0, 1, 2–5, 6–10, 11–14 and 15+ visits to the MLC in any given year. Fig. 8. View largeDownload slide Percentage of first-year service mathematics students who made 0, 1, 2–5, 6–10, 11–14 and 15+ visits to the MLC in any given year. Fig. 9. View largeDownload slide Number of students in each LC grade category who made a set number of visits to mathematics support. Fig. 9. View largeDownload slide Number of students in each LC grade category who made a set number of visits to mathematics support. This does not, however, convey much about the prior mathematical attainment of students who made these visits, and so the numbers at each LC grade level who attended the MLC were calculated and are shown in Fig. 9. This is a valuable way of looking at this data as it allows us to account for both engagement with the MLC and prior knowledge, as well as quantifying the number of students of each background that are regularly attending for additional support. From this, it can be seen that the largest numbers of students attending are all from OL cohorts, with OB by far the highest across the visit categories. Another important angle is to look at attendees as a percentage of the total number of students at each LC grade level, as shown in Fig. 10. From this, it can be seen that the spread of attendance by LC grade is as might be expected, with the highest percentage of non-attendees from the HA+HB bracket and the lowest percentage from the OC+OD bracket. This trend then reverses for a single visit to the MLC and continues almost uniformly through the greater number of visits, with the highest percentage of attendees from the OC+OD cohort (as a percentage of all students in the OC+OD grouping) and the lowest from HA+HB. Fig. 10. View largeDownload slide Percentage of students in each LC grade category (expressed as a percentage of all students in that LC grade category) who made a set number of visits to mathematics support. Fig. 10. View largeDownload slide Percentage of students in each LC grade category (expressed as a percentage of all students in that LC grade category) who made a set number of visits to mathematics support. 4.4 Statistical analysis of results To establish if there was a relationship between final mark and attendance at the MLC, the sample Pearson correlation coefficients between the two for each LC grade was done. Significant correlation coefficients were found for grades between HD and OC+OD, and these are presented in Table 1. Table 1 Significant correlation coefficients when final mark was correlated with attendance at the MLC LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  †Significant at the 0.05 level. Others are significant at the 0.01 level View Large Table 1 Significant correlation coefficients when final mark was correlated with attendance at the MLC LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  †Significant at the 0.05 level. Others are significant at the 0.01 level View Large In the former analyses, prior mathematical attainment was taken into account by grouping the students into their LC grades and levels. However, this type of model would consider a student who achieved a final mark of 87% to be fundamentally different in their attainment level from a student who received 88% or 86% which is, for the purpose of this study, an excessively fine-grained distinction. Thus, in the logistic regression described below, final marks will simply be categorized into a pass or a fail, allowing the use of binomial logistic regression, while also providing us with valuable information about the potential of mathematics support to impact upon student success in passing mathematical modules. The binomial logistic regression model used is defined as follows: let y$$_{i}$$ be an indicator of whether student i passed their module such that y$$_{i}$$ = 1 if passed and y$$_{i}$$ = 0 if not passed. The model is then defined in terms of the log-odds of student i passing the model as follows:   \begin{equation*}{\ln \left(\frac{P(Y_i=1)}{P(Y_i=0)}\right) =X_i\beta,}\end{equation*}where $$\boldsymbol{X}_{i}$$ denotes row i of a matrix whose columns contain the responses for the dependent variables which are multiplied by a vector of model parameters denoted by $$\beta$$. The dependent variable in the model was whether or not student i passed their module. Since this model presents the log-odds as a linear model as show in the equation, as is usual practise we report, in Table 2, exp($$\beta$$), the exponential of the estimated parameters (odds ratios) which represent the multiplicative effect of the independent variables on the odds of passing a module. Several possible regression models were considered with the following independent variables: diagnostic test scores, total visits to the MLC (in categories), LC mathematics grade, LC points and mathematics module taken. It had been feared that there would be a high correlation between LC grade and the diagnostic test scores but when Variance Inflation Factors were calculated for all independent variables, these were all lower than two, where only scores above 10 would be a general cause for concern in terms of multicollinearity. As a result, the full model was used for this analysis, and it was found that every category in each independent variable was significant. This model is a significantly better fit to the data than the null model (chi-square = 1975.335, df = 23, p <0.001), with 79.9% correct classification (compared with 70% for the null model). The Hosmer and Lemeshow test for the goodness of fit also suggests that the model is a good fit to the data as p= 0.535 (> 0.05) (chi-square = 7.014, df = 8). Nagelkerke’s R$$^{2}$$ suggests that the model explains roughly 42.4% of the variation in the outcome, which is acceptably good. Table 2 shows how much higher the odds of passing a module are for a student who engages a set number of times with mathematics support than for one who does not, as well as how much higher these odds are for students with a certain prior mathematical background over those with an OC or OD. Without a doubt, prior mathematical knowledge showed the largest impact by far in terms of the likelihood of a student not failing their module, as might have been expected. The odds of passing a module were roughly 3.5 times higher for a student with an OB compared with somebody with an OC or less with all else being equal. However, engagement with mathematics support also emerged as a significant influence upon examination outcomes. The odds of a student who only attended the MLC once passing their module were 1.63 times higher than for one who had never attended the MLC, assuming all other factors kept constant. The odds for a student who engaged frequently (15+ visits) passing were almost 14 times higher. The impact of the LC points was minimal overall, while the module in question and the diagnostic test scores did have a significant impact, as may have been expected. The fact that LC points did not emerge as a major predictor is not a surprise in that these are based on a student’s best six subjects at LC, which may not include mathematics at all. The diagnostic test is nowhere near as clear a predictor as the LC grade, but previous research (Cleary, 2007) has shown that a second, later test tends to be more closely correlated with eventual performance, as those who engage with study and support after an initial low score often end up doing well overall. 5. Conclusions We set out in this research paper to consider two main research questions. The first of these involved controlling for factors such as prior mathematical background and module studied and investigating the potential impact of usage of the mathematics support service upon student success in final examinations. The results of the binomial logistic regression on 12 years of data from the MLC in DCU shows that even a single visit to the MLC can have a positive impact upon a student’s chance of passing their first-year service mathematics module, with all other factors being equal. Over the course of those 12 years, we have seen fluctuations in attendance patterns from various modules, largely due to factors outside the control of mathematics support such as student numbers, prior mathematical background, changes in assessment patterns and so on. As a result, taking a longitudinal approach to such data provides us with more robust findings to reinforce the value of mathematics support in higher education. Our second research question considered whether it may be appropriate to consider even a single visit as engaging with mathematics support, given the varying levels of support that may be optimal for different students. As a result, for our data set, unlike previous research in this area, we considered students who had made a single visit to mathematics support as being different from those who had not attended at all, and our statistical analysis backed up this decision. We originally explored this as a result of anecdotal evidence of students attending with one particular area of difficulty and devoting a couple of hours in the centre to overcoming this but not subsequently returning. It would appear that for some such students, this one visit was still of significant help to them, an important point to note for those involved in quantitative analysis of mathematics support data. It would be interesting to see if this result can be replicated in other centres around Ireland and UK as it is likely to have an effect upon the future reporting of the impact of mathematics support upon student retention and pass rates in mathematics modules. Table 2 Table showing how the odds of a student passing their module are higher if they have (a) attended the MLC [total visits] times than a student who has never attended the MLC , assuming LC grade and module studied are kept constant and (b) if they have received [LC Grade] in their terminal mathematics examination in school than a student who received an OC or OD, assuming number of visits to the MLC and module studied are kept constant   Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001    Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001  View Large Table 2 Table showing how the odds of a student passing their module are higher if they have (a) attended the MLC [total visits] times than a student who has never attended the MLC , assuming LC grade and module studied are kept constant and (b) if they have received [LC Grade] in their terminal mathematics examination in school than a student who received an OC or OD, assuming number of visits to the MLC and module studied are kept constant   Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001    Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001  View Large Acknowledgements The authors would like to acknowledge the particularly helpful comments of Referee 2 and their contribution to the explanation of the binomial logistic regression model which has added greatly to the paper. Maria Jacob is currently enrolled in the Mathematics of Planet Earth Centre for Doctoral Training with University of Reading and Imperial College London. She is studying how to make forecasting maxima in peak energy demands efficiently and thus her research interests lie in Extreme Value Theory. She is motivated by applications of Mathematics in real world, particularly those that can help prepare for changing climate and society. She undertook the work involved in this paper while doing an undergraduate internship in Dublin City University. Dr Eabhnat Ní Fhloinn is the Director of the Mathematics Learning Centre and lecturer in the School of Mathematical Sciences at DCU. She holds a Ph.D. in Mathematics from Trinity College Dublin. She is a committee member of the Irish Mathematics Learning Support Network (IMLSN), and her research interests are in the areas of transition from second level to higher education Mathematics, Mathematics Support and Engineering Mathematics Education. References Berry, E., Mac An Bhaird, C. & O’shea, A. ( 2015) Investigating relationships between the usage of Mathematics Learning Support and performance of at-risk students. Teach. Math. Its Appl. , 34, 194– 204. Cleary, J. ( 2007) Diagnostic testing - an evaluation 1998– 2007. Proceedings of Second National Conference on Research in Mathematics Education (MEI2) , 14th–15th September 2007. Drumcondra, Dublin: St. Patrick's College, pp. 216– 228. Dowling, D. & Nolan, B. ( 2006) Measuring the effectiveness of a maths learning support centre - the DCU experience. Proceedings of CETL-MSOR Conference 2006, 11th - 12th September 2006. The Maths, Stats and OR Network: Birmingham, pp. 51-54. http://www.mathcentre.ac.uk/resources/uploaded/cetl-msor2006conference-proceedingsweb.pdf. Lee, S., Harrison, M. C., Pell, G. & Robinson, C. L. ( 2008) Predicting performance of first year engineering students and the importance of assessment tools therein. Engineering Education , 3, 44– 51. Google Scholar CrossRef Search ADS   Mac An Bhaird, C., Morgan, T. & O’shea, A. ( 2009) The impact of the mathematics support centre on the grades of first year students at the National University of Ireland Maynooth. Teach. Math. Its Appl. , 28, 117– 122. Matthews, J., Croft, T., Lawson, D. & Waller, D. ( 2013) Evaluation of mathematics support centres: a review of the literature. Teach. Math. Its Appl. , 32, 173– 190. Patel, C. & Little, J. ( 2006) Measuring maths study support. Teach. Math. Its Appl., 25, 131– 138. Pell, G. & Croft, T. ( 2008) Mathematics support—support for all? Teach. Math. Its Appl., 27, 167– 173. © The Author(s) 2018. Published by Oxford University Press on behalf of The Institute of Mathematics and its Applications. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) For permissions, please e-mail: journals. permissions@oup.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Teaching Mathematics and Its Applications: International Journal of the IMA Oxford University Press

# A quantitative, longitudinal analysis of the impact of mathematics support in an Irish university

, Volume Advance Article – May 11, 2018
14 pages

/lp/ou_press/a-quantitative-longitudinal-analysis-of-the-impact-of-mathematics-gEEr0NUanf
Publisher
Institute of Mathematics and its Applications
ISSN
0268-3679
eISSN
1471-6976
D.O.I.
10.1093/teamat/hry012
Publisher site
See Article on Publisher Site

### Abstract

Abstract Mathematics support has become embedded in a large number of higher education institutes in Ireland and UK in particular. Measuring the impact of such support is a challenging task, which can be attempted either qualitatively, through the use of surveys or focus groups, or quantitatively, looking at data such as number of visits and subsequent exam performance. Here, we consider a quantitative analysis of the impact of a mathematics support centre in an Irish university, based on data gathered over the past 12 years. A binary logistic regression was carried out which showed that, when prior mathematical achievement and module studied were kept constant, the odds of a student who attended mathematics support once passing their module were 1.63 times higher than for one who had never engaged with the service. The odds for those who attended 15 or more times were almost 14 times higher. This study also showed that there was a significant difference between those who never engaged with mathematics support and those who attended once, meaning that such cohorts should be considered differently, which has not traditionally been done in other studies conducted to this point. 1. Introduction Measuring the impact of mathematics support is a difficult process, given that it should, by its nature, support the traditional lecture-tutorial format for students who need additional help. Isolating the impact of this support from students’ learning experiences in lectures, tutorials, lab classes, online homework and so on is challenging. However, as mathematics support becomes embedded in higher education over a longer period of time, longitudinal data and larger sample sizes mean that a more reliable picture of this impact can be investigated from a quantitative point of view. The Mathematics Learning Centre (MLC) in Dublin City University (DCU) was set up in February 2004 with the aim of providing additional assistance to all undergraduate students undertaking any mathematics module as part of their degree. In 2006, some early indications of the effectiveness of the MLC were reported by Dowling & Nolan (2006). This work presented evidence that the MLC had a positive impact on student retention and that the pass rates of First Science students (who can go on to specialize in programmes such as Biotechnology, Environmental Science and Health, Analytical Science or Chemical and Pharmaceutical Sciences) and at-risk students were higher for those who had attended the MLC than those who had not. While the findings at the time were not conclusive, they were important in validating the role of and acquiring funding for the MLC, which is in operation to date. In fact, in 2014/2015, the MLC was relocated to the ground floor of the Library, to a location which is more accessible, more comfortable and has a higher seating capacity. Given this recent upgrade and the duration for which the MLC has been in operation, it is an appropriate time to conduct a longitudinal study of the impact of mathematics support within DCU to date. Therefore, in this paper, we will address the following research questions: (1) When controlling for factors such as prior mathematical background and module studied, what impact does usage of mathematics support services have upon student success in final examinations? (2) Given the varying levels of support that may be optimal for different students, is it appropriate to consider even a single visit as engaging with mathematics support? 2. Literature review A number of studies on the effectiveness of mathematics support have been conducted in Ireland and UK, with some relying on qualitative data such as surveys, focus groups and interviews and others taking a more quantitative approach, studying attendance rates and examination performance. A comprehensive review of all of these studies is outside the scope of this paper, but can instead be found in the study by Matthews et al. (2013). Here, we will focus in detail on some of the quantitative studies conducted to date. Patel & Little (2006) looked at pass rates of students who had availed of the MLC service versus those who had not and found that 92% of mathematics-related passes were associated with MLC use in comparison to 88% passes related to non-use of the MLC. Pell & Croft (2008) analysed data from five first-year modules in the Engineering department in Loughborough University. They found that 20% of students in all but one module attended the MLC between 2 and 9 times, whereas 3–8% of students used it 10 times or more. Surprisingly, however, 35% of students who had achieved an A* attended the support more than once and from this the authors concluded that the facility was being used more by students seeking excellence than by less-able students looking to avoid failure. Moreover, it is noteworthy that despite the success rate of students who did attend the MLC more than once, Pell and Croft reported that roughly 10% of students visited the MLC less frequently (once or never) and failed and some of these students, i.e. those who had failed with an E grade, could have passed with more regular visits. Mac An Bhaird et al. (2009) considered data for the MLC at Maynooth University and found significant differences in the pass rates between First Arts students taking mathematics who had attended the MLC more than once (89%) and those who had attended once or never (62%). To account for prior mathematical knowledge, the study grouped the students by the grade (A, B, C and D) achieved and level (Higher or Ordinary) undertaken for the Leaving Certificate (LC), the terminal examination taken at the end of secondary education in Ireland. When the authors considered the First Arts students, the mean mark in every subgroup was higher if they had attended the MLC more than once. For First Science students, the earlier results were not statistically significant but when diagnostic test scores were used to categorize at-risk students, there was a small but significant difference in pass rates as well as mean final mark between those who attended the MLC more than once and those who attended once or never. A subsequent study for the same MLC by Berry et al. (2015) further validated the advantage to at-risk students who used the MLC in two academic years 2011/2012 and 2012/2013. This report used multiple regression analysis to identify the predictors of final mark among First Science students using their LC points (note: a certain number of ‘points’ are awarded by the Higher Education Institutes (HEIs) for the grade achieved in a student’s best six subjects in the LC, with entry to higher education courses contingent on achieving a specified number of points), diagnostic test scores, number of visits to the MLC and the total duration spent in the MLC. The study found the LC points, diagnostic test scores and the total duration to be predictors for final mark in 2011/2012 and LC points, diagnostic test scores and total visits to be predictors in 2012/2013. To gain further insight, the paper also conducted a two-step cluster analysis from which it was found that the subgroup that performed least well in both years was the one which demonstrated low engagement with the MLC and had weak mathematical background. Lee et al. (2008) also considered linear regression for predicting exam performance among engineering students. The significant predictors were found to be diagnostic test scores, whether a student had visited the MLC or not and how many statistics modules the students had taken at A-level. They did argue, however, that not all diagnostic test scores may be good predictors, as these vary widely in design, and that the model would not be directly transferrable to other diagnostic tests, but that, nonetheless, their diagnostic test scores were an efficient and reliable way of identifying at-risk students. All of these papers recognize that accrediting student performance to engagement with mathematics support is non-trivial and other factors, which can be all but impossible to measure, are also predictors of student success. However, in a longitudinal study, with a large sample size, it is possible to investigate whether the contribution of mathematics support may be significant, as described in the next section. 3. Methodology To add to and enhance the existing body of work, data for first year students who enrolled in first year Mathematics service modules from 2004/2005 to 2015/2016, inclusive, were collected. This data set included over 10,000 students who undertook one of the following modules, which will be identified by the labels Modules A–E in this paper: IT Mathematics (Module A) Mathematics for Scientists (Module B) Calculus (Module C) Mathematics for Economics and Business (Module D) Accounting Mathematics 1 (Module E) In addition, information on academic performance in these modules i.e. students’ examination mark, final mark and examination result (pass, fail, etc.) and their prior mathematics background i.e. LC grade in Mathematics, the level at which they studied Mathematics (Higher or Ordinary) and their overall LC points were collected. The data also provided information on their course choice, department and faculty. From the MLC, data were collected on student attendance, the module for which they sought assistance, how many refresher and revision classes students attended (where relevant) and their diagnostic test scores (where available). These sources of information were collated to create the data set that has been used in this report. The data was inputted into Statistical Package for the Social Sciences (SPSS) and this was used for all the statistical tests undertaken. Initially, some generic analysis of how the student profile has changed was conducted. This included comparing the number of Higher Level (HL) with Ordinary Level (OL) students, mean final mark, attendance at the MLC, etc. Subsequently some initial correlation tests were conducted to assess if there was any relationship between attendance at the MLC and the final mark as a way to validate applying regression analysis, which was the last step. Binary logistic regression was applied to the data set to investigate if and how much the MLC has an impact on pass rates. The independent variables that were included in the regression were diagnostic score, LC points, total visits at the MLC, LC grade and final examination outcome. Diagnostic test scores were categorized in bands of 10 percentage points, but the first four categories were merged into two due to the low number of students in these bands, meaning that they were grouped 0–19, 20–39, 40–49, 50–59 and so on. The total number of visits to the MLC were grouped as 0, 1, 2–5, 6–10, 11–14 and 15 or greater, for reasons that will be explored in the next section. For the LC grade, students were categorized into Higher (H) or Ordinary (O) level, and then each level was split into A, B, C or D, representing the grade category achieved by the student. (In fact, each grade is subdivided, with A split into A1 and A2 and B, C and D split in three e.g. B1, B2, B3 and so on, but these grades were merged to avoid an excessive number of categories with a low number of entries.) In addition, HA and HB were combined, as were OC and OD, as the numbers in the extreme categories were too low for accurate analysis otherwise. Those not fitting into these categories were deemed to be missing data points. While over 10,000 students enrolled in at least one of the above modules, there are data missing for various reasons. For example, every year a certain portion of students do not take the diagnostic test as it is not mandatory. From the central data set, there were some students who did not have a LC grade, LC points or level at which Mathematics was taken at LC (e.g. international students or mature students) and these will be treated as missing data and left out of the analysis where necessary. Every effort has been made to ensure that the maximum number of data points is used in the following analysis. 4. Results and discussion 4.1 Student background 2004/2005–2015/2016 The data included information on 10,504 students over 12 years, from the academic years 2004/2005 to 2015/2016. Descriptive statistics on student intake is presented below. Figure 1 shows the total number of students enrolled in a first-year service mathematics module in DCU each year. Fig. 1. View largeDownload slide Number of students registered in first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Fig. 1. View largeDownload slide Number of students registered in first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). This was further categorized into the individual modules involved, which is presented in Fig. 2. The fluctuation in module numbers from year to year is dependent upon the intake into certain programmes as well as the movement of programmes from one module to another if some redesign of the programme had taken place. Fig. 2. View largeDownload slide Number of students registered in each of the first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Fig. 2. View largeDownload slide Number of students registered in each of the first-year service mathematics modules from the academic years 2004/2005 (shown as 2005) to 2015/2016 (shown as 2016). Categorizing students by the grade they achieved in LC Mathematics and the level at which they took the examination, there are clear changes in profile. For example, the number of students achieving an HD and HC has increased since the academic year 2012/2013 (labelled 2013 in Fig. 3). This is a response that has been observed nationwide and is likely to be due to an initiative whereby 25 bonus points have been awarded by all HEIs to all students who pass HL Mathematics since 2013. Equivalently, the number of OA and OB students dropped as these are the cohort of students most likely to be incentivized by the bonus points to take the HL paper instead of OL. Fig. 3. View largeDownload slide Profile of LC Mathematics levels and grades for incoming service mathematics students from 2004/2005–2015/2016, where H represents HL and O represents OL. Fig. 3. View largeDownload slide Profile of LC Mathematics levels and grades for incoming service mathematics students from 2004/2005–2015/2016, where H represents HL and O represents OL. In line with this, there have been changes in the absolute number of students enrolling in DCU first-year service mathematics modules who have studied mathematics at Higher or Ordinary levels, as shown in Fig. 4. Before 2013/2014, OL students outnumbered HL students but in the last three years, this trend has reversed. There was also a large increase in the HL students in 2012/2013 which marked the first year of the bonus points. This profile is again representative of the national picture. Fig. 4. View largeDownload slide Number of incoming service mathematics students who took HL or OL mathematics for LC from 2004/2005–2015/2016. Fig. 4. View largeDownload slide Number of incoming service mathematics students who took HL or OL mathematics for LC from 2004/2005–2015/2016. All first-year service mathematics students in DCU are also given a mathematics diagnostic test during Orientation Week, the results of which are shown in Fig. 5. Fig. 5. View largeDownload slide Percentage of incoming service mathematics students who achieved certain diagnostic test marks during Orientation Week from 2004/2005–2015/2016. Note that no data is available for 2013/2014, as the diagnostic test did not take place that year. Fig. 5. View largeDownload slide Percentage of incoming service mathematics students who achieved certain diagnostic test marks during Orientation Week from 2004/2005–2015/2016. Note that no data is available for 2013/2014, as the diagnostic test did not take place that year. It should be noted that there was no diagnostic test in 2013/2014, and also that the marking scheme changed in 2007/2008 to introduce negative marking. This was done with the aim of providing a more accurate assessment of student knowledge, as the multiple-choice test without penalties which was used prior to 2007/2008 may have artificially inflated some students’ marks, which is backed up by the fact that the number of students in the 80–100% band dramatically reduced once negative marking was brought in. 4.2 Student examination performance 2004/2005–2015/2016 To also gain a better understanding of student examination performance, the changes in pass rates over this period were also considered, as shown in Fig. 6. This figure excludes students who did not sit their final examinations due to illness or other reasons, roughly 2% on average. Fig. 6. View largeDownload slide Pass and fail rates for first-year service mathematics modules from 2004/2005–2015-2016. Fig. 6. View largeDownload slide Pass and fail rates for first-year service mathematics modules from 2004/2005–2015-2016. Fig. 7. View largeDownload slide Pass rates by first-year service mathematics module from 2004/2005–2015-2016. Fig. 7. View largeDownload slide Pass rates by first-year service mathematics module from 2004/2005–2015-2016. If these pass rates are broken down by module, as shown in Fig. 7, a variable picture emerges, depending on the module in question, with Module C appearing particularly changeable, but it must be borne in mind that in many years, the student numbers in this module were almost 10 times smaller than in the largest module considered (Module D), so such fluctuations are to be expected with smaller sample sizes. 4.3 Student engagement with mathematics support 2004/2005–2015/2016 Attendance at the MLC from first-year service mathematics students fluctuated vastly during the 12 years analysed here, with numbers ranging from around 500–1,500 visits in a year. These variations were due to a number of factors, ranging from raw student numbers in any given year to the types and number of assessments in certain modules (as peaks in attendance are usually noticed immediately prior to in-class tests, particularly in larger modules). In order to analyse the potential impact of mathematics support upon individual students, it was necessary to categorize students by the number of visits they made to mathematics support. As mentioned in the previous section, in order to address our second research question as to whether or not a single visit to mathematics support can have an impact upon student examination performance, students who did not engage with mathematics support in any way were categorized differently from those who visited once, as shown in Fig. 8. Fig. 8. View largeDownload slide Percentage of first-year service mathematics students who made 0, 1, 2–5, 6–10, 11–14 and 15+ visits to the MLC in any given year. Fig. 8. View largeDownload slide Percentage of first-year service mathematics students who made 0, 1, 2–5, 6–10, 11–14 and 15+ visits to the MLC in any given year. Fig. 9. View largeDownload slide Number of students in each LC grade category who made a set number of visits to mathematics support. Fig. 9. View largeDownload slide Number of students in each LC grade category who made a set number of visits to mathematics support. This does not, however, convey much about the prior mathematical attainment of students who made these visits, and so the numbers at each LC grade level who attended the MLC were calculated and are shown in Fig. 9. This is a valuable way of looking at this data as it allows us to account for both engagement with the MLC and prior knowledge, as well as quantifying the number of students of each background that are regularly attending for additional support. From this, it can be seen that the largest numbers of students attending are all from OL cohorts, with OB by far the highest across the visit categories. Another important angle is to look at attendees as a percentage of the total number of students at each LC grade level, as shown in Fig. 10. From this, it can be seen that the spread of attendance by LC grade is as might be expected, with the highest percentage of non-attendees from the HA+HB bracket and the lowest percentage from the OC+OD bracket. This trend then reverses for a single visit to the MLC and continues almost uniformly through the greater number of visits, with the highest percentage of attendees from the OC+OD cohort (as a percentage of all students in the OC+OD grouping) and the lowest from HA+HB. Fig. 10. View largeDownload slide Percentage of students in each LC grade category (expressed as a percentage of all students in that LC grade category) who made a set number of visits to mathematics support. Fig. 10. View largeDownload slide Percentage of students in each LC grade category (expressed as a percentage of all students in that LC grade category) who made a set number of visits to mathematics support. 4.4 Statistical analysis of results To establish if there was a relationship between final mark and attendance at the MLC, the sample Pearson correlation coefficients between the two for each LC grade was done. Significant correlation coefficients were found for grades between HD and OC+OD, and these are presented in Table 1. Table 1 Significant correlation coefficients when final mark was correlated with attendance at the MLC LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  †Significant at the 0.05 level. Others are significant at the 0.01 level View Large Table 1 Significant correlation coefficients when final mark was correlated with attendance at the MLC LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  LC grade  HD  OA  OB  OC+OD  Pearson correlation  0.069†  0.078  0.169  0.244  †Significant at the 0.05 level. Others are significant at the 0.01 level View Large In the former analyses, prior mathematical attainment was taken into account by grouping the students into their LC grades and levels. However, this type of model would consider a student who achieved a final mark of 87% to be fundamentally different in their attainment level from a student who received 88% or 86% which is, for the purpose of this study, an excessively fine-grained distinction. Thus, in the logistic regression described below, final marks will simply be categorized into a pass or a fail, allowing the use of binomial logistic regression, while also providing us with valuable information about the potential of mathematics support to impact upon student success in passing mathematical modules. The binomial logistic regression model used is defined as follows: let y$$_{i}$$ be an indicator of whether student i passed their module such that y$$_{i}$$ = 1 if passed and y$$_{i}$$ = 0 if not passed. The model is then defined in terms of the log-odds of student i passing the model as follows:   \begin{equation*}{\ln \left(\frac{P(Y_i=1)}{P(Y_i=0)}\right) =X_i\beta,}\end{equation*}where $$\boldsymbol{X}_{i}$$ denotes row i of a matrix whose columns contain the responses for the dependent variables which are multiplied by a vector of model parameters denoted by $$\beta$$. The dependent variable in the model was whether or not student i passed their module. Since this model presents the log-odds as a linear model as show in the equation, as is usual practise we report, in Table 2, exp($$\beta$$), the exponential of the estimated parameters (odds ratios) which represent the multiplicative effect of the independent variables on the odds of passing a module. Several possible regression models were considered with the following independent variables: diagnostic test scores, total visits to the MLC (in categories), LC mathematics grade, LC points and mathematics module taken. It had been feared that there would be a high correlation between LC grade and the diagnostic test scores but when Variance Inflation Factors were calculated for all independent variables, these were all lower than two, where only scores above 10 would be a general cause for concern in terms of multicollinearity. As a result, the full model was used for this analysis, and it was found that every category in each independent variable was significant. This model is a significantly better fit to the data than the null model (chi-square = 1975.335, df = 23, p <0.001), with 79.9% correct classification (compared with 70% for the null model). The Hosmer and Lemeshow test for the goodness of fit also suggests that the model is a good fit to the data as p= 0.535 (> 0.05) (chi-square = 7.014, df = 8). Nagelkerke’s R$$^{2}$$ suggests that the model explains roughly 42.4% of the variation in the outcome, which is acceptably good. Table 2 shows how much higher the odds of passing a module are for a student who engages a set number of times with mathematics support than for one who does not, as well as how much higher these odds are for students with a certain prior mathematical background over those with an OC or OD. Without a doubt, prior mathematical knowledge showed the largest impact by far in terms of the likelihood of a student not failing their module, as might have been expected. The odds of passing a module were roughly 3.5 times higher for a student with an OB compared with somebody with an OC or less with all else being equal. However, engagement with mathematics support also emerged as a significant influence upon examination outcomes. The odds of a student who only attended the MLC once passing their module were 1.63 times higher than for one who had never attended the MLC, assuming all other factors kept constant. The odds for a student who engaged frequently (15+ visits) passing were almost 14 times higher. The impact of the LC points was minimal overall, while the module in question and the diagnostic test scores did have a significant impact, as may have been expected. The fact that LC points did not emerge as a major predictor is not a surprise in that these are based on a student’s best six subjects at LC, which may not include mathematics at all. The diagnostic test is nowhere near as clear a predictor as the LC grade, but previous research (Cleary, 2007) has shown that a second, later test tends to be more closely correlated with eventual performance, as those who engage with study and support after an initial low score often end up doing well overall. 5. Conclusions We set out in this research paper to consider two main research questions. The first of these involved controlling for factors such as prior mathematical background and module studied and investigating the potential impact of usage of the mathematics support service upon student success in final examinations. The results of the binomial logistic regression on 12 years of data from the MLC in DCU shows that even a single visit to the MLC can have a positive impact upon a student’s chance of passing their first-year service mathematics module, with all other factors being equal. Over the course of those 12 years, we have seen fluctuations in attendance patterns from various modules, largely due to factors outside the control of mathematics support such as student numbers, prior mathematical background, changes in assessment patterns and so on. As a result, taking a longitudinal approach to such data provides us with more robust findings to reinforce the value of mathematics support in higher education. Our second research question considered whether it may be appropriate to consider even a single visit as engaging with mathematics support, given the varying levels of support that may be optimal for different students. As a result, for our data set, unlike previous research in this area, we considered students who had made a single visit to mathematics support as being different from those who had not attended at all, and our statistical analysis backed up this decision. We originally explored this as a result of anecdotal evidence of students attending with one particular area of difficulty and devoting a couple of hours in the centre to overcoming this but not subsequently returning. It would appear that for some such students, this one visit was still of significant help to them, an important point to note for those involved in quantitative analysis of mathematics support data. It would be interesting to see if this result can be replicated in other centres around Ireland and UK as it is likely to have an effect upon the future reporting of the impact of mathematics support upon student retention and pass rates in mathematics modules. Table 2 Table showing how the odds of a student passing their module are higher if they have (a) attended the MLC [total visits] times than a student who has never attended the MLC , assuming LC grade and module studied are kept constant and (b) if they have received [LC Grade] in their terminal mathematics examination in school than a student who received an OC or OD, assuming number of visits to the MLC and module studied are kept constant   Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001    Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001  View Large Table 2 Table showing how the odds of a student passing their module are higher if they have (a) attended the MLC [total visits] times than a student who has never attended the MLC , assuming LC grade and module studied are kept constant and (b) if they have received [LC Grade] in their terminal mathematics examination in school than a student who received an OC or OD, assuming number of visits to the MLC and module studied are kept constant   Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001    Exp(B)  95% confidence interval for Exp(B)  p-value      Lower  Upper    Total visits          1  1.632  1.348  1.976  <0.001  2–5  2.410  1.977  2.938  <0.001  6–10  4.301  2.962  6.246  <0.001  11–14  6.350  3.178  12.690  <0.001  15+  13.778  6.279  30.231  <0.001  LC Grade          HA+HB  164.012  96.798  277.896  <0.001  HC  34.250  24.629  47.630  <0.001  HD  12.265  8.984  16.744  <0.001  OA  11.077  8.408  14.592  <0.001  OB  3.501  2.788  4.397  <0.001  CAO Points            1.001  1.001  1.002  <0.001  Diagnostic Scores          [20,30)  1.636  1.174  2.279  0.004  [30,40)  1.317  0.961  1.805  0.086  [40,50)  1.149  0.847  1.558  0.371  [50,60)  1.604  1.179  2.182  0.003  [60,70)  1.912  1.399  2.614  <0.001  [70,80)  1.777  1.261  2.504  0.001  [80,90)  3.359  2.281  4.947  <0.001  [90,100]  3.212  2.000  5.159  <0.001  Module          Module A  0.710  0.525  0.960  0.026  Module B  0.732  0.548  0.978  0.035  Module C  0.096  0.068  0.136  <0.001  Module D  2.128  1.589  2.849  <0.001  View Large Acknowledgements The authors would like to acknowledge the particularly helpful comments of Referee 2 and their contribution to the explanation of the binomial logistic regression model which has added greatly to the paper. Maria Jacob is currently enrolled in the Mathematics of Planet Earth Centre for Doctoral Training with University of Reading and Imperial College London. She is studying how to make forecasting maxima in peak energy demands efficiently and thus her research interests lie in Extreme Value Theory. She is motivated by applications of Mathematics in real world, particularly those that can help prepare for changing climate and society. She undertook the work involved in this paper while doing an undergraduate internship in Dublin City University. Dr Eabhnat Ní Fhloinn is the Director of the Mathematics Learning Centre and lecturer in the School of Mathematical Sciences at DCU. She holds a Ph.D. in Mathematics from Trinity College Dublin. She is a committee member of the Irish Mathematics Learning Support Network (IMLSN), and her research interests are in the areas of transition from second level to higher education Mathematics, Mathematics Support and Engineering Mathematics Education. References Berry, E., Mac An Bhaird, C. & O’shea, A. ( 2015) Investigating relationships between the usage of Mathematics Learning Support and performance of at-risk students. Teach. Math. Its Appl. , 34, 194– 204. Cleary, J. ( 2007) Diagnostic testing - an evaluation 1998– 2007. Proceedings of Second National Conference on Research in Mathematics Education (MEI2) , 14th–15th September 2007. Drumcondra, Dublin: St. Patrick's College, pp. 216– 228. Dowling, D. & Nolan, B. ( 2006) Measuring the effectiveness of a maths learning support centre - the DCU experience. Proceedings of CETL-MSOR Conference 2006, 11th - 12th September 2006. The Maths, Stats and OR Network: Birmingham, pp. 51-54. http://www.mathcentre.ac.uk/resources/uploaded/cetl-msor2006conference-proceedingsweb.pdf. Lee, S., Harrison, M. C., Pell, G. & Robinson, C. L. ( 2008) Predicting performance of first year engineering students and the importance of assessment tools therein. Engineering Education , 3, 44– 51. Google Scholar CrossRef Search ADS   Mac An Bhaird, C., Morgan, T. & O’shea, A. ( 2009) The impact of the mathematics support centre on the grades of first year students at the National University of Ireland Maynooth. Teach. Math. Its Appl. , 28, 117– 122. Matthews, J., Croft, T., Lawson, D. & Waller, D. ( 2013) Evaluation of mathematics support centres: a review of the literature. Teach. Math. Its Appl. , 32, 173– 190. Patel, C. & Little, J. ( 2006) Measuring maths study support. Teach. Math. Its Appl., 25, 131– 138. Pell, G. & Croft, T. ( 2008) Mathematics support—support for all? Teach. Math. Its Appl., 27, 167– 173. © The Author(s) 2018. Published by Oxford University Press on behalf of The Institute of Mathematics and its Applications. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) For permissions, please e-mail: journals. permissions@oup.com

### Journal

Teaching Mathematics and Its Applications: International Journal of the IMAOxford University Press

Published: May 11, 2018

## You’re reading a free preview. Subscribe to read the entire article.

### DeepDyve is your personal research library

It’s your single place to instantly
that matters to you.

over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month ### Explore the DeepDyve Library ### Search Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly ### Organize Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place. ### Access Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals. ### Your journals are on DeepDyve Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more. All the latest content is available, no embargo periods. DeepDyve ### Freelancer DeepDyve ### Pro Price FREE$49/month
\$360/year

Save searches from
PubMed

Create lists to

Export lists, citations