ARTICLE DOI: 10.1057/s41599-018-0116-6 OPEN Buildup of speaking skills in an online learning community: a network-analytic exploration 1 1 2 3 1 Rasoul Shaﬁpour , Raiyan Abdul Baten , Md Kamrul Hasan , Gourab Ghoshal , Gonzalo Mateos & Mohammed Ehsan Hoque ABSTRACT Studies in learning communities have consistently found evidence that peer- interactions contribute to students’ performance outcomes. A particularly important com- petence in the modern context is the ability to communicate ideas effectively. One metric of this is speaking, which is an important skill in professional and casual settings. In this study, we explore peer-interaction effects in online networks on speaking skill development. In particular, we present an evidence for gradual buildup of skills in a small-group setting that has not been reported in the literature. Evaluating the development of such skills requires studying objective evidence, for which purpose, we introduce a novel dataset of six online communities consisting of 158 participants focusing on improving their speaking skills. They video-record speeches for 5 prompts in 10 days and exchange comments and performance- ratings with their peers. We ask (i) whether the participants’ ratings are affected by their interaction patterns with peers, and (ii) whether there is any gradual buildup of speaking skills in the communities towards homogeneity. To analyze the data, we employ tools from the emerging ﬁeld of Graph Signal Processing (GSP). GSP enjoys a distinction from Social Net- work Analysis in that the latter is concerned primarily with the connection structures of graphs, while the former studies signals on top of graphs. We study the performance ratings of the participants as graph signals atop underlying interaction topologies. Total variation analysis of the graph signals show that the participants’ rating differences decrease with time (slope = −0.04, p < 0.01), while average ratings increase (slope = 0.07, p < 0.05)—thereby gradually building up the ratings towards community-wide homogeneity. We provide evi- dence for peer-inﬂuence through a prediction formulation. Our consensus-based prediction model outperforms baseline network-agnostic regression models by about 23% in predicting performance ratings. This in turn shows that participants’ ratings are affected by their peers’ ratings and the associated interaction patterns, corroborating previous ﬁndings. Then, we formulate a consensus-based diffusion model that captures these observations of peer- inﬂuence from our analyses. We anticipate that this study will open up future avenues for a broader exploration of peer-inﬂuenced skill development mechanisms, and potentially help design innovative interventions in small-groups to maximize peer-effects. 1 2 Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA. Department of Computer Science, University of Rochester, Rochester, NY, USA. Department of Physics and Astronomy, University of Rochester, Rochester, NY, USA. These authors contributed equally: Rasoul Shaﬁpour, Raiyan Abdul Baten. Correspondence and requests for materials should be addressed to R.S. (email: rshaﬁpo@ur.rochester.edu) PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms 1 | | | 1234567890():,; ARTICLE PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 Introduction he idea that learners actively construct knowledge as they Participant B, and Participant B can also learn from Participant build mental frameworks to make sense of their environ- A’s public speaking approach, good practices, and mistakes by Tments has been widely studied under the cognitivism and watching the video. Moreover, the meta-cognitive task of constructivism theories of learning (Jones, 2015; Harasim, 2017). explaining something through feedback can also clarify Partici- For example, in a research on cognitive psychology, subjects were pant B’s own understanding (De Backer et al., 2015; Roscoe and shown the sentence, The window is not closed. Later, most of them Chi, 2008). In line with the collaborative model of peer-induced recalled the sentence as The window is open (Cross, 1998), indi- learning, one might expect these interactions to help the parti- cating that people build a mental image or a cognitive map, a cipants gradually build mental models of the good practices of schema, in the learning process, and later build new knowledge in speaking (Davidson and Major, 2014). Knowledge regarding connection to what they already know (Mayes and De Freitas, speaking skills has a tacit nature, which in turn can be developed 2013; Harasim, 2017). through observation, imitation, and practice (Busch, 2008; Chugh Peer interaction is established as an important way people et al., 2013). Furthermore, knowledge is a non-rival good and can learn (Topping et al., 2017; Johnson and Johnson, 2009). In the be traded without decreasing the level possessed by each trader. process of collaboration, learning occurs as individuals build and Therefore, such mutually beneﬁcial interactions can be antici- improve their mental models through discussion and information pated to lead to a gradual accumulation of speaking skills in the sharing with their peers in small groups (Davidson and Major, communities—as measured by the performance ratings—even- 2014; Leidner and Jarvenpaa, 1995). Research on learning com- tually driving the communities towards skill homogeneity as time munities has investigated how social and academic networks goes on. Whether or not such gradual buildup of skills happens in contribute to students’ learning experiences and outcomes a small group setting has not been reported in learning com- (Brouwer et al., 2017; Smith, 2015; Smith et al., 2009; Celant, munity literature, and that is a gap we address in this study. The 2013). In particular, study-related, task-dependent academic temporal nature of our dataset facilitates such exploration. support relationships have been shown to lead to signiﬁcantly Interactions in a learning platform are well suited to be mod- higher academic performance (Gašević et al., 2013; Frank et al., eled as a complex network, where the learners are denoted as 2008). In soft-skill development, Fidalgo-Blanco et al., (2015) nodes and their interactions as edges. This naturally leads to the explored peer-interaction effects in developing teamwork com- use of Social Network Analysis (SNA) tools to gain insights on petency in small groups, and found a direct relation between how various network parameters affect networked learning. For quantities of interaction and individual performance. example, network position parameters (closeness and degree In this paper, we explore the scope of peer-induced speaking centrality, prestige etc.) have been shown to inﬂuence ﬁnal grades skill development, namely, the inﬂuence of peer interactions in (Putnik et al., 2016; Cho et al., 2007) and cognitive learning small groups towards improving one’s speaking competencies. outcomes (Russo and Koesten, 2005) of the learners. However, The ability to communicate ideas and thoughts effectively is SNA is concerned with the structure of relational ties between a valued to be an important skill throughout professional and group of actors (Carolan 2013), and not with any intrinsic casual interaction settings. Given this, can individuals improve measures of ‘performance’ or ‘merits’ of those actors. For exam- their speaking skills by interacting with peers in small groups? If ple, if a person has a lot of followers on Twitter, the in-degree so, can such effects be extracted from data objectively, and then measure will be high, and one would say the person has a high in- correctly modeled? To the best of our knowledge, there have been degree centrality (in lay terms, enjoys a prominent position in the limited studies conducted in the past in this particular context. network) (Newman, 2010). This intuition of ‘importance’ is Part of the problem is in identifying or indeed generating relevant extracted from the connection topology, not from any intrinsic datasets. In particular, the monitoring and evaluation of peer measure of how good the person’s tweets actually are. Therefore, effects objectively require data with quantiﬁable evidence. How- in a learning setting, researchers usually apply correlation analysis ever, keeping a structured history of peer interactions had proved and other statistical methods to investigate the relations between difﬁcult in traditional ofﬂine studies, leading to the use of ques- SNA measures and performance/learning outcomes (Dado and tionnaires and self-reported data (Lomi et al., 2011; Huitsing Bodemer, 2017). Recent developments in the emerging ﬁeld of et al., 2012). Graph Signal Processing (GSP) provide a novel way to integrate With the recent advent of online communication, it is now such ‘performance’ measures (signals) with network graphs commonplace to track detailed speciﬁcs of the participants’ (Shuman et al., 2013). The idea is to attach a signal value to every interactions from their online footprint, such as from Massive node in the graph, and process the signals atop the underlying Open Online Courses’ (MOOC) discussion forums (Tawﬁk et al., graph structure. For example, Huang et al., (2016) used GSP to 2017; Brinton et al., 2014). Online learning communities and study brain imaging data, where the brain structure is modeled as other computer supported collaborative learning platforms thus a graph and the brain activities as signals. Deri and Moura (2016) facilitate objective studies of learner interaction and associated exploited GSP to study vehicle trajectories over road networks. outcomes (Jones, 2015;O’Malley, 2012; Dado and Bodemer, 2017; Other recent application domains include Neuroscience (Rui Russo and Koesten, 2005; Palonen and Hakkarainen, 2013). et al., 2017), imaging (Pang and Cheung, 2016; Thanou et al., Consequently, our ﬁrst contribution in this paper is in con- 2016), medical imaging (Kotzagiannidis and Dragotti, 2016), to structing a novel dataset of six online communities, where the name a few. In a learning setting, these developments in GSP participants focus on developing their speaking skills. The 158 open up the opportunity to model learners’ academic grades or participants in the six communities record video speeches in similar outcome measures as graph signals on top of a peer response to common job interview prompts, and subsequently interaction network. Consequently, our second contribution in exchange feedback comments and performance ratings with peers this paper is in mining evidences of peer-inﬂuence and buildup of in their respective communities. They respond to 5 prompts speaking skills using novel Graph Signal Processing based tech- across 10 days, thereby generating a unique temporal dataset of niques. More formally, we ask (1) whether the learners’ ratings comment-based interactions and speaking performance ratings. are affected by their interaction patterns and the ratings of the In the dataset, the participants leave a comment only after they peers they interact with, and (2) whether there is any gradual have watched a video of a peer. In such a pairwise interaction, buildup of speaking skills in the communities towards Participant A can potentially learn from the feedback given by homogeneity. 2 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms | | | PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 ARTICLE Towards answering the aforementioned research questions, we and stored in the platform. The study was conducted upon take two approaches. First, we measure the smoothness of the approval from the authors’ University IRB. Note, that in what is graph signals (speaking ratings) as the participants interact to follow, we use the terms groups and communities temporally across 5 prompts. We show that with time, people’s interchangeably. ratings come closer to the ratings of the peers they interact with and the ratings also increase on average. This suggests that the Automated feedback. For groups 2, 4, 6, the system generated communities in the dataset gradually approach homogeneity in automatic feedback on smile intensity, body movement/gesture, terms of the performance ratings of the interacting participants. loudness, pitch, unique word count, word cloud and instances of Second, we approach the mining of peer-inﬂuence as a prediction weak language use, as shown in Fig. 1. Groups 1, 3, 5 did not formulation. Let us introduce the idea through a stylized scenario. receive any automated feedback. In a classroom setting, a teacher can track the test scores of a student and make a prediction of his/her future score by running Feedback comments and ratings. In all six groups, the participants a linear regression through previous scores, providing a baseline exchanged feedback with their peers in their respective groups. trend. Instead, if the teacher also takes into account the scores of They were required to give feedback to at least three peers in each the student’s peers (those that the student interacts with), we prompt, whom they could choose at will from a feed of all their demonstrate that the prediction of future scores for both the peers’ videos. Each feedback comprised of (1) at least three student and the peers outperform the baseline. More formally, a comments and (2) performance ratings on a 1–5 Likert scale. network-consensus prediction model outperforms a network- The participants were not given any explicit instruction on agnostic model, a trend we demonstrate to be true in all six what aspects of speaking skills to give comments on. However, communities in our generated data. This corroborates previous the user interface allowed them to tag the comments whether they ﬁndings that interaction quantities and peers’ grades/performance were on friendliness, volume modulation or use of gestures. The outcomes impact learners’ own performance outcomes (Fidalgo- comments were mostly focused on speech delivery. Here, we do Blanco et al., 2015; Hoxby, 2000), and we show that they hold for not concern ourselves with an analysis of the speciﬁc contents of speaking skill development as well. the comments, instead refer to each comment as an ‘interaction’ We further proceed to model and simulate the peer-effects unit. observed in the dataset. From an economic lens, various models When uploading a video, participants could select 5 qualities for knowledge, information or opinion ﬂow across a network has from a list of 23 qualities on which they wanted performance been explored in previous literature (Golub and Sadler, 2017; ratings from their peers. For example, the video in Fig. 1 asks for Cowan and Jonard, 2004; Gale and Kariv, 2003). We introduce a ratings on the qualities of Attention Grabbing, Explanation of model of peer-induced knowledge propagation (in terms of Concepts, Credibility, Vocal Emphasis, Appropriate Pausing, as speaking performance ratings) based on consensus protocols for shown in segment D of the ﬁgure. In addition to these customized network synchronization and distributed decision making (Olfati- rating categories, all videos received ‘overall’ delivery perfor- Saber et al., 2007). Olfati-Saber and Murray (2004) studied such mance ratings from the peers, as shown in segment D of Fig. 1. consensus algorithms and presented relevant convergence ana- The participants were at liberty to use their judgments in giving lysis. We modify their protocol to suit the characteristics of the ratings to their peers. For our analysis, we use the average ‘overall’ communities of our dataset. We discuss how the proposed dif- rating each video received from the peers and refer to this average fusion dynamics capture the peer-inﬂuence and the gradual value as ‘peer rating’, ‘performance rating’ or ‘speaking rating’ buildup of performance ratings towards community-wide interchangeably in the sequel, considering it to be an objective homogeneity, as observed in our dataset. abstraction of the participant’s overall performance or skill level In this context, our contributions can be summarized as in delivering the speech. follows: Dataset summary. All of the 158 participants in the 6 groups Constructing a dataset of six online communities that allow completed the study by recording at least one video to all 5 studying longitudinal peer-effects in speaking skill develop- prompts—generating a total of 817 videos. The 6 groups have ment; 25,665 peer-generated ratings in total. Out of them, 5053 are Yielding new evidences for skills gradually building up in ‘overall’ performance ratings that we use in this study. Therefore, learning communities towards homogeneity, corroborating each video received ‘overall’ ratings from 6.19 peers on average. previous ﬁndings of positive impacts of peer-inﬂuence, and The dataset has 14,285 comments in total, with an average of modeling the observations via diffusion dynamics; 17.49 comments per video. Employing novel Graph Signal Processing tools for extracting the insights. Methodology In this section, we detail the methods of mining peer-inﬂuence evidence in the dataset. As argued in the Introduction, we are Dataset interested to see if the interactions impact the development of Data collection. The data comes from a ubiquitous online sys- speaking-related skills in the six groups. To reiterate, we examine tem, ROC Speak (Fung et al., 2015), that gives people semi- (1) whether the learners’ ratings are affected by their interaction automated feedback on public speaking. One hundred and ﬁfty patterns and the ratings of the peers they interact with, and (2) eight participants, aged 18 to 54 years, were hired from Amazon whether there is any gradual buildup of speaking skills in the Mechanical Turk. All of them were native English speakers, and communities towards homogeneity. came from a variety of professional and educational backgrounds. The participants were randomly assigned into 6 groups labeled 1 through 6, which had 26, 31, 26, 30, 22, and 23 participants Performance ratings as graph signals. Consider modeling an respectively. They were assigned a common goal of improving online community’s participants and their interactions through a their speaking skills, towards which they were given 5 common network graph. To that end, we naturally identify each participant job interview prompts in 10 days. Responses to these prompts with a node in the said graph. As discussed in the Introduction, were recorded via webcam as videos (typically 2 min in length) each comment-based interaction can potentially inﬂuence both PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms 3 | | | ARTICLE PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 Fig. 1 A snapshot of the ROC Speak feedback interface. The page shows automatically generated measurements for (a) smile intensity, gestural movements, (b) loudness, pitch, (c) unique word count, word recognition conﬁdence, a transcription of the speech, word cloud, and instances of weak language. Additionally, the peer-generated feedback are shown in (d) a Feedback Summary section, and (e) a top ranked comments section classiﬁed by usefulness and sentiment. In our study, we use the overall ratings given by the peers, as shown in segment (d) the commenter and the receiver. The receiver can beneﬁt from is a vertex-valued network process that can be represented as a the comment itself. On the other hand, the commenter can learn vector of size N supported on the nodes of G, where its ith from watching the peer’s video and from the meta-cognitive task component is the rating of node i. As explained in the Dataset of providing a feedback. Therefore, each interaction is repre- section, the participants are given overall performance ratings by sented via an undirected edge, acknowledging mutual beneﬁt for their peers, and we take a participant’s average overall rating in the two nodes. Exchanges of multiple comments are encoded each prompt as the graph signal value. Thus, we collect the ðpÞ through integer-valued edge weights; more formally, consider an ðpÞ ratings of the pth prompt in a vector r 2 R , where r is ith ðpÞ undirected, weighted graph GðN ; E; W Þ representing this net- participant’s rating. This representation is visualized in Fig. 2. work at pth prompt with a node set N of known cardinality N Under the natural assumption that signal properties are related to (i.e., the total number of participants in the group—each of the graph topology, the goal of Graph Signal Processing is to develop groups in the dataset have their own cardinality), and the edge set models and algorithms that fruitfully leverage this relational E of unordered pairs of elements in N . The so-called symmetric structure, making inferences about these signal values when they N ´ N ðpÞ weighted adjacency matrix is denoted by W 2 R , whose are only partially observed (Shuman et al., 2013). ijth element represents the total number of comments that par- ticipants i and j have given to each other in the pth prompt. Figure 2 illustrates the construction of W using a toy example. Total variation analysis. We explore whether the communities Since the interaction patterns tend to change from prompt to gradually approach homogeneity in performance ratings, by prompt, so does the connectivity pattern (i.e., the topology) of the conducting a total variation analysis. Total variation is a measure resulting graph and hence the explicit dependency of the weights of smoothness of the graph signals (ratings), the idea being, if (p) in W with respect to p. non-rival interactions allow participants to gradually achieve Next, we incorporate the participants’ rating information in the ratings closer to their peers, then ratings will become smoother or form of graph signals. Here, a graph signal (Shuman et al., 2013) more homogeneous across the network. 4 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms | | | PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 ARTICLE Fig. 2 a A network graph representation of an example community with N = 4 participants depicted as red nodes. The edges denote interactions in the form of feedback comments, with the edge weights corresponding to the number of interactions between nodes. As indicated by edge weights, participants 1 and 4 interact 2 times, while all other nonzero pairwise interactions take place only once. The blue bars on top of the nodes represent average speaking ratings as graph signals, and can take any value between 1 and 5. b The Adjacency matrix W captures the number of interactions between participants i and j in its ijth element. Graph signal r captures everyone’s speaking performance ratings. The diagonal degree matrix D has node i’s number of interactions in its iith entry. The Laplacian L is computed by D−W. In our dataset, W, r, D, and L vary across prompts p, and also across the six groups N ´ N ðpÞ Given the network adjacency matrix W 2 R , the degree past performance ratings of the participant, i.e., a model which is (number of links) of participant i at the pth prompt is the total agnostic to the network effects on the participant; and (ii) aug- number of comments exchanged in that prompt, deﬁned as menting the least-squares predictor with a smoothness regulariza- ðpÞ ðpÞ tion term encouraging network-wide consensus. In other words, the d :¼ W . Using these deﬁnitions, we construct the i j ij second model considers the ratings of one’s peers and the quantities (p) (p) Laplacian matrix for any prompt p according to L := D − of interaction along with one’s own records. It is important to note (p) (p) (p) W , where D := diag(W 1 ) is a diagonal matrix with that we do not seek to ﬁt the data to ﬁnd the best possible regression elements corresponding to the nodes’ degree, and 1 are the model for predicting future ratings, rather, we intend to compare vector of ones of length N (Chung, 1997). Figure 2 illustrates the prediction errors between network-agnostic and consensus-based construction of the degree and Laplacian matrices. For a graph regression models to illustrate our point that the network has an signal r, one can utilize the graph Laplacian to compute the total impact on individual outcomes. variation (TV) of the participants’ ratings thus, We use the ﬁrst four prompts in our data for training purposes, X and predict the 5th prompt’s ratings for all the participants in the ðpÞ TVðrÞ :¼ r Lr ¼ W r r ; ð1Þ six groups. Speciﬁcally, we collect the rating signal vectors r 2 ij i j i;j¼1;i>j R for prompts p = 1, 2, 3, 4 to form the training set and predict the ratings at prompt p = 5 using two different linear regression where r and r reﬂect the ratings of participants i and j,while models: i j weights W account for their volume of interaction. The total ij br ¼ β þ β p ð3Þ 0 1 variationinEq. (1) is often referred to as a smoothness measure of the signal r with respect to the graph G.Iftwo nodes i and j do not 2 and interact in a prompt, W is 0 and therefore (r − r ) makes no ij i j pﬃﬃﬃ contribution to the sum in Eq. (1), while increasing interaction leads br ¼ β þ β p þ β p; ð4Þ 0 1 2 to more contributions from the term (r − r ) .If r differs i j signiﬁcantly between pairs of nodes that show strong patterns of where br 2 R is the estimated ratings at pth (in our case, 5th) interaction, then TV(r) is expected to be large. On the other hand, if N prompt. For Eqs. (3) and (4), the parameters β ; β ; β 2 R are 0 1 2 ratings vary negligibly across connected nodes, then the graph signal learned through the following regularized least-squares criteria, is smooth and TV(r) takes on small values. Increasing convergence respectively: of ratings across interacting participants leads to a decreasing total variation. However, TV does not capture the overall trend of the 1 ðpÞ b b fβ ; β g¼ argmin r ðβ þ β pÞ 0 1 0 1 2m ratings—whether the participants’ ratings are improving or not. p¼1 β ;β 0 1 ð5Þ Therefore, to complement the total variation analysis, we calculate ðpÞ the network-wide average ratings at each prompt as þλ ðβ þ β pÞ L ðβ þ β pÞ 0 1 0 1 p¼1 ðpÞ ðpÞ ð2Þ r :¼ r ; and i¼1 pﬃﬃﬃ 2 1 ðpÞ b b b fβ ; β ; β g¼ argmin r β þ β p þ β p ðpÞ 0 1 2 2m 0 1 2 where r is the rating of the ith participant in the pth prompt. β ;β ;β p¼1 0 1 2 pﬃﬃﬃ þλ β þ β p þ β p Prediction under smoothness prior. In the Introduction, we 0 1 2 ð6Þ p¼1 presented a scenario of a teacher tracking the performance rating pﬃﬃﬃ ðpÞ ´ L β þ β p þ β p evolution of students to illustrate our approach of prediction for- 0 1 2 mulation towards shedding light on peer-effects. More formally, we 2 2 þμ β þ β : 1 2 formulate a convex optimization problem to predict future per- formance ratings. We compare prediction errors between two models: (i) an ordinary least-squares regression model using only PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms 5 | | | ARTICLE PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 pﬃﬃﬃ Here,c non-linear p term to capture any saturation effect at Table 1 Fifth prompt prediction errors comparison between limiting time. We have m = 4 since the training is done on the consensus-based (λ ≠ 0) and network-agnostic (λ = 0) ﬁrst four prompts’ data. Let us break down the objective functions frameworks, for regression models with (a) linear features Eqs. (5) and (6) for clarity. The ﬁrst summands in Eqs. (5) and (6) using Eq. (3) and (b) non-linear features using Eq. (4) are data ﬁdelity terms, which take into account each participant’s individual trajectory of learning across the ﬁrst m prompts, and (a) Prediction errors with (b) Prediction errors with ﬁts them to the respective linear regression models. Accordingly, linear features non-linear features the ﬁrst terms can be attributed to a person’s own talent or pace Consensus- Network- Consensus- Network- of learning, agnostic to the network. The second summands in based (λ ≠ 0) agnostic (λ based (λ ≠ 0) agnostic (λ Eqs. (5) and (6) incorporate the effects of the neighbors’ ratings = 0) = 0) and the amount of interaction into a participant’s learning curve. Group 1 9.85% 14.52% 7.83% 8.18% Notice that these terms are nothing else than smoothing Group 2 10.93% 18.67% 10.72% 12.75% regularizers of the form TVðbrÞ, hence encouraging p¼1 Group 3 12.42% 15.62% 9.95% 13.14% participant ratings prediction with small total variation. The Group 4 11.26% 15.4% 11.31% 12.81% tuning parameter λ > 0 balances the trade-off between faithfulness Group 5 12.29% 13.99% 10.02% 12.57% to the past ratings data and the smoothness (in a total variation Group 6 7.04% 11.43% 7.06% 10.47% b b sense) of the predicted graph signals r ¼ β þ β p and 0 1 Bold errors denote better prediction results. The optimization functions for the regression pﬃﬃﬃ b b b models are elaborated through Eqs. (5) and (6) in the Methods section br ¼ β þ β p þ β p, and can be chosen via model selection 0 1 2 techniques such as cross validation (Friedman et al., 2001). Parameter μ which is chosen along with λ via leave-one-out cross model the positive drift ϵ as a Gaussian random variable with a validation, prevents overﬁtting via a shrinkage mechanism which positive mean μ > 0. Superposition of random effects is well ensures that none of β and β get to dominate the objective 1 2 modeled by a Gaussian random variable by virtue of the Central functions disproportionately (Friedman et al., 2001). Equations Limit Theorem. The ratings received by participants lie between (5) and (6) are both convex, speciﬁcally unconstrained quadratic 1–5, so we introduce an explicit control procedure to bound the programs that can be solved efﬁciently (Boyd and Vandenberghe, ratings in our model. Thus the evolution of ratings can be 2004) via off-the-shelf software packages. Here we use CVX modeled via the Laplacian dynamics (Grant et al., 2008), a MATLAB-based modeling system for solving convex optimization problems. An outline of the ðpþ1Þ ðpÞ ðpÞ ðpÞ r ¼P r cL r þ ϵ ; ð7Þ ½ r ;r min max procedure is presented in Algorithm 1. (p) th where r is the ratings vector of p prompt, c ∈ (0, 1/d ) is the max diffusion constant and d is the maximum degree of nodes at max (p) the corresponding prompt, L is the Laplacian matrix of the graph G at the pth prompt, ϵ is a Gaussian random vector with mean μ > 0 and given variance σ , and P ðÞ is a projection ½ r ;r min max Setting λ = 0 in Eqs. (5) and (6) reduces them to ordinary operator onto the interval [r , r ]. For ROC Speak one has min max network-agnostic least-squares regression models. In simple r = 1 and r = 5. min max terms, a participant’s 5th prompt rating is then predicted by Figure 3 demonstrates the overall idea pictorially. To under- running linear regression over previous ratings, which is used as a stand the chosen dynamics, disregard the projection operator in baseline. For λ ≠ 0 chosen by cross-validation, the network-effects Eq. (7) for the sake of a simpler argument. Then notice that the are switched on in Eqs. (5) and (6), and the optimization (p+1) (p) (p) (p) update r = r − cL r represents a Laplacian-based net- procedure determines parameters that minimize the total work diffusion process (Olfati-Saber and Murray 2004), where variation of the participant ratings in combination with least- the future rating of a given participant depends on the ratings of squares regression. Peer effects are then determined by compar- (p) his/her peers in the current prompt (r ) and the nature of the ing the model with λ ≠ 0 to that with λ = 0. Relative prediction (p) interactions taking place in the learning community (L ). ð5Þ ð5Þ b b errors deﬁned as β þ 5β r = r and Focusing on the ith participant recursion in Eq. (7) (modulo the 0 1 pﬃﬃﬃ projection operator), one obtains the scalar update ð5Þ ð5Þ b b b β þ 5β þ 5β r = r respectively for Eqs. (5) and 0 1 2 ðpþ1Þ ðpÞ ðpÞ ðpÞ (6) are reported in Table 1. r ¼ð1 cd Þr þ c W r þ ϵ ; i i i ij j i ð8Þ j2V Modeling and simulation. We proceed to develop a model for ðpÞ the participant ratings’ evolution which facilitates simulation of where r is ith participant rating at the pth prompt. In obtaining the network process. We model the temporal update of the rat- (p) Eq. (8), we have used the deﬁnition of graph Laplacian, i.e., L : (p) ings r as a diffusion process on the graph G, where the trade of P ðpÞ ðpÞ ðpÞ (p) (p) = D − W . The quantity ð1 cd Þr þ c W r is a i i ij j feedback can take place without decreasing the individual level of weighted average of participant i and his/her neighbors’ ratings, knowledge/speaking ratings. We impose a positive drift to the with each neighbor’s weights being proportional to the number of ratings in order to model the fact that a learner can accumulate an interactions that s/he has with participant i. This way, the model understanding of skills (i.e., build mental models of better prac- captures peer-inﬂuenced buildup of ratings across the network tices in speaking) from various sources such as automated feed- where the diffusion constant c is a relatively small number. back, peer feedback, experience of watching peers’ videos, and (p+1) (p) Further intuition can be gained by interpreting r = r − cL other external resources. In a real network, people will have dif- (p) (p) ferent learning rates, their individual talents will differ, the r as a gradient-descent iteration to minimize the total T (p) external inﬂow of information will not be consistent, and in some variation functional TV(r):= r L r in Eq. (1). This suggests that less probable cases, they may forget information as well. To lump Eq. (7) will drive the ratings towards a consensus of minimum all these noisy yet positively skewed effects into one variable, we total variation. 6 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms | | | PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 ARTICLE We test the proposed model in Eq. (7) by running a numerical deviation σ = 0.1 based on the ROC Speak average ratings simulation that synthetically generates participant ratings across behavior, and select c = 0.01. prompts. We synthetically generate graphs with structural properties resembling our dataset. Given that in our platform, Results participants can, in principle, send communication at random, we Convergence of community-wide performance with increasing generate Erdös-Rényi random graphs for each prompt with the interactions. We evaluate the total variation of the rating signals average number of nodes in the communities (N = 26 nodes) (p) r across the ROC Speak network at the end of each prompt [cf. (Erdos and Rényi, 1960). Each edge is included in the graph with (1)]. We collect ith community’s TV of the 5 prompts in a vector probability p = 0.5, so that the expected number of edges p ð1Þ ð2Þ ð5Þ 2 TV = TV ; TV ; ¼ ; TV and plot the normalized total i i i matches the number of edges in the dataset. We set the variation as TV /||TV || over prompts in Fig. 4a, where ||·|| maps a i i positive drift term ϵto have a mean μ = 0.05 and standard vector to its Euclidean norm. The normalized total variations mostly decrease from the third prompt onwards. The dashed plot in Fig. 4a indicates the linear trend of the average across the six groups TV . A diminishing trend is apparent, with a i¼1 i moderate yet signiﬁcant slope of −0.04 (p < 0.01), suggesting that interacting participants gradually converge in terms of ratings. In addition, we plot the network-wide average ratings deﬁned in Eq. (2) as a function of prompts p in Fig. 4b. It is evident that all six groups have a trend of improvement in average ratings across prompts. The dashed plot indicates the average linear trend estimation across all six groups, which has a moderate and signiﬁcant slope of 0.07 (p < 0.05). Taken together, these ﬁndings provide strong evidence that communities gradually approach homogeneity in terms of (improved) performance with increasing interactions among participants. Improved prediction of ratings as a function of incorporating peer effects. As explained in the Methods section, we compare Fig. 3 An example demonstrating how participant 1’s rating in the (p + 1)th the predictions against the original ground-truth ratings of the ðpþ1Þ prompt, r , is calculated from the participant’s own rating in the pth ﬁnal prompt to report the prediction errors. The results are ðpÞ prompt, r ¼ 4, single interactions with participants 2, 3 and 4, and a summarized in Table 1. The ﬁrst and third columns of the table th positive drift ϵ . The p prompt ratings for the participants are shown by 1 show the prediction errors obtained by our proposed consensus- blue bars. Here, participants 2 and 4 both have ratings of 3 in the pth based predictors in Eqs. (5) and (6). The second and fourth prompt, and participant 3 has a rating of 5. The net diffusion effect with columns refer to the baseline predictors of ordinary least-squares peers takes participant 1’s rating from 4 down to 3.99 (denoted by a yellow regression, which are computed by making the smoothness reg- bar). However, a Gaussian random positive drift ϵ = 0.0676 (denoted by a ularization term λ = 0 in Eqs. (5) and (6). Table 1 shows that the green bar on top of the yellow bar) represents a sample realization of consensus-based predictors (λ ≠ 0) outperform their baseline participant 1's gathering of understanding of speaking skills from sources counterparts (λ = 0) in all six communities. In particular, the external to peer feedback, and pushes his rating in the (p + 1)th prompt to consensus-based approach relatively improves the predictions by ðpþ1Þ r = 3.99 + 0.0676= 4.0576. The magniﬁed plot reﬂects the 23% averaged over six groups. In other words, the ratings of one’s probability distribution of participant 1’s rating at (p + 1)th prompt which is peers and the associated interaction patterns help predict his/her centered around 3.99 + μ, with mean μ. This also shows that the rating in performance ratings better. This, in turn, supports the idea that th (p + 1) prompt can take any value in the interval [r , r ], but with a min max interactions with the neighbors indeed impact one’s learning higher probability it is closer to 3.99 + μ outcomes. 0. 7 4.6 Group 1 Group 3 Group 5 Group 2 Group 4 Group 6 0. 6 4.4 Linear Trend Estimation 0. 5 4.2 0. 4 0. 3 3.8 Group 1 Group 3 Group 5 0. 2 3.6 Group 2 Group 4 Group 6 Linear Trend Estimation 0. 1 3.4 12345 12 3 4 5 Prompt Prompt (a) Total variation of ratings across networks versus prompts. (b) Average ratings of the participant groups versus prompts. Fig. 4 Total variations and average ratings as functions of prompts. In a, the total variation decreases across prompts while b shows the network-wide average ratings increase. The dashed lines are the linear trend estimations across prompts PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms 7 | | | Normalized Total Variation Average Ratings ARTICLE PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 Capturing the dynamics of the interactions. To simulate the explorations in directional peer tutoring when it comes to proposed Laplacian dynamics-based model described in Eq. (7), developing oral expression in learning a second language— ð1Þ (1) we run iterations with r as initialization, where r ’s are drawn namely between two students in school or between a family uniformly at random from the interval Eqs. (1, 5). Figure 5a member and a student at home (Duran et al., 2016; Topping et al., shows two realizations of the total variation Eq. (1) as it evolves 2017). However, in the setting of small learning groups, the over prompts, superimposed to the mean evolution obtained after network-effects in developing one’s speaking skills received neg- averaging 1000 independent such realizations. The speciﬁc rea- ligible attention in literature—a context we presented our insights lizations show ﬂuctuations similar to those observed in Fig. 4a, in. Our novel dataset of six online communities thus allowed us to but the trend is diminishing towards zero meaning that ratings study the peer-effects in an objective manner. converge with temporal evolution. Likewise, Fig. 5b shows two We studied performance ratings as graph signals on top of realizations and the mean evolution of the network-wide average comment-based interaction networks. The emerging ﬁeld of GSP ðpÞ ratings r in Eq. (2) as a function of prompts. has received attention from a variety of ﬁelds such as Neu- The ﬁgures indicate that the simulation results accurately roscience and medical imaging, but GSP was not applied to a capture the positively skewed effects of interactions in the gradual learning setting previously. Taking a GSP-based prediction buildup of speaking skills. While the network model is admittedly approach helped us observe that the learners’ ratings had a simple, the synthesized sample paths qualitatively match the connection with their interaction patterns, as well as the rating trends in the data. This is further illustrated in Fig. 6, where signals of the peers they interacted with. Both parts of this participant-ratings are shown as blue bars demonstrating a observation agree with previous ﬁndings in other scopes of decrease in the variation of ratings as the simulation evolves in learning. For instance, Fidalgo-Blanco et al., (2015) had shown time, with the average ratings approaching the limiting value r . that there is a direct relation between quantities of interaction and max individual performance outcomes. Using Texas Schools Micro- data, it was shown that an exogenous change of 1 point in peers’ Discussion reading scores raises a student’s own score between 0.15 and 0.4 points (Hoxby, 2000). Our prediction formulation outperformed In this paper, we explored how participants’ performance ratings the baseline relatively by 23%, thus corroborating the same progressed as they interacted in online learning communities. In insights in the context of speaking skill development. particular, we asked whether the learners’ ratings were affected by Moreover, evidence for a gradual buildup of skills towards their interaction patterns and their peers’ ratings, and also whe- community-wide homogeneity was not reported in earlier ther the communities gradually approached homogeneity in learning community literature, and that is a gap we addressed. speaking performance ratings. We found positive evidence for both of the questions. Theoretically, this effect has long been anticipated. For instance, Cowan and Jonard (2004) simulated diffusion of knowledge and Developing competence in speaking skills has its importance across professional and personal settings. There have been some suggested its gradual buildup in networks. Following the 0.7 5 (1) A sample realization with TV(r )/||TV|| = 0.5987 Average of 1000 realizations 4.5 (1) 0.5 A sample realization with TV(r )/||TV|| = 0.6873 0.3 3.5 (1) A sample realization with r = 3.31 Average of 1000 realizations 0.1 (1) A sample realization with r = 2.65 2.5 10 20 30 40 50 60 70 10 20 30 40 50 60 70 Prompt Prompt (a) Simulation results of normalized total variation versus prompts. (b) Simulation results of average user ratings versus prompts. Fig. 5 Simulation results of normalized total variations and average ratings as functions of prompts. The normalized total variation across the network decreases in a, while the average ratings saturate to the ceiling in b (p) Fig. 6 A sample visualization of how the participants’ ratings r (blue bars) change as total variation diminishes and average rating saturates at limiting time (i.e., prompt p→∞). The network topology changes in every prompt as the participants randomly interact with each other. A decreasing TV(r) (p) measure is a direct manifestation of the signal r being smoother as p increases 8 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms | | | Normalized Total Variation Average Ratings PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 ARTICLE collaborative model of learning, this buildup is also reasonable, as Centola D (2010) The spread of behavior in an online social network experiment. Science 329(5996):1194–1197 repetitive interactions can be anticipated to help participants Cho H, Gay G, Davidson B, Ingraffea A (2007) Social networks, communication develop their tacit understandings of speaking skills gradually styles, and learning performance in a cscl community. Comput Educ 49 (Davidson and Major, 2014). In our dataset, total variation of the (2):309–329 rating signals going down (slope = −0.04, p < 0.01) and the Chugh R et al. (2013) Workplace dimensions: tacit knowledge sharing in uni- average ratings going up (slope = 0.07, p < 0.05) with time pro- versities. J Adv Manag Sci 1(1):24–28 Chung FK (1997) Spectral graph theory. CBMS regional conference series in vide evidence of skill-ratings building up among interacting mathematics. No. 92 participants. To the best of our knowledge, this is the ﬁrst data- Cowan R, Jonard N (2004) Network structure and the diffusion of knowledge. J driven evidence for this idea. We further proposed a network Econ Dyn Control 28(8):1557–1575 diffusion model and used it to simulate the rating buildup pro- Cross KP (1998) Why learning communities? why now? Campus 3(3):4–11 cess. The model is simple and derived from previous work Dado M, Bodemer D (2017) A review of methodological applications of social network analysis in computer-supported collaborative learning. Educ Res Rev (Olfati-Saber and Murray, 2004), but we modiﬁed the formula- 22:159–180 tion therein to capture the peer-learning effects as observed in our Davidson N, Major CH (2014) Boundary crossings: cooperative learning, colla- analysis of the data. borative learning, and problem-based learning. J Excellence College Teaching, It is important to stress that while our dataset is concerned 25(3, 4), 7–56 with speaking skill development and therefore our insights are De Backer L, Van Keer H, Valcke M (2015) Exploring evolutions in reciprocal peer tutoring groups’ socially shared metacognitive regulation and identifying its presented in that context, the analysis methods are not limited to metacognitive correlates. Learn Instr 38:63–78 speaking-skills only. Any dataset in a learning setting that comes Deri JA, Moura JMF (2016) New york city taxi analysis with graph signal pro- with quantiﬁable learning outcome measures (e.g., grades, scores, cessing. 2016 IEEE Global Conference on Signal and Information Processing ratings), as well as interaction measures (e.g., comments or other (GlobalSIP). pp 1275–1279 synchronous/asynchronous textual discussion, audio, video, or Duran D, Flores M, Oller M, Thomson-Garay L, Vera I (2016) Reading in pairs. Peer tutoring for reading and speaking in English as a foreign language. any suitable Internet-supported media based interaction) can be http://grupsderecerca.uab.cat/grai/en/node/3882 modeled and analyzed using our formulations. Erdos P, Rényi A (1960) On the evolution of random graphs. Publ Math Inst Hung In this paper, we did not take into account the contents of the Acad Sci 5(1):17–60 comments, which is left as a future work. We treated each com- Fidalgo-Blanco Á, Sein-Echaluce ML, García-Peñalvo FJ, Conde MÁ (2015) Using ment as an interaction unit. This still turned out to hold infor- learning analytics to improve teamwork assessment. Comput Human Behav 47:149–156 mation regarding skill-development in the communities, as shown Frank KA, Muller C, Schiller KS, Riegle-Crumb C, Mueller AS, Crosnoe R, Pearson through our mining of the data. Similarly, the exact speciﬁcs of the J (2008) The social dynamics of mathematics coursetaking in high school. speaking skills were not considered in this paper, such as the use Am J Sociol 113(6):1645–1696 of gestures, vocal variety, friendliness, clarity, speaking speed etc. Friedman J, Hastie T, Tibshirani R (2001) The elements of statistical learning, Vol. Rather, we used an average ‘overall’ rating that we assumed to be 1, Springer Series in Statistics. Springer, Berlin Fung M, Jin Y, Zhao R, Hoque ME (2015) Roc speak: Semi-automated personalized an objective abstraction of speaking performance or skills. Digging feedback on nonverbal behavior from recorded videos. Proceedings of the deeper into the comments and connecting them with speciﬁc 2015 ACM International Joint Conference on Pervasive and Ubiquitous attributes of speaking-skills (e.g., a comment on vocal variation Computing. ACM, pp 1167–1178 leading to better vocal performance) in the future might help us Gale D, Kariv S (2003) Bayesian learning in social networks. Games Econ Behav 45 uncover more insights on the peer-inﬂuence characteristics. (2):329–346 Gašević D, Zouaq A, Janzen R (2013) “Choose your classmates, your gpa is at In his study on behavior propagation, Centola (2010) showed stake!” the association of cross-class social ties and academic performance. that health behavior spreads farther and faster across clustered- Am Behav Sci 57(10):1460–1479 lattice networks than across corresponding random networks, Golub B, Sadler ED (2017) Learning in social networks since the former structure facilitates social reinforcement through Grant M, Boyd S, Ye Y (2008) Cvx: Matlab software for disciplined convex redundant network ties. Valente (2012) reviewed various social programming Harasim L (2017) Learning theory and online technologies. Taylor & Francis intervention approaches for accelerating behavior change. Hoxby C (2000) Peer effects in the classroom: Learning from gender and race Exploring the effects of various network structures and inter- variation. Technical report. National Bureau of Economic Research vention strategies to maximize skill development are some of our Huang W, Goldsberry L, Wymbs NF, Grafton ST, Bassett DS, Ribeiro A (2016) future courses of work. We thus anticipate that the current study Graph frequency analysis of brain signals. IEEE J Sel Top Signal Process 10 will open up many avenues for broader exploration of peer- (7):1189–1203 Huitsing G, Veenstra R, Sainio M, Salmivalli C (2012) “It must be me” or “it could inﬂuenced skill development in the future. be them?”: the impact of the social network position of bullies and victims on victims’ adjustment. Soc Netw 34(4):379–386 Johnson DW, Johnson RT (2009) An educational psychology success story: Received: 5 October 2017 Accepted: 16 April 2018 social interdependence theory and cooperative learning. Educ Res 38(5): 365–379 Jones C (2015) Networked learning: an educational paradigm for the age of digital networks. Springer Kotzagiannidis MS, Dragotti PL (2016) The graph fri framework-spline wavelet theory and sampling on circulant graphs. Acoustics, Speech and Signal References Processing (ICASSP), 2016 IEEE International Conference on, IEEE. pp Boyd S, Vandenberghe L (2004) Convex optimization, Cambridge University Press 6375–6379 Brinton CG, Chiang M, Jain S, Lam H, Liu Z, Wong FMF (2014) Learning about Leidner DE, Jarvenpaa SL (1995) The use of information technology to enhance social learning in moocs: From statistical analysis to generative model. IEEE management school education: a theoretical view. MIS Quarterly. pp 265–291 Trans Learn Technol 7(4):346–359 Lomi A, Snijders TA, Steglich CE, Torló VJ (2011) Why are some more peer than Brouwer J, Flache A, Jansen E, Hofman A, Steglich C (2017) Emergent achievement others? evidence from a longitudinal study of social networks and individual segregation in freshmen learning community networks. Higher Education p academic performance. Soc Sci Res 40(6):1506–1520 1–18 Mayes T, De Freitas S (2013) Technology-enhanced learning. Beetham, H. & Busch P (2008) Tacit knowledge in organizational learning. IGI Global Sharpe. pp 17–30 Carolan BV (2013) Social network analysis and education: theory, methods and Newman M (2010) Networks: an introduction. Oxford University Press applications. Sage Publications Olfati-Saber R, Fax JA, Murray RM (2007) Consensus and cooperation in net- Celant S (2013) The analysis of students academic achievement: the evaluation of worked multi-agent systems. Proc IEEE 95:215–233 peer effects through relational links. Qual Quant 47(2):615–631 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms 9 | | | ARTICLE PALGRAVE COMMUNICATIONS | DOI: 10.1057/s41599-018-0116-6 Olfati-Saber R, Murray RM (2004) Consensus problems in networks of agents with Acknowledgements switching topology and time-delays. IEEE Trans Autom Control 49 This work was supported by National Science Foundation Awards IIS-1464162 and CCF- (9):1520–1533 1750428, a Google Faculty Research Award, and Microsoft Azure for Research grant. O’Malley C (2012) Computer Supported Collaborative Learning, Vol. 128. Springer Science & Business Media Author contributions Palonen T, Hakkarainen K (2013) Patterns of interaction in computersupported R.S., R.A.B., M.K.H., G.G., G.M., and M.E.H. designed and performed the research and learning: a social network analysis. Fourth International Conference of the wrote the manuscript. All the ﬁgures are prepared by the authors. Learning Sciences, pp 334–339 Pang J, Cheung G (2016) Graph laplacian regularization for inverse imaging: Reprints and permission information is available online at http://www.nature.com/ analysis in the continuous domain. Preprint at arXiv:1604.07948 reprints Putnik G, Costa E, Alves C, Castro H, Varela L, Shah V (2016) Analysing the correlation between social network analysis measures and performance of Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims students in social network-based engineering education. Int J Technol Des inpublished maps and institutional afﬁliations. Educ 26(3):413–437 Roscoe RD, Chi MT (2008) Tutor learning: The role of explaining and responding to questions. Instr Sci 36(4):321–350 Rui L, Nejati H, Safavi SH, Cheung N-M (2017) Simultaneous low-rank compo- nent and graph estimation for high-dimensional graph signals: application to Additional information brain imaging. Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE Competing interests: The authors declare no competing interests. International Conference on, IEEE. pp 4134–4138 Russo TC, Koesten J (2005) Prestige, centrality, and learning: a social network Reprints and permission information is available online at http://www.nature.com/ analysis of an online class. Commun Educ 54(3):254–261 reprints Shuman DI, Narang SK, Frossard P, Ortega A, Vandergheynst P (2013) The emerging ﬁeld of signal processing on graphs: Extending high-dimensional Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims data analysis to networks and other irregular domains. IEEE Signal Process inpublished maps and institutional afﬁliations. Mag 30(3):83–98 Smith BL, MacGregor J, Matthews R, Gabelnick F (2009) Learning communities: reforming undergraduate education Smith RA (2015) Magnets and seekers: a network perspective on academic inte- Open Access This article is licensed under a Creative Commons gration inside two residential communities. J High Educ 86(6):893–922 Attribution 4.0 International License, which permits use, sharing, Tawﬁk AA, Reeves TD, Stich AE, Gill A, Hong C, McDade J, Pillutla VS, Zhou X, adaptation, distribution and reproduction in any medium or format, as long as you give Giabbanelli PJ (2017) The nature and level of learner-learner interaction in a appropriate credit to the original author(s) and the source, provide a link to the Creative chemistry massive open online course (mooc). J Comput High Educ 29 Commons license, and indicate if changes were made. The images or other third party (3):411–431 material in this article are included in the article’s Creative Commons license, unless Thanou D, Chou PA, Frossard P (2016) Graph-based compression of dynamic 3d indicated otherwise in a credit line to the material. If material is not included in the point cloud sequences. IEEE Trans Image Process 25(4):1765–1778 article’s Creative Commons license and your intended use is not permitted by statutory Topping K, Buchs C, Duran D, Van Keer H (2017) Effective peer learning: from regulation or exceeds the permitted use, you will need to obtain permission directly from principles to practical implementation, Taylor & Francis the copyright holder. To view a copy of this license, visit http://creativecommons.org/ Valente TW (2012) Network interventions. Science 337(6090):49–53 licenses/by/4.0/. Data availability © The Author(s) 2018 The dataset and the analysis codes can be found in: https://github.com/ROC-HCI/ ROCSpeak-Buildup-of-Skills 10 PALGRAVE COMMUNICATIONS (2018) 4:63 DOI: 10.1057/s41599-018-0116-6 www.nature.com/palcomms | | |
Palgrave Communications – Springer Journals
Published: Jun 5, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.
All for just $49/month
Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly
Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.
Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”
Daniel C.
“Whoa! It’s like Spotify but for academic articles.”
@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”
@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”
@JoseServera
DeepDyve Freelancer | DeepDyve Pro | |
---|---|---|
Price | FREE | $49/month |
Save searches from | ||
Create lists to | ||
Export lists, citations | ||
Read DeepDyve articles | Abstract access only | Unlimited access to over |
20 pages / month | ||
PDF Discount | 20% off | |
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.
ok to continue