Access the full text.
Sign up today, get DeepDyve free for 14 days.
Loading next page...
/lp/ou_press/a-new-pan-european-train-the-trainer-programme-for-bioinformatics-IbZME6MdSf
- Publisher
- Oxford University Press
- Copyright
- Copyright © 2022 Oxford University Press
- ISSN
- 1467-5463
- eISSN
- 1477-4054
- DOI
- 10.1093/bib/bbx112
- Publisher site
- See Article on Publisher Site
Abstract
Demand for training life scientists in bioinformatics methods, tools and resources and computational approaches is urgent and growing. To meet this demand, new trainers must be prepared with effective teaching practices for delivering short hands-on training sessions—a specific type of education that is not typically part of professional preparation of life scientists in many countries. A new Train-the-Trainer (TtT) programme was created by adapting existing models, using input from experienced trainers and experts in bioinformatics, and from educational and cognitive sciences. This pro- gramme was piloted across Europe from May 2016 to January 2017. Preparation included drafting the training materials, organizing sessions to pilot them and studying this paradigm for its potential to support the development and delivery of future bioinformatics training by participants. Seven pilot TtT sessions were carried out, and this manuscript describes the results of the pilot year. Lessons learned include (i) support is required for logistics, so that new instructors can focus on their teaching; (ii) institutions must provide incentives to include training opportunities for those who want/need to Allegra Via is a scientific researcher in bioinformatics at the CNR-IBPM (Italy). She is the ELIXIR-IT Training Coordinator, co-leader of the ELIXIR- EXCELERATE Train-the-Trainer subtask and a GOBLET member. Teresa K. Attwood is a Professor of Bioinformatics at the University of Manchester; she has taught introductory bioinformatics for 20 years, and has writ- ten several bioinformatics textbooks and reference works. She is currently ELIXIR-TeSS subtask lead, and GOBLET Chair. Pedro L. Fernandes coordinates Bioinformatics training at the Instituto Gulbenkian de Cie ˆ ncia, where he created and has run the GTPB programme since 1999. He currently chairs the Learning, Education and Training committee in GOBLET, is the ELIXIR Portugal Training Coordinator and leads the ELIXIR Train-the-Researcher subtask. Sarah Morgan is a Training Programme Manager at EMBL-EBI responsible for the external user training programme and trainer development. She is add- itionally the ELIXIR-EBI training coordinator, co-leader of the ELIXIR-EXCELERATE Train-the-Trainer subtask and member of the GOBLET standards committee. Maria Victoria Schneider is the Deputy Director of the EMBL Australia Bioinformatics Resource and an Associate Professor at the University of Melbourne; she also chairs the GOBLET Standards Committee and pioneered the Train the Training in bioinformatics with EMBL-EBI and Bioplatforms Australia (BPA). Patricia M. Palagi is a Team Leader of the SIB Training at the SIB Swiss Institute of Bioinformatics and ELIXIR-CH training coordinator. She is the co-leader of the ELIXIR-EXCELERATE Training Work Package and of the ELIXIR Training platform. She is also a GOBLET member. Gabriella Rustici runs the Bioinformatics Training Programme at the University of Cambridge, UK. She is the Deputy Head of the ELIXIR-UK Node and co- leads the ELIXIR-EXCELERATE training work package and the impact/quality subtask. She is a member of the GOBLET Standards Committee. Rochelle E. Tractenberg is a cognitive scientist and a research methodologist with accreditation as a Professional Statistician from the American Statistical Association. She serves on the GOBLET Learning, Education and Training committee. Submitted: 30 April 2017; Received (in revised form): 26 July 2017 V The Author 2017. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited. 405 Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 406 | Via et al. become new or better instructors; (iii) formal evaluation of the TtT materials is now a priority; (iv) a strategy is needed to recruit, train and certify new instructor trainers (faculty); and (v) future evaluations must assess utility. Additionally, defining a flexible but rigorous and reliable process of TtT ‘certification’ may incentivize participants and will be considered in future. Key words: Train-the-Trainers; pilot study; decision-making; degrees of freedom analysis; ELIXIR-GOBLET training new instructors later to apply a general ‘best’ practice model Introduction [12, 13, 18] to their own future training courses; on its own, Bioinformatics is a highly dynamic and rapidly evolving field topic-specific training will not address the increasing demand that requires some level of understanding of the life sciences for high-level skills that are responsive both to computational and of applied computation. This background must be aug- and technological innovations, and to their concomitant train- mented with an agility to respond to technological and compu- ing [as suggested by 1–3, 4, 7]. Moreover, domain experts are tational advances, by using, adapting or developing new skills. embedded in their own universities or institutes, where training Although there is a recognized global, unmet need for well- may not be their primary role. Offering training to those who do trained bioinformaticians across research domains [1–4], it is wish to become instructors can benefit their careers by exposing difficult to see how formal university programmes can meet them to, and encouraging them to adopt, ‘best’ practices to this need effectively: the field changes so quickly that lengthy improve the quality of their instruction [12–14]. degree-level education may not be sufficient. University educa- Train-the-Trainer (TtT) is a model for establishing and grow- tion is expensive in time and effort for students and instructors ing a pool of individuals who are driven by their interest and en- [4–5]; more importantly, it is too localized to meet the global de- thusiasm to become (better) instructors. A notable large-scale, mand. Clearly, it is neither possible nor desirable for all practis- worldwide TtT programme that has established formal training ing life scientists to complete new degrees; nevertheless, and assessment protocols was developed by Software Carpentry modern biological science requires at least awareness of, or (SWC) (http://software-carpentry.org) in 1998 and cloned for Data familiarity with, basic bioinformatics terminology, tools and Carpentry (DC) (http://www.datacarpentry.org/), its sister organ- resources. These can include databases and algorithms such ization, in 2014; the formal training model was first published in as those used in next-generation sequencing data ana- 2015 [4, 21]. The SWC and DC networks comprise volunteers who lyses, resources collecting macromolecular data (e.g. UniProt) share the mission of delivering high-quality training in software or genomic data (e.g. HapMap), graphical/visualization soft- development (SWC) or computational skills required for data ware (e.g. PyMol), databases of pharmaceutical interest management and analysis (DC). The TtT paradigms of both (e.g. DrugBank) and alignment tools (e.g. BLAST). Carpentries follow a pedagogical theory-based structure (https:// To address the growing demand to build bioinformatics and swcarpentry.github.io/instructor-training/), so that every trainer computational abilities, ‘point-of-need’ training is becoming learns the same approach and method, specifically based on more prevalent, and is likely to remain relevant even if bioinfor- theory and best practices for training. matics skills are incorporated into undergraduate curricula. Another training model was developed by the European Online opportunities have expanded dramatically, but (among Molecular Biology Laboratory-European Bioinformatics Institute many other challenges; [4]) the target skill set often requires (EMBL-EBI) (www.ebi.ac.uk/training/train-trainer), which has both tutored hands-on practice and real-time feedback [6–7], provided bioinformatics training and support to external and making some online training (e.g. asynchronous and/or lecture- internal audiences since 2007, and has been running a TtT pro- based ‘lessons’) less ideal. Locally available training in bioinfor- gramme since 2012. Externally, the programme prepares new matics has therefore gained importance. However, experts in a trainers to deliver a specific training course, which is based on field may themselves require training to competently teach/ an EMBL-EBI model [11]. Participants complete face-to-face train and assess learners [8, 9]; in fact, instructors must be pre- training, attend and observe the course of interest and are then pared with effective practices for delivering ‘point-of-need’, typ- supported to develop, deliver and assess a version of the course ically short-duration (e.g. half a day to 2 days), hands-on that is appropriate to their target audience in their local context. instruction. This is important because not only are ‘training’ Internally, the programme prepares new trainers to deliver the and ‘education’ fundamentally different [10–11], but learning extant EMBL-EBI user training programme. After face-to-face about teaching practices in either has not been a routine part of training, internal (i.e. located at EMBL-EBI) trainers are assigned the professional preparation of life scientists in Europe. We use a mentor, and training opportunities are identified in which the term ‘best’ practices, meaning teaching practices based on they can participate. long-standing ideas from cognitive science about how adults Each of these models has a specific focus: the SWC/DC TtT learn, e.g. [12–14] together with a consensus of experts in the programme was designed to ensure consistent delivery of a domain and expert trainers ([15–17, 18]; see Table 5 for the def- consolidated set of training materials, developed as a commu- initions adopted in this article). nity activity; the EMBL-EBI programme set out to enable exter- Training in ‘best’ practices for effective instruction is vital, nal trainers to deliver courses in a similar manner to EMBL-EBI but instruction and practise are also required to develop an courses [11]. Both models have a network of trainers who have understanding of the design, organization and evaluation of completed the TtT programme, which new instructors can ac- courses and teaching materials [5, 18]. To be truly ‘best’ prac- cess for expertise and support. tices, in both training and education, the delivered learning In 2015, ELIXIR (https://www.elixir-europe.org, https:// should be sustainable, i.e. should endure beyond the end of the f1000research.com/channels/elixir), in the context of the European instruction, and be applicable in other contexts [19, 20]. Union H2020 programme (https://ec.europa.eu/programmes/hori Experience with theory-based foundations is intended to help zon2020/), was granted additional funding, ELIXIR-EXCELERATE Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 Pilot results for abioinformatics Train-the-Trainer programme | 407 (https://www.elixir-europe.org/about-us/how-funded/eu-projects/ life sciences research, and to train scientists in its use.) (1—by excelerate) (EE), one of the objectives of which is to support a pan- phone), and from the domain of educational-psychology-in- European training programme to increase bioinformatics capacity higher-education (4). Representation from GOBLET included its and competency. This programme is not limited to the develop- Chair, and most of the other participants were also members ment of new courses and materials: it also specifically supports the of GOBLET. development of new instructors to increase training capacity and Participants in the TtT pilot sessions came from all over sustainability, through a tailored TtT programme. The EE-TtT pro- Europe, and were identified by their home ELIXIR node or gramme, its background and goals are described in detail in [22]. through institutional affiliations with the ELIXIR nodes in their The formal preparation of new trainers via the EE-TtT aims to give countries. Training sessions included 5–11 participants, and new instructors tools and tips for providing an enriching learning involved 2–3 TtT faculty (who had co-developed the materials experience to trainees, irrespective of topic, and to include best in collaboration with GOBLET) with one exception, where the practice guidance on course and training material development. TtT lead faculty member was the sole instructor. The EE-TtT programme also aims to build a network of in- structors to allow them to benefit from reciprocal support Analyses and discussion. The programme focuses on—but is not limited to—scientists in the ELIXIR community, aiming to promulgate The evaluation of the pilot programme is a series of matrices ‘best’ practices and promote active learning ideas, grounded in based on the degrees of freedom analysis (DoFA) method [25, 26]. educationally relevant, research-based theories of how people The DoFA method facilitates the analysis of qualitative data, learn [12, 13, 18, 23]. including observations, survey results and theory. The method A kick-off meeting was held in Hinxton, UK, in January 2016, was applied to capture evidence from the pilot project about both bringing together ELIXIR training coordinators and global collab- the design and feasibility (or practicality) of the EE-TtT pro- orators [including experts in cognitive and educational gramme, and to study potential metrics for its utility (participant- psychology, and representatives from SWC/DC and the Global perceived usefulness). Feasibility and utility were considered Organisation for Bioinformatics Learning, Education and from the perspectives of the programme (ELIXIR and individual Training (GOBLET; [24])] to discuss the desired features of the nodes) as well as faculty/participants. Finally, as the data were new EE-TtT programme [22]. being collated and this report prepared, a new model of the sus- The programme was further developed during the following tainability (or potential for endurance and/or transfer of learning) 4 months; four core topics were identified as the basis for the was published [20]; hence, a final analysis examined the align- EE-TtT course structure: (1) principles of learning and how they ment of the EE-TtT programme with this model of sustainability apply to training; (2) training techniques for enhancing learner of learning, as this is important for creating a programme that engagement and participation; (3) design of engaging sessions, trains instructors who may then go on to become TtT programme materials and courses; and (4) assessment and feedback in developers themselves. The results are therefore organized to training. These are described in detail in [22]. Seven pilot TtT present the design of the programme (results of the kick-off sessions were then held from May 2016 to January 2017. This meeting), the feasibility and utility of the materials and design, manuscript describes the evolution and features of this TtT and then the sustainability of learning the TtT programme can programme, and feasibility and utility results from the pilot promote. These aspects of the evaluation (feasibility; utility; sus- sessions. It also considers the sustainability of the learning TtT tainability of learning) are described in Table 5, together with de- ‘graduates’ experienced, alongside lessons learned from the scriptions of low, moderate and high levels of each of these pilot project. features. The results of the evaluation are summarized in Table 5. Methods Results Pilot project structure Design The EE-TtT pilot was created to identify and, through training, Table 1 is a DoFA matrix that shows the features of our two ‘qualify’ individuals to reliably and reproducibly train new users ‘model’ training programmes, as provided by SWC/DC and the of bioinformatics and computational biology tools, methods EMBL-EBI. Features (rows) serve as predictions [26] on which we and resources across Europe. The programme also aimed to scored each of the models; scoring was derived from the kick- seed a durable community that would support, nurture and off meeting as follows: scores of 0¼ model does not have this mentor EE-TtT participants going forward. A pilot study was feature; scores of 0.5¼ model has the feature, but it is either not required to assess whether these goals could be accomplished sufficiently explicit or not as completely integrated as the satisfactorily; in the kick-off meeting, it was decided to call TtT experts involved in the EE-TtT programme design intended it to course completers ‘new instructors’; those who were, or be- be; and scores of 1¼ model has the feature and the EE-TtT pro- came, qualified to train new EE-TtT participants (i.e. able to lead gramme can adopt that model’s implementation of it. Details the TtT workshops) were termed ‘TtT faculty’. about the execution of this DoFA analysis are given in Supplemental Table S1. Table 1 shows that not all of the desired features of the TtT participants EE-TtT programme were present in the two model training pro- Participants in the TtT kick-off meeting came from Europe and grammes, which supported the need for the pilot and the design the United States. Fifteen participants were invited from the of the EE-TtT programme itself. Although some features shown ELIXIR Training Coordinators Group (n¼ 10, representing Italy, in Table 1 were adoptable or adaptable from the existing mod- The Netherlands, Portugal, Switzerland, UK and EMBL-EBI), from els, as noted earlier, these two TtT programmes were developed DC (1) and CyVerse (CyVerse (http://www.cyverse.org) mission is for specific purposes, and could not simply be adopted whole to design, deploy and expand a national cyberinfrastructure for cloth. For example, the EE-TtT programme needs to build a Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 408 | Via et al. Table 1. DoFA: predictions of how/whether training models include the desired features of the EE-TtT programme Features desired for the EE-TtT programme: SWC/DC EMBL-EBI TtT Carpentry TtT Training paradigm is focused on theory and is evidence-based/evidence-informed 1 0.5 Explicit developmental trajectories—for the trainers themselves to continue to grow/refine 00 their knowledge, skills and abilities (KSAs) relating to training specifically Training paradigm seeks to build a community of ELIXIR trainers 0 0 Pedagogical and andragogical principles are explicit in training new trainers, so new in- 11 structors will also follow these principles New trainers are introduced to—and encouraged to only use—evidence-based principles of 1 0.5 learning Training paradigm includes formative assessment of the training KSAs that the programme 11 develops in new instructors New trainers are introduced to—and encouraged to use—Bloom’s taxonomy to develop learn- 11 ing outcomes Training paradigm embodies the target KSAs of effective training, course design and learning 00 assessment, together with an explicit developmental trajectory new instructors can con- tinue to build on Training paradigm uses and promotes the use of active learning techniques 1 1 Programme provides instruction on how to integrate technology, including virtual machines 0 0.5 (VMs) and cloud, in training delivery and development Programme involves TtT participants attending actual training courses to observe expert in- 0.5 1 structors in action, and follow-up discussion about observation, evaluation and develop- ment of reflection around their own teaching Programme includes post-TtT support (e.g. forum/blog/network/meetings/discussions), includ- 11 ing support for instructors’ development of their own pre-course assessment (selection) and evaluations Materials are FAIR 1 0.5 network and community of trainers specifically around ELIXIR attending and hosting a TtT session. The full nature of these services and platforms. While both existing models are commit- costs must be understood, and a model for recovering them ted to the creation of communities for the trainers they prepare, devised, if the TtT programme is to be scalable (i.e. feasible those communities are not specific to ELIXIR services, platforms going forward). and resources. Another important feature for the new pro- For scalability of the EE TtT programme, the pilot has un- gramme is that all training materials should be Findable, covered a particularly important feature in terms of support for Accessible, Interoperable and Re-usable (FAIR). That is, the ma- the interchange of EE-TtT faculty across sessions and countries. terials are intended to be made public (findable; accessible). Scalability will only succeed with new faculty—especially in This is true in one, but not the other, model. Finally, it is essen- local nodes—and specifically, the provision of ongoing support tial that the EE-TtT programme is based on theoretically driven (and possibly additional preparation) for those who have ‘best’ practices in educational psychology, pedagogy and andra- completed an EE-TtT workshop and wish to go on to deliver gogy, to yield interoperable and reusable teaching practices that such workshops themselves. As can be seen in Table 2 and can be easily applied for new topics, new instructors and new Table 5, the feasibility of a program must be understood from tools/technology. the perspectives of the funder (EE), the faculty and the attendees. While experience with the pilot EE-TtT programme has Feasibility shown that it can achieve its aim of ‘local capacity’ building, it has also underlined the need to develop a robust, sustainable Materials that implement all the desired characteristics out- cost model that would allow development, and incorporation, lined in Table 1 were developed to support 2 day training of new of new TtT faculty who can share the workload and allow unfet- instructors [22]. These were used by the three faculty members tered roll-out of the programme across Europe. As the EE-TtT who run the TtT programme to provide seven TtT sessions programme goes forward, all of these aspects need to be around Europe. Four key features of feasibility were identified. considered and evaluated. Feasibility going forward must be These are: (1) the financial costs to the organizers (hosts) of assessed from multiple perspectives (e.g. funder, faculty and each TtT session; (2) costs to trainees who attend the session; participants; see Table 5) and on a variety of dimensions of the (3) the number of participants; and the (4) feasibility of append- construct. ing the TtT session to an existing conference, workshop, etc. (i.e. requiring an additional 2 day commitment to an already planned trip by organizers, faculty and trainees). Table 5 de- Utility scribes these features of ‘feasibility’ at high, moderate and low levels. Table 2 presents the feasibility results for each of the We administered the same questionnaire (see Supplementary seven pilot sessions. Materials) to participants at the end of every EE-TtT pilot but As for any event, the TtT programme requires resourcing to one to collect feedback. In Pilot 4, we administered a different cover what can be considerable costs associated with both questionnaire (see Supplementary Materials). The analysis of Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 Pilot results for abioinformatics Train-the-Trainer programme | 409 Table 2. Feasibility results Feasibility factors: Financial feasibility Practical feasibility Costs covered using Costs that were not # of participants Stand-alone TtT course EE-TtT budget covered by the (# of participants from (1); co-organized with EE-TtT budget the hosting node) existing meeting (2) co-organized with other course (3)? Case: Pilot 1 Cambridge—May Travelþhotel costs for Coffee breaks and lunch; 11 (9) 1 2016 one faculty participants’ travel hotel costs Pilot 2 Cambridge—July Travel for one faculty Coffee breaks and lunch 9 (9) 1 Pilot 3 Oeiras—July 2016 Travelþhotel costs for Coffee breaks and lunch; 8 (2) 3 one faculty trainees’ travelþhotel costs Pilot 4 Rome—October Coffee breaks and lunch; Trainees’ travelþhotel 8 (0) 2 2016 travelþhotel costs for costs one faculty Pilot 5 Ljubljana— Travelþhotel costs for Coffee breaks and lunch 8 (4) 3 November 2016 one faculty One facultyþtrainees travelþhotel costs Pilot 6 Lausanne— Coffee breaks; travel Trainees travelþhotel 9 (6) 3 January 2017 hotel costs for one costs faculty Pilot 7 Oeiras—January None Coffee breaks and lunch 10 (7) 1 In general, coffee breaks, lunch and faculty travel costs were supported by the hosting node; they were not covered by the EE-TtT budget; Pilots 4, 6 and 7 were exceptions. responses made it possible to assess the utility, based on user course (3/7 pilots), 10 provided an explicit positive comment on perspectives, of the EE-TtT programme. the importance of sitting in on a training session as observer/ The overall satisfaction was ‘good/excellent’ for every pilot helper. This tension between utility and feasibility needs to be (data not shown). However, to better understand whether partici- considered for the EE-TtT programme going forward. pants perceived actual ‘utility’ from their engagement in this pro- Additionally, the features of perceived ‘utility’ that we ex- gramme, we analysed the comments (given by <100% of tracted from this informal thematic analysis of responses to the participants in each pilot), to try to glean features of ‘perceived’ open-ended course-evaluation questions must be considered utility that could be incorporated into future workshop evalu- for future evaluations: TtT evaluations should solicit actionable ations. We also included the yes/no responses to the question, input about course utility and how to improve courses across all ‘would you recommend the course?’ in this analysis as a ‘sum- respondents. Further, the utility results in Table 3 are an infor- mary indicator’ of perceived utility. This item helps us to interpret mal analysis of what aspects of utility the participants per- whether what we sought as evidence of participant-perceived util- ceived; collecting such valuable input should not depend on ity is plausible—based on the assumption that recommendations participants’ inclination to supply open-ended commentary in a would not be made for a course that has no utility (as opposed to field labelled ‘other comments’. Therefore, adding items to ask any utility). Table 3 therefore presents counts, by pilot session all participants to evaluate the utility of the TtT course is a pri- (rows), for how many of those who gave written additional feed- ority for the scaled-up EE-TtT programme. Whether, or that, back included a comment that is aligned with one or more of the they have been inspired to think differently at the end of a themes (columns) we extracted from an informal analysis of the course is important, but not sufficient, feedback: a further step narrative responses. is required to understand and evaluate true utility—specifically, Considering Tables 2 and 3, we can compare and contrast we need to ascertain that they did something with the new ‘feasibility’ and ‘utility’. Specifically, the logistics for stand-alone knowledge and/or inspiration. Therefore, participant follow-up TtT sessions are harder and more costly than for those that can should also be put in place to ask whether they have done be aligned with existing conferences or concurrent courses. things differently in their post-course teaching and/or ‘Feasibility’ is greatest when the TtT workshop is co-organized assessment. with another meeting that all participants (faculty and new in- structors) have funding to attend, but the greatest ‘utility’ arises when the TtT workshop is co-organized with another course be- Sustainability cause participants have the opportunity to observe an experi- enced trainer teaching, which strengthens the training in many ‘Sustainable learning’ is defined as learning that continues be- important ways. In this regard, of the 19 respondents to evalu- yond the end of formal instruction, and can be described by four ations distributed in TtT workshops co-organized with another distinct features or dimensions that were originally identified in Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 410 | Via et al. Table 3. Perceptions of ‘utility’ gleaned from informal analysis of narrative comments in evaluations by participants across pilot sessions Comment: I learned new I was inspired by Useful for excel- Opportunities Useful practice Useful practice Would you rec- and relevant new ways of lence in future and a new venue and feedback on and feedback on ommend the things about thinking training that I to exchange presentation session/course course? (Y: Yes teaching and offer/provide ideas on teaching skills preparation N: No MB: Maybe) learning and learning with peers and TtT faculty Case : Pilot 1 (n¼9) 1 2341Y:8/9 N: 0 MB: 1/9 Pilot 2 (n ¼11) 2455Y: 10/11 N: 1/11 MB: 0 Pilot 3 (n ¼7) 4 2 Y: 5/7 N: 0 MB: 2/7 Pilot 4 (n ¼7)– –––––Y:5/7 N: 0 MB: 2/7 Pilot 5 (n ¼6) 1 6 1 11Y:6/6 N: 0 MB: 0 Pilot 6 (n ¼6) 3 1 2 1 Y: 5/6 N: 0 MB: 1/6 Pilot 7 (n ¼9)2 22113Y:7 N: 1 MB: 1 The number of individuals who did complete the course evaluation is given in parentheses. Table cells report the count of individuals who expressed a comment that is aligned with one or more of the themes (columns) we extracted from an informal analysis of the narrative responses to the feedback questionnaire we administered at the end of each TtT workshop (see Supplementary Materials). The version of this evaluation for Pilot 4 did not capture open-ended comments. The evaluation in this pilot workshop included the explicit question: ‘I was inspired to new ways of thinking’. In total, 6/6 people answered: ‘Agree completely’. 2006 [19] and first demonstrated to be perceptible to students in The results in Table 4 suggest that the EE-TtT programme has 2017 [20]. The dimensions are: potential to provide sustainable learning for the trainers who complete it. This is an important feature of a TtT programme 1. Lifelong learning (an additional level of depth, or dimension, because it can promote ongoing professional development, and that you bring to a course or experience unrelated to the (pri- lifelong learning around teaching and instruction—irrespective mary) topic) of what content or techniques need to be taught. The documen- 2. Changing your learning behaviour as a result of the specific tation of sustainability of the learning obtained via the EE-TtT learning: Describe how your learning (fact-finding, thinking, programme would further support the focus on evidence and understanding of something or approach to learning some- theory in the structure of the training that is provided. Adding thing new) changed items to the TtT course evaluation, or follow-up evaluations, that 3. A process of personal development continuing beyond the can assess the sustainability of the TtT learning would be useful course: Something you did, or initiated, for your own sense for shaping this program towards this characteristic. of learning (i.e. not taking a course as part of your pro- With this definition of sustainablelearningin mind (seealso gramme, but a learning or training experience that you Table 5), we can revisit the objective of the EE-TtT programme to sought, created or identified—not already planned) create a durable community of supportive instructors and faculty. 4. Deconstruction/reconstruction: An idea or concept that you This objective would enhance the practicality of the programme thought you understood, but recognized you did not truly because supporting new instructors and nurturing future TtT fac- understand (deconstruction), so sought deeper understand- ulty make it more likely that the programme would become self- ing, and discovered an error in your original understanding sustaining, as are the Carpentry communities. Establishing a com- that you remedied or sought to remedy (reconstruction) munity is only indirectly representative of ‘feasibility’, however. We used a DoFA to evaluate the alignment of the features of At this stage of the TtT programme’s development (immediately post-pilot), it is not possible for participants to perceive utility the EE-TtT programme with the four dimensions of sustainable learning—the alignment was achieved by the first and last au- from such a feature—nor has there been opportunity yet to evalu- ate that perception. Nevertheless, the community aspect of the thors’ independent evaluation, which was then discussed in a conference call to ensure consensus (disagreement was on two programme aligns with the sustainability of learning that partici- pation in the TtT programme is intended to deliver. Although this cells only). The final (consensus) version is explored in Table 4. Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 Pilot results for abioinformatics Train-the-Trainer programme | 411 Table 4. Alignment of the EE-TtT programme with dimensions of sustainable learning Sustainability dimensions: Lifelong learning Changing your A process of per- Deconstruction/ learning behav- sonal develop- reconstruction iour as a result of ment continuing the specific beyond the learning course TtT programme features: Training paradigm is focused on theory and is evidence-based/XX X X evidence-informed Developmental trajectories—for the trainers themselves to XX X X continue to grow/refine their KSAs relating to training specifically Training paradigm seeks to build a community of ELIXIR XX X trainers Pedagogical and andragogical principles are explicit in training XX X new trainers, so new instructors will also follow these principles New trainers are introduced to—and encouraged to only use— XX X evidence-based principles of learning Training paradigm includes a formative assessment of the X training KSAs that the programme develops in new instructors New trainers are introduced to—and encouraged to use— XX Bloom’s taxonomy to develop learning outcomes Training paradigm embodies the target KSAs of effective train-XX X X ing, course design and learning assessment, together with an explicit developmental trajectory new instructors can continue to build on Training paradigm uses and promotes the use of active learn- XX ing techniques Programme provides instruction on how to integrate technol- ogy, including VMs and Cloud, in training delivery and development Programme involves TtT participants attending actual trainingXX X X courses to observe expert instructors in action, and follow- up discussion about observation, evaluation and metacogni- tive development around their own teaching Programme includes post-TtT support (e.g. forum/blog/net- XX X work/meetings/discussions), including support for in- structors’ development of their own pre-course assessment (for participant selection) and evaluations (for their continu- ing professional development) Materials are FAIR X X X X was not considered when the pilot was being developed, sustain- associated with sending newly qualified instructors to teach able learning is an important attribute that the programme can in other countries. claim. As the programme scales up, attention to the cultivation of 2. The ability to recruit trainees for EE-TtT sessions depends this community will be a priority. on the availability of funds (for the individual participants or nodes seeking to send participants) to support trainee participation. The EE-TtT pilot programme was necessarily constrained by grant-limited funds; however, the exercise highlighted features that must be factored into future cost Other lessons learned models. Other results from the pilot study have significant implications 3. Another constraint on the ability to recruit trainees is that for the future of the programme overall. These lessons relate to, the priority they (or their home groups/institutions) give to or derive from, the definitions of scalability, feasibility and util- training must often be balanced against competing commit- ity described in Table 5. ments. Scientists need to be convinced that enhancing their training skills is likely to be useful to their careers. 1. An important goal of the EE-TtT programme is to build train- 4. A formal certification or recognition process is one possible ing capacity in countries where the ability to provide bio- strategy that could incentivize participation. This requires informatics training is not yet well developed. However, specific mechanisms via which trainees may be qualified, such countries may have difficulties in organizing or hosting and maintain their qualifications. The training model used a course; they may also be unable to cover the costs Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 412 | Via et al. by the Carpentries requires that their trained instructors Meanwhile, ancillary (positive) feedback has arisen from further must teach two SWC/DC workshops every 2 years to main- piloting of the materials in two new TtT programmes, one for tain their certification. In our TtT kick-off meeting, we iden- high school teachers and one for university faculty and doctoral tified ongoing skills development, but not certification, as an students, currently being run by ELIXIR-Italy. important feature. However, as recognition is one of the The EE-TtT programme was designed to have the essential principal themes that the joint ELIXIR-GOBLET training features shown in Table 1; these differ sufficiently from other strategy (https://www.elixir-europe.org/news/elixir-and-gob programmes to have justified the development of a new one. let-publish-joint-training-strategy) sets out to address—and Moreover, Table 4 shows that the new programme, as de- the EE-TtT programme already collaborates with GOBLET on veloped, is aligned with features of sustainable learning. material development—the natural next step will be to Sustainability of the learning that TtT delivers should be an further collaborate on defining a process to promote trainer evaluated feature of the scaled-up EE-TtT programme. To date, recognition and integrate this into the EE-TtT programme in no TtT program has focused on or emphasized the sustainabil- future, if it is deemed a priority. ity of the learning that was delivered. 5. The EE-TtT pilot programme had only three ‘independent’ Some of the lessons learned from this pilot relate to the (confident to run a workshop with no assistance) TtT faculty grant-limited nature of its current funding model: while stimu- members available, and highlighted the need to develop a lating local capacity building, it limited the mobility of experi- structured programme to develop new faculty. As part of a enced and new TtT faculty across Europe. This highlighted the trial process, one participant completed a TtT workshop, need to develop a robust funding model to scale the TtT and then his home ELIXIR node hosted another workshop in programme up successfully. Building a viable model of cost re- which he served as an ‘assistant’, but expressed a sense of covery for training is a challenge but is nevertheless essential. insufficient preparation to teach a workshop independently. This pilot was initiated with grant funding, but both feasibility Two other people attended a workshop (Pilot 6) with the aim and scalability require ongoing investment. In this respect, we of becoming TtT faculty, but the trial faculty development can learn from the SWC/DC and GOBLET approaches, and draw process stipulates that, in addition to attending the TtT on the experience of dedicated training programmes, such as workshop, participants must then also serve as ‘assistants’ those provided by EMBL-EBI, EMBL-ABR, the Swiss Institute of or ‘co-trainers’ in at least another course. This model for Bioinformatics, the Gulbenkian Institute and the University of preparing new EE-TtT faculty is therefore not robust enough Cambridge, as examples. There have also been many virtual to achieve the objective of developing—and retaining—new training platform efforts [27, 28] including a distance (‘e-’) learn- TtT faculty. We intend to explore the Carpentry model ing platform being developed via ELIXIR at the Slovenian node further for ways to refine EE-TtT faculty development. [29]. The SWC/DC successfully delivers instructor training ses- sions online, allowing faculty to teach a large number of partici- Table 5 summarizes the findings of the evaluation of the pilot pants sitting in different countries, and overcoming scalability in terms of feasibility, utility, sustainability and scalability. The issues while also providing the hands-on, synchronous engage- results of Tables 1–4 are summarized by integrating the evidence/ ment that is so essential to successful learning of complex pro- lessons learned with the key constructs of the evaluation. gramming and software-based tools and methods. As these are As can be seen, the results about feasibility depend on the features that make distributed training effective, the SWC/DC perspective: there is some low, moderate and high feasibility model is worth emulating to ensure that the EE-TtT follows suit from the ELIXIR point of view, only moderate feasibility for the (e.g. avoiding asynchronous and/or lecture-based programmes). participant’s perspective and both low and moderate feasibility The pilot programme has also yielded actionable informa- from the faculty point of view. Similarly, scalability results are tion about planning and hosting future TtT workshops, and has low and moderate. However, utility, sustainability and the highlighted important choices about whether to prioritize utility alignment with ‘best’ practices are moderate (utility) to high. or feasibility. Scheduling stand-alone TtT sessions is more costly (time/money), but was not perceived by participants to be more or less useful; feasibility was greatest when TtT sessions Discussion and conclusions were scheduled alongside scientific meetings or conferences; but the greatest utility arose when the workshops were co- This pilot study has successfully identified strengths and weak- organized with other courses. Given this observation, utility nesses of the EE-TtT programme. Strengths include the intense would need to be explicitly assessed if e-learning approaches commitment of faculty; alignment of the design with features are used in future TtT offerings. of sustainable learning; evidence of participants’ perception of There were some challenges filling the TtT workshops. This utility of the TtT workshops; and great deal of concrete action- is not really surprising, as most life scientists receive grant able input as to next steps for further TtT workshop offerings funding to support their research and not their professional de- and how best to evaluate them. velopment as instructors; it might therefore have been doubly Weaknesses arise from the fact that the TtT faculty and the difficult to justify taking time away from their research and first cohorts of workshop participants were highly enthusiastic funding their participation in TtT workshops. Moreover, while early adopters, so little information was gleaned from the wider members of ELIXIR have an interest in increasing national train- community. For the TtT programme to be self-sustaining, and ing capacity, engagement in training and especially professional to encourage the development of a network and community development are seldom the top priorities of life scientists. around training new bioinformatics instructors, the identified Hence, strategies may be required to convince researchers that challenges in feasibility/scalability must be addressed so that TtT workshops can make significant contributions both to their the program can be shown to work more broadly. work and to the scientific community more broadly. The pilot has highlighted several aspects on which to focus One approach to incentivizing participation is to introduce a as the EE-TtT programme evolves, as detailed in the ‘Results’ formal process of certification or qualification. The Carpentries’ section. We plan to complete a formal evaluation of the pro- training models (Table 1) could be a useful reference point; and gramme and its materials before formulating the next stage. Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 Pilot results for abioinformatics Train-the-Trainer programme | 413 Table 5. Key constructs for the TtT pilot program evaluation and summary of pilot results (in black ink) at ‘low’, ‘moderate’ and ‘high’ levels Construct, definition ‘low’ levels ‘moderate’ ‘high’ • • • Feasibility (also practicality): From Too costly Breaks even/moderately costly Affordable (or fully funded) • • • ELIXIR’s point of view: Is creating and Low interest/participation Moderate interest/Participation Consistent, high interest and • • maintaining (and growing) the program No faculty available to provide training Sufficient faculty <for now> participation • • • technically achievable and financially Unacceptable levels of finances and Current funding and program are suffi- Faculty engaged and new faculty in supportable? time required cient (no expansion is possible) training Evaluation and refinement of program can continue • • • Feasibility (also practicality): From TtT Incentives to design/run new events Moderate/positive incentives to design/ Strongly positive incentives to design/ faculty point of view: Is participating as low or negative run new events run new events • • • a faculty logistically achievable and fi- Time commitment too great Time commitment acceptable Time commitment acceptable. • • • nancially reasonable? Disincentives to faculty participation No disincentives to faculty participation Incentives to faculty participation from from home institution from home institution home institution. Feasibility (also practicality): From TtT Too expensive to attend (time; money; Costs in time, money, and effort are bal- Costs in time, money and effort are far participant’s point of view: Is partici- effort); benefits of participation are not anced with benefits of participation. lower than benefits of participation pating logistically and financially obvious reasonable? Scalability: From ELIXIR’s point of view: Increasing program will greatly increase Increasing program will barely increase Increasing program will decrease costs Can the program grow larger, effect- costs and requirements of time and costs and requirements of time and ex- and requirements of time and expert- ively train new trainers to meet the expertise pertise—but they will not go down ise, as expertise is built in new trainees growing demand for training—of new trainers and of new TtT faculty? Utility: Participant-perceived usefulness: Low satisfaction with training; no inter- Moderate satisfaction with training High satisfaction with training and con- Is completing this training beneficial or est in transferring new knowledge Some interest in transferring crete plans to actively transfer the helpful to me at all? knowledge knowledge Sustainability of the learning provided No alignment with sustainable learning, Some alignment with sustainable learn- Strong alignment with sustainable [13,14]: Potential for endurance and/or limited chance for endurance or trans- ing, moderate chance for endurance or learning, significant chance to and even transfer of learning from the formal fer of knowledge transfer of knowledge plans for both endurance and transfer setting to any other problem or setting of knowledge. ‘Best’ practices: Methods (of teaching) ‘best’ is defined based on costs and not ‘best’ is defined based on consensus of ‘best’ is defined based on consensus of that are generally accepted to be super- outcomes. Identified as ‘best’ because it expert trainers. Identified as ‘best’ be- experts in the domain, expert trainers, ior to alternatives based on consistency is standardized cause it matches expert experiences and is aligned with strong evidence (and level) of success base. Identified as ‘best’ because it is evidence-based and also achievable Note: Greyed out ¼ not observed in the pilot results. Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 414 | Via et al. although the committee ultimately decided that the approach 2. Barone L, Williams J, Micklos D. Unmet needs for analyzing would not be successful, GOBLET’s work around certification biological big data: a survey of 704 NSF principal investiga- and “badging” is also important to leverage, given the signifi- tors. bioRxiv 2017. doi: https://doi.org/10.1101/108555. cant overlap of ELIXIR and GOBLET membership within the TtT 3. Brazas MD, Brooksbank C, Jimenez RC, et al. A global perspec- programme (as faculty, participants and, thereby, future fac- tive on bioinformatics training needs. bioRxiv 2017. doi: http:// ulty). Whether a certification process is a priority will have to be dx.doi.org/10.1101/098996. 4. Teal TK, Cranston KA, Lapp H, et al. Data carpentry: work- determined in the programme’s next stages. Other possibilities include ‘fellowship’ awards and/or scholarship funding for shops to increase data literacy for researchers. Int J Digit which individuals can apply, which would add both the training Curation 2015;20(1):135–43. and an ‘award’ to their CV. Options for incentivizing participa- 5. Dinsdale E, Elgin SCR, Grandgenett N, et al. NIBLSE: a network for integrating bioinformatics into life sciences education. tion to be considered cannot create additional burdens (which could further disincentivize new participants). CBE Life Sci Educ 2015;14(4):le3. Overall, these results show significant strengths and remedi- 6. Reisdorph N, Stearman R, Kechris K, et al. Hands-on work- able weaknesses in the pilot EE-TtT programme, and we are shops as an effective means of learning advanced technolo- committed to its further development. With the engagement of gies including genomics, proteomics and bioinformatics. ELIXIR Training Coordinators, TtT faculty and contributors and Genomics Proteomics Bioinformatics 2013;11(6):368–77. 7. Jordan K. MOOC completion rates: the data. 2013. http:// GOBLET—which has as a core mission the ‘professionalisation’ of bioinformatics training—we will gain additional motivation, www.katyjordan.com/MOOCproject.html (26 March 2017, support and expert input in evaluating the EE-TtT programme, date last accessed). to help scale it up and drive it forward. 8. Shachak A, Ophir R, Rubin E. Applying instructional design theories to bioinformatics education in microarray analysis and primer design workshops. Cell Biol Educ 2005; Key Points 4(3):199–206. 9. Tanner K, Allen D. Approaches to biology teaching and learn- To increase bioinformatics training capacity across ing: on integrating pedagogical training in to the graduate ex- ELIXIR, a TtT programme has been created and piloted. periences of future science faculty. CBE Life Sci Educ 2006;5(1): Groundwork for the pilot involved designing a formal 1–6. and theoretically based training paradigm that can sup- 10. Tractenberg RE. Core curriculum issues in big data research/ port the preparation of future trainers to develop learn- analysis. In: L. A. Schintler and C. L. McNeely (eds). ing opportunities in the bioinformatics resources, tools Encyclopedia of Big Data. New York, NY: Springer, 2017. doi: and methods that European life scientists need. 10.1007/978-3-319-32001-4_285-1. Feedback from seven pilot TtT sessions will support the 11. Schneider MV, Watson J, Attwood T, et al. Bioinformatics train- ongoing success of the programme, including develop- ing: a review of challenges, actions and support requirements. ment of new faculty. Brief Bioinform 2010;11(6):544–51. Pilot results suggest that while the TtT programme is 12. Ambrose SA, Bridges MW, DiPietro M, et al. How Learning feasible, useful and consistent with principles of sus- Works: Seven Research-Based Principles for Smart Teaching. San tainable learning, issues of scalability remain to be Francisco, CA: Jossey-Bass, 2010. addressed. 13. Fink LD. Creating Significant Learning Experiences: An Integrated Key outcomes have highlighted the need for: (i) support Approach to Designing College Courses, 2nd edn. San Francisco, for organizational/hosting logistics, to allow instructors CA: Jossey-Bass Higher & Adult Education, John Wiley & Sons to focus their efforts on teaching; (ii) incentives for sci- Inc (Verlag), 2013. entists to engage with opportunities to become new or 14. Bain K. What the Best College Teachers Do. Cambridge, MA: better instructors; (iii) rigorous evaluation of the TtT Harvard University Press, 2004. materials to ensure they support sustainable learning; 15. Via A, De Las Rivas J, Attwood TK, et al. Ten simple rules for (iv) strategies to recruit, train and certify new instructor developing a short bioinformatics training course. PLoS trainers; and (v) actionable items to be added to the Comput Biol 2011;7(10):e1002245. evaluations to promote improvements to the pro- 16. Via A, Blicher T, Bongcam-Rudloff E, et al. Best practices in gramme overall. bioinformatics training for life scientists. Brief Bioinform 2013; 14(5):528–37. 17. Revote J, Watson-Haigh NS, Quenette S, et al. Development of Supplementary Data a cloud-based bioinformatics training platform. Brief Bioinform 2017;18(3):537–44. Supplementary data are available online at http://bib.oxford 18. Patel VL, Yoskowitz NA, Arocha JF, et al. Cognitive and learn- journals.org/. ing sciences in biomedical and health instructional design: a review with lessons for biomedical informatics education. Funding J Biomed Inform 2009;42(1):176–97. 19. Knapper C. Lifelong learning means effective and sustainable This work was funded by the European Union’s Horizon learning: reasons, ideas, concrete measures. In: Seminar 2020 research and innovation programme ELIXIR – Presented at 25th International Course on Vocational Training and EXCELERATE, grant agreement no 676559. Education in Agriculture, Ontario, Canada. 2006. http://www.ciea.ch/ documents/s06_ref_knapper_e.pdf (26 July 2017, date last References accessed). 20. Tractenberg RE, FitzGerald KT, Collmann J. Evidence of 1. Chang J. Core services: reward bioinformaticians. Nature 2015; sustainable learning with the Mastery Rubric for 520:151–2. Downloaded from https://academic.oup.com/bib/article/20/2/405/4237465 by DeepDyve user on 12 July 2022 Pilot results for abioinformatics Train-the-Trainer programme | 415 26. Tractenberg RE. Degrees of freedom analysis in educational Ethical Reasoning. Educ Sci 2017;7(1):2. doi:10.3390/educsci 7010002. research and decision-making: leveraging qualitative and 21. Wilson G. Software carpentry: lessons learned. F1000Res 2016; survey data to promote excellence in bioinformatics training 3:62. doi: 10.12688/f1000research.3-62.v2. and education. Brief Bioinform 2017, in press. 22. Morgan SL, Palagi PM, Fernandes PL, et al. The ELIXIR- 27. Watson-Haigh NS, Shang CA, Haimel M, et al. Next-gener- EXCELERATE Train-the-Trainer pilot programme: empower ation sequencing: a challenge to meet the increasing demand researchers to deliver high-quality training. F1000Res 2017, 6: for training workshops in Australia. Brief Bioinform 2013;14(5): 1557. doi: 10.12688/f1000research.12332.1. 563–74. 23. Best practices in training. DEPOCEI Report, 2013. http://www. 28. Watson-Haigh NS, Revote J, Suchecki R, et al. Towards an depocei.org/wp-content/uploads/2013/06/DEPOCEI_Final-report_ open, collaborative, reusable framework for sharing hands- TRAINING-BEST-PRACTICES.pdf (26 July 2017, date last on bioinformatics training workshops. Brief Bioinform 2017; accessed) 18(2):348–55. 24. Attwood TK, Bongcam-Rudloff E, Brazas MD, et al. GOBLET: 29. Leskosek B, Dimec J, Javorsek JJ, et al. ELIXIR Slovenia the global organisation for bioinformatics learning, education eLearning Platform (EeLP). In: Poster presented at ELIXIR All Hands Meeting 8-9 March 2-16, Barcelona, Spain. 2016; https:// and training. PLoS Comput Biol 2015;11:e1004143. 25. Campbell DT. Degrees of freedom in the case study. Comp f1000research.com/posters/5-338 (26 July 2017, date last Political Stud 1975;8:178–93. accessed).
Journal
Briefings in Bioinformatics – Oxford University Press
Published: Mar 25, 2019