Abstract Evaluating cross-disciplinary collaboration has generally been undertaken using disciplinary standards. However, this practice is increasingly being found to be inadequate due to the often contradictory nature of the methods used. It has been suggested that methods that consider the unique integrative nature of these studies be employed. This study describes the tracking of ideas method that was developed to consider the integration of ideas in group knowledge products developed by a cross-disciplinary group. The cross-disciplinary group was from the New Zealand disability field who used an eight-phase approach to brainstorm ideas over the course of a weekend on how to build an inclusive society for all New Zealanders. It was found that this new method was effective for tracking the ideas through numerous different artefacts and simplifying the complex path of those ideas. These artefacts included the worldviews, paradigms of disability and concept maps of the participants, the activity sheets from the group activities, the activity topics, and other artefacts made available to the groups. The findings from the tracking of ideas method were generally corroborated by the participant’s reflections. Further research is needed to test the tracking of ideas method and to corroborate the findings with participants’ perspectives. 1. Introduction Research indicates that many researchers favour evaluating cross-disciplinary studies using disciplinary standards (Boix-Mansilla 2006). Disciplines have their own clear guidelines for determining the effectiveness and quality of research dependent, to a large degree, on the philosophical basis of the fields of study and the types of research methodologies used (Sulkunen 2008; Huutoniemi 2010). For example, researchers from the positivist or post-positivist philosophies generally use quantitative methodologies in cross-disciplinary studies, such as surveys, social network analysis, and bibliometric analysis in a search for generalizable ‘truths’ (Willis 2007; Stokols et al. 2010). Those from the interpretivist philosophies, on the other hand, seeking to understand a particular context, tend towards more qualitative methodologies, such as investigator interviews, self-directed discussions, narrative analysis, and peer reviews (Willis 2007; Stokols et al. 2010). In particular, funding agencies like to evaluate cross-disciplinary studies using panels that have representatives from the different constituent disciplines (Grigg 1999). Huutoniemi (2010), however, argues that the use of these disciplinary standards leaves the projects and researchers vulnerable, as they have to meet the criteria of both types of study, which are often contradictory. To overcome the problems of meeting competing criteria, some authors consider that cross-disciplinary studies should be evaluated in different ways that recognize their unique interactive nature (Spaapen, Dijstelbloem and Wamelink 2007). It is suggested that the effectiveness and rigour of the endeavour would then be related to the outcomes and degree of integration achieved (Spaapen, Dijstelbloem and Wamelink 2007). Integration can be of ideas (perspectives; concepts; theories), methods (tools and techniques), information, or the group (Porter, Roessner and Heberger 2008). The integration of ideas or information develops as a collective construction in the form of new ideas and knowledge (Defila and Di Giulio 1999; Fiore et al. 2010; Stokols et al. 2010). The collective constructions can take the form of integrated care plans for patients, integrative models, publications such as co-authored journal articles, new innovative policies, new training programmes, scientific innovations, or an integrated understanding of, or resolutions to, complex real-world issues (Marzano, Carss and Bell 2006; Stokols 2006; Endberg 2007; Holmes, Lehman and Hade 2008; Lowe and Phillipson 2009; Stokols et al., 2010; Clark 2011). Fiore et al. (2010) consider that a sign that integration of knowledge has occurred is when the knowledge product developed by the group does not come from any one individual but emanates from the group dialogue. However, the question is how to evaluate integration within these collective constructions. Wagner et al. (2011) consider that evaluation of integration either examines the ongoing social and cognitive processes of knowledge integration or the internal workings of the knowledge system as represented by the outputs of collaboration using bibliographic analysis. Bibliometric analysis considers the number of different disciplines represented in co-authored articles (Porter, Roessner and Heberger 2008). Some authors also look at the cross-disciplinary nature of the references cited in journal articles to consider the integration or specialization of the content. Both of these methods assume that if a range of disciplines is present, then cognitive integration of ideas or knowledge has occurred. However, the presence of a variety of disciplines does not necessarily mean that cognitive integration is present (Porter, Roessner and Heberger 2008). Belcher et al. (2016) agree and question the overemphasis on measuring academic outputs as a sign of integration. Therefore, this article looks at a study that sought to consider how integration of ideas occurred through the development of collective constructions within a cross-disciplinary group from the New Zealand disability field (Budd 2014). 2. Method This study involved a two-step process. The first process was to develop the ‘Tracking of Ideas’ method. The second process was to trial the method and consider its usefulness. 2.1 Development of the ‘Tracking of Ideas’ method To develop the tracking of ideas method, three data analysis methods were researched, including conversation analysis by Hutchby and Wooffitt (2008), meme-tracking by Leskovec, Backstrom, and Kleinberg (2009), and the mobility of ideas by Allen-Robertson and Beer (2010). Conversation analysis involving the detailed analysis of talk-in-interaction (Hutchby and Wooffitt 2008) was quickly eliminated, since it requires all interactions to be transcribed. This would be impractical when working with a cross-disciplinary group across multiple activities. Meme-tracking was a method developed to track ideas that emerged in the media using online environments. The first search looks for the exact words and phrases. This is then followed by a second search using variations of the original words or phrases. These searches help to show not only where the ideas come from and go to but also how they mutate (Leskovec, Backstrom and Kleinberg 2009). Mobility of ideas is similar to meme-tracking and involves tracking ideas across time and space. The first analysis seeks to track the ideas in chronological time to discover where the ideas originate and how frequently they are used. The second analysis seeks to map where in terms of geographical location the ideas are used. This method seeks to discover the trends in usage and consider the networks of association that may exist. The one major drawback of this method is that while it is easy to track unique ideas, ideas that are in common usage are much harder to map (Allen-Robertson and Beer 2010). It was decided that a combination of meme-tracking and mobility of ideas would be used to develop the tracking of ideas method. This method was designed to track the ideas generated by the participants and groups to determine whether the final knowledge products emanated from the group’s interaction or from one individual. This method was also used to consider where and how often ideas emerged. 2.2 Trialling the ‘Tracking of Ideas’ method The tracking of ideas method was trialled with a cross-disciplinary group of 19 academics and practitioners from the New Zealand disability sector. This group came together for a brainstorming weekend to consider how to build an inclusive society for all New Zealanders, including those experiencing disability in line with the New Zealand Disability Strategy (Minister for Disability Issues, 2001). Participants were from a range of disciplines or knowledge areas including physiotherapy, mental health, nursing, sociology, disability studies, education, disability rights advocacy, rehabilitation (including drug and alcohol rehabilitation), and social work. An eight-phase approach was used to help facilitate cross-disciplinary collaboration within this group. The eight phases of the approach were: building a conducive environment; initiating the intra-individual process; preparing to build a common goal and vision; building a common frame of reference; developing cross-disciplinary understanding; redirecting tension; building collective constructions and identity; and preparing for collective action. A range of activities was used to implement each phase, including reflections on participants’ worldviews and paradigms of disability; the development of ground rules and concept maps; a study of the paradigms of disability in the movie Happy Feet; a Timelines activity to consider the major events in the disability field; a Timelines and Typology activity to consider when and how the different paradigms of disability emerged and impacted events; and activities adapted from the Whole Systems Change Approaches of Future Search and Appreciative Inquiry (AI). These activities included a Future Search activity involving role plays and questions concerning what the different paradigms were proud and ashamed of, an AI Discover activity to explore positive experiences of inclusion, an AI Dream activity to develop a dream of how New Zealand might look if building an inclusive society was successful, and an AI Design activity to plan how to make that dream a reality. Participants were allocated identifying letters and worked either as individuals or in randomly selected groups for the first five phases and then came together in three consistent groups of four to six people in phases 6–8. Three participants had to leave the weekend before Phase 6, which left 16. The consistent groups that came together in Phases 6–8 were given names that comprised their participant’s identifying letters and were SNARQK, PHOTBD, and LEGJ. Prior to the weekend participants were asked to record an understanding of their worldview and their paradigm of disability, which they recorded in a journal identified by their participant letter. One of the first activities at the weekend was to develop a concept map of their understanding of what helps to build an inclusive society, and these were also recorded in their journals. A number of activity sheets were developed as part of the discussions in Phases 1–6. Participants identified their contributions to these activity sheets using their identifying letters. Two knowledge products were collaboratively developed by each of groups SNARQK, PHOTBD, and LEGJ in Phases 7 and 8. In Phase 7, each group developed a creative presentation of a dream of how they envisaged an inclusive society in New Zealand to be. These creative activities were described and recorded by the group members. In Phase 8, each group developed a design using an Ishikawa (Fishbone) planning tool to determine how the dream they developed in Phase 7 could become a reality. The journal reflections on the participants’ worldviews and paradigms of disability, the activity sheets, and the group knowledge products all formed the data for the study. The tracking of ideas method applied in this study used the data from all the individual and group knowledge products that were collected throughout the study. The main ideas were extracted from the final two group knowledge products developed by the three small groups in Phases 6–8 of the approach. These ideas were then tracked back through the activities to see where they originated. This search used the activity sheets and the information from each individual’s worldview and paradigm of disability reflections, and their concept maps. The search was undertaken using the computer find tool using the exact words or similar words derived from a thesaurus from the group knowledge products. For example, if employment was an idea in a group knowledge product, then similar words, such as career, jobs, occupation, and work, were also searched for. Diagrams were then plotted to show how the ideas emerged from the individuals’ perspectives and/or through the group work to determine the level of integration of ideas that had occurred as well as to identify where the ideas had come from. These diagrams were designed to reduce the complexity of the flow of ideas. The findings from the tracking of ideas method were also compared to the collaborative experiences of the participants during the activities as recorded in their journals. 3. Findings The ideas documented in Phase 8 of the approach were recorded for each group as shown in Figs 1–3. The boxes across the top of the figures represent the activities in the different phases. The boxes within the larger group box represent the participants in the groups. These participant boxes represent ideas that originated in a participant’s worldview reflection, paradigm of disability reflection, and/or concept map. Boxes under activities were named according to the members of the group working on that activity where a particular idea originated. The boxes on the far right represent the main ideas from the AI Design activity. The unidirectional and bidirectional arrows represent the flow of ideas from one participant or activity or idea to another. Broken lines represent a possible flow of an idea. Figure 1. View largeDownload slide Tracking of Ideas Group SNARQK. Figure 1. View largeDownload slide Tracking of Ideas Group SNARQK. Figure 2. View largeDownload slide Tracking of Ideas Group PHTOBD. Figure 2. View largeDownload slide Tracking of Ideas Group PHTOBD. Figure 3. View largeDownload slide Tracking of Ideas Group LEGJ. Figure 3. View largeDownload slide Tracking of Ideas Group LEGJ. 3.1 Group SNARQK Participants in Group SNARQK came from the disciplines/knowledge bases of nursing, physiotherapy, disability studies, rehabilitation, and education. Participants, S, N, R, Q, and K, recorded their worldviews and paradigms and A, R, Q, and K recorded their concept maps. Group SNARQK developed two group knowledge products, one during the AI Dream and the other during the AI Design activities. Group SNARQK expressed their dream for an inclusive society in Phase 7 as a song and dance. Each participant in the group danced using their own steps and rhythms, thus expressing diversity and unity at the same time. The idea of unity in diversity was also reflected in the words of the song, which was sung to the tune of Imagine by John Lennon. The dream provided the base metaphor for the final design of an intervention to facilitate an inclusive society in Phase 8 with the intervention being a more detailed version of the dream. The group also choose a first action step to help them implement the resolution developed. As demonstrated in Fig. 1, the ideas for the dream and intervention emanated from a number of different places. For example, values, such as openness, honesty, accepting all people, life is more important than material things, and a happy life, were carried through from Participant R’s worldview to Group SNARQK’s AI Discover and AI Dream activities. Another idea, barriers, was also carried through from Participant R and N’s paradigms of disability to the AI Dream activity. One idea, connections between people or shared kaupapa [topic], appears in Participant K’s concept map and the AI Discover activity. Some ideas, such as enjoyment, no discrimination, positive attitudes, accepting and understanding differences, focus on abilities, supportive relationships, respect and caring communities from the concept maps, and flexibility from the AI Discover, were also carried forward to the AI Dream activity. The concept of Happy Feet, from one of the earlier activities, was also carried through to the AI Dream activity. The AI Design for Group SNARQK had a number of different foci based around the main concept of Safe Neighbourhoods, including changing the language of disability through the use of popular media, involving people with disability in leadership and decision-making, building alliances and partnerships with other groups who are discriminated against, building purpose and opportunities for people with disabilities to gain work and be involved through a ‘Fair-Go’ Employment Conference, and address the structures that provide funding and services for people with disabilities. When asked what would be their first action step the group unanimously stated that they were going to organize a ‘Fair-Go’ employment conference. As can be seen in Fig. 1, the main theme of safe neighbourhoods was carried forward from the AI Dream activity and the concept maps of Participants Q, K, and R. For example, in their concept map, Participant Q talks about secure neighbourhoods and supportive relationships, and Participant K talks about supportive, caring communities, and belonging. Participant R talks about love and acceptance in their worldview and about integrated communities, where there is no discrimination, positive attitudes, acceptance, and understanding difference in their concept map. As shown in Fig. 1, the ideas around changing the language of disability in the AI Design came predominantly from the paradigms of disability of Participants N, R, and S. For example, Participant N speaks about disability being reduced through communication with others about their attitudes, Participant R talks about the greatest barriers for people society labels as disabled are the attitudes of others, and Participant S states disabled people are most often handicapped by the negative attitudes of others. Although as individual ideas these statements may not have been shared directly by the group, they would no doubt have informed Participants N and R’s contributions to group discussions. The subject of the critique of language was discussed in one of the Timelines and Typology groups consisting of Participants D, K, and O. Participant O, for example, states in their worldview that language is not neutral but is embedded in relations of power. This may suggest that Participant O’s ideas from her worldview were shared in the Timelines and Typology activity with Participants D and K. Participant K may then have shared the ideas about language during the AI Design activity. The idea of people with disabilities in leadership and decision-making is another strong perspective of the AI Design of Group SNARQK. This idea picks up on the themes of power sharing from the group’s AI Dream and the idea of empowerment from their AI Discover activity. These ideas were also reflected in the concept map of Participant Q who states that people with disabilities should have opportunities to participate in civil activities. Participant H recorded the idea of leadership of a person with disabilities on one of the Timelines activity sheets that Participants K, N, and S were part of, and Participant N recorded the associated concept of empowerment on another Timeline sheet. These recorded ideas may then have been shared by Participants K, N, and S during the group work and been integrated into the group knowledge building process. Building alliances and partnerships with other groups who are discriminated against was also another theme in Group SNARQK’s AI Design. This idea originated in the AI Discover activity of the group but does not directly appear in any of the other activities. The comments I have always been an advocate for social justice issues, initially environmental and anti-racist issues from Participant K’s worldview and I loathe injustice and especially persecution on spurious grounds relating to personal attributes from Participant N’s worldview indicate that they both have an interest in social justice and may have shared these ideas in group discussions, which were later adopted by the group. Funding and support are two more themes that appear in Group SNARQK’s AI Design with the statements, re-position the money spent on disability issues, and funding needs to be questioned—Accident Compensation Corporation (ACC)—Social Welfare need reviewing and reorganizefamily support systems … support they might need. The idea of funding is closely linked to support in Participant R’s concept map where it states adequate support systems, guiding strategies, provision of resources to meet needs, while supportive relationships is mentioned in Participant Q’s concept map. Participant K, in her concept map, also links the two ideas of funding and support, stating, free education, free healthcare, free social services … supportive, caring communities. These ideas of funding and support are then carried through to the AI Dream in the words of the song systems support us … supports need on demand. An interesting idea developed as part of Group SNARQK’s AI Design is that of the ‘Fair-Go’ Conference for employment issues. This idea has two main strands. The first strand is employment and develops from the group’s AI Dream. When the activity sheets were scanned for the word employment, three references to supported employment were found within different Timeline activity sheets all recorded by Participant J. There would appear to be no direct link to employment issues in the recorded ideas of the participants of the group, but they may have had access to the employment ideas through studying the different Timelines activity sheets that were accessible to the participants throughout the weekend. The concept of ‘Fair-Go’ is slightly different, since it is a unique concept that formed a major part of Group SNARQK’s resolution developed in the AI Design activity. The concept of ‘Fair-Go’ did not appear in any of Group SNARQK’s individual or group artefacts. Fair-Go Thinking, however, did appear in Group PHOTBD’s AI Discover activity. All the groups’ AI Discover outcomes were shared and available, and so the idea of ‘Fair-Go’ may well have been discussed and adopted by members of Group SNARQK who then integrated the idea into their final AI Design. It would appear from tracking the ideas of Group SNARQK that the two knowledge products developed by the group were definitely the synthesis of more than one participant and that the ideas had been drawn from a number of different sources, including other participants, activities, activity sheets, and formal and informal discussions with others. These findings would seem to indicate that Group SNARQK did maximize the benefits of not only their ideas and perspectives but also those shared within the wider group as well as ideas and concepts from the activities themselves, such as Happy Feet, to develop integrated knowledge products or collective constructions. The integration of ideas as identified by the tracking of ideas method was further confirmed by the members of Group SNARQK. For example, Participant R considered that it was through their discussions that they were able to use expertise from all the group and the ideas came together. Following on from the previous discussion, the ideas just flowed and we felt were well captured in our fish chart. The passion developed as the ideas evolved and the idea of a conference to start the ball rolling gelled. We were able to utilise the expertise of the various group members in the process. (R) 3.2 Group PHOTBD Participants in Group PHOTBD came from the disciplines/knowledge bases of rehabilitation, sociology, education, and physiotherapy. All six members of Group PHOTBD documented their paradigms of disability and all but Participant O their concept maps. Four participants, T, O, B, and D recorded their worldviews. Group PHOTBD developed two group knowledge products during the AI Dream and the AI Design activities. In the AI Dream activity for an inclusive society, Group PHOTBD designed an event entitled I have a dream: He Moenga au. Each participant in the group wrote short quotes on post-it notes that they then stuck on a large piece of paper. When they did their presentation, people were asked to pass the paper around and select a post-it note, read it out, and pass it on. People could also add their own post-it notes if they wanted to. Therefore, it can be seen that while all the participants of Group PHOTBD contributed to the activity, there was little integration of ideas during this AI Dream activity. As can be seen in Fig. 2, a number of factors were carried through from the group’s individual ideas to the AI Discover activity. These ideas included respect, equality, tolerance of difference, and the importance of systems. For example, respect and equality were both evidenced in the concept map for Participant H, who stated an acknowledgement that some people may require more support than others and that that’s not unfair, respect for all and equality of opportunity. Tolerance of difference was mentioned in the concept maps of Participants T, D, and P. For example, Participant T considered that difference needs to be respected and tolerated, Participant D talked about valuing difference/diversity, and Participant P talked about tolerance of differences. Tolerance was then carried through to the AI Dream where quotes included embracing difference and accept and embrace my difference. The importance of systems also emerged as one of the ideas in the AI Discover and seemed to derive from the concept map of Participant P who talked about fully democratic systems. The importance of systems also flowed through the AI Dream where one post-it note stated, The New Zealand Social Service System should ensure that all can live in dignity and a reasonable standard of living, and another stated, the measure of society is the mechanisms used to create vulnerabilities. The idea of social justice was evidenced in the worldviews of Participants D and O. Participant D, for example, stated I grew up in a home where principles and social justice and fair play were articulated and enacted, while Participant O talked about relations of power that impact society. Social justice is also contained within Participants P and D’s concept maps. Participant P stated, disability stems from the failure of a structured social environment to adjust to the needs and aspirations of disabled citizens, while Participant D stated I like Simi Linton’s suggestion of attending to the mechanisms society uses to create vulnerable ‘others’—rather than vulnerability as a given. The theme of social justice was also carried through the AI Discover via the wish to have a fair and just society to the AI Dream in the quotes get up stand up, stand up for your rights, Bob Marley, to each according to their needs, from each according to their means, and no one should have so much money they can buy the life of another; no one should be so poor that they need to sell their own, Wilberforce. In the AI Design activity, Group PHOTDB used the Ishikawa (Fishbone) diagram to document their aims, which were politicization, there are alternatives and dismantle the neo-liberal state. They approached the task on three levels: macro, meso, and micro. The idea for these levels was taken from the information shared by the researcher at the beginning of the activity. The group clearly integrated and used these levels to provide an overarching framework to structure the development of their AI Design. When Group PHOTBD was asked what would be the first action step, they stated that no group decision had been made but that they would all do what was stated in their AI Design within their own spheres of influence. This action plan was highly individualized and was not refined by the group. As can be seen from Fig. 2, political action seems to be a definite theme in the AI Design taking the form of politicizecauses at community level, be smart about who you lobby in Parliament, and create political and economic alternatives. This theme of political action emerged from the worldviews of Participants O and D. Participant O stated social theory enabled me to see how the personal was political and political solutions are sought, while Participant D stated I was a feminist/activist. This theme continued through the paradigms of disability for Participants H and T with Participant H advocating for activism, while Participant T quoted a sociopolitical definition of disability: Disability is removed when disabling factors such as [list given] … are reduced. A lot in common with feminist, racism. (H) Socio-political definition of disability—disability stems from the failure of a structured social environment to adjust to the needs and aspirations of disabled citizens. (T)Political access is also mentioned in Participant T’s concept map. Having appeared in the individual ideas, however, this theme is not carried through the AI Discover or the AI Dream and only re-emerges in the AI Design. Associated with this theme of political action is dismantle the neo-liberal state. The ideas about neo-liberalism were carried forward from Participant O’s worldview through to the AI Discover and then onto the AI Design: [Worldview Reflection] I lived the Neo-Liberal reforms, user pays, charging for specialists’ appointments, prescriptions, the rationalisation of health care. Public vs Private … . [Three wishes of the AI Discover activity] Demise of Neo-Liberalism and replacement with a fair and just society. (O)Another idea linked to political action is legislation, reflected in the statements consistency with key legislation—New Zealand Disability Strategy (NZDS)andUnited Nations Convention on the Rights of the Child (UNROC) and implementing the enabling legislation that we already have in the AI Design. The importance of legislation is drawn from the AI Discover activity where it was stated that what was important was inclusion policies local (national) and international. The group made up of Participants D, S, and E also mentioned different legislation in a number of the Timelines, and Timelines and Typology activities. As part of this group, Participant D might well have taken the idea about legislation from the Timelines activity into the AI Discover or AI Design. The idea of strategic relationships over issues, alliances, lobby groups, strengthen coalitions, and personal alliances all point to the importance of developing partnerships with others. This would seem to have flowed through from one of the quotes no man is an island in the AI Dream, which in turn appears to have come from Participant T’s worldview, where again the same words no man is an island were used. The ideas of interdependence and relationships also appeared in Participant D’s concept map. The idea of the importance of individuals was another dominant theme in Group PHOTBD’s AI Design, with the micro-level suggestions, individuals can make a difference, personal alliances, and speak up, speak out. The theme of the importance of individuals was also reflected in the action point for the group, which recommended that they should each do all of the resolution suggestions within their own spheres of influence. This theme was also reflected in the quotes, the power of one, what is the most precious thing in the world? He Tāngata, He Tāngata He Tāngata [the people, the people, the people], and the only way that evil can triumph is for good people to do nothing from the AI Dream. These ideas in the AI Dream in turn were drawn from the AI Discover where it was stated individuals (especially individual efforts) may have powerful impacts, and their loss from the system can be damaging to the process. Closely linked to the importance of individuals was the idea of leadership by people with disabilities, represented by the statement strengthen representation at government level in the meso-level of the AI Design. The idea of people with disabilities being in leadership is a strong theme from Participant H’s concept map. This idea was also reflected in the comment visible participation and political access from Participant T’s concept map. Overall, it would seem from tracking the ideas of Group PHOTBD that, while the ideas of all the members were used in the AI Dream, the ideas were juxtaposed rather than integrated. From Fig. 2, it can be seen that while the ideas of Participants H, O, T, and D were all integrated into the AI Design, no ideas were carried forward from Participants P and B. The final action point was also highly individualized and had not been refined by the group. The group did, however, use the structure of the different levels of reality to provide a framework for their ideas. Therefore, it can be seen that Group PHOTBD, rather than developing fully integrated knowledge products, developed ones that either juxtaposed ideas or did not include some of the members’ ideas. The lack of integration of ideas by Group PHOTBD identified by the tracking of ideas method was further confirmed by the group participants’ accounts of the collaboration. For example, Participant B stated I was so excluded and uncomfortable within our group, while Participant O considered that the group had—a sort of shared vision but lots of different opinions about how to get there. 3.3 Group LEGJ Participants in Group LEGJ came from the disciplines/knowledge bases of nursing, physiotherapy, and social work. All the participants in Group LEGJ documented their worldviews. Participants L, E, and J documented their paradigms of disability, and Participants L, E, and G documented their concept maps. Group LEGJ developed two group knowledge products during the AI Dream and the AI Design activities. Group LEGJ’s activity for the AI Dream was an unspoken role-play in which they all joined a circle in the midst of the bigger group. Some of the group faced inwards, some out, and some to the side. At the centre was the Circle of Friends statue with a candle in the middle that was taken from the reflection table. Once their circle was formed, they then kept inviting others from the wider group to come and join the circle and the circle enlarged. There were no words spoken by group members during the activity. At the end of the activity, the group gave the following summary. People were invited to join circles not squares. People were allowed to go in and out as they chose. Can be facing in or sideways or however. Whenever guys stepped out, like XXX came out with me and the circle grew and came bigger again. No one was excluded, no one put in a corner. Not intentional, just happened. Invited but didn’t have to join. (Summary of activity given by Group LEGJ)The idea of tolerance from Participant G’s concept map and acceptance from Participant E’s concept map were both reflected in the AI Dream through the idea of all being invited but free to express individuality by standing sideways, backwards, or facing inwards. Group LEGJ’s AI Design activity was not structured around the Ishikawa (Fishbone) diagram, but statements were placed on post-it notes and grouped together on a sheet. Although no overall aim was written on the activity sheet, the core theme, as described by Participant E, was citizenship. The core central theme here was around achieving citizenship—1) from a legal perspective Top down 2) from an individual perspective Bottom up. Citizenship and being part of a society is dependent on knowing rights (arrow), a central depository. You don’t know what you don’t know. So the development of a central depository—legislation but of who is who and what they do and how they are linked. Taught about in schools—in libraries—101 things you didn’t know that you needed to know. (E)The statements citizenship guaranteed, supported and connectedness, natural networks, and strengths collectivity and flexibility represented the themes of citizenship and belonging. As shown in Fig. 3, this theme of citizenship flows from the inclusivity of the circle that was the group’s AI Dream. Citizenship can also be clearly traced from Participant L’s concept map where they stated, citizenship leads to social participation, civic participation and community participation and then through one of the Timelines activity sheets where Participant L stated revival of the notion of citizenship, social inclusion. When asked what the first step was going to be, the group commented that this would be the development of an information portal. They also stated that none of them had the technical skills to undertake the task and so they would need to find someone who could help with this aspect of the action point. The concept of the information portal picks up on the theme of access that is mentioned in the following statements from the AI Design. These statements include resources, multiple connection points—signposts to different systems, first point of contact, open access to information, access point, open access to experiences, and open access to information. This idea of access again flows from the open invitation to join the circle in the AI Dream but did not really appear in the AI Discover activity for Group LEGJ as shown in Fig. 3. The idea of access would seem to have flowed through from the paradigm of disability of Participant E where it was stated, disability removed by equal opportunity, access, which they also shared in the Timelines and Typology group made up of Participants D, S, and E. Knowledge of Legal Rights was also a dominant theme in Group LEGJ’s AI Design, represented by the statement rights and experience. This theme picks up on many of the entries on the Timelines, and Timelines and Typology activities and appears in Group LEGJ’s AI Discover activity in the comments rights, justice, fairness, and stand up for what important. It also reflects the quote stand up for your rights by Bob Marley from the AI Dream of Group PHOTBD. The idea of the importance, strength, resilience, and courage of individuals is expressed in Group LEGJ’s AI Design in the statements self-responsibility, strengths building, building on current strengths of capacity, strengths, willingness, empathy, and courage. This theme of strength and resilience was reflected in multiple sources, for example, focus on strengths was found in Group SNARQK’s AI Discover. The statement history despite its wrenching pain cannot be unlived, but if faced with courage need not be lived again was found in Group PHOTBD’s AI Dream. Courage was also stated in Group LEGJ’s own AI Discover activity sheet, while sense of self, from Participant G’s concept map, was carried forward to the AI Discover activity. Self-efficacy, being brave, empathy, and self-determination were carried forward from the AI Discover to the AI Design activity. From the tracking of ideas, it would seem that the AI Dream for Group LEGJ was a cohesive concept that used ideas from the group members. The group not only incorporated others into their activity but also used an artefact from the reflection table as a metaphor for their AI Dream. The AI Design further developed the concept from the AI Dream and incorporated the ideas from all of the group members as well as the activities and ideas from both Group SNARQK and Group PHOTBD. The action point from the group was refined and determined by the group, but the AI Design did not use the levels and scales or the Ishikawa (Fishbone) diagram to structure their ideas. The integrated nature of the work was reinforced by Participant G who considered that the group worked well together and produced a collective construction. Though we all came from different backgrounds, knowledge and experience we collectively embraced the thought of an information portal responsive to the individual, guided by and for the individual. Each person’s response elicited more buy-in from the rest and momentum was evident. No gatekeeping was evident and people stayed with the picture rather than the script. (G)Participant E also considered that the group integrated their ideas saying Interdisciplinary, Trying to make into a coherent whole. Participant L, however, considered that no new ideas were generated and that the resolutions/knowledge products were not new and thought that the exercise had been too rushed. They also considered that they had not fully contributed to the exercise, that the group had not fully explored the nuances of the different perspectives, and that what had appeared to be consensus was just a rushed outcome. This would seem to indicate that although Participant L’s idea of citizenship was one of the main ideas in the Group LEGJ’s AI Design, the collective identity that had been evident in the AI Dream was not evident in the AI Design. I didn’t think I was able to participate fully in this exercise because I felt that the four of us started off from different grounds but as soon as we knew time was running out, I felt that we were just trying to come out with some sort of outcome. I would have liked to unpack some of the issues further. (L)Overall, while the tracking of the ideas and the comments of some of the group indicated that Group LEGJ did integrate their ideas and the ideas from other groups and activities to develop collective constructions, one group member considered that they were not included and did not have their ideas integrated despite having their ideas used within the group knowledge products. 3.4 Summary of group knowledge products In summary, it can be seen from the analysis of the knowledge products developed that two of the groups, SNARQK and LEGJ, did integrate ideas from multiple sources. These sources included participants, other groups’ knowledge products, and ideas including knowledge shared in the group activities, activity themes, e.g. Happy Feet, levels and scales of reality, and artefacts, e.g. the Circle of Friends. These ideas were shared and developed throughout the activities at the weekend. Two of the groups, SNARQK and LEGJ, were able to build on the abstract concepts developed for the AI Dream activity in their AI Design activities and then outline a clear plan of action to deliver their dreams. The analysis also showed, however, that Group PHOTBD did not integrate ideas but only juxtapositioned them in the AI Dream and had highly individualized ideas in the AI Design and action point. The opinions of the participants generally corroborated the tracking of ideas method except in Group LEGJ where one participant considered they were excluded, despite her ideas being one of the main ones used by the group. Groups PHOTBD and SNARQK used the structure provided, e.g. the Ishikawa (Fishbone) diagram, to structure their ideas and knowledge products, while Group LEGJ used their own methods. 4. Discussion and conclusion Evaluating cross-disciplinary studies is an ongoing issue due to the competing criteria of different research traditions (Boix-Mansilla 2006; Huutoniemi 2010). Based on the concept proposed by Spaapen, Dijstelbloem, and Wamelink (2007), the study discussed in this article sought to consider the integration of ideas in the group knowledge products produced by three small cross-disciplinary groups. Since no specific method was found to evaluate the integration of ideas a new method, the tracking of ideas method, was developed. 4.1 Development of the ‘Tracking of Ideas’ method The tracking of ideas method based on the methods of meme-tracking and mobility of ideas sought to evaluate the integration of ideas in the collective constructions or group knowledge products developed by the three small cross-disciplinary groups in the study. Overall, it was designed to help consider whether the ideas that formed the basis of the group knowledge products in the study emanated from more than one person and developed as a result of the group’s interactions. In line with meme-tracking, the tracking of ideas method sought to track exact words or phrases such as employment. It also tracked variations of exact words or phrases, such as strategic relationships, alliances, coalitions, and lobby groups. Although meme-tracking is often employed to consider where mutations occur (Leskovec, Backstrom and Kleinberg 2009), this was not the focus of the study and so the variations were used merely to track the idea rather than to consider how mutations occurred. Mobility of ideas is generally used to track ideas across time and space (Allen-Robertson and Beer 2010). The tracking of ideas method indeed did seek to track the ideas across time and space, not in terms of worldwide geography or long-term usage but in terms of activities and groups across a weekend event. It is interesting to note that when the tracking the ideas method was employed, it was much easier to track unique concepts such as ‘Fair-Go’ and Happy Feet rather than ideas such as employment that were common to a number of participants, which supports the findings of Allen-Robertson and Beer (2010) and their experience of using the mobility of ideas method. Overall, it can be seen that the meme-tracking method was useful to incorporate tracking of exact words as well as similar words, while the mobility of ideas method helped the ideas to be tracked over the course of the weekend and from activity to activity. Together they helped to develop a method of tracking ideas that was able to be used as a data analysis tool to consider the integration of ideas in the cross-disciplinary study reported in this article. 4.2 Integration of ideas As was stated earlier, the tracking of ideas method, developed in the study and discussed in this article, sought to consider if the ideas embedded in the group knowledge products emanated from the group dialogue in line with Fiore et al.’s (2010) thinking. The findings of the study show that the tracking of ideas method was useful to track the ideas and to consider whether the ideas came from one person or from the dialogue of the group as well as helped to reduce the complexity of the paths of those ideas. For example, Figs 1–3 all showed that the ideas in the final group knowledge products originated from a number of sources including individual group members, participants from other groups and activities. It was particularly interesting to see how the unique concept of ‘Fair Go’ was expressed by one group and then taken up and used by another group as a key concept in their group knowledge product. The method also clearly showed which group members did not contribute such as Participant B in Group PHOTBD. This non-participation also supports Porter, Roessner, and Heberger’s (2008) contention that the presence of different disciplines in a group does not necessarily mean that integration of ideas occurs. While the tracking of ideas was generally a good method to consider the integration of ideas, it may not give the whole picture. For example, although the ideas of Participant L were integrated into Group LEGJ’s collective constructions, they did not think that they had contributed to the group work. This raises the question, can an integration of ideas be said to have occurred if one of those individuals whose ideas have been integrated considers that they were not part of a group and their ideas were not heard. This may suggest that in order for cross-disciplinary collaboration to have occurred there would need to be an integration of the members of the cross-disciplinary group in the form of a collective identity as well as an integration of ideas. If this is the case, then future evaluation of cross-disciplinary studies could involve the use of the tracking of ideas method as well as considering the perspectives of the group members in terms of their contribution to the collective and the establishment of a collective identity. In conclusion, it can be said that the tracking of ideas method was useful to consider the integration of ideas and reduce the complexity of the paths of those ideas in the study described in this article. The method was also found to be valuable to determine the origin of ideas across the time and space of the study and was able to show where the ideas came from and where and how they were integrated. The findings of the study described in this article suggest that it might also be a potentially useful tool to consider the integration of ideas in other cross-disciplinary studies. It was also found that the evaluation could be further strengthened through a consideration of the participants’ perspectives of the integration of not only ideas but also the group as a whole and the development of a collective identity. Therefore, further research is needed, to consider the effectiveness of the tracking of ideas methods with methods to consider the integration of the group in cross-disciplinary studies. Conflict of interest statement. None declared. References Allen-Robertson J., Beer D. ( 2010) ‘ Mobile Ideas: Tracking a Concept Through Time and Space’, Mobilities , 5/ 4: 529– 45. Google Scholar CrossRef Search ADS Belcher B. et al. ( 2016) ‘ Defining and Assessing Research Quality in a Transdisciplinary Context’, Research Evaluation , 25: 1– 17. Google Scholar CrossRef Search ADS Boix-Mansilla V. ( 2006) ‘ Assessing Expert Interdisciplinary Work at the Frontier: An Empirical Exploration’, Research Evaluation , 15: 17– 29. Google Scholar CrossRef Search ADS Budd J. ( 2014) A design-based Research Study to Promote Cross-Disciplinary Collaboration Using a Case Form the New Zealand Disability Field . Palmerston North, New Zealand: Massey University. Clark P. ( 2011) ‘ Examining the Interface Between Interprofessional Practice and Education: Lessons Learned from Norway for Promoting Teamwork’, Journal of Interprofessional Care , 25: 26– 32. Google Scholar CrossRef Search ADS PubMed Defila R., Di Giulio A. ( 1999) ‘Evaluating Transdisciplinary Research’, in Panorama , I, pp. 4– 27. Berne, Switzerland: Swiss National Science Foundation Newsletter. Endberg M. ( 2007) ‘ Educating the Workforce for the 21st Century: A Cross-Disciplinary Analysis of the Impact of the Undergraduate Experience on Students' Development of a Pluralistic Orientation’, Research in Higher Education , 48/ 3: 283– 317. Google Scholar CrossRef Search ADS Fiore S. et al. ( 2010) ‘ Towards an Understanding of Metacognition in Teams: Predicting Processes in Complex Collaborative Contexts’, Human Factors , 52/ 2: 203– 24. Google Scholar CrossRef Search ADS PubMed Grigg L. ( 1999), ‘Cross-Disciplinary Research: A Discussion Paper’, Commissioned Report No. 16, Canberra: Australian Research Council. Holmes J., Lehman A., Hade E. ( 2008) ‘ Challenges for Multi-Level Health Disparities Research in a Transdisciplinary Environment’, American Journal of Preventive Medicine , 35/ 2S: S182– 92. Google Scholar CrossRef Search ADS PubMed Hutchby I., Wooffitt R. ( 2008), Conversation Analysis . Cambridge, UK: Polity. Huutoniemi K. ( 2010), ‘Evaluating Interdisciplinary Research’, in Frodeman R. (ed.) The Oxford Handbook of Interdisciplinarity , pp 309– 20, Oxford: Oxford University Press. Leskovec J., Backstrom L., Kleinberg J. ( 2009), ‘Meme-Tracking and the Dynamics of the News Cycle’, in Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pp. 497– 506. Defilia, Paris: ACM. Lowe P., Phillipson J. ( 2009) ‘ Barriers to Research Collaboration Across Disciplines: Scientific Paradigms and Institutional Practices’, Environment and Planning , 41: 1171– 84. Google Scholar CrossRef Search ADS Marzano M., Carss D., Bell S. ( 2006) ‘ Working to Make Interdisciplinarity Work: Investing in Communication and Interpersonal Relationships’, Journal of Agricultural Economics , 57/ 2: 185– 97. Google Scholar CrossRef Search ADS Minister for Disability Issues. ( 2001), The New Zealand Disability Strategy. Making a World of Difference . Wellington, New Zealand: Ministry of Health. Porter A., Roessner D., Heberger A. ( 2008) ‘ How Interdisciplinary is a Given Body of Research?’, Research Evaluation , 17/ 4: 273– 82. Google Scholar CrossRef Search ADS Spaapen J., Dijstelbloem H., Wamelink F. ( 2007), Evaluating Research in Context: A Method for Comprehensive Assessment , 2nd edn. The Hague: Consultative Committee of Sector Councils for Research and Development (COS). Stokols D. et al. ( 2010), ‘Cross-Disciplinary Team Science Initiatives: Research, Training, and Translation’, in Frodeman R., Klein J. T., Mitcham C. (eds) Oxford Handbook of Interdisciplinarity . Oxford: Oxford University Press. Stokols D. ( 2006) ‘ Toward a Science of Transdisciplinary Action Research’, American Journal of Community Psychology , 38: 63– 77. Google Scholar CrossRef Search ADS PubMed Sulkunen P. ( 2008), ‘Social Research and Social Practice in Post-Positivist Society’, in Alasuutari P., Bickman L., Brannen J. (eds) The SAGE Handbook of Social Research Methods , pp. 68– 80. London, UK: SAGE Publications Ltd. Google Scholar CrossRef Search ADS Wagner C. et al. ( 2011) ‘ Approaches to Understanding and Measuring Interdisciplinary Scientific Research (IDR): A Review of the Literature’, Journal of Informetrics , 165: 14– 26. Google Scholar CrossRef Search ADS Willis J. ( 2007) Foundations of Qualitative Research: Interpretive and Critical Approaches . Thousand Oaks: SAGE Publications Ltd. Google Scholar CrossRef Search ADS © The Author 2017. Published by Oxford University Press. All rights reserved. For Permissions, please email: email@example.com
Research Evaluation – Oxford University Press
Published: Jan 1, 2018
It’s your single place to instantly
discover and read the research
that matters to you.
Enjoy affordable access to
over 12 million articles from more than
10,000 peer-reviewed journals.
All for just $49/month
Read as many articles as you need. Full articles with original layout, charts and figures. Read online, from anywhere.
Keep up with your field with Personalized Recommendations and Follow Journals to get automatic updates.
It’s easy to organize your research with our built-in tools.
Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.
All the latest content is available, no embargo periods.
“Hi guys, I cannot tell you how much I love this resource. Incredible. I really believe you've hit the nail on the head with this site in regards to solving the research-purchase issue.”Daniel C.
“Whoa! It’s like Spotify but for academic articles.”@Phil_Robichaud
“I must say, @deepdyve is a fabulous solution to the independent researcher's problem of #access to #information.”@deepthiw
“My last article couldn't be possible without the platform @deepdyve that makes journal papers cheaper.”@JoseServera