TY - JOUR
AU - Szabo, Victoria
AB - Abstract Evaluating scholarly work in extended reality for the humanities requires understanding of both scientific and humanistic standards for the work. This article shows how humanities values and interests intersect, and at times conflict, with norms developed for the medium in scientific contexts, and reflects on the risks humanists take in exploring new approaches to scholarly practice. The discussion draws upon the work of a recent Digital Humanities (DH) institute that reflected on how the Extended Reality (XR) goals of immersiveness, completeness, and realism compete with valuing the visible traces of ambiguity, uncertainly, incompleteness, and foregrounding argument and documentation. It considers approaches to creating evaluation guidelines for XR as an extension of existing DH guidelines published by scholarly societies, and suggests ways in which multiple of these approaches to evaluation might converge in, on the one hand, multi-modal and reflective XR documentation practices that include written supplements, and on the other, in broadening the scope of what kinds of creative and storytelling work fit into academic evaluation contexts in the humanities. Evaluating scholarly work in Extended Reality (XR) poses a challenge to conventional academic practice in ways that are both familiar to Digital Humanities (DH) practitioners, and which amplify some of the biggest challenges to its wider acceptance as scholarly work. This talk outlines some of the Evaluation Best Practices elaborated at the Virtual and Augmented Reality Digital Humanities Institute (V/AR-DHI) sponsored by the US National Endowment for the Humanities from 2018 to 2020 and hosted at Duke University (Scholarly Communications Institute, 2014; Virtual and Augmented Reality Digital Humanities Institute, 2019). At the Institute, in addition to sharing projects and best practices in the field, we identified a collective goal to make our work more visible and legible within our institutions as academic scholarship. We plan to augment the general guidelines provided by scholarly organizations like the Modern Language Association, College Art Association, and American Historical Association around digital scholarship to account for the specific challenges XR offers, drawing also upon our own and others’ discipline-specific work to enrich the discussion. Over the course of the next year, we will continue to develop our ideas and to share them in conference settings and online, with the goal of a standards document with a rich and extensive set of examples included. This article outlines some initial findings and challenges for the group as we move forward. The initial NEH V/AR-DHI Institute group participants were selected for the range of their interests, as well as their experience in the field. Participants ranged from graduate and early career scholars to full professors, and from a variety of disciplines. We grouped their projects into four main categories of XR intervention as follows: historical reconstruction and representation; digital storytelling and pedagogy; data visualization; and media arts. Because our work was supported by a national grant, our Institute participants came from US institutions, though several have international connections from their own personal histories and/or their academic partnerships.1 The facilitator group, led by Projector Director Victoria Szabo and Co-Director Philip Stern, was equally diverse, and included people from a variety of backgrounds and roles at Duke and the University of Padua. These participants included faculty, staff, and students from various US institutions, some focused more on the humanities, some on digital technologies, and some who would resist those characterizations as being unnecessarily reductive.2 Facilitating the VARDHI Institute and its subsequent conversations has highlighted the extent to which thinking about XR involves thinking across a wide range of use cases and value systems, but also with a lot of transdisciplinary common ground, with shared interests in strategies of representation, interpretation, interface design, aesthetics, and embedded critique. Our participants have come from Art History, Archeology, History, Journalism, Languages, Literature, Media Arts, Music as well as Scientific Visualization, Computer Science, and Architectural Engineering. Their application interests include historical reconstruction, cultural heritage, digital storytelling, language instruction, representation of fictive spaces, media performance, and data visualization. Content creation includes 3D models of buildings, objects, and landscapes, 360 videos, abstract data visualizations, and hybrid media installations. We work in immersive CAVE spaces, high- and low-end headsets, cardboards and mobile devices, and on 2D screens and the web, and with Internet of Things devices and haptics. It became clear from our time together that what we are talking about XR is not only a varied landscape in terms of disciplines, subject areas, and methods, but also in how the final outputs are experienced, distributed, and understood in terms of corresponding research projects. One concern that struck many of us throughout the institute—a subject we returned to again and again, no matter the ostensible topic of discussion—was the extent to which participants felt anxiety about doing this kind of work, what it would ‘count’ for, and how it would be understood in an institutional context. Is it worth the effort, the risk? This is part of why we decided to focus one of our ‘deliverables’ on thinking through the evaluation process, and by extension, what constitutes the good in the field. Even with all the variety of the emerging XR world, we recognize a need for specificity that can be shared with our campus interlocutors and beyond. This will help us create and teach with XR effectively, conduct robust peer review, perform institutional evaluation, and intervene in broader cultural conversations around the medium and its impact. The seeds for the V/AR-DHI evaluation standards project were sown in a prior gathering, as noted earlier. A group at the Scholarly Communications Institute held in Chapel Hill, NC in 2014 (also a diverse group) came together to discuss issues around cross-disciplinary standards for Scientific Visualization.3 That group, organized by Paul Fyfe of North Carolina State University, converged around three key meta-principles for evaluation of scholarly visualization projects: ‘Integrity, Integration, and Impact’. ‘Integrity’ relates to the ability of the material to interpret and transmit source materials, and the extent to which it surfaces an original argument. ‘Interaction’ focuses on the extent to which the interactive elements aid in interpretation and discovery. ‘Impact’ refers to the work’s originality, the extent to which it affects the conversation in the disciplinary field, as well as for the broader visualization community. As we have learned at SCI, and which our experience with the V/AR-DHI Institute confirms, the necessity of dialog between scientific and humanistic communities becomes especially tangible within the collaborative contexts where XR work is being done. Our hope is that by surfacing a range of evaluation measures that draw upon the values of both communities, that we can further a conversation around what XR can do for us in the humanities, and what we in the humanities can do for it as an emerging media form. Building upon the baseline established at SCI, we have identified a range of additional XR-specific evaluation measures to consider. These include user interface and experience design, graphic design and accessibility, response time, accuracy, the presence or absence of negative effects (motion sickness), biometric measures of ‘presence’ in virtual spaces, memorability, comparative analysis with other forms, emotional impact, and more. But to what extent can and should practitioners in the humanities using XR be judged according to standards developed in scientific fields, or even in digital media and design? Julia Flanders talks about the ‘productive unease of 21st c. digital scholarship’ as centered in part around a new awareness of the process of representation in Defining Digital Humanities (Flanders 2009). Perhaps, nowhere is this concern more visibly a challenge than in the translation into technologies that push into the dimensional, immersive, and in some contexts, increasingly photo-realistic. Aside from affirming the importance of cross-disciplinary collaboration, the V/AR-DHI experience of sharing our work and objectives as case study examples also highlighted where the humanities values butted up against evaluation standards that focused on the production of seamless and convincing ‘realities’. Our experience highlighted the challenge of simultaneously living up to the expectations of multiple disciplines and focusing on our own research interests and values. If our goals were different, should not some of our measures also vary? With this in mind, some of the ways in which we have striven to think through evaluation standards attempt to push back against the ‘worst’ or most seductive tendencies of the medium—and those most likely to draw criticism from those not actively working in the area. For XR as a humanities endeavor, therefore, some complementary evaluation standards address the extent to which the project also pushes against immersive flow of the medium. We might look for features which: signal ambiguity and conflicts, represent uncertainty and change over time, highlight the subjectivity and position of the observer, provide access to underlying data, offer visible engagement with a scholarly argument; provide contextualization and documentation, and demonstrate adherence to established standards. At the same time as we seek to acknowledge humanistic approaches to knowledge production in how we approach XR, we also do not want to lose what is most powerful about these media forms: their ability to engage, and to produce immersive experiences and affective and visceral responses. To the extent that maintaining critical distance and experiencing XR can be antithetical objectives, how then do we also pay attention to the most powerful aspects of the medium? This question leads into investigating the standards of boundary disciplines, like participatory journalism, documentary studies, game studies, performance studies, or exploring the contradictions inherent in creative non-fiction, biography, or adaptation. Where does creative expression intersect with the critical? The technical? With these ideas in mind, for example, one new project I have begun working on with a colleague is Visualizing Lovecraft, looking to the fictive places, spaces, and structures of his fiction as a way into understanding both its underlying power and engagement with real and imagined places, and the potential of immersive media construction as a different kind of literary criticism engaged with complex notions of space and time. As the VARDHI group continues its exploration of XR evaluation standards this coming year, we are led back to the questions of value, of risk, in engaging with these media forms in our research, teaching, and outreach. But another risk also exists, the risk that by not engaging with this emerging media force, we will lose sight of not only its scholarly potential, but also the nature of its cultural impact, and our ability to influence its reception and use. As with other kinds of DH, embrace of XR will require defining appropriate, and at times bespoke, standards for evaluation, as well as engagement with collaborative authorship practices. It will likely also require a fundamentally hybrid approach to publication, a need that continues to build with each new digital humanities method and product. We will continue to look to the both-and. Text is not going away, but being supplemented, complemented, perhaps sidelined. But text too has its affordances and implicit values, as doing this kind of work increasingly brings to our attention through its differences.4 And perhaps here is where the experience of making becomes most central. At times what is most important is not the final product, but the process of creation itself—the study, the abstraction, the conversation, and the transformation that leads us to know our ‘materials’ more fully and enables us to examine their movements and meanings more intimately. We know this from working with our students on DH projects; it can also be true for us as scholars as well. In many cases, as we have seen, the boundary between teaching and research blurs as students become part of research teams inside and outside of class settings. This elision is also a challenge for evaluation standards, but important for the design and experience of higher education itself. Similarly, the raw expenses of exploring XR—in terms of time, expertise, equipment, and professional risk—also necessitate that the boutique exploration, the incomplete implementation, and the suggestive experiment should be acknowledged as valuable contributions alongside flashier, more polished projects that would require substantial material investments to be adopted for wider use. For reasons of both media experience and technological obsolescence, it is almost inevitable that translation from the original media forms of XR will need to happen. Questions of equity around interface and socio-economic access also demand attention here. This is also true for other forms of DH, but it is especially true of XR, which operates as a kind of performance, with all the ephemerality and situatedness that implies. Of course, strategies for rezzing such installations are also important areas of research, as our media librarians and game studies, museum studies, and e-lit colleagues well know. Scholarly XR will continue to need a text (and image and audio and video) track, whether as a complementary work or integrated into the project itself. As I have been discussing, XR authorship is taking place in a wide range of humanities disciplines, for various purposes, and with a wide range of interdisciplinary partners, tools, and methods in play. As we develop case study examples out of our own collective experience and research, we will also be reaching out to the broader community for comment. If the evaluation methods we devise together can both address the real issues XR raises—and also embrace its immersive and affective strengths, as I have explored elsewhere5—then our hope is that we can work together towards a critical approach to XR that helps establish it as a viable approach to digital humanities scholarship at the same time as it becomes an increasingly important object of critical inquiry. Funding This work was supported by the Office of the Digital Humanities at the National Endowment for the Humanities, USA [HT-256969-17]. Footnotes 1 Participants included Shane Denson, Tahir Hemphill, Mona Kasra, Micki Kaufman, Fotini Kondyli, Lynne Ramey, Kathleen M. Ryan, Margaret Schedel, Filippo Screpanti, John Shelton, and Justin Underhill. 2 Facilitators included Victoria Szabo (PD), Philip Stern (Co-PD), Paolo Borin, Hannah Jacobs, Matthew Kenney, Regis Kopper, Aaron Kutnick, Sylvia K. Miller, Cosimo Monteleone, Mark Olson, Angelina Liu, Amanda Starling Gould, Edward Triplett, and David Zielinski. 3 Evaluating Scholarly Visualization Projects was a team project at the Scholarly Communications Institute in 2014. Team members included Paul Fyfe (Lead), Victoria Szabo, Markus Wust, David Zielinski, Angela Zoss, Anna Bentkowska-Kafel. 4 Examples, such as Scalar, JITP, arts.code, Hyperrhiz, the work of Media Commons, IDAH, Kairos, Leonardo w/MIT, and ARTEKA, in addition to the efforts of various other university press partnerships, such as Manifold at Minnesota are all possible platforms. 5 I recently discussed the potential of augmented reality for digital heritage in terms of ‘comprehension’ and ‘apprehension’ for the Routledge Companion to Media Studies and Digital Humanities edited by Jentery Sayers (Szabo, 2018). References Flanders J. ( 2009 ). The productive unease of 21st-century Digital Scholarship . Digital Humanities Quarterly , 3 ( 3 ). http://www.digitalhumanities.org/dhq/vol/3/3/000055/000055.html (accessed 4 November 2019) Google Scholar OpenURL Placeholder Text WorldCat Scholarly Communications Institute ( 2014 ). Online. https://trianglesci.org/2014-institute/ (accessed 4 November 2019). Szabo V. ( 2018 ). Apprehending the past: augmented reality, archives, and cultural memory. In Sayers J (ed.), The Routledge Companion to Media Studies and Digital Humanities . New York : Routledge , pp. 372 – 83 . Google Scholar Crossref Search ADS Google Preview WorldCat COPAC Virtual and Augmented Reality Digital Humanities Institute ( 2019 ). Online.http://vardhi.org (accessed 4 November 2019). © The Author(s) 2021. Published by Oxford University Press on behalf of EADH. All rights reserved. For permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
TI - Evaluating XR: Standards for an emerging DH medium
JO - Digital Scholarship in the Humanities
DO - 10.1093/llc/fqab037
DA - 2021-11-05
UR - https://www.deepdyve.com/lp/oxford-university-press/evaluating-xr-standards-for-an-emerging-dh-medium-pjftxEp3PC
SP - ii273
EP - ii276
VL - 36
IS - Supplement_2
DP - DeepDyve
ER -