|
Evaluating Digital Scholarship
NINES/NEH Summer Institutes: 2011-2012

Guidelines for Promotion and Tenure Committees in Judging Digital Work

Adapted from an original authored by Cristina Della Coletta, and revised by Cristina Della Coletta, Jo Anne Harris, Andy Jewell, Meredith Martin, Brad Pasanek, Grant Wythoff and the NINES Summer Institute 2011 Group.

Permalink for this paragraph 0 Digital technology provides ever more sophisticated tools that transform traditional ways of archiving, interpreting, accessing, and disseminating knowledge.  These new tools have occasioned debate on the form and assessment of knowledge production. Therefore we propose guidelines to help evaluate the scholarly contribution of the project.

Permalink for this paragraph 0 The term “guidelines” offers a framework that is less prescriptive than that of principles, standards, and rules, while furnishing a map of shared criteria that orients the process of creation and evaluation of digital scholarship. Flexible guidelines are especially important for scholars working with rapidly transforming digital technologies.

Permalink for this paragraph 0 The following guidelines help guarantee procedural clarity and fairness of assessment for both candidates and evaluators involved in processes of hiring, reappointment, tenure and promotion.   Abiding by the following three procedural parameters ensures informed fairness.

Permalink for this paragraph 0 a) Review and assess the project in the medium in which it was created.

Permalink for this paragraph 0 b) Recognize the intrinsically collaborative nature of digital projects.

Permalink for this paragraph 0 c) Consult specialists in relevant disciplines regarding the various components of the project.

Permalink for this paragraph 0 In order to respond to potentially clashing needs of flexibility and normativity, we recommend that the guidelines be framed as a set of general questions rather than a list of prescriptive statements:

A. Organization, Production, and Integration of Knowledge.

Permalink for this paragraph 0 1) What is the nature of the community that conceptualizes, organizes, and produces this scholarship?

Permalink for this paragraph 0 a. (Who?) Software engineers? Informatics experts? Interface designers? Archivists and librarians? Humanists and social scientists?

Permalink for this paragraph 0 b. (How?) Hierarchical collaboration (“main investigators” and collaborators); “horizontal” collaboration? Other options?

Permalink for this paragraph 0 c. (Why?) What kind of conceptualization of knowledge does this collaborative set up create?

Permalink for this paragraph 0 2) What is the content of the digital project?

Permalink for this paragraph 0 a. What decisions and choices have been made regarding the representation of the materials?

Permalink for this paragraph 0 b. What is included? Left out? Why?

Permalink for this paragraph 0 c. How does the project represent various discourses and rhetorical choices?

Permalink for this paragraph 0 3) If non-digital materials have been digitized, how? Have relevant “best practices and standards” been followed?

Permalink for this paragraph 0 4) How do the elements of the project interact and integrate with one another?

Permalink for this paragraph 0 a. What are the elements of the project? (multimedia options; commentaries, annotations, interpretive essays, multiple interfaces for different audiences, etc.)

Permalink for this paragraph 0 b. Is there a legible intentionality behind the structure of the data?

Permalink for this paragraph 0 c. Does the interface communicate effectively?

Permalink for this paragraph 0 As a result of the choices listed above, does the digital project organize and produce original and innovative knowledge (knowledge as discovery) compared to other formats and other media, in one or more disciplines?

B. Dissemination of Knowledge

Permalink for this paragraph 0 5) What is/are the intended audience(s) for this project?

Permalink for this paragraph 0 a. Is the project readily available for its target audience(s)?

Permalink for this paragraph 0 b. Is it equally effective to reach all targeted audiences (for example, in multilingual/multicultural projects?)

Permalink for this paragraph 0 c. Are there potentially valuable unintended audiences?

Permalink for this paragraph 0 6) What kinds of disciplinary fields and professional communities participate in discussions that involve this project?

Permalink for this paragraph 0 a. Is the project linked to or affiliated with other projects?

Permalink for this paragraph 0 b. Do other projects acknowledge this project?

Permalink for this paragraph 0 7) Who hosts the project?

Permalink for this paragraph 0 a. A University server?

Permalink for this paragraph 0 b. A commercial host? (see section on maintenance)

Permalink for this paragraph 0 c. What’s the rationale for this choice?

Permalink for this paragraph 0 C.  Evaluation and Self-Reflection

Permalink for this paragraph 1 8 ) What funding and grants did the project obtain?

Permalink for this paragraph 0 9) Are there formal and/or informal processes of self-evaluation built into the project?

Permalink for this paragraph 0 a. What feedback/consultation system has been implemented (outside consultants?)

Permalink for this paragraph 0 b. Have need-analysis, interface design, and usability, been discussed, documented, revised in the course of the project development?

Permalink for this paragraph 0 c. Have design experiments/tests have been conducted?

Permalink for this paragraph 0 d. Which ones?

Permalink for this paragraph 0 10) Has the project been submitted for peer-review and review?

Permalink for this paragraph 0 a. In which venues?

Permalink for this paragraph 0 b. How often?

Permalink for this paragraph 1 11) Has the digital project been presented and demonstrated at conferences, symposia, and invited presentations? Which ones?

Permalink for this paragraph 1 12) Have papers or studies on the digital project been published?

Permalink for this paragraph 0 a. Where?

Permalink for this paragraph 0 b. By whom?

Permalink for this paragraph 0 13) Has the candidate submitted a list of qualified potential evaluators who will be able to understand the scope and significance of the project?

Permalink for this paragraph 0 14) Does the project include an explicit reflective essay that documents all these steps?

Permalink for this paragraph 0 As a result of the choices listed above, does the digital project have the potential of being part of a wider and more sophisticated evaluation and self-evaluation system compared to those available in other formats and other media?

D. Maintenance, Sustainability, and Future Plans

Permalink for this paragraph 0 15) What is the project’s sustainability plan?

Permalink for this paragraph 0 a) Where is the project hosted?

Permalink for this paragraph 0 b) Is the server hosting the project likely to be adequately maintained over time?

Permalink for this paragraph 0 c) What are the accessibility plans in the long run?

Permalink for this paragraph 0 16) Has the technical set up for the project been documented and implemented in ways that promote sustainability and accessibility?

Permalink for this paragraph 0 a. Can new materials be added effectively?

Permalink for this paragraph 0 b. Can problems be identified and fixed easily?

Permalink for this paragraph 0 c. Can it be easily migrated to new platforms?

Permalink for this paragraph 0 d. Is the project’s data as open and accessible as possible?

Permalink for this paragraph 0 17) What are the plans for long-term testing, evaluation, future development, and transformation of the project?

Permalink for this paragraph 0 18) Based on the current trajectory and long-term potential of the project, what is its future?

Permalink for this paragraph 0 As a result of the choices listed above, does the digital project have the potential of evolving and improving over time in ways that are not possible or available in other formats and other media?

Permalink for this paragraph 0 We recommend the following resources:

Permalink for this paragraph 0 • Modern Language Association, Guidelines for Editors of Scholarly Editions

Permalink for this paragraph 0 • University of Nebraska-Lincoln (CDRH), Recommendations for Digital Humanities Projects

page 3