Cross-posted at the Hypothesis blog. Updated Dec. 2nd, 2017
Scholars are increasingly being called on – by journal editors, funders, and each other – to “show their work.” Social science is only fully understandable and evaluable if researchers share the data and analysis that underpin their conclusions. Making qualitative social science transparent poses several knotty problems. The Qualitative Data Repository (QDR) and Hypothesis have partnered to meet this challenge by developing a new way to cite, supplement, and share the data underpinning published work – Annotation for Transparent Inquiry (ATI). ATI builds on “active citation,” an earlier approach to achieving transparency in qualitative research pioneered by Moravcsik (e.g., 2010, 2012a, 2012b, 2014a, 2014b, 2014c, 2016).
The Challenge: Achieving Transparency in Qualitative Research
Three aspects of qualitative inquiry complicate transparency. First, qualitative data are multi-format and non-numeric (text, audio, video, pictures). Second, they are analyzed and used to support claims individually or in small groups: each insight drawn from one or a handful of cited sources (e.g., books, archival documents, interview transcripts, newspaper articles, video clips, etc.) serves as a distinct input to the analysis. Third, data, analysis, and conclusions are typically densely interwoven across the span of a book or article.
Quantitative social science does not face the same challenges. Quantitative work involves the computational analysis of numeric data arranged in a matrix and approached as an aggregate body of information. The analysis is typically summarized in tabular form in the text or appendix of published work. To make quantitative publications transparent, scholars share the study dataset (and relevant information about its creation) and supplemental materials such as the code used for analysis.
Making qualitative research similarly transparent requires resolving at least two problems: safely sharing non-numeric data that may come in multiple forms, and placing those data adjacent to the claims and conclusion in the text that they support. Traditionally, qualitative researchers showed at least some of their work in extended footnotes in which they cited the data they relied upon; provided supplemental information about how the data were analyzed and support their points; and provided extracts from those materials. Traditional footnotes are a sub-optimal solution, however. Tight space constraints severely limit what can be included, a problem made even more acute by the increasing use of in-text citation styles. Moreover, even where extracts of the evidence are included in long-form footnotes, there is no systematic way to ensure that available underlying sources are held and curated in ways that make them accessible and useful to scholars.
The Solution: Annotation for Transparent Inquiry (ATI)
Annotation for Transparent Inquiry (ATI), developed through a partnership between QDR and Hypothesis, uses author-generated web annotations on academic publications. Annotations provide information about data analysis, excerpts from data sources, and links to underlying sources, housed in a data repository. The approach harnesses the power of open web annotations, displayed by Hypothesis. Authors annotate their work and deposit underlying data sources with QDR. The repository curates these deposits and converts them into a set of web annotations on the published article, and creates a data project (the aggregate of the underlying data sources). The annotations can be viewed alongside the article using the Hypothesis client, and interested readers can access the underlying data sources archived at QDR.
The new collaboration between Hypothesis and QDR is already bearing fruit. You can see an example of scholarship annotated using ATI here. This is a working paper by Sam Handlin (Department of Political Science, University of Utah), “The Politics of Polarization: Governance and Party System Change in Latin America, 1990-2010,” published by the Kellogg Institute at Notre Dame University. The annotations you see on the side are served by Hypothesis. QDR curated the annotations and provides access to the underlying files, e.g. for this annotation.
Further, working with the Agile Humanities Agency, QDR has developed the function to use the g #annotations:query:<search phrase> at the end of a link to only show a subset of annotations on a given page using the Hypothesis proxy service. QDR uses this feature to present links to the set of annotations that make up the qualitative data underlying an article by limiting the view to annotations created from QDR’s Hypothesis account. You can see this at work in the link to Sam Handlin’s paper above.
Looking ahead, QDR will hold two workshops in late 2017 and early 2018 focused on evaluating and further developing ATI. The workshops are funded by a grant that the Robert Wood Johnson Foundation has awarded to the Qualitative Data Repository to pilot and promulgate ATI and to encourage its use.
Further, QDR and Hypothesis are hoping to address the challenges created a large share of academic literature in the hard and social sciences residing behind a paywall. Access is provided to particular IP-ranges known to be associated with institutions that pay for access. Finding user-friendly solutions to allow viewing annotations on paywalled material is therefore high on our agenda. We hope to draw on our partnership with a wide range of academic publishers in the “Annotating All Knowledge” coalition to develop those solutions. While our immediate interest is motivated by rendering qualitative research transparent, the annotation of academic literature will benefit a much broader scholarly community.
QDR and Hypothesis will also work towards facilitating third-party authentication to the Hypothesis platform. For QDR, the ability to authenticate users against its own user base is critical to limit access to sensitive material that may be stored in annotations, e.g. in the form of interview excerpts.