Digital Archives and Special Collections

“Can LLMs Categorize the Specialized Documents from Web Archives in a Better Way?”

The explosive growth of web archives presents a significant challenge: manually curating specialized document collections from this vast data. Existing approaches rely on supervised techniques, but recent advancements in Large Language Models (LLMs) offer new possibilities for automating collection creation. Large Language Models (LLMs) are demonstrating impressive performance on various tasks even without fine-tuning. This paper investigates the effectiveness of prompt design in achieving results comparable to fine-tuned models. We explore different prompting techniques for collecting specialized documents from web archives like UNT.edu, Michigan.gov, and Texas.gov. We then analyze the performance of LLMs under various prompt configurations. Our findings highlight the significant impact of incorporating task descriptions within prompts. Additionally, including the document type as justification for the search scope leads to demonstrably better results. This research suggests that well-crafted prompts can unlock the potential of LLMs for specialized tasks, potentially reducing reliance on resource-intensive fine-tuning. This research paves the way for automating specialized collection creation using LLMs and prompt engineering.

https://dl.acm.org/doi/10.1145/3677389.3702591

“Developing Practices for FAIR and Linked Data in Heritage Science”

Heritage Science has a lot to gain from the Open Science movement but faces major challenges due to the interdisciplinary nature of the field, as a vast array of technological and scientific methods can be applied to any imaginable material. Historical and cultural contexts are as significant as the methods and material properties, which is something the scientific templates for research data management rarely take into account. While the FAIR data principles are a good foundation, they do not offer enough practical help to researchers facing increasing demands from funders and collaborators. In order to identify the issues and needs that arise “on the ground floor”, the staff at the Heritage Laboratory at the Swedish National Heritage Board took part in a series of workshops with case studies. The results were used to develop guides for good data practices and a list of recommended online vocabularies for standardised descriptions, necessary for findable and interoperable data. However, the project also identified areas where there is a lack of useful vocabularies and the consequences this could have for discoverability of heritage studies on materials from areas of the world that have historically been marginalised by Western culture. If Heritage Science as a global field of study is to reach its full potential this must be addressed.

https://doi.org/10.1038/s40494-025-01598-x

“Building Trustworthy AI Solutions: Integrating Artificial Intelligence Literacy into Records Management and Archival Systems”

This paper explores the essential role of Artificial Intelligence (AI) competencies and literacy in the fields of records management and archival practices, within the framework of the InterPARES Trust AI project. . . . The study employs two complementary approaches: (1) a detailed competency framework developed through literature reviews, interviews with archival professionals who have applied AI to the processing of records, and validation workshops with practitioners; and (2) a comprehensive AI literacy framework derived from multiple case studies and theoretical discussions. . . . Findings indicate that archival professionals can leverage AI in their work practices by acquiring basic AI literacy, practical AI skills, data-related skills, tool-testing and evaluation, adaptation of AI to their workflows, and by actively engaging in collaborative projects with information technology (IT) developers.

https://doi.org/10.48550/arXiv.2307.14852

“Datafication and Cultural Heritage Collections Data Infrastructures: Critical Perspectives on Documentation, Cataloguing and Data-sharing in Cultural Heritage Institutions”

The role of cultural heritage collections within the research ecosystem is rapidly changing. From often-passive primary source or reference point for humanities research, cultural heritage collections are now becoming integral part of large-scale interdisciplinary inquiries using computational-driven methods and tools. This new status for cultural heritage collections, in the ‘collections-as-data’ era, would not be possible without foundational work that was and is still going on ‘behind the scenes’ in cultural heritage institutions through cataloguing, documentation and curation of cultural heritage records. This article assesses the landscape for cultural heritage collections data infrastructure in the UK through an empirical and critical perspective, presenting insights on the infrastructure that cultural heritage organisations use to record and manage their collections, exploring the range of systems being used, the levels of complexity or ease at which collections data can be accessed, and the shape of interactions between software suppliers, cultural heritage organisations, and third-party partners. The paper goes on to include a critical analysis of the findings based on the sector’s approach to ‘3s’, that is standards, skill sets and scale, and how that applies to different cultural heritage organisations throughout the data lifecycle, from data creation, stewardship to sharing and re-using.

https://doi.org/10.5334/johd.277

“Copyright and Licencing for Cultural Heritage Collections as Data”

Cultural Heritage (CH) institutions have been exploring innovative ways to publish digital collections to facilitate reuse, through initiatives like Collections as data and the International GLAM Labs Community. When making a digital collection available for computational use, it is crucial to have reusable and machine-readable open licences and copyright terms. While existing studies address copyright for digital collections, this study focuses specifically on the unique requirements of collections as data. This research highlights both the legal and technical aspects of copyright concerning collections as data. It discusses permissible uses of copyrighted collections, emphasising the need for interoperable, machine-readable licences and open licences. By reviewing current literature and examples, this study presents best practices and examples to help CH institutions better navigate copyright and licencing issues, ultimately enhancing their ability to convert their content into collections as data for computational research.

https://doi.org/10.5334/johd.263

“Conceptualizing Aggregate-Level Description in Web Archives”

Web archives collections are often excluded from archival science discussions, and their description instead focuses on bibliographic approaches to item-level metadata. This article argues that web archives are best understood using approaches of archival description, focusing on a case study of the Danish Netarchive, a long-running national web archive. By capturing and preserving web sites for the purposes of legal deposit, the Netarchive creates and maintains historical records of the web. Examining the Netarchive’s systems and activities through the lens of archival representation, this article develops a typology of representational artifacts that support this work, including the use of database entities, wiki documentation, classification and management via Jira issues, and codes, identifiers, and structures embedded in network protocols themselves. The analysis considers how meaningful aggregations can be understood via these representational schemes, systems and architectures, and how the nature of born-networked records challenges concepts of singular, hierarchical orderings of records aggregations. The closing discussion proposes new modes of description that address these multiple interconnected systems, and raises questions about what this might mean for aggregate-level description in the context of digital and born-networked records more broadly.

https://doi.org/10.5334/johd.265

"Starting with the Digital Doesn’t Make it Easier: Developing Transparent Born Digital Acquisition Policies for Archives "

As organizations continue to overwhelmingly abandon all forms of paper-based record keeping, libraries are still adapting to increased offers of born digital archival donations. Simple misunderstandings or disconnects between the units facilitating donations and maintaining born-digital collections creates pain-points for donor relations and can result in a lack of transparency over how their records may be processed. To facilitate better donor transparency and cross-area collaboration over born digital records, Special Collections and archives need comprehensive policies and shifts in training and collaboration paradigms. This paper analyses the intersections of born digital archiving, collection development polices, donor relations, human-supported AI tools, and digital records education within American academic libraries to propose a functional toolkit for born digital acquisitions. Unrealistic expectations of collection processing, retention, growth, and publication onto openly accessible platforms can quickly overwhelm a libraries’ digital collections’ team due to size, need for digital forensics work, copyright limitations, or other capacity-related issues. Intertwined within this discussion is an additional discourse over the need to carefully curate our digital spaces not only for practical cost reasons, but due to the environmental costs of massive data storage solutions. Through an analysis of the elements stated above, the paper will reflect on the need to integrate born digital materials into archival acquisition procedures and provide practical solutions to meet this need.

https://tinyurl.com/8r3ucesb

"Text Analysis of Archival Finding Aids; Collection Scoping and Beyond"

In this study, we examine the suitability of text analysis as a method for analyzing collection scope strengths across a repository’s physical archival holdings. We apply a tool for text analysis called Leximancer to analyze a corpus of archival finding aids to explore topical coverage. Leximancer results were highly aligned with the baseline subject heading analysis that we performed, but the concepts, themes, and co-occurring topic pairs surfaced by Leximancer suggest areas of collection strength and potential focus for new acquisitions. We discuss the potential applications of text analysis for internal library use including collection development, as well as potential implications for wider description, discovery, and access. Text analysis can accurately surface topical strengths and directly lead to insights that can inform future acquisition decisions and archival collection development policies.

https://tinyurl.com/mr45f8e7

"Capturing Captions: Using AI to Identify and Analyse Image Captions in a Large Dataset of Historical Book Illustrations"

This article outlines how AI methods can be used to identify image captions in a large dataset of digitised historical book illustrations. This dataset includes over a million images from 68,000 books published between the eighteenth and early twentieth centuries, covering works of literature, history, geography, and philosophy. The article has two primary objectives. First, it suggests the added value of captions in making digitized illustrations more searchable by picture content in online archives. To further this objective, we describe the methods we have used to identify captions, which can effectively be re-purposed and applied in different contexts. Second, we suggest how this research leads to new understandings of the semantics and significance of the captions of historical book illustrations. The findings discussed here mark a critical intervention in the fields of digital humanities, book history, and illustration studies.

https://tinyurl.com/bdvjespp

"AI and Medical Images: Addressing Ethical Challenges to Provide Responsible Access to Historical Medical Illustrations"

This article examines the ethical considerations and broader issues around access to digitised historical medical images. These illustrations and, later, photographs are often extremely sensitive, representing disability, disease, gender, and race in potentially harmful and problematic ways. In particular, the original metadata for such images can include demeaning and sometimes racist terms. Some of these images show sexually explicit and violent content, as well as content that was obtained without informed consent. Hiding these sensitive images can be tempting, and yet, archives are meant to be used, not locked away. Through a series of interviews with 10 archivists, librarians, and researchers based in the UK and US, the authors show that improved access to medical illustrations is essential to produce new knowledge in the humanities and medical research, as well as to bridge the gap between historical and modern understandings of the human body. Improving access to medical illustration can also help to address the "gender data gap", which has acquired mainstream visibility thanks to the work of activists such as Caroline Criado-Perez, the author of Invisible Women: Data Bias in a World Designed for Men.

https://tinyurl.com/3jek7ey4

Paywall: "Social Media Archiving in Practice: A Troubled Landscape in Review"

This article looks at several notable examples of major social media data loss that have already taken place in recent history and examines one of the largest institutional attempts to preserve social media thus far in the Library of Congress’ Twitter Archive. By exploring some of the key challenges that arose during this attempt which ultimately grounded the project, this article aims to better understand what continues to keep the practice of social media archiving at bay, and what large scale changes might be necessary to make any further progress in the field.

https://doi.org/10.1080/0361526X.2024.2367405

"Improving Accessibility of Digitization Outputs: Eodopen Project Research Findings"

In this contribution, the authors present the results of a trial implementation among EODOPEN partners regarding their digitization workflows, used delivery file formats and the resulting quality of OCR results, depending on the type of digitization output file format. It was shown that partners using the OCR tool ABBYY FineReader Professional and producing scanning outputs in tagged PDF and PDF/UA formats achieved better results according to set criteria.

https://doi.org/10.1108/DLP-09-2023-0080

"The Puzzle of Large-Scale Digital Collections: Have We Reached an Inflection Point?"

Shared Collections allows institutions either to have JSTOR harvest their digital collections of documents, photos, and other special collections from a local Digital Asset Management System, or to create and share those same collections through JSTOR’s collection management tool. . . . While Shared Collections appears to represent a significant advance, the jury will be out for some time. The fundamental issues facing DPLA and Shared Collections are simply difficult, and the struggles with them have little or nothing to do with the skills or intentions of the capable people of both organizations. It is both a tough economic problem and an outcome of what we might call "rugged individualism in heritage collections": while shared descriptive efforts have been in place for books for more than a century, many standards for heritage collections have emerged since 2000. It’s a symptom of under-investment in cultural heritage in the United States.

https://rb.gy/597nkq

"Artificial Intelligence’s Role in Digitally Preserving Historic Archives"

The term "Artificial Intelligence" (AI) is increasingly permeating public consciousness as it has gained more popularity in recent years, especially within the landscape of academia and libraries. AI in libraries has been a trending subject of interest for some time, as within the library there are numerous departments that serve a role in collectively contributing to the library’s mission. Consequently, it is imperative to consider AI’s influence on the digital preservation of historic documents. This paper delves into the historical evolution of preservation methods driven by technological advancements as, throughout history, libraries, archives, and museums have grappled with the challenge of preserving historical collections, while many of the traditional preservation methods are costly and involve a lot of manual (human) effort. AI being the catalyst for transformation could change this reality and perhaps redefine the process of preservation; thus, this paper explores the emerging trend of incorporating AI technology into preservation practices and provides predictions regarding the transformative role of Artificial Intelligence in preservation for the future. With that in mind, this paper addresses the following questions: could AI be what changes or creates a paradigm shift in how preservation is done?; and could it be the thing that will change the way history is safeguarded?

https://doi.org/10.1515/pdtc-2023-0050

Paywall: "Changes in Digital Collections and Their Metadata: A Longitudinal Study of UIUC Digital Library"

This article showcases the evolution of digital collections and their metadata at the University of Illinois Urbana-Champaign (UIUC) Library in the last 20 years. It discusses the growth of its collections and their characteristics, examines historical changes in the use of metadata elements, and explores responses to the changing nature of digitized and born-digital materials. Based on a large-scale data analysis of the digital collections and their metadata housed in UIUC Digital Library, the paper also examines the challenges and opportunities of the curation and management of digital collections and digital libraries in the future.

https://doi.org/10.1080/19386389.2024.2338015

Paywall: "For the People: How We Make Online LAM [Library, Archive, and Museum] Collections More Democratized"

The article critiques the misconception that online collections democratize artifact information for public consumption and explores the ways in which LAM institutions fall short of living up to their democratic ideals when it comes to digital collections projects. Inspired by others with similar critiques, the authors discuss how LAM institutions can better fulfill the ideal of accessible and equitable access to their collections. The article emphasizes the importance of five areas of digital collections projects: system design, metadata practices, digitization selection and prioritization, labor, and user participation and engagement.

https://doi.org/10.1080/1941126X.2024.2306042

2024 Fedora Technology Assessment Report

The Fedora Program Team, in collaboration with the Technology Working Group, designed a project to understand the specific Fedora-related priorities of using institutions, along with the capacity and available resources of both individuals and institutions to contribute to the Fedora community between 2024 and 2026. They collaborated with the Research and Innovation Division at Lyrasis to survey Fedora users. Responses were collected between November 2023 and January 31, 2024, and analyzed by Leigh A. Grinstead, Senior Digital Services Consultant from Lyrasis, an independent, nonprofit, research group.

https://tinyurl.com/2s4b4rec

"Responsible AI at the Vanderbilt Television News Archive: A Case Study"

We provide an overview of the use of machine-learning and artificial intelligence at the Vanderbilt Television News Archive (VTNA). After surveying our major initiatives to date, which include the full transcription of the collection using a custom language model deployed on Amazon Web Services (AWS), we address some ethical considerations we encountered, including the possibility of staff downsizing and misidentification of individuals in news recordings.

https://doi.org/10.7191/jeslib.805

"Islandora for Archival Access and Discovery"

This article is a case study describing the implementation of Islandora 2 to create a public online portal for the discovery, access, and use of archives and special collections materials at the University of Nevada, Las Vegas. The authors will explain how the goal of providing users with a unified point of access across diverse data (including finding aids, digital objects, and agents) led to the selection of Islandora 2 and they will discuss the benefits and challenges of using this open source software. They will describe the various steps of implementation, including custom development, migration from CONTENTdm, integration with ArchivesSpace, and developing new skills and workflows to use Islandora most effectively. As hindsight always provides additional perspective, the case study will also offer reflection on lessons learned since the launch, insights on open-source repository sustainability, and priorities for future development.

https://journal.code4lib.org/articles/17929

Partial Paywall: The Nordic Model of Digital Archiving

Bringing together contributions from practitioners and academics to offer a range of international case studies, this book offers practical solutions for archivists in terms of governance, technologies and processes. It highlights and analyses the cornerstones of the Nordic model of archiving: reliance on standards; powerful regulatory instruments — especially in public sector archiving, including legislation; and collaboration between archivists and government agencies, and among different tiers of central and local government.

One of four open access chapters: "The Nordic Model of Digital Archiving."

https://doi.org/10.4324/9781003325406

"E-preservation of Old and Rare Books: A Structured Approach for Creating a Digital Collection "

Antique books, old and rare documents are fragile and vulnerable to different hazards. Preserving them for an extended period is a real challenge. From ancient times people started expressing their knowledge by writing and keeping records and subsequently started collecting and storing these at later ages as antique materials. These can be seen in different museums, libraries, archives, individual households, and other places all over the world. Preserving and conserving these antique, old and rare books, documents etc. in good condition is a challenge for librarians, conservators, preservation administrators or persons associated with storing these. In this paper, details of the digital preservation of such a collection available in the Directorate of Historical and Antiquarian Studies (DHAS), Guwahati, Assam, India, are discussed. DHAS is a Government of Assam wing and is mainly mandated to collect, preserve and research historical and antiquarian resources. The collection of DHAS is one of the oldest collections and has been serving as a study and research centre in Assam since 1928. A special drive has been taken for the digital preservation of an identified part of the collection, with grant support from the National Archive of India. This paper discusses the entire project process starting from the project proposal formulation to the structuring of the digital collection. The paper sequentially discusses the different steps of the entire work of digitization of a collection of 241 old and rare books from the main collection of DHAS.

http://www.ijdc.net/article/view/855

"Finding the Right Platform: A Crosswalk of Academy-Owned and Open-Source Digital Publishing Platforms"

A key responsibility for many library publishers is to collaborate with authors to determine the best mechanisms for sharing and publishing research. Librarians are often asked to assist with a wide range of research outputs and publication types, including eBooks, digital humanities (DH) projects, scholarly journals, archival and thematic collections, and community projects. These projects can exist on a variety of platforms both for profit and academy owned. Additionally, over the past decade, more and more academy owned platforms have been created to support both library publishing programs. Library publishers who wish to emphasize open access and open-source publishing can feel overwhelmed by the proliferation of available academy-owned or -affiliated publishing platforms. For many of these platforms, documentation exists but can be difficult to locate and interpret. While experienced users can usually find and evaluate the available resources for a particular platform, this kind of documentation is often less useful to authors and librarians who are just starting a new publishing project and want to determine if a given platform will work for them. Because of the challenges involved in identifying and evaluating the various platforms, we created this comparative crosswalk to help library publishers (and potentially authors) determine which platforms are right for their services and authors’ needs.

https://hcommons.org/deposits/item/hc:59231/

"Creating a Full Multitenant Back End User Experience in Omeka S with the Teams Module"

When Omeka S appeared as a beta release in 2016, it offered the opportunity for researchers or larger organizations to publish multiple Omeka sites from the same installation. Multisite functionality was and continues to be a major advance for what had become the premiere platform for scholarly digital exhibits produced by libraries, museums, researchers, and students. However, while geared to larger institutional contexts, Omeka S poses some user experience challenges on the back end for larger organizations with numerous users creating different sites. These challenges include a "cluttered" effect for many users seeing resources they do not need to access and data integrity challenges due to the possibility of users editing resources that other users need in their current state. The University of Illinois Library, drawing on two local use cases as well as two additional external use cases, developed the Teams module to address these challenges. This article describes the needs leading to the decision to create the module, the project requirement gathering process, and the implementation and ongoing development of Teams. The module and findings are likely to be of interest to other institutions adopting Omeka S but also, more generally, to libraries seeking to contribute successfully to larger open-source initiatives.

https://journal.code4lib.org/articles/17389

"Data Sharing Implementation in Top 10 Ophthalmology Journals in 2021"

Background/Aims: Deidentified individual participant data (IPD) sharing has been implemented in the International Committee of Medical Journal Editors journals since 2017. However, there were some published clinical trials that did not follow the new implemented policy. This study examines the number of clinical trials that endorsed IPD sharing policy among top ophthalmology journals.

Method: All published original articles in 2021 in 10 highest-ranking ophthalmology journals according to the 2020 journal impact factor were included. Clinical trials were determined by the WHO definition of clinical trials. Each article was then thoroughly searched for the IPD sharing statement either in the manuscript or in the clinical trial registry. We collected the number of published clinical trials that implemented IPD sharing policy as our primary outcome.

Results: 1852 published articles in top 10 ophthalmology journals were identified, and 9.45% were clinical trials. Of these clinical trials, 44% had clinical trial registrations and 49.14% declared IPD sharing statements. Only 42 (48.83%) clinical trials were willing to share IPD, and 5 (10.21%) of these share IPD via an online repository platform. In terms of sharing period, 37 clinical trials were willing to share right after the publication and only 2 showed the ending of sharing period.

Conclusion: This report shows that the number of clinical trials in top ophthalmology journals that endorsed the IPD sharing policy and the number of registrations is lower than half even though the policy has been implemented for several years. Future updates are necessary as policy evolves.

http://dx.doi.org/10.1136/bmjophth-2023-001276

Paywall: "Human-AI Interaction for Exploratory Search & Recommender Systems with Application to Cultural Heritage "

This dissertation introduces three primary contributions through publicly deployed sys- tems and datasets. First, we demonstrate how the construction of large-scale cultural heritage datasets using machine learning can answer interdisciplinary questions in library & information science and the humanities (Chapter 2). Second, based on the feedback of users of these cultural heritage datasets, we introduce open faceted search, an extension of faceted search that leverages human-AI interaction affordances to empower users to define their own facets in an open domain fashion (Chapter 3). Third, encountering similar challenges with the deluge of scientific papers, we explore the question of how to improve recommender systems through human-AI interaction and tackle the broad challenge of advice taking for opaque machine learners (Chapter 4).

https://tinyurl.com/yc59txc5