Our Research Exchange series features the work of researchers across The National Archives in which they discuss their new discoveries and their work’s potential for impact. The purpose of the series is to highlight and demystify the research we do and open it up to new ideas and discussion.
This month and next, to coincide with our upcoming Annual Digital Lecture, we will be publishing a series of blogs focusing on different aspects of digital research happening at The National Archives.
Our staff will be writing and reflecting on their research, discussing topics such as using AI in archives, using 3D Virtual Tech to access heritage collections, archiving citizen research projects, and much more.
To kick off the series, Agile Delivery Manager Hazel Jell shares her experience of using Agile in an archival setting.
Large Digital Humanities (DH) research projects often require collaboration between a multidisciplinary team of technical experts (e.g. software engineers, data scientists, AI experts, designers, user researchers) and humanities researchers, to combine expertise to answer complex questions in their fields.
Typically, researchers define tool or software requirements upfront, which are implemented by technical experts and delivered as a resource, before being evaluated at the end of a project (see footnote 1). This reflects the traditional waterfall approach to software development – a linear process working to achieve a pre-defined scope which is delivered to the customer at the end of a project.
While this approach works well in some environments, it is inflexible and can discourage innovation. Where projects push boundaries in digital and humanities fields, they require a more collaborative approach, with frequent communication to experiment with digital methods in the process of the project rather than as an output.
The linear approach is not suited to a dynamic environment with experimentation and potential surprises.
So, when working in a field such as DH, could agile be the answer?
What is agile?
Simply, ‘agile is the ability to create and respond to change’.
Agile was born in the software development world, based on values and principles in the Agile Manifesto. It is an iterative and collaborative approach to software development and project management, which focuses on the individuals as a self-organising multidisciplinary team responding to the needs of the customers.
Agile embraces change and uncertainty, with short time-boxed iterations of work (known as ‘sprints’) allowing for frequent review, feedback, revision and fine-tuning. It is a way of delivering value quickly and frequently.
There are many frameworks to help implement agile, such as scrum, which we are using in the ‘Our Heritage, Our Stories’ project.
Agile is spreading across sectors outside of software development. So, why not digital humanities?
‘Our Heritage, Our Stories’: A case study in agile and DH
‘Our Heritage, Our Stories’ (OHOS) (footnote 2) is making previously hard-to-find and unlinkable community-generated digital content (CGDC) discoverable, through a new public-facing ‘Observatory’ where anyone can access, reuse, and remix this content.
I lead the team of research software engineers (RSEs) at The National Archives as Agile Delivery Manager, responsible for the development of the Observatory. We have implemented agile for The National Archives’ workstream, covering research and design as well as development.
The scrum framework we’ve adopted focuses on delivering specific goals in sprints, with meetings and roles to help structure and manage work. Project work is defined in user stories – a defined increment of work that can be completed independently of any other and within one sprint. User stories present features or requirements from a user’s point of view and how they will provide value. RSEs respond by building the feature over the course of the sprint.
‘Story types’ help us distinguish differences between ‘definitions of done’ – an agreed understanding of expectations for all user stories with a checklist – for research or development. Research answers a question or acquires new knowledge, and development focuses on the completion of code for the Observatory. We’re working in three-week sprints to allow time for research, experimentation, and creativity than the more common two-week sprints.
This approach allows us to be adaptable and flexible, delivering iteratively, incorporating feedback, and responsive to outcomes of research in the wider project team. We share and celebrate our achievements after each sprint, and plan goals for the next.
Having the tools to support our way of working is important. We use GitHub and Amazon Web Services (AWS) to ensure RSEs work on the most recent version of code, and that changes are reviewed and tested as routine. For research and design, we use collaborative tools such as Miro and Figma to share knowledge, ideas, and feedback.
Ten months into OHOS, we’re reflecting on our experience so far. Having such clearly defined tasks could be seen to hinder creativity, but in practice this is not the case. Experimentation and creativity are core to research of this kind, and having time-boxed sprints supports our research while helping us avoid going down ‘rabbit holes’.
There are some reservations that agile may not allow freedom for exploratory research, but the overwhelming feelings are positive. Flexibility, visibility and communication were the key benefits highlighted.
Sprints empower individuals to ‘own’ their work, providing a useful cadence with no ‘hiding places’ for difficulties. Responsibilities are clearly delineated and the project shares an understanding of complementary skillsets needed to achieve project goals.
Agile ‘events’ including daily stand-ups and end-of-sprint ‘demos’ are essential in providing clear goals, visibility and sprint progress. Other workstream teams have found the technical vocabulary of agile slightly opaque, but found that demos communicate achievements and challenges in a transparent, accessible manner.
Agile has enabled us to build a supportive and collegial working environment, developing a shared and iterated understanding of how long work will take. The whole team can feed back on our work, regardless of levels of technical knowledge.
Agile is how we have broken down an enormous task into manageable chunks, co-creating requirements as the research unfolds.
As the project responds to the challenge of siloed CGDC, our agile approach breaks down silos in the project and enables collaboration.
Pip Willcox, Head of Research at The National Archives and Co-Investigator for OHOS, responded: ‘the surprise is quite what a natural fit agile is for research software engineering … I can’t see myself working in any other way.’
It has been suggested that as a field DH should pay more attention to project management, and develop its own models, principles and methods (footnote 3). While we are early in our agile journey, the benefits to us are obvious. Over the remaining years of the project, we will continue to adapt our method as we iteratively and collaboratively build the Observatory.
- Heyer, G, Kahmann, C & Kantner, C, (2019). Generic tools and individual research needs in the Digital Humanities – Can agile development help?. In: Draude, C, Lange, M & Sick, B (Hrsg.), INFORMATIK 2019: 50 Jahre Gesellschaft für Informatik – Informatik für Gesellschaft (Workshop-Beiträge). Bonn: Gesellschaft für Informatik e.V.. (S. 175-180). DOI: 10.18420/inf2019_ws19.
- Our Heritage, Our Stories (OHOS) is a Discovery project in the AHRC’s Towards a National Collection programme (AH/W00321X/1). OHOS is a multidisciplinary team of researchers in digital humanities, archives, history, linguistics, and computer science, and research software engineers (RSEs), from the Universities of Glasgow and Manchester, and The National Archives.
- Tabak, E 2017. ‘A Hybrid Model for Managing DH Projects.’ DHQ: Digital Humanities Quarterly 11.1. http://www.digitalhumanities.org/dhq/vol/11/1/000284/000284.html.