Category Archives: Development

Documenting Decisions to Build Buy-In

Posted on by

by Jeremy Friesen 

Introduction

In any long-running project at Hesburgh Libraries, our developer teams make countless decisions every day. Some decisions are big and some are small. — some affect a few people while others have an impact on the entire organization.

Inevitably, these decisions evolve over time. 

Sometimes we have to adjust or even reverse a decision after gathering more information or gaining experience with a tool or software. We don’t sweat the ebb and flow — we welcome it. Embracing decision-making as an evolutionary process is one of our guiding principles for a healthy team culture.

We also realize that decisions are only as good as the documentation and communication processes that underpin them.

Photograph of a horse from various angles

Documenting our decisions from every angle helps us understand where we’re going and why. Image: Eadweard Muybridge, “Eagle” Walking, Free, plate 576 from Animal Locomotion, 1845-1904, albumen silver print. The Janos Scholz collection of 19th century photography, Snite Museum of Art, University of Notre Dame, 1981.031.543.

To this end, we use consistent documentation and transparent communication to serve as a two-way roadmap for new challenges, team discussions, and retrospectives in the midst of a rapidly changing landscape. 

These decision documents also help to facilitate conversations with stakeholders and build enduring relationships with project partners.

The cases below illustrate how decision documents and transparent communications during the MARBLE project have contributed to team success and project impact.

A tale of two documented decisions

One of the goals of the MARBLE software development project funded by the Andrew W. Mellon Foundation is to create a unified discovery for digitized cultural heritage collections held by the Snite Museum of Art and Hesburgh Libraries at the University of Notre Dame. 

Our immediate aim is to make these objects discoverable in the context of a collaborative digital collections platform. We also surmised that we may want to someday make the digitized objects discoverable through our general library catalog. 

Given these aspirations, we decided to leverage the library-wide discovery system as our search and discovery interface. library-wide discovery system is a vendor-supplied search index software used by many libraries around the world as their primary catalog. 

On the surface, this decision ran contrary to another goal of our project: to develop and release open-source software.

To reconcile these apparent contradictions and keep our road map intact, I wrote a decision document supporting the use of our library-wide discovery system and how we would proceed with delivering open-source software. We clarified that we would use an API to interact with our search index. (An API, or Application Programming Interface, is a protocol or specification that allows information to transfer from one system to another. Using an API is a simple and common practice in developing new software.) 

In this case, we viewed the decision to interact with an API as a way to support other institutions or potential adopters that don’t use our discovery system. In other words, our solution would be built in such a way that another institution could connect with their search index of choice.

We shared our draft decision with project leadership, stakeholders, and developer teams to solicit feedback. From the feedback, we amended the draft document to reflect any new considerations, questions, and challenges.

With a decision document firm in hand, we began working on implementing our solution. We gathered help from other library-wide discovery system adopters. (Thank you, Northwestern Libraries!). We dove deeper into our usage of library-wide discovery system, expanding our expertise and understanding of a technology we have long used.

Then we hit a wall. 

Our user interviews identified full-text search as a key desired feature. According to library-wide discovery system documentation, this functionality should have worked. But, it didn’t, and we entered into a “waiting on vendor response” holding pattern.

While waiting, one of our developers explored ElasticSearch as another option.. After only a  few afternoons of work and testing, ElasticSearch proved to be a viable alternative. Within a week, we referenced our documents. We reassessed our prior decision to leverage our library-wide discovery system and chose to pivot towards ElasticSearch. 

Drawing of a dancer

Pivoting on a decision takes balance and flexibility. Image: Edgar Degas, Study of a Ballet Dancer, ca. 1880-1885, brown conte crayon and pink chalk on paper. Gift of John D. Reilly ND’63, ’64 B.S., Snite Museum of Art, University of Notre Dame, 2004.053.004.

Again, I wrote up a decision document outlining the rationale, process, and lessons learned. For example:

  • We found that ElasticSearch allowed us to implement the full-text search feature.
  • ElasticSearch also performed faster searches
  • There existed open-source ReactJS components for facet rendering, something we were going to need to create in our previous approach.
  • Since ElasticSearch is open-source, our own developers can work out bugs instead of waiting on a vendor. 
  • Our decision to explore our existing library-wide discovery system also produced useful outcomes in that we have a deeper understanding of how to better leverage our library-wide discovery system in our current workflows.
  • The quick swap from one system to another confirmed for us that we have a robust architecture. 
  • Finally, we have postponed the goal of ensuring that all campus cultural heritage content is in our library search index, but our software design will make this work easier going forward.

Amazon Web Services: Are you being serverless?

Another problem we encountered during the development of the MARBLE project was choosing an International Image Interoperability Framework (IIIF) compliant image server. 

Early in the project, we chose to implement Cantaloupe from the list of known server options. With that decision documented and shared, we built blueprints to deploy our Cantaloupe instance into Amazon Web Services (AWS) as a Fargate container.

This worked to get us started.

However, as we added more and more images to Cantaloupe, we encountered problems such as spikes in response times, incidents of high error rates, numerous restarts. We soon discovered the root cause: Cantaloupe’s architecture conflicts with AWS’s Fargate container implementation.

Our options were to move to a more expensive AWS service or look for something else and a possible contender emerged.

Our colleagues at Northwestern University, David Schober and Michael Klein, presented “Building node-iiif: A performant, standards-compliant IIIF service in < 500 lines of code” at Open Repositories 2019. After a quick conversation, they pointed us to their implementation, a serverless service.

Learning from our community is crucial to the development process.
Image: Flemish, The Lawyer’s Office, after Marinus van Reymerswaele, 1535-1590, oil on cradled panel. Gift of Dr. and Mrs. Dudley B. Kean, Snite Museum of Art, University of Notre Dame, 1954.005.

As has become our practice, we documented a plan to experiment with the serverless implementation. 

  • We kept Cantaloupe running for our pre-beta site, while we tested and expanded on Northwestern’s implementation.
  • On October 8th, we made the decision to move away from Cantaloupe. 
  • On November 7th, we switched from using Cantaloupe to using Northwestern’s IIIF-Serverless in our pre-beta instance. This was done without downtime or disruption to our site. 
  • Based on our findings we believe we’ll be able to reduce our image server costs by two orders of magnitude.

You can see our archived image-server repository and a snapshot of the blueprints to build this out in AWS. Here is the code commit that moved our blueprints from Cantaloupe to Serverless. You can also look at our documentation evaluating migrating from Cantaloupe to serverless.

Conclusion

The key takeaway is that it’s worth taking the time to document decisions and have consistent communications. 

It’s true that not every decision necessitates thorough documentation. However, the decisions that require widespread buy-in, impact a key tool or process, or re-orient project goals deserve an organization-wide commitment to this evolving decision-making process. 

For me, decision documents should identify the problem that needs to be solved and includes context, considerations, and constraints. Teams should build decision documents by seeking the input of those with a significant stake in this problem.

Because we have taken the time to document milestones and decisions, our project is modeling how to have a more robust memory of a particular problem and attempted solutions. We are able to be visionary and more agile as we create solutions to meet stakeholder needs.

Simply said, decision documents make all the difference.

And, as a bonus, it was much easier to write this blog post. So, go forth and document! 

 

IIIF at Notre Dame, or, the heart of MARBLE

Posted on by

By Abigail Shelton and Jon Hartzler 

What if you could compare a Rembrandt van Rijn print of David and Goliath held at the Snite Museum of Art in South Bend, IN, with the same print at the National Gallery of Art in Washington, D.C., online, at any time of day? 

Or, zoom in so close that you could see the paint texture on this Vincent Van Gogh self-portrait hanging across the country at the Harvard Art Museum? 

The answer is that you can!  With the help of an open-source image sharing system called IIIF or the International Image Interoperability Framework. 

This set of specifications, pronounced triple-eye-eff, provides a standardized way of storing and displaying images. This allows institutions using IIIF to easily share images. IIIF also has a suite of enhanced features that offers end-users a powerful research experience. 

IIIF is at the heart of the Museums, Archives, Rare Books and Library Exploration (MARBLE) platform in development at the University of Notre Dame.

What is IIIF? 

If you are new to IIIF, a helpful way to think about it might be the analogy of a travel adapter. A recent blog post written by museum developers describes IIIF technology in this way:

  • Picture your preparations for an international trip. 
  • You gather all of your travel essentials and realize that you don’t know anything about the power outlets in your destination country. 
  • So, you look it up and find that they use the same outlet as you do at home. 
  • What a relief! No need to be concerned with bulky adaptors. 
  • You just know it will work—”it’s one less thing to worry about,” as stated in the aforementioned post.

Similarly, when libraries and museums use IIIF compatible images, it’s easier for developers and audiences to work with these images, both locally and across institutions. Developers don’t need to build custom “adaptors” because IIIF standardizes the way images are delivered and displayed. Users have a seamless viewing experience. Every time.

And now for a more technical explanation. The IIIF specification defines two application programming interfaces (APIs); one for image delivery and one for presentation. An API is a software program that allows information to transfer between applications. 

The image delivery API provides a standardized way for web applications to pull images from where they are stored (server) to where they are displayed (website). This makes it easy to retrieve images in different sizes (thumbnail vs. full-size), different orientations (upside down), and different color scales (greyscale, black and white, color, etc.). 

The presentation API provides a structure for how images are displayed online. It sets rules for how images are ordered (e.g. pages in book), and what information is presented to the user about the images (e.g. descriptive metadata). The presentation API is what allows developers to display images in IIIF compatible viewers (like Universal Viewer or Mirador) on a webpage.  

The combination of these two API standards means that images can be easily used and reused in predictable and reliable ways. Developers no longer need to build highly customized applications for a diverse array of image formats. Or, in the travel analogy, no need for bulky adaptors for every new destination!  

How is the University of Notre Dame using IIIF?

IIIF is at the heart of the MARBLE project. Our IIIF server is the primary way that images are presented to users through the website. The process works a little like this: 

  • Library and museum colleagues deposit their images in a Google Drive directory, along with metadata that describes what the image is and how it should be displayed. 
  • Once the deposit is complete, our open-source IIIF Pipeline automatically detects the new images and information and creates a IIIF compatible document (IIIF manifest). This document, full of standardized image metadata, allows us to use the same image in multiple ways, across multiple platforms, without duplicating our efforts. 

For example, the IIIF document for Paul’s Wood’s Absolution Under Fire could be searchable in the MARBLE site, highlighted in a digital exhibit on the artist at Notre Dame, and featured in a specialized collection website highlighting all of the Civil War materials on campus.  With IIIF, this can all happen at the same time, without the need to re-enter any metadata or copy any images.

Priest ministering to soldiers on battlefield

Paul Henry Wood (American, 1872-1892), Absolution Under Fire, 1891, oil on canvas. Gift of the artist, 1976.057. Snite Museum of Art.

We’ve also embedded a IIIF image viewer called Mirador in the website to empower users to examine objects closely and compare them side-by-side. Ultimately, we’d also like to enable users to annotate images in the viewer and then save, search, and share those annotations for personal research or classroom uses. 

Several of our staff are also involved in the IIIF international community, helping to spread the word and develop new features for this open-source software. IIIF is an open-source application meaning that there is a group of volunteers from institutions around the world contributing to its development. Notre Dame colleagues have attended the annual conference, regional working group meetings, and participate in regular community conference calls. Our involvement in the IIIF community means that we’re learning from our peers and bringing ideas back to campus for how to showcase our collections in creative ways.

What can I do with IIIF as a student, educator, scholar, practitioner, ________?

So glad you asked! When the MARBLE portal goes live in 2021, you’ll be able to use Notre Dame’s unique collections in a variety of ways, including: 

close up of family photo album

Detail of Le Rossignol Family Photo Album, Rare Books and Special Collections Department, Hesburgh Libraries, University of Notre Dame.

  • You’ll be able to examine artworks and textual materials far closer than you could ever safely do in person. 
  • As an educator, you could project these images in your classroom to help students see details they may have missed during their museum or library visits. 
  • Students will be able to zoom into images while working on course assignments outside of open hours for the archives or museum. 
  • You will also be able to display multiple images side-by-side, either from the Notre Dame collections or from any other institution with IIIF compliant collections. This could allow you to compare details of multiple paintings by the same artist or read different versions of a text together. 
  • Using the same feature, you could digitally reunite collections of objects or even pages from libraries or museums around the world.
a photo album next to a handwritten letter

Le Rossignol Family Photo Album (leftside) next to a Letter to Ethel Le Rossignol (rightside) in Mirador viewer. Both from Rare Books and Special Collections Department, Hesburgh Libraries, University of Notre Dame.

Here’s an example project where four European libraries combined their early Qurʾānic fragments to recreate original codices using IIIF. 

While IIIF opens up a myriad of possibilities for international collaboration, it will also allow us to reunite collections on our campus. For instance, the Snite Museum of Art’s current exhibition of Irish art features a number of broadsides, prints, and volumes from the Rare Books and Special Collections department alongside paintings and photographs from the museum collection. 

The only way for audiences to see these objects together is to see the exhibition in person before it closes in December (strongly encouraged!). But what if these connections could be preserved? By making these materials available online in a IIIF compatible way, curators can maintain and highlight relationships between collections beyond the close of a special exhibition.  

IIIF also powers fun! From art puzzles to Google Chrome plug-ins to mindfulness apps, the possibilities are endless. We’re excited to see how our users experiment with IIIF and create new ways of seeing and experiencing Notre Dame’s unique collections.