Today we are going to tackle two of the newer areas of inquiry for the Digital Arts & Humanities: image analysis and artificial intelligence. The past few years have seen an explosion of interest in both due to recent increases in the processing power of computers, the sophistication of machine learning algorithms, and the intense reaction and even backlash against the frightening dystopian futures these developments make possible. Political races rigged by Deep Fake misinformation videos putting words into candidates mouths? Artists no longer able to make a living since AI art generators can produce passable imitations of their work without permission for essentially zero cost?
This is not the future we were promised.
One of the biggest current debates in the field — and the culture at large, for that matter — is whether these developments are a net good or evil, and what the appropriate responses should be. In the spirit of not accepting black boxes and peeking under the hood that informs this class, we are going to inform our own views on this debate through gaining a practical understanding of a small subset of these tools: the IIIF standards for sharing images online and AI tools to colorize black and white images. We’ll use these to make colorized versions of black and white photos in the Carleton College Archives Digital Collections and wrestle with the ethical concerns the exercise raises.
DeOldify
The tool that we are going to be using for this is called DeOldify. It is an open-source AI image manipulation tool in active development, so the results work today but may not tomorrow! This has a goal of colorizing and “restoring” old images and film footage, but it’s application is controversial and raises a number of ethical issues which you’ll address in your reflective post.
First, check out the Github Repository for DeOldify and quickly scan through the examples and documentation offered here.
There are a lot more examples of the applications of this tool, like Old Ireland in Colour and the MyHeritage genealogical service.
- What is your gut reaction to this process?
- What questions would you like to know the answers to?
In order to use the process ourselves, we need to get a high resolution version of a Carleton College historic image, and to do that, we need to first take a detour into the wild world of the International Image Interoperability Framework.
International Image Interoperability Framework (triple-eye-eff)
The International Image Interoperability Framework or IIIF is a community of people developing a set of open standards for sharing high-quality, scalable images and other digital objects online. The aims are to break through silos by encouraging everyone to adopt standards that presume their images will be linked, viewed, zoomed, annotated, and repurposed, without the need to make redundant copies or negotiate paywalls and copyright notices. IIIF Images allow for the creative mixing and annotating of content on the web and open exciting new possibilities for image-based projects that are only just beginning to be realized.
Check out some of the demo projects highlighted on the IIIF home page to see what the community is up to.
In-class Exercise: The Image API
In practice, IIIF consists of a series of APIs (Application Programming Interfaces) for hosting and viewing images and other audio/visual materials on the world wide web. We will focus on the Image API today to see how we can take an image shared by Carleton’s archives and repurpose it for a different use case.
The IIIF Image API solves the problems of needing high resolution images for close study that also load quickly and work well on the web by providing:
- A standard URL structure for accessing images (and therefore easy to support)
- Enough information to allow others to easily develop zoom viewers and other tools
- Cacheable and Scalable URLs
- Human hackable URLs
This last element is quite helpful and will make our task easier today. The basic URL structure contains a predictable format with set parameters we can change manually to get display variants of the image live in the browser. The template for a IIIF Image API request looks like this
TEMPLATE: {scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format}
EXAMPLE: https://example.org/image-service/abcd1234/full/max/0/default.jpg
One we have the identifier for a given image, we can play around with the other parameters to return a certain region of the image (i.e. cropping), a different size (scaling), any 360 degree rotation, an alternative quality of the image (e.g. color, grayscale or black and white, if provided by the host), and a different image format (gif, jpg, png, tif).
To see how this works:
- Open the free IIIF Image Inspector tool and load one of its example images, like Canvas of Cornfields at Sunset
- Click the OPEN button to see what it looks like with the defaults
- Now use the options below the image to construct a URL that shows an image that is, for example:
- square cropped
- 512px wide
- upside down
- gray scale
- has a format of png
- OPEN the new image and compare the two tabs
The amazing thing here is that you didn’t need to download and open the image in a third party image editing app to do any of that. It happens in the browser purely by manipulating the URL!
Leave the Image Inspector open, since we’ll come back to it soon.
In Class Exercise: Colorized Interiors of Historic Carleton
To see how these tools work together, we’re going to create a collection of AI colorized versions of black and white images of Carleton interiors in days gone by, to try to get a better sense of the lived experience of past students.
The inspiration for this exercise is a fantastic project from the Ghent Center for Digital Humanities looking at exploring the art history of domestic interiors through colorized forensic Belgian crime scene photographs: CRIME SCENES: Interbelluminterieurs in België door de lens van de forensische fotografie.
We’ll use the same tools they used (DeOldify, IIIF, and Omeka) on a selection of images from Carleton that show interiors — though not (as far as I know) crime scenes!
Carleton College Archives Photos and IIIF
A nice thing about the IIIF standard is that a lot of tools are starting to incorporate it out of the box.
Carleton uses a CMS called CONTENTdm to store some of its digital collections, and it (being a product of library-land from the folks who brought you WorldCat) supports IIIF images natively. Because of that, we can get the URL of images in our collection and use our knowledge of the IIIF Image API to get the best resolution version we can for colorizing.
Process Overview
Here are the steps we are going to follow to begin to build a collection of colorized historic Carleton photos:
- Find an image record of an interior scene that you want to colorize from our CONTENTdm catalog
- Get the IIIF image URL to that image using your DevTools inspector skills
- Use the IIIF Image Inspector tool to get a full-size IIIF URL to the full region, full size, default quality jpg of the image (or a smaller cropped region, if pasted in a scrapbook).
- Use the DeOldify AI colorizer to create a colorized variant of the image
- Add the colorized version as a new item to the collection on our Omeka site
1. SOURCES: The Archives Collection
The button and image below link to a search of Carleton’s Digital Collections for Interior and Scrapbook
Look through the photos to find an image that you want to colorize
You can also look in the scrapbooks, which contain many images pasted onto larger pages, like the one at right
With IIIF, we can crop and rotate these fairly easily to get clean versions.
2 PROCESS: IIIF Image URL
Carleton’s ContentDM doesn’t expose the IIIF URL in the right api format for us, so we need to use our DevTools web inspector to get it.
To do this, we’ll use the Network Tab, which monitors all page load activities, like loading an external image URL
SAMPLE IMAGE:
Here’s the image I chose from the collection to demonstrate the steps of this exercise:
Campus view (New Women’s Dormitory, Gridley, Skinner, Goodsell), viewed from the northeast, across Lyman Lakes
IIIF IMAGE URL
- Open DevTools and go to the Network tab (see image below)
- Click on the image to open it in the ContentDM IIIF viewer so you can get the full IIIF URL for the image
- Click the Img filter to more easily find the right URLs for the default.jpg file
- Click on default.jpg and you should see the Request URL: in the Headers tab.
Copy the Request URL to your clipboard, which should look something like this: https://cdm17227.contentdm.oclc.org/digital/iiif/CCAP/843/full/839,/0/default.jpg
3 PROCESS: IIIF Image Manipulation
We will know use our knowledge of the IIIF Image API to get a high resolution, cropped version of the image.
FULL SIZE IIIF URL:
- Reopen the IIIF Image Inspector tool
- Paste the IIIF link in the tool to validate you got a working link
- Change the size parameter to get the full resolution image, e.g.
https://cdm17227.contentdm.oclc.org/digital/iiif/CCAP/843/full/full/0/default.jpg - If necessary, adjust the region and rotation parameters to get a clean image
4 PROCESS: Colorize with DeOldify
To use DeOldify, you have to run a series of python scripts. Some of you could do that, but the overhead of teaching it and setting up all your laptops is too high for this class.
We will use a Google Colab notebook, which will let us use a virtual cloud environment entirely in the browser.
COLORIZING IMAGE:
Use the Image Colorizer Google Colab notebook to create a colorized variant of your image.
Follow the instructions in the DeOldify Tutorial video from the Old Ireland in Color project here.
ETHICAL CONSIDERATIONS
We want to make it clear that AI tools did this work, so:
- Leave the watermarked check box checked to stamp the output with “created by AI”
- Feel free to play around with the Render Factor slider to get results that look good to you, but make sure to record the render factor setting used.
5 PRESENTATION: Omeka Collection
We going to gather the results of our efforts in a collection on our Omeka site for display
Log in to Digital Objects of Carleton
OMEKA ITEM OF COLORIZED IMAGE:
Add a New item for your colorized image to our Omeka collection using the following metadata values:
- Collection:
Colorized Images - Dublin Core (example page)
- Title:
COLORIZED: {title field from CONTENTdm} - Subject:
{Subject field from CONTENTdm} - Description: Record your process here by pasting the following and changing any Options as appropriate
Image from the Carleton College archives, colorized using AI tool DeOldify (https://github.com/jantic/DeOldify) on a Google Colab notebook (https://bit.ly/deoldifyimages) with options: Model, Artistic; Watermarked, Yes; Render factor, 35 - Creator:
Stork, Harvey E.or{Creator field from CONTENTdm}if a scrapbook - Source:
{CONTENTdm URL of original image} - Publisher:
Carleton College Archives - Date:
{date field from CONTENTdm} - Contributor:
{Your name} - Rights:
Carleton College encourages the use of these materials for research, teaching, and private study according to our Terms of Use, available at https://apps.carleton.edu/digitalcollections/about/terms/ - Format:
AI colorized image - Identifier:
{Local identifier field from CONTENTdm}
- Title:
- Item Type Metadata
- Item Type:
Still Image - Original Format:
Photograph - Physical Dimensions:
{dimensions field from CONTENTdm}
- Item Type:
- Files: Upload your colorized photo of the object
- Tags:
Interiors, plus any other free tags that make sense (e.g. “campus, buildings, etc.) - Save
- NB: CHECK THE PUBLIC BOX.
YOU’RE DONE! Phew
Lab Assignment (due Sunday)
Your assignment for this week is to think about how easy it was to use AI to create these colorized images, and use that process to reflect more broadly on the issues raised by the readings.
- Finish your Omeka Item, if you haven’t already
- Write a post that raises at least one ethical issue arising from the combination of artificial intelligence and image manipulation.
- Include at least two quotations from the readings that help you think through or position your argument.
- Illustrate your post with a side-by-side comparison of your original and colorized images from today’s exercise (you should be able to add these to WP via the public URLs in ContentDM and Omeka)
- Add a link to your Omeka Item
Post to Moodle as Lab Assignment Week 4
