Computed Curation

by Philipp Schmitt

Curating photography with neural networks.

Computed Curation is a computer-generated accordeon photobook

Computed Curation is a computer-generated accordeon photobook

Computed Curation is a photobook created by a computer. Taking the human editor out of the loop, it uses machine learning and computer vision tools to curate a series of photos from an archive of pictures.

Considering both image content and composition — but through the sober eyes of neural networks, vectors and pixels — the algorithms uncover unexpected connections and interpretations that a human editor might have missed.

The book consists of 207 photographs

To browse the book open the web version here or download the PDF.

Machine learning based image recognition tools are already adept at recognizing training images (umbrella, dog on a beach, car), but quickly expose their flaws and biases when challenged with more complex input. In Computed Curation, these flaws surface in often bizarre and sometimes poetic captions, tags and connections. Moreover, by urging the viewer to constantly speculate on the logic behind its arrangement, the book teaches how to see the world through the eyes of an algorithm.

Looking at multiple pages reveals similiarities between pictures

You can buy the book at Bromide Publishing House Ltd.

Looking at multiple pages reveals similiarities between pictures

Use the slideshow to flick through some of the pages.

Use the slideshow to flick through some of the pages.

Use the slideshow to flick through some of the pages.

Use the slideshow to flick through some of the pages.

Use the slideshow to flick through some of the pages.

Use the slideshow to flick through some of the pages.

Process

The book features 207 photos taken between 2013 to 2017. Metadata is collected through Google’s Cloud Vision API (tags, colors), Microsoft’s Cognitive Services API (captions) and Adobe Lightroom (date, location). Composition is analyzed using HOGs1.

Considering more than 850 variables for each photo, a t-SNE2 algorithm arranges the pictures in two-dimensional space according to similiarities in content, color and composition. A genetic TSP algorithm3 computes a shortest path through the arrangement, thereby defining the page order.

The book layout is generated using a custom Node.js application and rendered in InDesign using basil.js. The book is printed on a HP Indigo digital press and hand-bound as an accordion book with a total length of 29 meters (95ft).

Computed Curation is a computer-generated accordeon book with a total length of 29 meters (95ft).

Footnotes

Credits

Concept & Development: Philipp Schmitt
Photography: Philipp Schmitt
Book Design: Philipp Schmitt & Margot Fabre
Typography: Px Grotesk by Optimo Type
Printing: CedarGraphics
Special Thanks: Mathias Bernhard, Daniel Schwarz, Moe Vazifeh, Sebastiano Milardo, Ian Seiferling
Open Source: node.js, mongoDB, hog-descriptor, word2vec, GA-TSP, tsne-js, basil.js

  1. Histogram of oriented gradients; an algorithm used in computer vision for object detection. For future iterations I plan to look into convolutional neural networks. ↩

  2. t-distributed stochastic neighbor embedding; a popular algorithm to reduce data with thousands or millions of dimensions (i.e. attributes) to two or three dimensions percievable by humans. A visual explanation. ↩

  3. The travelling salesman problem (TSP) is a classic computer science problem: Given a list of cities, what is the shortest possible route that visits each city exactly once? (I simplified!) ↩