Algorithmia Blog - Deploying AI at scale

Use Smart Thumbnail to Perfectly Crop Images Programmatically

Image Cropping API
It’s not easy to dynamically scale and crop images of different sizes or aspect ratios. Typically, this is resolved in one of two ways: manipulate images by hand (slow, high quality, expensive), or indiscriminately crop-to-fit from the middle-out (fast, low quality, cheap).

Smart Thumbnail is a microservice developers can use to programmatically manipulate images at scale while ensuring that every image is perfectly cropped.

Since web services and social networks can’t employ hundreds of people to manually crop and resize images, they rely on “dumb” algorithms to generate these thumbnails. This is why we experience thumbnails throughout the web of decapitated people, mountains without peaks, or subjects missing from the frame entirely.

If you wanted to extract structured data from websites, or wanted to crawl, scrape, and analyze websites, Smart Thumbnail would be the tool you’d want for handling images manipulation.

Here are two examples of Smart Thumbnail fixing images poorly cropped images:

Face detection and cropping before and after Saliency image cropping before and after

What is Smart Thumbnail

Smart Thumbnail is a microservice that can automatically resize and crop images with the most relevant parts framed and retained.

This microservice takes advantage of both facial recognition with OpenCV, as well as a deep convolutional network for saliency predication to “learn” where in an image to scale and crop.

For example, the most relevant part of an image with people would probably revolve around their faces.

Smart Thumbnail will use OpenCV to recognize the faces in the image and then crops the image to your desired thumbnail size with the face in the frame. No more decapitated people.

But what if image doesn’t contain people? Good question!

Smart Thumbnail will recognize that, and then use saliency via SalNet, an implementation of the “Shallow and Deep Convolutional Networks for Saliency Prediction” research.

(The technical explanation goes like this: By default, Smart Thumbnail takes the average of the centroid of the largest face it detects, if one is detected, as well as the centroid of saliency in the image to determine where to crop. If no face is detected, Smart Thumbnail will use the centroid of saliency to figure out what the most important part of the image is, and ensure it’s retained when scaled and cropped.)

Saliency represents the most prominent part of an image, like a person in a painting. The visual representation of saliency could best be seen as a heatmap like this:

Image Saliency
Whether or not your image contains people, places, or things, Smart Thumbnail can intelligently crop and resize your images into thumbnails.

Why You Need Smart Thumbnail

By using Smart Thumbnail, you ensure that thumbnails on your site are always perfectly cropped, even if the most salient object isn’t already centered.

When you leverage a microservice like Smart Thumbnail, you can scale your ability to manipulate images on your site programmatically, and avoid embarrassing thumbnails like the examples above.

Or, just look at this real life examples of a poorly cropped image:

Poorly cropped tweet before and afterLow quality thumbnails like this erode trust and undermine professionalism.

How to Use Smart Thumbnail

There are a few ways to leverage Smart Thumbnail in your apps and services. The API requires an array with an input URL, output URL, thumbnail width, and thumbnail height.

The input requires either a URL or an Algorithmia Data URI. Your output will be the binary representation of the image, unless you specify a destination for a file when making the API call. This will be saved as a PNG.

The height and width are the desired output size of the image.

To start making calls, you’ll need a free API key from Algorithmia.

Below is the sample input/output. We’ll be using this image for reference:
Original Uncropped Image

Sample Input

import Algorithmia

input = [
"https://hd.unsplash.com/photo-1466840787022-48e0ec048c8a",
"data://.algo/temp/test.png",
500,
500
]
client = Algorithmia.client('API KEY HERE')
algo = client.algo('opencv/SmartThumbnail/2.1.1')
print algo.pipe(input)

Sample Output

data://.algo/temp/arnold.png

Smart Thumbnail Output Image

That was easy. Now you have a solution for dynamically generating thumbnails next time you’re working on a web development project, app, or service.

To take things a step further, trying using one of our deep learning models, like ArtsyNet, Illustration Tagger, or the Places classifier to enrich your images with metadata and tags.

Give a try and let us know what you think @Algorithmia.

Product manager at Algorithmia helping to give developers super powers.

More Posts - Website

Follow Me:
TwitterFacebookLinkedIn