Algorithmia

Deep Filter: Getting Started With Style Transfer

Algorithm Spotlight: Deep Filter There’s been a lot of interest lately in a deep learning technique called style transfer, which is the process of reimagining one image in the style of another.

It’s fun way to convert photos or images into the style of a masterpiece painting, drawing, etc. For instance, you could apply the artistic style of Van Gogh’s Starry Night to an otherwise boring photo of the Grand Canal in Venice, Italy.

Van Gogh’s Starry Night style to an image of the Grand Canal in Venice, Italy.

The above image was created using Deep Filter, an image processing algorithm that consists of several deep learning models trained on various styles of artistic images. Essentially, each model is a filter the algorithm can use to manipulate images.

What is Deep Filter?

Deep Filter is an implementation of the paper Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. We chose to create our filters based on this paper because it tends to quickly produced high-quality results. We can stylize an image in only a few seconds. Some of the other papers we considered were slow, taking several minutes to stylize an image, or the quality of the output was subpar (our opinion).

Illustration of Feed-forward Synthesis of Textures and Stylized Images.

Demystifying How Deep Filter Works

The model tries to learn how to create images similar in style to a reference style, while also maintaining the same general structure of the original image (objects like humans, faces, objects, etc.). This is what we mean by “style transfer” — we’re applying the style, textures, colors, etc. of a reference image onto a new image while retaining the features of the original.

It does this by extracting low-level features that describe the artistic style in a style image. It then extracts the high-level features from the given test image, like objects, faces, houses, etc. The model then learns (by tweaking its weights in the neural network model) to make the given test image similar in style, but by preserving the objects (aka. high-level statistics-features).

Example of (a) low-level, (b) mid-level and (c) high-level features.

We’ve found that it takes around 50,000 iterations during the training process to produce a model that can successfully stylize an image. Of course, there’s more that goes into training these models, which we’ll share in a future blog post.

Since Deep Filter is microservice, you’re able to call it from our API. Your call consists of the path to an image, a save path, and the style you want to apply.

Your request first gets picked up by a GPU server in our cloud. We execute the model for the filter you selected, and then return the stylized image to the location you specified.

Because we utilize GPU cloud computing, we can scale our availability to keep up with ever increasing demand for making selfies look cool and trippy.

Why You Need Deep Filter

We’re all familiar with Instagram and Snapchat filters. These are a fun way to stylize an image. This past year an app called Prisma came out, which popularized style transfer.

While it’s hard to say, pragmatically, that anybody “needs” style transfer, we believe that pastiche (French for art that imitates the style of other art) is an important way for celebrating and commenting on art – a sort of art imitating art type of thing. It’s also fun to convert photos and images into the style of masterpiece paintings, drawings, etc.

How To Use Deep Filter

To get started using Deep Filter, you’ll need a free API key from Algorithmia. After creating your account, go to your profile page and navigate to the Credentials tab. There you will find your API key. Copy that to a safe location, so we can use it later.

Now, let’s create a collection in the Data API called DeepFilterTest. We’re going to save our stylized image here.

Next, upload an image you want to stylize to this folder. I’ve picked this photo of Nikola Tesla from Wikipedia.

Don’t forget to install our Python client: pip install algorithmia

Now by running the sample code (in Python) below, we’ll be able to get our stylized image in a few seconds!

Sample API Call

import Algorithmia

client = Algorithmia.client("YOUR_API_KEY_GOES_HERE")

input = {
"images": ["https://en.wikipedia.org/wiki/Nikola_Tesla#/media/File:N.Tesla.JPG"],
"savePaths": ["data://.my/DeepFilterTest/stylized_image.jpg"],
"filterName": "space_pizza"
}

if not client.dir("data://.my/DeepFilterTest").exists():
client.dir("data://.my/DeepFilterTest").create()

result = client.algo("deeplearning/DeepFilter").pipe(input).result

print result

Nikola Tesla American Inventor

Sample Output

{
"savePaths": [
"data://.my/DeepFilterTest/stylized_image.jpg"
]
}

Style Transfer and Nikola Tesla Your stylized image is now in your DeepFilterTest collection.

Deep Filter is an algorithm that utilizes the power of deep learning to stylize your photos into cool, trippy, and fun photos. It uses a special technique called style transfer that looks for low-level features in the style image, and applies them to the high-level features (aka. Objects, building, humans etc.) in the original image.

The magic and heavy lifting is handled by the Algorithmia API, so you can just focus on developing your app without worrying about scalability issues.

Please let us know if you’ve used any of the filters in your app or service via @Algorithmia on Twitter!

For more, the Deep Filter algorithm page contains all the relevant documentation and information for stylizing photos.

Deep Learning on Algorithmia:

Algorithm Engineer at Algorithmia, helps make complicated things simpler. Believes that Machine Intelligence will have a huge impact on our lives in the days to come, and hopes to have a defining role in shaping this new future.

More Posts

Follow Me:
TwitterLinkedIn