Algorithmia

Automatic Scene Detection

Hundreds of thousands of videos are uploaded each day to Youtube, Facebook, Instagram, Snapchat and other sites. One of the many issues that these services face is the extraction of useful metadata. At Algorithmia, we’ve been automating a lot of this process, enabling rapid auto-tagging and feature detection of videos. But there’s still more work to be done…

For example, if you wanted to put an ad in the middle of a video, as an advertiser you’d probably prefer to show the ad in between scene cuts, where it would be less intrusive. Or perhaps you’re curating a stock footage gallery but are working with multi-scene footage which needs to be split up. In both of these cases, you don’t want to manually scrub through many thousands of videos each day to determine the best insertion or cut-points.

Scene Detection is an algorithm which automates this task at scale, and is now available for you to use via Algorithmia.  Let’s dig into the details of how we detect scene changes in videos…

What is Scene Detection?

threshold-param-example.png

Scene detection is used for detecting transitions between shots in a video to split it into basic temporal segments. It helps video editors to automate the process of quickly splitting videos in bulk rather than editing it frame by frame by hand.

This brings down the process from a few hours just to a few minutes.

Demystifying: How does it work?

The algorithm essentially has 2 different methods for detecting scene changes.

The content method compares each frame sequentially looking for changes in content, which is useful for detecting quick cuts between scenes.  This is the default method.

The threshold method compares each frame to a set black level.  It is faster than the content method, but useful only when there are cuts and fades to/from black.

Why you need to use scene detection?

example_output_video_listing.png

If you’re a professional who edits videos on a consistent basis, or operates a service that needs to extract metadata from videos, Scene Detection can be of enormous use.

As a professional video editor, you can spend less time manually detecting scene changes by hand, and instead spend more time on other parts of your job.  For your streaming media service, you can extract scene detection metadata information for better advertisement engagement, helping help you create less intrusive ads.  As a developer, you can enhance your app/service by including smart scene detection.

And, since Scene Detection runs inside Algorithmia’s cloud, you won’t need to think about scalability or infrastructure (especially if you have a lot of users or spiky usage). All of that is automagically handled by the Algorithmia platform.

Scene Detection in 5 easy steps

  1. Create an account on Algorithmia (includes 5,000 free credits each month).
  2. Go to your profile page, click the Credentials tab, and find your API key.
  3. Find a test video. You can use a public URL (such as this YouTube video), or upload one to our hosted data storage.
  4. Install the Python Algorithmia client using the command “pip install algorithmia“.
  5. Copy the the sample code below, replace YOUR_API_KEY with your own key, and run it to extract the scenes from your video!
import Algorithmia

client = Algorithmia.client("YOUR_API_KEY")

input = {
  "video": "https://www.youtube.com/watch?v=OMgIPnCnlbQ",
  "detector": "content",
  "output_collection": "data://.algo/media/SceneDetection/temp"
}

result = client.algo("media/SceneDetection/0.1.5").pipe(input).result

print result

Sample Video Input

Sample Output

{
  "framerate": 23.976023976023978,
  "output_collection_videos": [
    "data://.algo/media/SceneDetection/temp/output_0.mp4",
    "data://.algo/media/SceneDetection/temp/output_1.mp4",
    "data://.algo/media/SceneDetection/temp/output_2.mp4",
    "data://.algo/media/SceneDetection/temp/output_3.mp4",
    "data://.algo/media/SceneDetection/temp/output_4.mp4",
    "data://.algo/media/SceneDetection/temp/output_5.mp4",
    "data://.algo/media/SceneDetection/temp/output_6.mp4",
    "data://.algo/media/SceneDetection/temp/output_7.mp4",
    "data://.algo/media/SceneDetection/temp/output_8.mp4",
    "data://.algo/media/SceneDetection/temp/output_9.mp4",
    "data://.algo/media/SceneDetection/temp/output_10.mp4",
    "data://.algo/media/SceneDetection/temp/output_11.mp4",
    "data://.algo/media/SceneDetection/temp/output_12.mp4",
    "data://.algo/media/SceneDetection/temp/output_13.mp4",
    "data://.algo/media/SceneDetection/temp/output_14.mp4",
    "data://.algo/media/SceneDetection/temp/output_15.mp4",
    "data://.algo/media/SceneDetection/temp/output_16.mp4",
    "data://.algo/media/SceneDetection/temp/output_17.mp4",
    "data://.algo/media/SceneDetection/temp/output_18.mp4",
    "data://.algo/media/SceneDetection/temp/output_19.mp4",
    "data://.algo/media/SceneDetection/temp/output_20.mp4"
  ],
  "scenes": [
    "00:00:03.753",
    "00:00:08.758",
    "00:00:10.802",
    "00:00:15.598",
    "00:00:27.110",
    "00:00:34.117",
    "00:00:36.536",
    "00:00:42.500",
    "00:00:44.002",
    "00:00:45.837",
    "00:00:48.965",
    "00:00:51.134",
    "00:00:52.552",
    "00:00:53.428",
    "00:00:55.638",
    "00:00:56.931",
    "00:01:10.779",
    "00:01:18.036",
    "00:01:19.913",
    "00:01:21.998"
  ]
}

As you can see above, the algorithm has returned the timestamps for the cuts, the file locations of the subclipped videos,  and the overall frame rate for the video. You can download the subclips from your hosted data page, under “Algorithm Data” and “media/SceneDetection”.

Conclusion

Whether you’re a professional video editor, a media provider, or a developer, it is now incredibly easy to add automated scene detection to your workflow.

We’d love to hear how you plan to use the Scene Detection microservice!

Algorithm Engineer at Algorithmia, helps make complicated things simpler. Believes that Machine Intelligence will have a huge impact on our lives in the days to come, and hopes to have a defining role in shaping this new future.

More Posts

Follow Me:
TwitterLinkedIn