All posts by Diego Oppenheimer

Machine Learning with Humans in the Loop

TL;DR The most accurate machine learning systems to date are those that use a “human-in-the-loop” computing paradigm.

Though we have seen huge advances in the quality and accuracy of pure machine-driven systems, they tend to fall short of acceptable accuracy rates. The combination of machine-driven classification enhanced by human correction, on the other hand, provides a clear path forward in acceptable accuracy. Below we will describe a real-world use case of building and scaling these type of systems.

Read More…

Making University Research Discoverable and Accessible

Distribution, Reach, and Monetization of University Research and AlgorithmsOne of the most rewarding parts of working at Algorithmia is that we get to collaborate with amazing university researchers across the globe.

Last May, Richard Zhang, Philip Isola, and Alexei A. Efros from the University of Berkeley Vision Lab published their work “Colorful Image Colorization.” This paper describes a novel use of a convolutional neural net (learn more about deep learning) for colorizing black and white pictures.

Read More…

Turn Your Deep Learning Model into a Serverless Microservice

Hosting Deep Learning Models

Introducing Cloud Hosted Deep Learning Models

At Algorithmia, we believe in democratizing access to state-of-the-art algorithmic intelligence. That’s why we’re introducing a solution for hosting and distributing trained deep learning models on Algorithmia using GPUs in the cloud.

Today, researchers and developers can train their neural nets locally, and deploy them to Algorithmia’s scalable, cloud infrastructure, where they become smart API endpoints for other developers to use. Read More…

How the Algorithm Economy and Containers are Changing the Way We Build and Deploy Apps Today

The algorithm economy creates a new value chain

In the age of Big Data, algorithms give companies a competitive advantage. Today’s most important technology companies all have algorithmic intelligence built into the core of their product: Google Search, Facebook News Feed, Amazon’s and Netflix’s recommendation engines.

“Data is inherently dumb,” Peter Sondergaard, senior vice president at Gartner and global head of Research, said in The Internet of Things Will Give Rise To The Algorithm Economy. “It doesn’t actually do anything unless you know how to use it.”

Google, Facebook, Amazon, Netflix and others have built both the systems needed to acquire a mountain of data (i.e. search history, engagement metrics, purchase history, etc), as well as the algorithms responsible for extracting actionable insights from that data. As a result, these companies are using algorithms to create value, and impact millions of people a day.

“Algorithms are where the real value lies,” Sondergaard said. “Algorithms define action.”

For many technology companies, they’ve done a good job of capturing data, but they’ve come up short on doing anything valuable with that data. Thankfully, there are two fundamental shifts happening in technology right now that are leading to the democratization of algorithmic intelligence, and changing the way we build and deploy smart apps today:

  1. The Algorithm Economy
  2. Containers

The confluence of the algorithm economy and containers creates a new value chain, where algorithms as a service can be discovered and made accessible to all developers through a simple REST API. Algorithms as containerized microservices ensure both interoperability and portability, allowing for code to be written in any programming language, and then seamlessly united across a single API.

By containerizing algorithms, we ensure that code is always “on,” and always available, as well as being able to auto-scale to meet the needs of the application, without ever having to configure, manage, or maintain servers and infrastructure. Containerized algorithms shorten the time for any development team to go from concept, to prototype, to production-ready app.

Algorithms running in containers as microservices is a strategy for companies looking to discover actionable insights in their data. This structure makes software development more agile and efficient. It reduces the infrastructure needed, and abstracts an application’s various functions into microservices to make the entire system more resilient.


The Algorithm Economy

Algorithm marketplaces and containers create microservices
The “algorithm economy” is a term established by Gartner to describe the next wave of innovation, where developers can produce, distribute, and commercialize their code. The algorithm economy is not about buying and selling complete apps, but rather functional, easy to integrate algorithms that enable developers to build smarter apps, quicker and cheaper than before.

Algorithms are the building blocks of any application. They provide the business logic needed to turn inputs into useful outputs. Similar to Lego blocks, algorithms can be stacked together in new and novel ways to manipulate data, extract key insights, and solve problems efficiently. The upshot is that these same algorithms are flexible, and easily reused and reconfigured to provide value in a variety of circumstances.

For example, we created a microservice at Algorithmia called Analyze Tweets, which searches Twitter for a keyword, determining the sentiment and LDA topics for each tweet that matches the search term. This microservice stacks our Retrieve Tweets With Keywords algorithm with our Social Sentiment Analysis and LDA algorithms to create a simple, plug-and-play utility.

The three underlying algorithms could just as easily be restacked to create a new use case. For instance, you could create an Analyze Hacker News microservice that uses the Scrape Hacker News and URL2Text algorithms to extract the text for the top HN posts. Then, you’d simply pass the text for each post to the Social Sentiment Analysis, and LDA algorithms to determine the sentiment and topics of all the top posts on HN.

The algorithm economy also allows for the commercialization of world class research that historically would have been published, but largely under-utilized. In the algorithm economy, this research is turned into functional, running code, and made available for others to use. The ability to produce, distribute, and discover algorithms fosters a community around algorithm development, where creators can interact with the app developers putting their research to work.

Algorithm marketplaces function as the global meeting place for researchers, engineers, and organizations to come together to make tomorrow’s apps today.


Containers

Putting algorithms in containers enables the algorithm economy

Containers are changing how developers build and deploy distributed applications. In particular, containers are a form of lightweight virtualization that can hold all the application logic, and run as an isolated process with all the dependencies, libraries, and configuration files bundled into a single package that runs in the cloud.

“Instead of making an application or a service the endpoint of a build, you’re building containers that wrap applications, services, and all their dependencies,” Simon Bisson at InfoWorld said in How Containers Change Everything. “Any time you make a change, you build a new container; and you test and deploy that container as a whole, not as an individual element.”

Containers create a reliable environment where software can run when moved from one environment to another, allowing developers to write code once, and run it in any environment with predictable results — all without having to provision servers or manage infrastructure.

This is a shot across the bow for large, monolithic code bases. “[Monoliths are] being replaced by microservices architectures, which decompose large applications – with all the functionality built-in – into smaller, purpose-driven services that communicate with each other through common REST APIs,” Lucas Carlson from InfoWorld said in 4 Ways Docker Fundamentally Changes Application Development.

The hallmark of microservice architectures is that the various functions of an app are unbundled into a series of decentralized modules, each organized around a specific business capability.

Martin Fowler, the co-author of the Agile Manifesto, describes microservices as “an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.”

By decoupling services from a monolith, each microservice becomes independently deployable, and acts as a smart endpoint of the API. “There is a bare minimum of centralized management of these services,” Fowler said in Microservices: A Definition of this New Architectural Term, “which may be written in different programming languages and use different data storage technologies.”

Similar to the algorithm economy, containers are like Legos for cloud-based application development. “This changes cloud development practices,” Carlson said, “by putting larger-scale architectures like those used at Facebook and Twitter within the reach of smaller development teams.”


tl;dr

  • The algorithm economy and containers are changing the way developers build and ship code.
  • The algorithm economy allows for the building blocks of algorithmic intelligence to be made accessible, and discoverable through marketplaces and communities.
  • Containerizing algorithms enables them to be packaged as microservices, making them accessible via an API, and hosted on scalable, serverless infrastructure in the cloud.

A Year In Review: A Letter From Algorithmia CEO Diego Oppenheimer

Diego Oppenheimer, CEO AlgorithmiaIt’s been an exciting and incredibly productive year here at Algorithmia. As we kickoff 2016, I want to look back at everything we accomplished in 2015.

First, I want to thank our community of more than 15,000 algorithm and application developers for their support – it has been an amazing experience. We’re honored by everybody that signed-up, used us in your applications, and added algorithms to the platform. We’re truly thankful for this unique community.

These are the major milestones we crossed in 2015:

Algorithmia Algorithms

MARCH

Algorithmia Launches

Algorithmia leaves private beta and publicly launches, introducing a community built around the algorithm economy, where state-of-the-art algorithms are always live, and accessible by everyone. The algorithm marketplace launches with over 800 algorithms, 3,500 users, and an API for leveraging the building blocks of human intelligence in your apps.

Algorithmia Algorithms

APRIL

How to Build Your Own Google

Algorithms on Algorithmia are like Legos, where they can be mixed and matched to explore the web algorithmically. In this demo, we use Algorithmia to implement PageRank, the algorithm Google was originally based on, to crawl a site, retrieve it as a graph, analyze the connectivity of the graph, and then analyze each node for its contents.

Algorithmia Algorithms

JUNE

Using Artificial Intelligence to Detect Nudity

We combine artificial intelligence with University research to teach an algorithm to determine if there is nudity in an image. The result is our Nudity Detection algorithm, which uses a combination of nose, face, and skin color detection algorithms to identify if there are people in an image, and if any of them are nude.

Don’t believe us? Try out the demo at isitnude.com.

Algorithmia Algorithms

JULY

Navigate Product Hunt Like a Pro

We build a Chrome Extension that uses FP-Growth and Keyword Set Similarity algorithms to surface related products on Product Hunt from users who’ve upvoted the product you’re browsing (i.e. collaborative filtering). Get the Chrome Extension here, and start discovering better products on Product Hunt today.

Algorithmia Algorithms

SEPTEMBER

AWS Lambda Partnership

Algorithmia teams up with AWS Lambda, enabling developers to build intelligent, serverless apps in minutes by leveraging our built-in Node.js blueprint. In this demo, we show you how to quickly make a serverless photo app to create digital art in less than 300 lines of code using AWS Lambda and Algorithmia.

The #1 Big Data Startup

A panel of judges from Strata + Hadoop World select Algorithmia as the winner of the Startup Showcase based on the team, technology, and innovation from a pool of the 12 leading big data startups.

Algorithmia Algorithms

OCTOBER

Supporting the Innovators of Tomorrow

Algorithmia joins more than 600 student hackers at DubHacks, the largest collegiate hackathon in the Pacific Northwest, as sponsor and participant to help teams build projects, and create solutions to real-world problems.

Content Recommendations Made Simple

We launch Algorithmia Recommends, a free content recommendation plugin for any WordPress or Drupal blog, or website to increase their engagement and retention. Algorithmia Recommends is built on the Breadth First Site Map web crawler algorithm, and powerful natural language processing algorithms Keywords For Document Set and Keyword Set Similarity to find and categorize all the pages on your website in order to help your users find content that’s most relevant to their interests.

Algorithmia Algorithms

NOVEMBER

Free eBook Published

Algorithmia launches its first eBook, Five Algorithms Every Web Developer Can Use and Understand, which teaches you how to harness the power of algorithms so you can make every app a smart app. In this short primer, we cover PageRank, Language Detection, Nudity Detection, Sentiment Analysis, and TF-IDF, and how you could implement it today.

Algorithmia Algorithms

DECEMBER

GeekWire: Algorithmia Top Seattle Startup

Algorithmia joins GeekWire’s prestigious, annual list of the 10 most promising early-stage startups in the Seattle area. Our business model gets translated onto a giant six-foot by six-foot cocktail napkin, which is unveiled during GeekWire Gala at the Museum of History & Industry (MOHAI).

One of the 10 Coolest Big Data Startups

Algorithmia caps off an incredible year with another award, this time CRN names us one of the coolest big data startups.

Algorithmia Algorithms

On top of those highlights, we also overhauled the Algorithmia UI, introducing a new experience to help users explore algorithms, discover solutions, and help users get started faster. Additionally, we improved our documentation to make using Algorithmia a snap, as well as added clients for CLI, Java, JavaScript, Python, and Scala.

This could have not been achieved without our amazing team Anthony, Besir, John, Jonathan, Kenny, Liz, Matt, Patrick and Zeynep. As well as our awesome 2015 interns Ahmad and Mark.

We’re just getting started, and 2016 is shaping up to be even bigger as we add more algorithms and use cases.

We’re looking forward to another incredible year, and we’re excited to have you along for the journey.

– Diego M. Oppenheimer, CEO

The Future of Algorithm Development

A year ago, we shared with you our belief that Algorithm Development is Broken, where we laid out the problems in the world of algorithm development, and our vision of how to make things better.

Since then, we’ve been working hard to create our vision of the world’s first open marketplace for algorithms – and you, the developers of the world, have enthusiastically joined us to help active the algorithm economy.

Today, we are proud to open our doors to the world, and deliver on our promise of a global, always-available, open marketplace for algorithms. For the first time, algorithm developers can publish their algorithms as a live API and applications can be made smarter by taking advantage of these intelligent algorithms.

Algorithm Development with Algorithmia

  • Live: easy access to any algorithm, callable at any time
  • No dependencies: no downloads, no installs, no stress
  • Interoperable: call any algorithm regardless of programming language
  • Composable: algorithms are building blocks, build something bigger!
  • Invisible but powerful infrastructure: don’t worry about containers, networks, or scaling; we make your algorithm scale to hundreds of machines
  • No operations: we scale, meter, optimize and bill for you

With our public launch we are enabling payments and go from being a repository of algorithms, to a true marketplace:

  • For algorithm developers, Algorithmia provides a unique opportunity to increase the impact of their work and collect the rewards associated with it.
  • For application developers, Algorithmia provides the largest repository of live algorithms ever assembled, supported by a vibrant community of developers.

So far, users have contributed over 800 algorithms, all under one API that grows every day. The most exciting thing from our point of view is watching the Algorithmia Marketplace grow and expand in its functionality.

In just the last few months, Algorithmia has gained the ability to:

Thanks to the thousands of developers who joined our cause, contributed algorithms, and gave us countless hours of feedback so far. We are very excited to be opening our doors, and we look forward to helping you create something amazing!

Interested? Join the largest marketplace for algorithms in the world.