Apache Big Data Seville 2016 – Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

There are many resources available for using Apache Spark to build collaborative filtering models. However, there are relatively few for how to build a large-scale, end-to-end recommender system.

This talk will show how to create such a system, using Apache Kafka, Spark Streaming and Elasticsearch for data ingestion, real-time analytics and data storage, Spark DataFrames and ML pipelines for data aggregation and model building, and Elasticsearch for model management, serving and data visualization. We will also explore techniques for scaling model serving, using Spark Streaming for real-time model updates, and how to incorporate state-of-the-art models into this framework.

The talk will be technical and developer-focused, highlighting experiences from building real-world recommender systems, and providing example code (which will be available as open source).

More information about this talk

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s