FeatherCast

The voice of The Apache Software Foundation

Apache Big Data Seville 2016 – Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

December 20, 2016
asfinfra

Building a Scalable Recommendation Engine with Apache Spark, Apache Kafka and Elasticsearch – Nick Pentreath

There are many resources available for using Apache Spark to build collaborative filtering models. However, there are relatively few for how to build a large-scale, end-to-end recommender system.

This talk will show how to create such a system, using Apache Kafka, Spark Streaming and Elasticsearch for data ingestion, real-time analytics and data storage, Spark DataFrames and ML pipelines for data aggregation and model building, and Elasticsearch for model management, serving and data visualization. We will also explore techniques for scaling model serving, using Spark Streaming for real-time model updates, and how to incorporate state-of-the-art models into this framework.

The talk will be technical and developer-focused, highlighting experiences from building real-world recommender systems, and providing example code (which will be available as open source).

More information about this talk

Leave a Reply

Powered by WordPress.com.
%d bloggers like this: