FeatherCast

‘At LinkedIn, we generate a staggering 2 trillion events to Kafka each day. nThe ingested events are processed by our streaming platform powered by Apache Samza. nWe currently have over 3000 applications in production that leverage Samza at LinkedIn. nThe use-cases include the LinkedIn feed, anomaly detection, combating fraud, profile notifications, realtime analytics and many more. In this talk, we will describe our journey in building and scaling a distributed stream nprocessing system over these years. Drawing from our experience running 24×7 applicationsnat LinkedIn, we motivate the key-challenges common across them, how we naddressed them and the lessons learnt along the way. Specific challenges include – massive scale, accuracy of results, multi-language support, ndeveloper productivity, high-performance data-access and fast recovery. Lastly, we will nalso share strategic areas for our future work in stream processing.’