FeatherCast

The voice of The Apache Software Foundation

Samza 1.0: How we scaled stream processing at LinkedIn Jagadish Venkatraman

September 12, 2019
timothyarthur

‘At LinkedIn, we generate a staggering 2 trillion events to Kafka each day. nThe ingested events are processed by our streaming platform powered by Apache Samza. nWe currently have over 3000 applications in production that leverage Samza at LinkedIn. nThe use-cases include the LinkedIn feed, anomaly detection, combating fraud, profile notifications, realtime analytics and many more. In this talk, we will describe our journey in building and scaling a distributed stream nprocessing system over these years. Drawing from our experience running 24×7 applicationsnat LinkedIn, we motivate the key-challenges common across them, how we naddressed them and the lessons learnt along the way. Specific challenges include – massive scale, accuracy of results, multi-language support, ndeveloper productivity, high-performance data-access and fast recovery. Lastly, we will nalso share strategic areas for our future work in stream processing.’

Leave a Reply

Required fields are marked *.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.
%d bloggers like this: