FeatherCast

The voice of The Apache Software Foundation

Pricing Lyft rides with Apache Beam – a case study in migrating from a worker-based workflow to streaming Rakesh Kumar

September 13, 2019
timothyarthur

Ride-sharing is a two-sided marketplace; balancing supply and demand with price in real time is critical to maintaining an efficient system. Dynamic pricing creates fairness for drivers (by raising rates when there is a lot of demand) and maintains good experiences for passengers (by satisfying pick-up time SLAs). This complex system makes real-time decisions using various data sources; machine learning models; and a streaming infrastructure for low latency, reliability and scalability. In this streaming infrastructure, our system consumes a massive number of events from different sources to make these pricing decisions.

Reacting to these events in a cron scheduler based legacy infrastructure with inherent latency becomes a challenge, especially where timely reactions are required to balance market conditions. By leveraging Apache Beam, Lyft’s Streaming Platform powers pricing by bringing together the best of two worlds: ML models in Python and Apache Flink on JVM as the streaming engine.Topics covered in this talk include:
* A brief discussion of dynamic pricing, including motivation and high-level problem formulation
* Comparison of legacy architecture and new streaming architecture
* Overview of streaming platform architecture and technology stack
* Major gains from streaming architecture
* Lessons learned

Leave a Reply

Required fields are marked *.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.
%d bloggers like this: