FeatherCast

The AWS S3 protocol is the defacto interface for modern object stores. Ozone supports S3 protocol as the first-class notion in Ozone. For all practical purposes, a user of S3 can start using Ozone without any change to code or tools. S3 protocol support offered by Ozone is strongly consistent, so users don’t need to run sidekick tools like S3Guard when running big data applications like Apache Spark, Apache YARN or Apache Hive. This talk is a deep dive into how Ozone supports S3, and how easy it is to use the cloud tools against Ozone. We will demo S3 over Ozone with different off-the-shelf tools like Goofys, Kubernetes CSI, AWS CLI and also demonstrate how to use applications like Apache Spark, Apache Hive and YARN. We will also discuss some of the challenges faced during building S3 compatible rest server. For example, how we did the mapping of S3 semantics to ozone semantics, how we solved the S3 security issues and mapped them to ozone world. With S3 support Ozone has become a generic object store: it can be used with thousands of s3 compatible tools while it will also provide Hadoop compatible file system access.