The voice of The Apache Software Foundation

Building S3 over Ozone : Making a Cloud Native File System Bharat Viswanadham Anu Engineer

September 12, 2019

The AWS S3 protocol is the defacto interface for modern object stores. Ozone supports S3 protocol as the first-class notion in Ozone. For all practical purposes, a user of S3 can start using Ozone without any change to code or tools. S3 protocol support offered by Ozone is strongly consistent, so users don’t need to run sidekick tools like S3Guard when running big data applications like Apache Spark, Apache YARN or Apache Hive. This talk is a deep dive into how Ozone supports S3, and how easy it is to use the cloud tools against Ozone. We will demo S3 over Ozone with different off-the-shelf tools like Goofys, Kubernetes CSI, AWS CLI and also demonstrate how to use applications like Apache Spark, Apache Hive and YARN. We will also discuss some of the challenges faced during building S3 compatible rest server. For example, how we did the mapping of S3 semantics to ozone semantics, how we solved the S3 security issues and mapped them to ozone world. With S3 support Ozone has become a generic object store: it can be used with thousands of s3 compatible tools while it will also provide Hadoop compatible file system access.

Leave a Reply

Required fields are marked *.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.
%d bloggers like this: