Overview

Ozone provides S3 compatible REST interface to use the object store data with any S3 compatible tools. S3 buckets are stored under the /s3v volume.

Getting started

S3 Gateway is a separated component which provides the S3 compatible APIs. It should be started additional to the regular Ozone components. You can start a Docker based cluster, including the S3 Gateway from the release package.

Go to the compose/ozone directory, and start the server:

docker-compose up -d --scale datanode=3

Alternatively, use the bundled run.sh script, which sets the number of Datanodes with the OZONE_DATANODES environment variable:

OZONE_DATANODES=3 ./run.sh -d

You can access the S3 Gateway at http://localhost:9878

URL Schema

Ozone S3 Gateway supports both the virtual-host-style URL S3 bucket addresses (eg. http://bucketname.host:9878) and the path-style addresses (eg. http://host:9878/bucketname) By default it uses the path-style addressing. To use virtual host style URLs set your main domain name in your ozone-site.xml:

<property>
   <name>ozone.s3g.domain.name</name>
   <value>s3g.internal</value>
</property>

Implemented REST Endpoints

The Ozone S3 Gateway implements a substantial subset of the Amazon S3 REST API. The tables below summarize each API’s support status, its feature, and any relevant notes, including known deviations from AWS S3 behavior.

Bucket Operations

API Name	Feature	Note
✅ ListBuckets	Lists all buckets owned by the authenticated user.	Without pagination parameters, returns the full bucket list. Supports pagination with max-buckets (up to 10000) and continuation-token.
✅ ListDirectoryBuckets	Lists the directory buckets owned by the authenticated user.	Ozone treats FSO buckets as directory buckets. The S3 Gateway serves this API on GET / when the request is signed with the s3express service name. Supports max-directory-buckets (up to 1000) and continuation-token.
✅ CreateBucket	Creates a new bucket.	Non-compliant behavior: The default bucket ACL may include extra group permissions instead of being strictly private. Bucket names must adhere to S3 naming conventions.
✅ HeadBucket	Checks for the existence of a bucket.	Returns a 200 status if the bucket exists.
✅ DeleteBucket	Deletes a bucket.	Bucket must be empty before deletion.
✅ GetBucketAcl	Retrieves the access control list of a bucket.	Converts the Ozone native ACLs of the bucket to S3 grants.
✅ PutBucketAcl	Sets the access control list of a bucket.	Accepts grants in the request body or in x-amz-grant-* headers. Each request replaces all existing ACLs on the bucket. Non-compliant behavior: Only user grantees (id=) are supported; other grantee types, such as groups, return NotImplemented.

Object Operations

API Name	Feature	Note
✅ GetObject	Retrieves the contents of an object.	Supports Range requests (returns 206 Partial Content) and the partNumber parameter for objects uploaded with multipart upload. A missing object returns 404 with a NoSuchKey XML error body. See Conditional requests.
✅ HeadObject	Retrieves metadata for an object.	Supports the partNumber parameter. A missing object returns 404 without a body, as defined for HEAD requests. See Conditional requests.
✅ PutObject	Uploads a new object.	Supports both single PUT and multipart uploads. In FSO buckets, a zero-length PUT to a key that ends with / creates a directory (controlled by ozone.s3g.fso.directory.creation, enabled by default). See Conditional requests. Non-compliant behavior: ACL headers in the request, such as x-amz-acl, are ignored.
✅ DeleteObject	Deletes a single object.	Returns 204 No Content, also when the object does not exist, as AWS S3 does. Supports If-Match; if the object does not exist or its ETag does not match, returns 412 Precondition Failed.
✅ DeleteObjects	Deletes multiple objects in one request.	Bulk delete operation with aggregated error reporting.
✅ CopyObject	Copies an object (intra- or inter-bucket).	Supports metadata directives (COPY/REPLACE). See Conditional requests. Non-compliant behavior: Self-copying for metadata updates may differ slightly from AWS S3.
✅ ListObjects	Lists objects in a bucket (version 1).	Served by the same GET /{bucket} endpoint as ListObjectsV2. Used when list-type is absent or 1. Supports prefix, delimiter, and marker.
✅ ListObjectsV2	Lists objects in a bucket with filtering and pagination.	Used when list-type=2. Supports prefix, delimiter, start-after, and continuation tokens. Object owners are returned only when fetch-owner=true.
✅ ListObjectVersions	Lists the versions of the objects in a bucket.	Bucket versioning is not supported, so each object is listed once, with VersionId set to the literal string null. DeleteObject, DeleteObjects and the object tagging operations accept null as the version ID for the current object.

Bucket Tagging Operations

API Name	Feature	Note
✅ PutBucketTagging	Sets the tag set for a bucket.	Fully supported; each request replaces all bucket tags and does not append to the existing tag set.
✅ GetBucketTagging	Retrieves the tag set of a bucket.	Returns the tag set as defined by the S3 API.
✅ DeleteBucketTagging	Removes the tag set from a bucket.	Clears all tags associated with the bucket.

Bucket Lifecycle Operations

API Name	Feature	Note
✅ PutBucketLifecycleConfiguration	Sets the lifecycle configuration of a bucket.	Supports the Expiration and AbortIncompleteMultipartUpload actions, with prefix, tag, and And filters. Non-compliant behavior: Transition actions return NotImplemented.
✅ GetBucketLifecycleConfiguration	Retrieves the lifecycle configuration of a bucket.	Returns the configuration as defined by the S3 API.
✅ DeleteBucketLifecycle	Removes the lifecycle configuration from a bucket.	Removes all lifecycle rules of the bucket.

Lifecycle rules take effect only when the lifecycle service is enabled on the OM (ozone.lifecycle.service.enabled, disabled by default). For details, see Object Lifecycle Management.

Object Tagging Operations

API Name	Feature	Note
✅ PutObjectTagging	Adds or updates tags on an object.	Fully supported; updates replace any existing tag set.
✅ GetObjectTagging	Retrieves the tag set of an object.	Returns the tag set as defined by the S3 API.
✅ DeleteObjectTagging	Removes the tag set from an object.	Clears all tags associated with the object.

Object Attributes Operations

API Name	Feature	Note
✅ GetObjectAttributes	Returns selected metadata about an object without the object body.	Served as GET /{bucket}/{key}?attributes. Use the x-amz-object-attributes header to request ETag, ObjectSize, StorageClass, ObjectParts, and Checksum. For multipart objects, x-amz-max-parts and x-amz-part-number-marker paginate part metadata. Non-compliant behavior: Checksum is accepted in the request but not returned, because Ozone does not persist x-amz-checksum-* on upload. The versionId query parameter and SSE-C headers are ignored. On OBS and LEGACY buckets, ObjectParts omits per-part Part elements unless an additional checksum was stored at upload time; FSO buckets always return part details.

Multipart Upload Operations

API Name	Feature	Note
✅ CreateMultipartUpload	Starts a multipart upload session.	Returns an UploadId for subsequent parts.
✅ UploadPart	Uploads an individual part for a multipart upload.	Part numbering and ETag computation follow AWS semantics.
✅ UploadPartCopy	Uploads a part by copying data from an existing object.	Uses the x-amz-copy-source header, and supports x-amz-copy-source-range to copy a byte range of the source object.
✅ ListMultipartUploads	Lists in-progress multipart uploads in a bucket.	Up to 1000 multipart uploads are returned in a batch by default.
✅ ListParts	Lists uploaded parts for an in-progress multipart upload.	Supports pagination if many parts exist.
✅ CompleteMultipartUpload	Assembles all uploaded parts into a final object.	Returns an ETag that follows AWS multipart rules. See Conditional requests.
✅ AbortMultipartUpload	Aborts an in-progress multipart upload.	Frees up storage used by uploaded parts. The OM automatically aborts incomplete multipart uploads that are older than ozone.om.open.mpu.expire.threshold (default 30 days). To abort them earlier for a bucket, use a lifecycle rule with AbortIncompleteMultipartUpload.

Additional Operations

API Name	Feature	Note
✅ Generate Presigned URL	Generates a temporary URL for accessing an object.	Uses AWS Signature V4. Non-compliant behavior: The generated URL may include a fixed default region rather than dynamically reflecting the bucket’s location. The S3 Gateway accepts presigned URLs (AWS Signature V4 query authentication) for all supported operations, such as GetObject, PutObject, DeleteObject, HeadObject, HeadBucket, and the multipart upload operations.

Conditional Requests

The S3 Gateway partially supports conditional requests:

• GetObject and HeadObject: Support If-Match, If-None-Match, If-Modified-Since, and If-Unmodified-Since. A failed If-Match or If-Unmodified-Since returns 412 Precondition Failed. A matching If-None-Match or a failed If-Modified-Since returns 304 Not Modified.
• CopyObject: Supports x-amz-copy-source-if-match, x-amz-copy-source-if-none-match, x-amz-copy-source-if-modified-since, and x-amz-copy-source-if-unmodified-since on the source object. A failed condition returns 412 Precondition Failed.
• PutObject, CopyObject, and CompleteMultipartUpload: Support If-Match with an ETag, to write only if the existing object matches, and If-None-Match: *, to write only if the object does not exist. If-None-Match accepts a specific ETag, to write only if the existing object does not match it; this is a Ceph S3 extension, not an AWS S3 feature. A failed condition returns 412 Precondition Failed. If-Match and If-None-Match cannot be used together.
• DeleteObject: Supports If-Match.

Operations Not Implemented

The S3 Gateway returns 501 NotImplemented for the operations below, except where noted as partial support:

API Name	Note
❌ GetBucketLocation
❌ GetObjectAcl, PutObjectAcl	Use bucket ACLs or Ozone native ACLs instead.
❌ GetObjectTorrent
❌ Bucket CORS, bucket policy, website, encryption, replication, notification, and logging configuration	Ozone uses its own permission model (Ozone native ACLs or Apache Ranger) instead of bucket policies.
❌ Object Lock (PutObjectLockConfiguration, GetObjectLockConfiguration, PutObjectRetention, GetObjectRetention, PutObjectLegalHold, GetObjectLegalHold)	In the design discussion phase. Tracked in HDDS-15945.
❌ GetBucketVersioning, PutBucketVersioning	In the implementation phase. See the design document and the AWS versioning overview.
❌ DeleteObject, DeleteObjects, PutObjectTagging, GetObjectTagging, and DeleteObjectTagging with a versionId other than null	Bucket versioning is not supported. null is accepted as the version ID of the current object.

Additional Notes

• Not supported: S3 Select, Server-Side Encryption headers (SSE-S3, SSE-KMS, SSE-C), cross-region replication, and S3 event notifications.
• Bucket owner check: Bucket and object operations honor the x-amz-expected-bucket-owner header, and copy operations also honor x-amz-source-expected-bucket-owner. If the header does not match the bucket owner, the request fails with 403 Access Denied.
• Temporary credentials: The S3 Gateway can issue temporary credentials with the AssumeRole API, and supports GetCallerIdentity, on a separate STS endpoint (port 9880, set by ozone.s3g.sts.http-address). The STS endpoint is disabled by default; enable it with ozone.s3g.sts.http.enabled. For details, see Ozone S3 Security Token Service (STS).
• While Ozone S3 Gateway provides extensive support for common S3 operations, users should be aware of the above non-compliant behaviors and limitations when integrating with applications expecting full AWS S3 functionality.
• While Ozone S3 Gateway does not support S3 Server-Side Encryption, it does support encrypted buckets using Apache Ranger KMS. For more information, see the Transparent Data Encryption documentation.

Security

If security is not enabled, you can use any AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

If security is enabled, you can get the key and the secret with the ozone s3 getsecret command (*Kerberos based authentication is required).

kinit -kt /etc/security/keytabs/testuser.keytab testuser/scm@EXAMPLE.COM
ozone s3 getsecret
awsAccessKey=testuser/scm@EXAMPLE.COM
awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999

note

Starting in Ozone 1.4.0, the secret will be shown only once when generated with getsecret. If the secret is lost, the user would have to revokesecret first before regenerating a new secret with getsecret.

Now, you can use the key and the secret to access the S3 endpoint:

export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
aws s3api --endpoint http://localhost:9878 create-bucket --bucket bucket1

To invalidate/revoke the secret, use ozone s3 revokesecret command. Parameter '-y' can be appended to skip the interactive confirmation.

ozone s3 revokesecret
Enter 'y' to confirm S3 secret revocation for 'testuser/scm@EXAMPLE.COM': y
S3 secret revoked.

Ozone Manager administrators can run ozone s3 getsecret and ozone s3 revokesecret command with -u parameter to specify another users.

# Obtained Kerberos TGT for testuser/scm@EXAMPLE.COM with kinit,
# testuser/scm@EXAMPLE.COM is an OM admin.
ozone s3 getsecret -u om/om@EXAMPLE.COM
awsAccessKey=om/om@EXAMPLE.COM
awsSecret=1e9379d0424cce6669b1a501ff14834e46dee004ee868b41a313b49eabcfb68f

ozone s3 revokesecret -u om/om@EXAMPLE.COM -y
S3 secret revoked.

Health check

The S3 Gateway serves two unauthenticated health endpoints on its web admin server (default port 19878), separate from the S3 listener (9878). Because they are served on the admin port, they do not collide with S3 bucket names and do not require AWS SigV4 signing, so a load balancer can poll them directly.

Liveness

curl http://localhost:19878/health/live
OK

Returns 200 OK while the gateway process is up and its web admin server can serve requests. It does not check OM reachability.

Readiness

curl http://localhost:19878/health/ready
READY

Returns 200 READY only when the gateway can reach OM, and 503 NOT READY during startup or while OM is unreachable. A background probe refreshes OM reachability off the request path, so the endpoint always responds immediately and cannot be blocked by a slow OM. The probe cadence and its timeout are tunable with ozone.s3g.health-check.probe.interval and ozone.s3g.health-check.probe.timeout.

When S3 Gateway is horizontally scaled, point the load balancer's HTTP health check at the readiness endpoint so it only routes S3 traffic to gateways that can serve it. Both endpoints can be turned off with ozone.s3g.health-check.enabled=false, which removes them entirely. If you disable the endpoints, also remove any probes on /health/live and /health/ready from your deployment, such as Kubernetes liveness and readiness probes; otherwise failed liveness checks restart the pods.

Expose any volume

Ozone has one more element in the name-space hierarchy compared to S3: the volumes. By default, all the buckets of the /s3v volume can be accessed with S3 interface but only the (Ozone) buckets of the /s3v volumes are exposed.

To make any other buckets available with the S3 interface a "symbolic linked" bucket can be created:

ozone sh volume create /s3v
ozone sh volume create /vol1

ozone sh bucket create /vol1/bucket1
ozone sh bucket link /vol1/bucket1 /s3v/common-bucket

This example expose the /vol1/bucket1 Ozone bucket as an S3 compatible common-bucket via the S3 interface.

note

The implementation details of the bucket-linking feature can be found in the design doc.

Clients

AWS CLI

aws CLI could be used by specifying the custom REST endpoint.

aws s3api --endpoint http://localhost:9878 create-bucket --bucket buckettest

Or

aws s3 ls --endpoint http://localhost:9878 s3://buckettest

Supported AWS CLI Versions

• AWS CLI v1 (1.37.0 and above) and AWS CLI v2 (2.23.0 and above) are supported. These are the versions in which AWS enabled default data integrity protections that automatically calculate a CRC32 checksum for S3 uploads; Ozone added handling for that flow in 2.1 and later.
• AWS has announced AWS CLI v1 enters maintenance mode on July 15, 2026; users are encouraged to upgrade to AWS CLI v2. Ozone may drop support for AWS CLI v1 at some point in the future.

Compatible third-party applications

Ozone's S3 Gateway enables integration with a wide range of cloud-native and analytics applications. Here are some examples of tools and platforms known to work with Ozone (in alphabetical order):

• Clickhouse
• Fluentd can send logs directly to Ozone via the S3 Gateway.
• JuiceFS
• Starburst (Starburst also supports Ozone ofs://)
• Teradata NOS
• CyberDuck See the tutorial page

This list is not exhaustive—any application that supports the S3 protocol can potentially connect to Ozone, making it easy to adopt Ozone in modern data pipelines and cloud-native workflows.