5 posts tagged with "Ozone"

If you have ever stared at a wall of logs trying to figure out why a single request took ten seconds, you know the struggle: each service tells its own story, but none of them show you the full journey.

Apache Ozone is a distributed system by design. A single request can touch the Ozone Manager (OM), Storage Container Manager (SCM), and multiple Datanodes. When things slow down, metrics might tell you something is wrong but they rarely show the full path. The logs might tell you where something complained, but they're hard to interpret.

Distributed tracing changes that. It maps the end-to-end path of a request with timing for every hop. Ozone now exports these paths using OpenTelemetry over OTLP, so you can pull them into tools like Jaeger and actually see what happened.

An increasing number of database systems are moving storage to S3 in shared-everything architectures to reduce cost, dependencies, and operational complexity. In the Hadoop 🐘 era, we typically used ZooKeeper and HDFS as the control plane and data plane. Modern systems are moving the control plane to self-managed consensus groups or RDBMS-backed catalogs, while moving the data plane onto AWS S3 or S3-compatible storage.

Shared-everything systems usually have two pain points: communication overhead and coordination. To reduce write latency, systems often use inline data writes, background flush, and LSN-based union reads. To reduce read latency, they add multi-layer caches, such as self-managed or OS-managed in-memory caches and on-disk caches. Coordination is harder: multiple clients may read the same metadata, make decisions locally, and then try to update the same object. Without a storage-level compare-and-set primitive, applications often need an external lock service, catalog database, or consensus system just to avoid lost updates.

Because Apache Ozone exposes S3, HCFS, HttpFS, and Java APIs as part of its multi-protocol story, conditional requests have become increasingly important. This work is now nearly complete.

Recently I created https://ozone.s3.peterxcli.dev/, an automated update dashboard for Apache Ozone S3 compatibility. It runs Ozone against real S3 compatibility suites, publishes the result every day, and lets you search test cases, inspect failures, read source snippets, open logs, and check whether your PR helps the compatibility story. It is fully open source at https://github.com/peterxcli/ozone-s3-compatibility.

Ever replaced a drive on a Datanode only to watch it become an I/O hotspot? Or seen one disk hit 95% usage while others on the same machine sit idle? These imbalances create performance bottlenecks and increase failure risk. Apache Ozone's new intra-node Disk Balancer is designed to fix this—automatically.

Here's a step‑by‑step guide to mounting and managing your Apache Ozone object store's S3 interface using CyberDuck.