Release 2.1.0 available

Joshua-Trees-haiku

Release Note

Apache Ozone 2.1.0 adds 805 new features, improvements and bug fixes on top of Ozone 2.0.

Notable Changes

Introduced a container reconciliation protocol to resolve mismatched container states and verify replica integrity.

  • HDDS-12940 | Ozone Snapshot Phase 3: Scale up Snapshot Operations

Enhanced Ozone Snapshot scalability and continued stability improvements.

  • HDDS-8387 | Container and volume scanners phase II

Enhanced storage volume failure handling by introducing a ‘degraded’ state to preserve data availability, along with improved health observability.

  • HDDS-12564 | Handling disk issues in Datanodes - Phase II

Enhanced Datanode disk space management to prevent out-of-space failures.

New Configuration Options

  • OZONE_SERVER_OPTS: A new environment variable was added to allow setting common Java options for all Ozone server processes at once.
  • ozone.compaction.service.enabled: Enable or disable a background job that periodically compacts rocksdb tables flagged for compaction. The default is false.
  • ozone.om.compaction.service.run.interval: A background job that periodically compacts rocksdb tables flagged for compaction. The default is 6 hours.
  • ozone.om.compaction.service.timeout: A timeout value of compaction service. If this is set greater than 0, the service will stop waiting for compaction completion after this time. Unit could be defined with postfix (ns,ms,s,m,h,d). The default is 10 min.
  • ozone.om.compaction.service.columnfamilies: A comma separated, no spaces list of all the column families that are compacted by the compaction service. If this is empty, no column families are compacted. The default is “keyTable,fileTable,directoryTable,deletedTable,deletedDirectoryTable,multipartInfoTable”.
  • ozone.om.snapshot.prune.compaction.backup.batch.size: This property controls Prune SST files in Compaction backup directory in batches every ozone.om.snapshot.compaction.dag.prune.daemon.run.interval. The default is 2000.

These properties provide more control over the Ratis write buffer, which can be adjusted to optimize performance based on workload and hardware.

  • ozone.om.ratis.server.pending.write.byte-limit: Sets the maximum byte size of all pending write requests. The default is 64MB.
  • ozone.om.ratis.server.pending.write.element-limit: Sets the maximum number of pending write requests. The default is 4096.

Incompatible change

  • HDDS-12152 | Ozone now requires the Hadoop 3.4 runtime. Support for Hadoop 3.1.2 is deprecated.

Default Value Changes

  • hdds.scm.block.deletion.per-interval.max: Default is now 500,000 blocks per interval.
  • hdds.datanode.volume.choosing.policy: Default changed from RoundRobinVolumeChoosingPolicy to CapacityVolumeChoosingPolicy.
  • hdds.datanode.volume.min.free.space: Default changed from 5GB to 20GB.
  • hdds.datanode.volume.min.free.space.percent: Default changed to 0.1%.
  • hdds.datanode.block.deleting.limit.per.interval: Default is now 20,000 blocks per interval.
  • hdds.scm.safemode.min.datanode: Default changed from 1 to 3.
  • hdds.secret.key.expiry.duration: Default is now 9 days. This change ensures that tokens remain valid for their full configured duration, preventing premature key invalidation and improving stable authentication.
  • fs.trash.classname: Class name updated to org.apache.hadoop.fs.ozone.OzoneTrashPolicy.
  • ozone.key.deleting.limit.per.task: Default is now 50,000 keys per task.
  • ozone.s3g.https-address: Default changed from None to 0.0.0.0:9879.
  • ozone.om.fs.snapshot.max.limit: Default is now 10,000 snapshots.
  • ozone.om.ratis.segment.size: Default is now 64MB (was 4MB).
  • ozone.scm.ha.dbtransactionbuffer.flush.interval: Default is now 1 min.
  • ozone.scm.ha.ratis.segment.size: Default is now 64MB (was 4MB).

Removed Configurations:

  • hdds.rest.http-address: This configuration is removed since HDDS-12761.
  • hdds.scm.safemode.pipeline-availability.check: This configuration is removed since HDDS-11799.

Changelog

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

[2.1.0] - 2025-12-31

Added

  • Added design documentation and initial integration steps for OpenTelemetry. (HDDS-13679)
  • Implemented an in-memory manager for Snapshot Local Data. (HDDS-13627)
  • Added support for Ranger to authorize STS tokens. (HDDS-13848)
  • Added latency metrics for key creation operations and deletion services to Grafana dashboards. (HDDS-13510)
  • Include openssl native library check in the ozone debug checknative command output. (HDDS-13266)
  • Apache Ozone is migrated from OpenTracing to OpenTelemetry for distributed tracing. (HDDS-13680)
  • Enable rpm builds for Ozone. To build rpm packages, do: mvn clean package -DskipTests=true -Prpm. Optionally specify -Drpm.targetArch=aarch64 for ARM builds. (HDDS-13439)
  • Ozone supports creating deb package for Debian. Example command: mvn clean package -Pdeb The package file ozone_<ozone_version>-<linux_distro>_.deb will be created under folder path hadoop-ozone/dist/target. The Ozone binary will be installed under /opt/ozone directory. (HDDS-13414)
  • Added a “Deletion Progress” section to the OM Web UI to track Key Deleting Service metrics (count and size) from the last 24 hours, along with service status and schedule details. (HDDS-12887)
  • Added a throttling mechanism to SCM to prevent excessive memory usage and long GC pauses during large-scale block deletions. (HDDS-12087)
  • Added support for Listener Ozone Managers (OM). Listener OMs are read-only, non-voting nodes that replicate logs from the leader to serve read requests, improving read performance and offloading voting OMs.(HDDS-11523)
  • Exposed new metrics under StorageVolumeScannerMetrics and VolumeInfoMetrics on DataNodes to improve observability of storage volume scanning activity and health. (HDDS-8783)
  • Added specific audit logs for background deletion services and internal OM requests. (HDDS-13370) (HDDS-11314)
  • Added logic to immediately trigger heartbeat and container close when a volume is full. (HDDS-13045)
  • Added support for uploading and deleting objects using presigned URLs. (HDDS-5195) (HDDS-13663)
  • Added support for custom metadata headers with uppercase characters. (HDDS-13429)
  • Added support for S3 STANDARD_IA storage class to accept EC replication config and owner field in listBuckets. (HDDS-1097)
  • Added new Grafana dashboards for low-level RocksDB operations and Deletion progress. (HDDS-12446) (HDDS-13259)
  • Added a “Mismatched Replica” tab to the Containers Page and improved mismatch API. (HDDS-12395)
  • Added ozone admin commands for listing containers based on health state and sorting datanodes. (HDDS-12653) (HDDS-13095)
  • Implemented AWS STS AssumeRole support to generate short-lived temporary credentials. (HDDS-13323)

Changed

  • Metadata Storage: Moved container-related metadata and OM task metadata from Derby DB to Ozone Manager (OM) RocksDB. (HDDS-13669)
  • Locking: Refactored OzoneManagerLock to handle hierarchical resource locking and implemented a pool-based lock manager. (HDDS-13797) (HDDS-13798)
  • Bumped Apache Ratis to version 3.2.0. (HDDS-12070)
  • Bumped AWS SDK to version 2.35.5. (HDDS-13793)
  • Bumped Jetty to 9.4.58. (HDDS-13585)
  • Bumped minimum Maven version to 3.6.3. (HDDS-13364)
  • Recon Sync: Separated Recon OM synchronization from task processing to improve stability. (HDDS-8633)
  • The default backup SST pruning interval is now 10 minutes. This means old backup files will be deleted more frequently, improving disk space usage if Ozone Snapshot is enabled. (HDDS-13638)
  • Ozone now requires the Hadoop 3.4 runtime. Support for Hadoop 3.1.2 is deprecated. (HDDS-12152)
  • Renamed the previous functionality of the --all option in ozone sh snapshot lsDiff to --all-status. The --all option now behaves as a listing limit, setting the count to Integer.MAX_VALUE. (HDDS-8802)

Fixed

  • Snapshot Locking: Fixed OMLockDetails usage where it incorrectly returned a ThreadLocal Object. (HDDS-13978) (HDDS-13004)
  • Recon Startup: Improved Recon bootup times by handling failed delta tasks from previous runs and making task reprocessing non-blocking. (HDDS-13791)
  • Corrected pagination semantics for listMultipartUploads and ListObjectsV2. (HDDS-13290)
  • Fixed Datanode decommission failing when other nodes are offline in Ratis replication. (HDDS-13544)
  • Fixed memory leaks in NSSummary and issues with Derby during schema upgrade. (HDDS-8565) (HDDS-12968)

Security

  • Ranger: Added artifacts for Ranger to authorize STS tokens. (HDDS-13848)

Download tar.gz

(checksum signature)

Download src

(checksum signature)

Documentation

2025 Dec 31