Skip to main content

Streaming Write Pipeline

This document discusses the Streaming Write Pipeline feature in Ozone. It is implemented with the Ratis Streaming API.

Write Pipeline Versions

Note that the existing Ozone Write Pipeline is implemented with the Ratis Async API. We refer to the new Streaming Write Pipeline as Write Pipeline V2 and the existing Async Write Pipeline as Write Pipeline V1.

The Streaming Write Pipeline V2 increases the performance by providing better network topology awareness and removing the performance bottlenecks in V1. The V2 implementation also avoids unnecessary buffer copying (by Netty zero copy) and has a better utilization of the CPUs and the disks in each Datanode.

For detailed architectural information about write pipelines, see the Write Pipelines documentation.

Configuration Properties​

Set the following properties to the Ozone configuration file ozone-site.xml.

Enable Streaming Write Pipeline​

To enable the Streaming Write Pipeline feature, set the following property to true:

<property>
  <name>hdds.container.ratis.datastream.enabled</name>
  <value>true</value>
  <description>Enable data stream of container</description>
</property>

Configure Datastream Port​

Datanodes listen to the following port for the streaming traffic:

<property>
  <name>hdds.container.ratis.datastream.port</name>
  <value>9855</value>
  <description>The datastream port number of container</description>
</property>

Enable Filesystem Streaming​

To use Streaming in FileSystem API, set the following property to true:

<property>
  <name>ozone.fs.datastream.enabled</name>
  <value>true</value>
  <description>
    Enable filesystem write via ratis streaming.
  </description>
</property>

Client APIs​

OzoneDataStreamOutput​

The new OzoneDataStreamOutput class is very similar to the existing OzoneOutputStream class, except that OzoneDataStreamOutput uses ByteBuffer as a parameter in the write methods while OzoneOutputStream uses byte[]. The reason of using a ByteBuffer, instead of a byte[], is to support zero buffer copying. A typical write method is shown below:

OzoneDataStreamOutput:

public void write(ByteBuffer b, int off, int len) throws IOException;

OzoneOutputStream:

public void write(byte[] b, int off, int len) throws IOException;

Using ByteBuffer enables zero-copy operations, reducing CPU overhead and improving throughput.

OzoneBucket​

The following new methods are added to OzoneBucket for creating keys using the Streaming Write Pipeline.

createStreamKey​

public OzoneDataStreamOutput createStreamKey(String key, long size)
    throws IOException;
public OzoneDataStreamOutput createStreamKey(String key, long size,
    ReplicationConfig replicationConfig, Map<String, String> keyMetadata)
    throws IOException;
public OzoneDataStreamOutput createStreamKey(String key, long size,
    ReplicationConfig replicationConfig, Map<String, String> keyMetadata,
    Map<String, String> tags) throws IOException;

createMultipartStreamKey​

For multipart uploads:

public OzoneDataStreamOutput createMultipartStreamKey(String key, long size,
    int partNumber, String uploadID) throws IOException;

Note that the methods above have the same parameter list as the existing createKey and createMultipartKey methods.

Example​

Below is an example to create a key from a local file using a memory-mapped buffer:

// Create a memory-mapped buffer from a local file:
final FileChannel channel = ...  // local file channel
final long length = ...          // length of the data
final ByteBuffer mapped = channel.map(FileChannel.MapMode.READ_ONLY, 0, length);

// Create an OzoneDataStreamOutput
final OzoneBucket bucket = ...   // an Ozone bucket
final String key = ...           // the key name
final OzoneDataStreamOutput out = bucket.createStreamKey(key, length);

// Write the memory-mapped buffer to the key output
out.write(mapped);

// close
out.close();      // In practice, use try-with-resource to close it.
channel.close();  // In practice, use try-with-resource to close it.

References​