From Source
- x86 or ARM64
- Linux or MacOS
- JDK 1.8 or higher
- Maven 3.6 or later
- Internet connection for first build (to fetch all Maven and Ozone dependencies)
You will also need to install the following dependencies to build native code (optional):
- gcc
- cmake
This is a guide on how to build the ozone sources. If you are not planning to build sources yourself, you can safely skip this page.
If you want to build from sources, Please untar the source tarball (or clone the latest code from the git repository).
ARM-based Linux
If you are using an ARM-based Linux, patch protobuf 2.5.0 and build it from source.
sudo yum install -y make cmake gcc g++ patch
# or for Ubuntu and Debian:
# sudo apt-get install -y make cmake gcc g++ patch
# Download protobuf 2.5.0 tarball
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz | tar zx
cd protobuf-2.5.0
# patch protobuf 2.5.0
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
patch -p1 < protobuf-2.5.0-arm64.patch
# build protobuf
./configure --disable-shared
make
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x
cp $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe
ARM-based Apple Silicon (Apple M1 … etc)
PROTOBUF_VERSION="3.7.1"
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-all-${PROTOBUF_VERSION}.tar.gz | tar zx
cd protobuf-${PROTOBUF_VERSION}
./configure --disable-shared
make -j
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
cd ..
# Download protobuf 2.5.0 tarball
PROTOBUF_VERSION="2.5.0"
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz | tar zx
cd protobuf-${PROTOBUF_VERSION}
# patch protobuf 2.5.0
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
patch -p1 < protobuf-2.5.0-arm64.patch
# build protobuf
./configure --disable-shared
make
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
Build Ozone
Run the ozone build command.
mvn clean package -DskipTests=true
This will build an ozone-\<version\>
directory in your hadoop-ozone/dist/target
directory.
You can copy this tarball and use this instead of binary artifacts that are provided along with the official release.
Depending on the network speed, the build can take anywhere from 10 minutes to 20 minutes.
To create tarball file distribution, use the -Pdist
profile:
mvn clean package -DskipTests=true -Pdist
Other useful Maven build options
- Use
-DskipShade
to skip shaded Ozone FS jar file creation. Saves time, but you can’t test integration with other software that uses Ozone as a Hadoop-compatible file system. - Use
-DskipRecon
to skip building Recon Web UI. It saves about 2 minutes. - Use
-Dmaven.javadoc.skip=true
to skip building javadocs. - Use
-Drocks_tools_native
to build the RocksDB native code for the Ozone Snapshot feature. This is optional and not required for building Ozone. It is only needed if you want to build the RocksDB native code for Ozone.
How to run Ozone from build
When you have the new distribution, you can start a local cluster with docker-compose.
cd hadoop-ozone/dist/target/ozone-X.X.X...
cd compose/ozone
docker-compose up -d
How to test the build
compose
subfolder contains multiple type of example setup (secure, non-secure, HA, Yarn). They can be tested with the help of robotframework with executing test.sh
in any of the directories.