Use this guide after the Quick Start to run, access, inspect, and configure your ZooKage cluster. The file paths and examples below refer to ZooKage v0.5.0, the version used in the Quick Start.

Start and stop

  • ./bin/up launches the cluster components.
  • ./bin/down deletes the resources created by ZooKage.

Connect and use

Use ./bin/ssh [pod [container]] to open a shell in a pod or container. Without arguments, it connects to client-node-0.

The following commands are available on client-node:

  • aws
  • beeline
  • gohdfs (colinmarc/hdfs)
  • hbase
  • hdfs
  • ozone
  • spark-shell
  • spark-sql
  • trino
  • yarn

Check the cluster

  • ./bin/logs [pod] prints logs from the given pod, or from all pods when no pod is specified.
  • ./bin/kubectl <args...> runs kubectl in the zookage namespace.

For example, list the current pod statuses with ./bin/kubectl get pods:

$ ./bin/kubectl get pods
NAME                                READY   STATUS      RESTARTS   AGE
hdfs-datanode-0                     1/1     Running     0          7m19s
hdfs-datanode-1                     1/1     Running     0          7m19s
hdfs-datanode-2                     1/1     Running     0          7m19s
...

Web UI

You can access web UIs through a browser on your host machine when the corresponding components are enabled. This feature is not stable and you may sometimes have to reboot Docker Desktop when the following ports are inaccessible.

Configure the cluster

Edit kustomization.yaml to choose components and image versions, or change configuration files to tune individual services.

Choose components

ZooKage includes HDFS, Hive, Tez, and YARN in its default sandbox cluster. Add or remove components by editing resources in kustomization.yaml. For example, to test Spark SQL on Ozone, enable both components:

 resources:
-# - base/ozone
-# - base/spark
+- base/ozone
+- base/spark

Choose versions

The Docker images used in ZooKage are interchangeable. Replace a newTag in kustomization.yaml to use a different version. For example:

 - name: zookage-spark
   newName: zookage/zookage-spark
-  newTag: 4.1.1-zookage-0.5
+  newTag: 4.0.1-zookage-0.4

Supported versions

ZooKage provides pre-built images for these components:

The following list contains one valid combination. These image tags use the -zookage-0.5 suffix, such as 3.4.1-zookage-0.5.

Hadoop HBase Hive Ozone Ranger Spark Tez Trino ZooKeeper
3.4.1 2.6.3 4.2.0 2.0.0 2.7.0 4.1.1 0.10.5 479 3.9.4

You can also build your own image to test local changes.

Change configurations

Service configuration files are in kubernetes/base/common/config. For example, tune HDFS parameters in hdfs-site.xml.

Set up HA

You can enable High Availability for HDFS and YARN to test HA features locally. This is not designed to deploy a production-ready HA cluster.

Enable ZooKeeper in resources and the HA patches in patches in kustomization.yaml:

 resources:
-# - base/zookeeper
+- base/zookeeper
 patches:
-# - path: patches/ha/hdfs.yaml
-# - path: patches/ha/yarn.yaml
+- path: patches/ha/hdfs.yaml
+- path: patches/ha/yarn.yaml

In both hdfs-site.xml and yarn-site.xml, comment out the section between the For non-HA markers and uncomment the section between the For HA markers. The HDFS and YARN patches configure the corresponding workloads.