StorageOS with Kubernetes best practices

StorageOS Pod placement

StorageOS must run on all nodes that will contribute storage capacity to the cluster or that will host Pods which use StorageOS volumes. For production environments, it is recommended to avoid placing StorageOS Pods on Master nodes.

StorageOS is deployed with a DaemonSet controller, and therefore tolerates the standard unschedulable taint placed on master nodes, and cordoned nodes (see the Kubernetes docs for more details). To avoid scheduling StorageOS pods on master nodes, you can add an arbitrary taint to them for which the StorageOS DaemonSet won’t have a toleration.

Be sure to appropriately modify the JOIN variable defined for the DaemonSet to avoid newly bootstrapping nodes trying to contact non-existent pods on Master nodes. If installing with our Cluster Operator, this is handled for you automatically.

StorageOS API username/password

StorageOS uses a Kubernetes secret to define the API credentials. For standard installations (non CSI), the API credentials are used by Kubernetes to communicate with StorageOS.

The API grants full access to StorageOS functionality, therefore we recommend that the default administrative password of ‘storageos’ is reset to something unique and strong.

You can change the default parameters by encoding the apiUsername and apiPassword values (in base64) into the storageos-api secret.

To generate a unique password, a technique such as the following, which generates a pseudo-random 24 character string, may be used:

# Generate strong password
PASSWORD=$(cat -e /dev/urandom | tr -dc '[email protected]#$%^&*()_+~' | fold -w 24 | head -n 1)

# Convert password to base64 representation for embedding in a K8S secret
BASE64PASSWORD=$(echo -n $PASSWORD | base64)

Note that the Kubernetes secret containing a strong password must be created before bootstrapping the cluster. Multiple installation procedures use this Secret to create a StorageOS account when the cluster first starts.

Use an external etcd cluster

StorageOS uses the etcd distributed key-value store to store essential cluster metadata and manage distributed configuration state. An embedded etcd instance is included in the StorageOS container, but for production environments and testing of production workloads, we recommend deploying using an external etcd cluster. For more details about, and an example of, how to run etcd, see the External Etcd Operations page.

It is highly recommended to use external etcd for cloud environments and place the etcd cluster on stable nodes. Placing the etcd on nodes that are recycled often might affect the normal operations of StorageOS.

Setup of storage on the hosts

We recommend creating a separate filesystem for StorageOS to mitigate risk of filling the root filesystem on nodes. This has to be done for each node in the cluster.

Follow the managing host storage best practices page for more details.

Resource reservations

StorageOS resource consumption depends on the workloads and the StorageOS features in use.

For production environments, it is recommended to test StorageOS using realistic workloads and tune resources accordingly.

The recommended minimum memory reservation for the StorageOS Pods is 235Mb. However it is recommended to prepare nodes so StorageOS can operate at least with 1-2GB of memory. StorageOS frees memory when possible.

StorageOS Pods resource allocation will impact directly on the availability of volumes in case of eviction or resource limit triggered restart. It is recommended to not limit StorageOS Pods.

StorageOS implements a storage engine, therefore limiting CPU consumption might affect the I/O throughput of your volumes.

Port blocking

StorageOS exposes ports to operate. It is recommended that the ports are not accessible from outside the scope of your cluster.