External key-value store
StorageOS uses a key-value store to hold cluster metadata across the
distributed platform. The key-value backend can be
embedded option is the default for ease of deployment and in this mode
StorageOS uses an internal etcd cluster. For testing that approximates a
real life workload we recommend the use of external etcd.
embedded etcd, the first StorageOS nodes that start, either the
first node or first three nodes, act as etcd servers. This is done in order to
rest of the nodes act as etcd clients, keeping consistency of the cluster by
communicating with the etcd servers. Whether a StorageOS node acts as an etcd server
or client cannot be changed once a StorageOS container has started.
It is recommended to use an external etcd cluster for production deployments. To use an etcd cluster provisioned and maintained outside the scope of StorageOS, StorageOS will locate the etcd endpoint using environment variables. When performing an installation using the StorageOS cluster operator these variables can be set in the StorageOSCluster resource definition. Installations using the Helm chart can set the variables in the values.yaml file.
KV_BACKEND: Set to
KV_ADDR: Comma separated list of etcd targets, in the form host[:port].
Etcd availability is mandatory for proper functioning of StorageOS. In particular, changes to the cluster; creation/deletion of volumes, adding/removing replicas e.g., require access to the etcd cluster. In the event that etcd becomes unavailable, StorageOS clusters become read only, allowing access to data but preventing metadata changes.
When using an external
etcd, the user must maintain the availability and
integrity of the etcd cluster, so StorageOS can use it as a service.
It is highly recommended to keep the cluster backed up and ensure high availability of its data. It is also important to keep the latency between StorageOS nodes and the etcd replicas low. Deploying an etcd cluster in a different data center or region can make StorageOS detect etcd nodes as unavailable because of latency.
StorageOS cannot be held responsible for supporting an external etcd cluster.
Suggested Deployment Models
Depending on the orchestrator used, methods of passing environment variables to StorageOS containers differ. The following examples work with both Kubernetes and Openshift.
Etcd can be deployed in Kubernetes using the official etcd-operator.
The official etcd-operator repository also has a backup deployment operator that can help backup etcd data. Make sure you take frequent backups of the etcd cluster as it holds all the StorageOS cluster metadata.
To deploy StorageOS on a Kubernetes cluster using an etcd cluster running outside Kubernetes, you can look at the following repository, where etcd endpoints are referenced using a Kubernetes external service.