Setting up Prometheus to monitor StorageOS

CoreOS has created a Kubernetes Operator for installing Prometheus. The operator uses ServiceMonitor custom resources (CRs) to scrape IP addresses defined in Kubernetes Endpoints. This article is intended to provide a quick guide to monitoring the StorageOS metrics endpoint and can be used with our example Grafana dashboard.

Scripted Prometheus Install

There is a install-prometheus.sh script that will perform the installation of Prometheus using the Prometheus operator and a ServiceMonitor monitoring StorageOS. If you wish to use it then follow steps 1 and 2 before skipping to installing Grafana. If you do not wish to use the script then go to the Prometheus Operator installation.

N.B. The standard installation of Prometheus and Grafana does not have persistent storage enabled. If you wish to use persistent storage then please see the README.md in the Prometheus directory from Step 1.

Clone the StorageOS deploy repository and move into the prometheus-operator directory

git clone https://github.com/storageos/deploy.git storageos
cd storageos/k8s/examples/prometheus

Run the install-prometheus.sh script.
```
./install-prometheus.sh
```

Install the Prometheus Operator

Clone the StorageOS deploy repository and move into the prometheus-operator directory
```
git clone https://github.com/coreos/prometheus-operator.git prometheus-operator
```

Deploy the quick start bundle.yaml

kubectl create -f prometheus-operator/bundle.yaml

Verify that the Prometheus operator is running.

kubectl get pods -l apps.kubernetes.io/name=prometheus-operator

Install Prometheus

Now that the Prometheus Operator is installed, a Prometheus CR can be created which the Prometheus operator will act upon to configure a Prometheus StatefulSet.

Clone the StorageOS deploy repo

git clone https://github.com/storageos/deploy.git storageos
cd storageos/k8s/examples/prometheus

If your cluster uses RBAC then create the necessary Cluster role and service account for Prometheus.
```
kubectl create -f prometheus-rbac.yaml
```
Create a Prometheus CR that defines a Prometheus StatefulSet.
```
kubectl create -f prometheus-cr.yaml
```
Create a ServiceMonitor CR that directs Prometheus to scrape the Endpoints defined in the storageos Endpoints resource. Prometheus will scrape the /metrics URL of the Endpoints and collect the metrics.
```
kubectl create -f storageos-serviceMonitor.yaml
```
In order to view the Prometheus UI in the browser port forward the local port to the Prometheus pod port.
```
kubectl port-forward prometheus-prometheus-storageos-0 9090
```
The Prometheus UI can now be seen in the browser at localhost:9090
Now that the Prometheus UI is available StorageOS metrics can be queried from the Graph page. A complete list of StorageOS metrics can be found here

Install Grafana

Grafana is a popular solution for visualising metrics. At the time of writing (30/04/2019) there is no Grafana operator so instead a helm installation is used. If a helm installation will not work then the helm generated manifests can be used.

Install Grafana
```
helm install stable/grafana
```
Grafana can query the Prometheus pod for metrics, through a Service. The Prometheus operator automatically creates a service in any namespace that a Prometheus resource is created in. Setup a Grafana data source that points at the Prometheus service that was created. The URL to use will depend on the namespace that Grafana is installed into.

If the Grafana pod runs in the same namespace as the Prometheus pod then the URL is: http://prometheus-operated:9090 otherwise it’s http://prometheus-operated.$NAMESPACE.svc:9090

When creating the data source make sure to set the scrape interval.
Once the Prometheus data source has been created have a look at the example StorageOS dashboard for ideas about how to monitor your cluster.