Setting up Prometheus to monitor StorageOS

CoreOS has created a Kubernetes Operator for installing Prometheus. The operator uses ServiceMonitor custom resources (CRs) to scrape IP addresses defined in Kubernetes Endpoints. This article is intended to provide a quick guide to monitoring the StorageOS metrics endpoint and can be used with our example Grafana dashboard.

Scripted Prometheus Install

There is a install-prometheus.sh script that will perform the installation of Prometheus using the Prometheus operator and a ServiceMonitor monitoring StorageOS. If you wish to use it then follow steps 1 and 2 before skipping to installing Grafana. If you do not wish to use the script then go to the Prometheus Operator installation.

N.B. The standard installation of Prometheus and Grafana does not have persistent storage enabled. If you wish to use persistent storage then please see the README.md in the Prometheus directory from Step 1.

  1. Clone the StorageOS deploy repository and move into the prometheus-operator directory
    git clone https://github.com/storageos/deploy.git storageos
    cd storageos/k8s/examples/prometheus
    
  2. Run the install-prometheus.sh script.
    ./install-prometheus.sh
    

Install the Prometheus Operator

  1. Clone the StorageOS deploy repository and move into the prometheus-operator directory
    git clone https://github.com/coreos/prometheus-operator.git prometheus-operator
    
  2. Deploy the quick start bundle.yaml
    kubectl create -f prometheus-operator/bundle.yaml
    
  3. Verify that the Prometheus operator is running.
    kubectl get pods -l apps.kubernetes.io/name=prometheus-operator
    

Install Prometheus

Now that the Prometheus Operator is installed, a Prometheus CR can be created which the Prometheus operator will act upon to configure a Prometheus StatefulSet.

  1. Clone the StorageOS deploy repo
    git clone https://github.com/storageos/deploy.git storageos
    cd storageos/k8s/examples/prometheus
    
  2. If your cluster uses RBAC then create the necessary Cluster role and service account for Prometheus.
    kubectl create -f prometheus-rbac.yaml
    
  3. Create a Prometheus CR that defines a Prometheus StatefulSet.
    kubectl create -f prometheus-cr.yaml
    
  4. Create a ServiceMonitor CR that directs Prometheus to scrape the Endpoints defined in the storageos Endpoints resource. Prometheus will scrape the /metrics URL of the Endpoints and collect the metrics.
    kubectl create -f storageos-serviceMonitor.yaml
    
  5. In order to view the Prometheus UI in the browser port forward the local port to the Prometheus pod port.
    kubectl port-forward prometheus-prometheus-storageos-0 9090
    

    The Prometheus UI can now be seen in the browser at localhost:9090

  6. Now that the Prometheus UI is available StorageOS metrics can be queried from the Graph page. A complete list of StorageOS metrics can be found here

Install Grafana

Grafana is a popular solution for visualising metrics. At the time of writing (30/04/2019) there is no Grafana operator so instead a helm installation is used. If a helm installation will not work then the helm generated manifests can be used.

  1. Install Grafana

    helm install stable/grafana
    
  2. Grafana can query the Prometheus pod for metrics, through a Service. The Prometheus operator automatically creates a service in any namespace that a Prometheus resource is created in. Setup a Grafana data source that points at the Prometheus service that was created. The URL to use will depend on the namespace that Grafana is installed into.

    If the Grafana pod runs in the same namespace as the Prometheus pod then the URL is: http://prometheus-operated:9090 otherwise it’s http://prometheus-operated.$NAMESPACE.svc:9090

    When creating the data source make sure to set the scrape interval.

  3. Once the Prometheus data source has been created have a look at the example StorageOS dashboard for ideas about how to monitor your cluster.