Prometheus

Prometheus with StorageOS

Prometheus is a popular application used for event monitoring and alerting in Kubernetes.

Before you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.

Deploying Prometheus on Kubernetes

This is the Prometheus use case for StorageOS. Following are the steps for creating a Prometheus StatefulSet and using StorageOS to provide persistent storage.

1. You can find the latest files in the StorageOS use cases repository

   git clone https://github.com/storageos/use-cases.git storageos-usecases
   

   Prometheus Custom Resource definition

    apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
      name: prometheus-storageos
      labels:
        app: prometheus-operator
    spec:
      ...
      storage:
        volumeClaimTemplate:
          metadata:
            name: data
            labels:
              env: prod
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: storageos-replicated
          resources:
            requests:
              storage: 1Gi
   

   This excerpt is from the Prometheus Custom Resource definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs due to the VolumeClaimTemplate in the Prometheus StatefulSet. The Prometheus StatefulSet is created by the Prometheus Operator, triggered by the creation of the Prometheus resource.

2. Move into the Prometheus examples folder and install the Prometheus Operator and create a Prometheus resource.

   $ cd storageos-usecases/prometheus
   $ ./install-prometheus.sh
   

3. Confirm Prometheus is up and running.

   $ kubectl get pods -w -l app=prometheus
   NAME                                READY   STATUS              RESTARTS   AGE
   prometheus-prometheus-storageos-0   3/3     READY               0          1m
   

4. You can see the created PVC using.

   $ kubectl get pvc
   NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
   data-prometheus-prometheus-storageos-0   Bound    pvc-b6c17c0a-e76b-4a0b-8fc6-46c0e1629210   1Gi        RWO            storageos-replicated   65m
   

5. In the Prometheus deployment script, a service monitor is created. A Service Monitor is a special object that the Prometheus operator uses to create configuration for Endpoints for Prometheus to scrape. Although the name implies that a Service Monitor defines Kubernetes services that will be scraped, Prometheus actually targets the Endpoints that the services point to. The new Prometheus instance will use the storageos-etcd service monitor to start scraping metrics from the storageos-etcd pods. Assuming the storageos cluster was setup using ETCD as pods. For more information about service monitors, have a look at the upstream documentation.

   $ kubectl get servicemonitor                       
   NAME             AGE
   storageos-etcd   5d1h
   

6. The Prometheus web ui can be accessed by port-forwarding the Prometheus pods port to localhost.

   $ kubectl port-forward prometheus-prometheus-storageos-0 9090
   

   Then launch a web browser and go to localhost:9090 to access the Prometheus web ui. You can confirm that the storageos-etcd target is configured there.

Configuration

In the storageos-usecases/prometheus/manifests/prometheus directory there are other example Service Monitors. For more information about Prometheus, check out the prometheus documentation.