InfluxDB

InfluxDB with StorageOS

InfluxDB is a popular open source time series database application optimised for managing datasets consisting of many small measurements. Its advantages include the ability to handle very high write and query loads. Its uses include monitoring, analytics and the recording and analysis of data from sensors.

Before you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.

Deploying InfluxDB on Kubernetes

  1. You can find the latest files in the StorageOS use cases repository

    git clone https://github.com/storageos/use-cases.git storageos-usecases
    cd storageos-usecases
    

    StatefulSet definition:

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: influxdb
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: influxdb
      serviceName: influxdb
      ...
        spec:
          serviceAccountName: influxdb
            ...
            volumeMounts:
            - mountPath: /var/lib/influxdb
              name: data
          ...
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: "fast" # StorageOS storageClass
          resources:
            requests:
              storage: 20Gi
    

    This excerpt is from the StatefulSet definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.

  2. Create the InfluxDB objects

    kubectl create -f ./influxdb
    
  3. Confirm InfluxDB is up and running.

    $ kubectl get pods
    NAME                    READY    STATUS     RESTARTS   AGE
    influxdb-client         1/1      Running    0          1m
    influxdb-0              1/1      Running    0          1m
    
  4. Connect to the InfluxDB client pod, then to the InfluxDB server through the service (this reflects the common kubernetes pattern of maintaining a client pod to conveniently inspect a resource interactively). The default user (admin) and password (admin) are defined in the StatefulSet.

    $ kubectl exec -it influxdb-client -- bash
    [email protected]:/# influx -host influxdb-0.influxdb
    Connected to http://influxdb-0.influxdb:8086 version 1.8.2
    InfluxDB shell version: 1.8.2
    > auth
    username: admin
    password: 
    > show databases
    name: databases
    name
     _internal
    
    > CREATE DATABASE weather;
    > USE weather
    Using database weather
    > INSERT temperature,location=London value=26.4
    > INSERT temperature,location=London value=24.9
    > INSERT temperature,location=London value=22.2
    > INSERT temperature,location=London value=14.7
    > INSERT temperature,location=London value=19.5
    > INSERT temperature,location=Paris value=27.1
    > INSERT temperature,location=Paris value=27.5
    > INSERT temperature,location=Paris value=21.3
    > INSERT temperature,location=Paris value=26.7
    > INSERT temperature,location=Paris value=30.0
    > SELECT MEAN(*) FROM "temperature" GROUP BY "location"
    name: temperature
    tags: location=London
    time mean_value
    
    0    25.65
    
    name: temperature
    tags: location=Paris
    time mean_value
    
    0    26.90
    

In the above steps we have inserted some time series data on the temperature at two locations, and calculated the mean for both. InfluxDB offers a variety of such aggregations (see the docs here), allowing convenient analysis of time series data.

Configuration

If you need custom startup options, you can edit or add to the environment variables within the 20-statefulset.yaml file.

Backups

In this example of how to perform backups of an InfluxDB database on a Kubernetes cluster, we write the output backup file to an Amazon Web Services (AWS) S3 bucket. Other approaches, such as backing up to internal servers or other StorageOS volumes, are possible. For this example to run successfully, Base64-encoded AWS credentials and an S3 bucket name should be inserted into the data field of the backup/50-secret-config.yaml file.

$ echo -n '<your-aws-access-key-id>' | base64
XXXXXXXXXXXX
$ echo -n '<your-aws-secret-access-key>' | base64
XXXXXXXXXXXX
$ echo -n '<your-aws-default-region>' | base64
XXXXXXXXXXXX
$ echo -n '<your-S3-bucket-name>' | base64
XXXXXXXXXXXX

Secret definition:

apiVersion: v1
kind: Secret
metadata:
  name: backup-pod-environment
type: Opaque
data:
  AWS_ACCESS_KEY_ID: XXXXXXXXXXXX
  AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXX
  AWS_DEFAULT_REGION: XXXXXXXXXXXX
  BUCKET_NAME: XXXXXXXXXXXX
  DB_NAME: d2VhdGhlcg==
  DB_HOST: aW5mbHV4ZGItMC5pbmZsdXhkYjo4MDg4

To perform the backup, create the secret and job from the manifest files in the backup directory.

kubectl create -f ./influxdb/backup/

Confirm that the backup pod has been created, and the backup performed successfully.

NAME           READY   STATUS      RESTARTS   AGE
backup-ks976   0/1     Completed   0          1m
client         1/1     Running     0          10m
influxdb-0     1/1     Running     0          10m
mysql-0        1/1     Running     0          10m

The files generated by the backup operation should now be available in you S3 bucket. Re-perform the operation at any time by deleting and re-creating the backup job.

$ kubectl delete -f "influxdb/40-backup-job.yaml"
job.batch "backup" deleted
$ kubectl create -f "influxdb/40-backup-job.yaml"
job.batch/backup created