Troubleshooting

Pod in pending because of mount error

Issue:

The output of docker describe pod $POD_ID contains no such file or directory and references the StorageOS volume device file.

[email protected]:~# docker -n storageos describe $POD_ID
(...)
Events:
  (...)
  Normal   Scheduled         11s                default-scheduler  Successfully assigned default/d1 to node3
  Warning  FailedMount       4s (x4 over 9s)    kubelet, node3     MountVolume.SetUp failed for volume "pvc-f2a49198-c00c-11e8-ba01-0800278dc04d" : stat /var/lib/storageos/volumes/d9df3549-26c0-4cfc-62b4-724b443069a1: no such file or directory

Reason:

Mount propagation is not enabled.

Doublecheck:

SSH into the one of the nodes and check if /var/lib/storageos/volumes is empty. If so, exec into any StorageOS pod and check the same directory.

[email protected]:~# ls /var/lib/storageos/volumes/
[email protected]:~# 
[email protected]:~# docker exec $POD_ID -c storageos -- ls -l /var/lib/storageos/volumes
bst-196004
d529b340-0189-15c7-f8f3-33bfc4cf03fa
ff537c5b-e295-e518-a340-0b6308b69f74

If the directory inside the container and the device files are visible, disabled mount propagation is the cause.

Solution:

Enable mount propagation both for Docker and docker, following the prerequisites page

StorageOs plugin not found

Issue:

Docker logs indicate that it cannot find StorageOS plugin, even though the StorageOS container is running.

Reason:

Docker Swarm installations don’t identify plugins that are registered as containers, rather than docker plugins. Therefore some swarm installations don’t load plugins as expected. Docker use lazy loading for those plugins that start as containers. StorageOS creates a socket for which Docker can communicate to the storage engine and provision volumes.

Solution:

Run a docker cli command related to a volume in order to trigger Docker’s lazy plugin evaluation.

docker volume ls