TL;DR

Sometimes Kubernetes Persistent Volumes are a bit too… persistent.

It occurred to me that some Persistent Volumes in Kubernetes were not properly released. It turned out that these were associated to Volume Attachments that were not released as well, even in the face of the associated volumes not existing any more in Openstack/Cinder. Go figure.

This program goes through the list of Persistent Volumes and detects the hanging ones, printing out the name of the Volume Attachment they are associated to (in addition to their name too). In this context, hanging is defined as having a reclaim policy of Delete (i.e. the persistent volume should be deleted after it is released by a claim) and a status in phase Released. This usually appears as Terminated in the normal output of kubectl get pv.

Local version here.

The astute reader will surely recognize this line of code as familiar:

         my $ref_to_key = reduce(sub { \($$a->{$b}) }, \$_, @path);

It’s the same trick as described in previous post Pointer to element.

I’m not sure of how scalable this solution is: if there are a lot of Persistent Volumes and/or Volume Attachments, I fear that JSON::PP might not be up to the task. I’ll have to do some testing one of these days, but until then it works fine in the clusters I’m playing with.

Cheers!