Hanging Persistent Volumes in Kubernetes

TL;DR

Sometimes Kubernetes Persistent Volumes are a bit too… persistent.

It occurred to me that some Persistent Volumes in Kubernetes were not properly released. It turned out that these were associated to Volume Attachments that were not released as well, even in the face of the associated volumes not existing any more in Openstack/Cinder. Go figure.

This program goes through the list of Persistent Volumes and detects the hanging ones, printing out the name of the Volume Attachment they are associated to (in addition to their name too). In this context, hanging is defined as having a reclaim policy of Delete (i.e. the persistent volume should be deleted after it is released by a claim) and a status in phase Released. This usually appears as Terminated in the normal output of kubectl get pv.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
#!/usr/bin/env perl
use strict;
use warnings;
use JSON::PP 'decode_json';
use List::Util 'reduce';

$|++;

print {*STDERR} "# persistent volumes that seem to be hanging (Delete/Released a.k.a. Terminating)\n";
print {*STDERR} "#    together with the associated volume-attachment\n";
print {*STDERR} "# <persistent-volume-name> <volume-attachment-name>\n";

my $vas = get_data('volumeattachment')->{items};
my $va_for = index_AoH_by_key($vas, qw< spec source persistentVolumeName >);
my $pvs = get_data('persistentvolume')->{items};
for my $pv (@$pvs) {
   next unless
      ($pv->{spec}{persistentVolumeReclaimPolicy} eq 'Delete')
      && ($pv->{status}{phase} eq 'Released');
   my $pv_name = $pv->{metadata}{name};
   my $va_name = $va_for->{$pv_name}{metadata}{name};
   print {*STDOUT} "$pv_name $va_name\n"
}

sub index_AoH_by_key {
   my ($data, @path) = @_;
   return {
      map {
         my $ref_to_key = reduce(sub { \($$a->{$b}) }, \$_, @path);
         $$ref_to_key => $_;
      } @$data
   };
}

sub get_data {
   return decode_json(slurp_utf8('-|', 'kubectl', 'get', @_, '-o', 'json'));
}

sub slurp_utf8 {
   my ($mode, @rest) = @_;
   open my $fh, $mode, @rest or die "open(@_): $!\n";
   binmode $fh, ':encoding(utf-8)';
   local $/;
   return <$fh>;
}

Local version here.

The astute reader will surely recognize this line of code as familiar:

         my $ref_to_key = reduce(sub { \($$a->{$b}) }, \$_, @path);

It’s the same trick as described in previous post Pointer to element.

I’m not sure of how scalable this solution is: if there are a lot of Persistent Volumes and/or Volume Attachments, I fear that JSON::PP might not be up to the task. I’ll have to do some testing one of these days, but until then it works fine in the clusters I’m playing with.

Cheers!


Comments? Octodon, , GitHub, Reddit, or drop me a line!