Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow PVC expansion #670

Merged
merged 2 commits into from
Jul 15, 2024
Merged

Allow PVC expansion #670

merged 2 commits into from
Jul 15, 2024

Conversation

burmanm
Copy link
Contributor

@burmanm burmanm commented Jun 27, 2024

What this PR does:
Support modifying the CassandraDataVolumeClaimSpec sizes if the underlying StorageClass supports it and an annotation cassandra.datastax.com/allow-storage-changes: trueis placed on the CassandraDatacenter. The operator will automatically increase the size of the PVC and wait for the expansion to finish.

Which issue(s) this PR fixes:
Fixes #263
Fixes #668

Checklist

  • Changes manually tested
  • Automated Tests added/updated
  • Documentation added/updated
  • CHANGELOG.md updated (not required for documentation PRs)
  • CLA Signed: DataStax CLA

@burmanm
Copy link
Contributor Author

burmanm commented Jun 28, 2024

TODO Add new experimental opt-in for this feature (so no one accidentially messes up their cluster). Was thinking cassandra.datastax.com/allow-pvc-expansion. Or prefix with experimental.cassandra.datastax.com to indicate it's not really battle tested yet?

@burmanm burmanm marked this pull request as ready for review July 4, 2024 11:27
@burmanm burmanm requested a review from a team as a code owner July 4, 2024 11:27
Copy link

@rzvoncek rzvoncek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've read to the changes and did not find anything wrong. But I must admit I wasn't able to actually evaluate all of it as it's a bit too much for me.
There are 3 tests failing that I re-triggered.

@burmanm
Copy link
Contributor Author

burmanm commented Jul 10, 2024

Those two tests are expected to fail at this point.

@burmanm burmanm force-pushed the pvc_expansion branch 2 times, most recently from 7b066ba to 56fca3e Compare July 11, 2024 13:49
verifyDatacenterDeleted(ctx, dcName).Should(Succeed())
})
})
// This isn't functional with envtest at the moment, fails with (only in envtest):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: remove the commented code

@@ -121,6 +121,8 @@ func (rc *ReconciliationContext) IsValid(dc *api.CassandraDatacenter) error {
return errs[0]
}

// TODO Verify if we can expand the PVC or should we reject changes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: we're not doing it in this PR? Any potential problem if we don't do the checks like the cluster breaking?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do check, this is just another place where I was planning to do the same check previously. I'll remove this comment.

burmanm added 2 commits July 15, 2024 19:01
… ability to verify if StorageClass supports resizing

Add resize functionality, new DatacenterCondition to indicate Resizing is happening

Patch the PVC and reconcile in 10 seconds

Fix existing tests

Add validation test

Implement unit tests and fix the implementation of expansion

Let VoluemResize actually just continue instead of waiting for the PVC resize, we will do a requeue and check there before recreating the StatefulSet

Add RBAC role for StorageClasses

Fix lint issues, update versions of tools

Add e2e test for the PVC expansion using TopoLVM

Disable StorageConfig webhook validation failure

Improve failure logs with dump of pvc and pv also

Move datacenterCondition reset for ResizingVolumes

Log also StatefulSets, add annotation to allow StorageConfig changes

Add validation if PVC expansion is allowed, modify the behavior to keep the existing PersistentVolumeClaims in the StS, but modifying the sizes only

Update controller-runtime to 0.17.4 and update some logging

Update docker/build-push-action from v4 to v6

Add CHANGELOG, return all the other e2e tests, update Cassandra/DSE versions

Modify the annotation check to happen in the CheckVolumeClaimSizes() instead of CheckRackPodTemplate() to ensure rest of the validations would still always happen. Also, modify the envtests to use more re-usable AsyncAssertions.

Add new events to indicate when Datacenter was set to Valid: False
@burmanm burmanm merged commit 74ac276 into k8ssandra:master Jul 15, 2024
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants