For organizations leveraging Kubernetes and Rancher, efficient secret management across multiple clusters is a common concern. This blog post uses SUSE’s Fleet to explore a custom operator solution that streamlines secret distribution in multi-cluster environments.
As organizations scale Kubernetes infrastructure, managing secrets across multiple clusters becomes increasingly complex. While it’s possible to configure each downstream cluster to communicate directly with a central secret store like HashiCorp Vault, this approach can become unwieldy as the number of clusters grows. Using the Kubernetes JWT authentication method with Vault requires careful management of roles and policies for each cluster. Alternatively, using the AppRole authentication method, while more straightforward to set up, falls short of providing the needed level of security.
Clusters managed by Rancher leverage the Fleet agent for various aspects of configuration. Using a custom operator, we can enable replication of secrets in targeted Rancher managed clusters. This accounts for secrets created manually in the Rancher cluster or externally managed secrets from tools like HashiCorp Vault or External Secrets.
We’ve developed a customer Kubernetes operator that works with Fleet to distribute secrets across clusters. This operator, the Fleet Handshake Operator, listens for defined Kubernetes secrets and creates Fleet Bundles to distribute to downstream clusters.
Fleet Handshake utilizes the cluster hosting the Fleet controller, typically deployed within Rancher, as the central source of truth for secrets. Fleet agents then consume these secrets in downstream clusters. This approach solves the “secret zero” problem by leveraging Rancher as the JWT authentication to Vault and Fleet to manage the connectivity to downstream clusters. Moreover, it provides a centralized point of control for secret distribution while maintaining security.
The Fleet Handshake Operator is built around a custom resource definition (CRD) called FleetHandshake
. This CRD defines the structure for specifying which secrets should be synchronized and to which target clusters. The main controller, FleetHandshakeReconciler
, handles the reconciliation loop for these custom resources. When a secret is created or updated, the reconciler, which listens for the changes, will then update the Bundle
resource it manages to distribute downstream.
Let’s dive into the critical components of the operator:
The Reconcile
function implements the operator’s core logic. It first gets the secret resource and the custom FleetHandshake. It then upserts the bundle attribute. If all is successful, it updates the fleet handshake to synced.
func (r *FleetHandshakeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// Fetch the FleetHandshake resource
var fleetHandshake rancherv1.FleetHandshake
if err := r.Get(ctx, req.NamespacedName, &fleetHandshake); err != nil {
// Handle error or return if not found
}
// Retrieve the target secret
var secret corev1.Secret
if err := r.Get(ctx, types.NamespacedName{Namespace: fleetHandshake.Namespace, Name: fleetHandshake.Spec.SecretName}, &secret); err != nil {
// Handle error or return if not found. if not found, a status of mising will be set
}
// Check to see if a fleet bundle exists
var existingBundle fleetv1alpha1api.Bundle
err := r.Get(ctx, types.NamespacedName{Name: bundle.Name, Namespace: bundle.Namespace}, existingBundle)
if err != nil && errors.IsNotFound(err) {
// Update existing Bundle if content has changed
if !reflect.DeepEqual(existingBundle.Spec, bundle.Spec) {
if err := r.Update(ctx, bundle); err != nil {
// Handle bundle error
}
}
} else {
if err := r.Create(ctx, bundle); err != nil {
// Handle bundle error
}
}
}
fleetHandshake.Status.Status = "Synced"
if err := r.Status().Update(ctx, &fleetHandshake); err != nil {
// Handle error
}
return ctrl.Result{}, nil
}
A Bundle–a resource housing the secret’s content–is distributed by Fleet to the respective targets. The owner reference plays a significant role here, tying the bundle to the fleet handshake resource. This synchronization ensures that the lifecycles align seamlessly, enhancing the efficiency of the process. Importantly, deleting the handshake prompts the Kubernetes API to remove the secrets downstream.
bundle := &fleetv1alpha1api.Bundle{
ObjectMeta: metav1.ObjectMeta{
Name: fleetHandshake.Name,
Namespace: fleetHandshake.Namespace,
OwnerReferences: []metav1.OwnerReference{{
APIVersion: fleetHandshake.APIVersion,
Kind: fleetHandshake.Kind,
Name: fleetHandshake.Name,
UID: fleetHandshake.UID,
}},
},
Spec: fleetv1alpha1api.BundleSpec{
Resources: []fleetv1alpha1api.BundleResource{
{
Name: fmt.Sprintf("%s.json", secret.Name),
Content: string(jsonSecret),
},
},
Targets: fleetHandshake.Spec.Targets,
},
}
The Fleet Handshake Operator provides a powerful solution for organizations seeking to streamline their secret management across multiple Kubernetes clusters. By leveraging Suse’s Fleet and implementing a custom operator, we can achieve a scalable, secure, and automated approach to secret distribution. This implementation serves as a testament to the pivotal role custom operators play in extending and enhancing the capabilities of existing Kubernetes ecosystem tools, providing tailored solutions for complex operational challenges. As Kubernetes environments become complex, the role of such custom operators in maintaining operational efficiency and security becomes increasingly significant.
To explore this solution and install it within your rancher instance, visit our GitHub repository at https://github.com/rptcloud/fleet-handshake, or checkout our visual walkthrough of the Fleet Handshake Operator’s capabilities.
About River Point Technology: River Point Technology (RPT) is an award-winning cloud consulting, training, and enablement provider, partnering with the Fortune 500 to accelerate their digital transformation and infrastructure automation journeys and redefine the art of the possible. Our world-class team of IT, cloud, and DevOps experts helps organizations leverage the cloud for transformative growth through prescriptive methodologies, best- in-class services, and our trademarked Value Creation Technology process. From consulting and training to comprehensive year-long RPT Accelerator programs, River Point Technology empowers enterprises to achieve Day 2 success in the cloud and maximize their technology investments.