Sealing your Secrets

Let's create a new secret catalog-sealed-db. We'll create a new file new-catalog-db.yaml with the same keys and values as the catalog-db Secret.

~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml

apiVersion: v1
kind: Secret
metadata:
  name: catalog-sealed-db
  namespace: catalog
type: Opaque
data:
  password: ZGVmYXVsdF9wYXNzd29yZA==
  username: Y2F0YWxvZ191c2Vy

Now, let’s create SealedSecret YAML manifests with kubeseal.

~$kubeseal --format=yaml < ~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml \

> /tmp/sealed-catalog-db.yaml

Alternatively, the public key can be fetched from the controller and use it offline to seal your Secrets:

~$kubeseal --fetch-cert > /tmp/public-key-cert.pem

~$kubeseal --cert=/tmp/public-key-cert.pem --format=yaml < ~/environment/eks-workshop/modules/security/sealed-secrets/new-catalog-db.yaml \

> /tmp/sealed-catalog-db.yaml

It will create a sealed-secret with the following content:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: catalog-sealed-db
  namespace: catalog
spec:
  encryptedData:
    password: AgBe(...)R91c
    username: AgBu(...)Ykc=
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: catalog-sealed-db
      namespace: catalog
    type: Opaque

Let's deploy the SealedSecret to your EKS cluster:

~$kubectl apply -f /tmp/sealed-catalog-db.yaml

The controller logs shows that it picks up the SealedSecret custom resource that was just deployed, unseals it to create a regular Secret.

~$kubectl logs deployments/sealed-secrets-controller -n kube-system

2022/11/07 04:28:27 Updating catalog/catalog-sealed-db

2022/11/07 04:28:27 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"catalog", Name:"catalog-sealed-db", UID:"a2ae3aef-f475-40e9-918c-697cd8cfc67d", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"23351", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

Verify that the catalog-sealed-db Secret unsealed from the SealedSecret was deployed by the controller to the secure-secrets namespace.

~$kubectl get secret -n catalog catalog-sealed-db

NAME                       TYPE     DATA   AGE

catalog-sealed-db          Opaque   4      7m51s

Let's redeploy the catalog deployment that reads from the above Secret. We have updated the catalog deployment to read the catalog-sealed-db Secret as follows:

Kustomize Patch

~/environment/eks-workshop/modules/security/sealed-secrets/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalog
spec:
  template:
    spec:
      containers:
        - name: catalog
          env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: catalog-sealed-db
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: catalog-sealed-db

Deployment/catalog

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/created-by: eks-workshop
    app.kubernetes.io/type: app
  name: catalog
  namespace: catalog
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: service
      app.kubernetes.io/instance: catalog
      app.kubernetes.io/name: catalog
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/component: service
        app.kubernetes.io/created-by: eks-workshop
        app.kubernetes.io/instance: catalog
        app.kubernetes.io/name: catalog
    spec:
      containers:
        - env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: catalog-sealed-db
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: catalog-sealed-db
          envFrom:
            - configMapRef:
                name: catalog
          image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 3
          name: catalog
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            successThreshold: 3
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 512Mi
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          volumeMounts:
            - mountPath: /tmp
              name: tmp-volume
      securityContext:
        fsGroup: 1000
      serviceAccountName: catalog
      volumes:
        - emptyDir:
            medium: Memory
          name: tmp-volume

Diff

             - name: DB_USER
               valueFrom:
                 secretKeyRef:
                   key: username
-                  name: catalog-db
+                  name: catalog-sealed-db
             - name: DB_PASSWORD
               valueFrom:
                 secretKeyRef:
                   key: password
-                  name: catalog-db
+                  name: catalog-sealed-db
           envFrom:
             - configMapRef:
                 name: catalog
           image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0

~$kubectl apply -k ~/environment/eks-workshop/modules/security/sealed-secrets

~$kubectl rollout status -n catalog deployment/catalog --timeout 30s

The catalog-sealed-db which is a SealedSecret resource is safe to be stored in a Git repository along with YAML manifests pertaining to other Kubernetes resources such as DaemonSets, Deployments, ConfigMaps etc. deployed in the cluster. You can then use a GitOps workflow to manage the deployment of these resources to your cluster.