How to implement ArgoCD-Vault-Plugin with AWS Secrets Manager

One of the main difficult with the GitOps mentality is how to store properly our secrets. We all want the benefits of keeping every cluster manifest inside a git repository, but we also don’t want the risk of exposing our credentials within.

There is a bulk of solutions for that, HashiCorp Vault, Sealed Secrets and so on. All solutions have their pro and cons, and we should look at our current set of cluster to determine the best approach.

Here I will guide how to implement AWS Secrets Manager with ArgoCD-Vault-Plugin

argocd-vault-plugin + aws secrets manager

To demonstrate here, we will start everything from scratch.

First you need to have docker running with our computer. If you don’t have yet, please follow this link here.

Once you have docker running, you can install minikube or Kind to test it. Either way, both solutions works just fine. Another solution that helps understand our current cluster is Lens. Most of the DevOps engineer comes from the command line way of work, but I assume that some times is hard to visualize our current configuration.

Here is our steps to accomplish during this guide lines.

1. Install ArgoCD
2. Install ArgoCD-Vault-Plugin
3. Deploy Application with Secrets.
4. Verify the secrets injection.

1. Install Argo CD

$ kubectl create namespace argocd
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Here we will create a new namespace argocd first, where Argo CD services and application resources will live in and later deploy the installation manifest.

Once the process is complete, to access the argo-cd server you have to forward the right port first with the following command.

$ kubectl port-forward svc/argocd-server -n argocd 8080:443

You also have to extract the initial password to login in the UI.

$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

This command will show you the initial admin password. If you are planning for production, please change this!

After your first login you will see the main page here and we can say that the first step is complete. 🎉🎉🎉🎉

2. Install ArgoCD-Vault-Plugin

First we have to patch our current argocd-repo-server to add the new configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: argocd-repo-server
        env:
          - name: AWS_REGION
            value: us-east-1
          - name: AVP_TYPE
            value: awssecretsmanager
          - name: AWS_ACCESS_KEY_ID
            value: AWS_ACCESS_KEY_ID_VALUE
          - name: AWS_SECRET_ACCESS_KEY
            value: AWS_SECRET_ACCESS_KEY_VALUE
        volumeMounts:
        - name: custom-tools
          mountPath: /usr/local/bin/argocd-vault-plugin
          subPath: argocd-vault-plugin
volumes:
      - name: custom-tools
        emptyDir: {}
      initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]
# Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
        env:
          - name: AVP_VERSION
            value: "1.9.0"
        args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.9.0/argocd-vault-plugin_1.9.0_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
# Not strictly necessary, but required for passing AVP configuration from a secret and for using Kubernetes auth to Hashicorp Vault
      automountServiceAccountToken: true

Here I’m using the version 1.9.0, but please depending when you are reading this article, you should update the version.

This manifest will download the argocd-vault-plugin binary and make it available to the argocd-repo-server to use before the actual deployment. This way it will get the secrets, inject inside the manifest and only later it will apply to the cluster.

$ kubectl patch deployment argocd-repo-server -n argocd --patch-file argocd-repo-server.yaml

Now, we need to register the plugin in the argo-cm.yaml in order to invoke the plugin with the application deployment.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  configManagementPlugins: |
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]
    - name: argocd-vault-plugin-helm
      generate:
        command: ["sh", "-c"]
        args: ["helm template . | argocd-vault-plugin generate -"]
    # This lets you pass args to the Helm invocation as described here: https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#with-helm
    - name: argocd-vault-plugin-helm-with-args
      generate:
        command: ["sh", "-c"]
        args: ["helm template ${helm_args} . | argocd-vault-plugin generate -"]
    - name: argocd-vault-plugin-kustomize
      generate:
        command: ["sh", "-c"]
        args: ["kustomize build . | argocd-vault-plugin generate -"]

Here we will declare the plugin to work with simple manifest, helm, helm with arguments and also with kustomize.

To apply this patch, use this command line.

$ kubectl patch configmap argocd-cm -n argocd --patch-file patch.yaml

Now, it’s possible to see our plugins in the UI. 🎉🎉🎉

3. Deploy Application with Secrets

Now that we have the plugin binary running within our argocd-repo-server and the plugins register to use it, it’s time run our first secrets injection! 💉

First, we need to be sure that we have inside our AWS Secrets Manager a secret to be deliver. Inside the AWS Console, go to Secrets Manager and generate a new key/value.

In this case, we will use secret name repository as prod/secret. But here could be any combination of it. A good exercise will be having dev/staging/prod separation.

With our secrets in the Secrets Manager, now we can deploy one Application to see if it will inject our secrets correctly.

Inside our git repository, we need to declare our application manifest and point inside our ArgoCD to deploy it for us.

I’ve create on repository to host my manifest, if you want please check it out here.

You can follow the UI for build it or use the following manifest.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-secret
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: .
    repoURL: 'https://github.com/vitorpavani/argocd-vault-plugin'
    targetRevision: HEAD
    plugin:
      name: argocd-vault-plugin
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

After generating the Application, in the UI or using the following manifest, you should be able to see the following Application.

4. Verify the secrets injection

Now that we deploy one application with secrets, we need to be sure that the value of the key isn’t expose and still the secrets manifest is hosted inside the git repository.

kind: Secret
apiVersion: v1
metadata:
  name: example-secret
  annotations:
    avp.kubernetes.io/path: prod/secret
type: Opaque
stringData:
  sample-secret: <foo>

Here we can see that the placeholder “<foo>” is just and reference for the actual secrets. If we scan using the git leaks or another tool, you won’t have any problem.

Now inside the Lens we can see what is actually deployed in our cluster and we should see the secrets with the real value in it.

Notice that we never type the real value of the key in our manifest, but the vault plugin change it with the value retrieve from AWS Secrets Manager.

Congrats! We have now deploy our first secrets in our git repository without exposing it self. 🎉🎉🎉🎉🎉🎉