Doppler Blog Managing Kubernetes Secrets with the External Secrets Operator and Doppler

API keys, credentials, and other types of sensitive information are the primary way your application calls external services and interacts with the world outside its own runtime. Ensuring that your secrets are secure is one of the most important operational tasks that you need to address, but it's not just rigorous security that's important providing an ergonomic, audited, and maintainable flow for the management of those secrets is crucial as well. If managing credentials is confusing or difficult, mistakes can undo all of the careful work that goes into keeping secrets secure.Container orchestrators like Kubernetes offer helpful abstractions to address the need for sensitive values with native Secrets. However, a Kubernetes Secret is a somewhat rudimentary object it lacks encryption by default and normally exists as a plain key/value within etcd. By contrast, services like Vault or Doppler are intentionally designed to provide strong guarantees and well-defined access controls. Can we combine the operational power of Kubernetes with the assurance of a fully managed secret provider?Yes! The External Secrets Operator is a Kubernetes operator that bridges the gap between Kubernetes' native secret support and external systems that provide a canonical source of truth for secret storage. It does this by leveraging custom resources that define how to retrieve external secrets in order to manage the lifecycle of Secret resources in your Kubernetes cluster. In this tutorial, we'll use Doppler as the external secret provider to illustrate using external secrets in a real-world application.OutcomesThe end goal of this guide will be to leverage well-managed secrets in an application deployed in Kubernetes. To achieve this, we'll use:The Kubernetes External Secrets OperatorDoppler to store and manage external secretsminikube or kind for a local Kubernetes APIAt the conclusion of this tutorial, you'll understand how to store and manage secrets in Doppler, use those same secrets as native Kubernetes secrets, and ultimately use them in a running application. Let's begin!PrerequisitesThis article assumes that you'll follow along with your own Kubernetes cluster. You may choose to operate in an existing cluster (potentially within a sandboxed namespace), but if you'd like to use a sandbox instead, we suggest using either minikube or kind to create a local Kubernetes installation which will provide a safe environment for testing. In order to provide a clean and pristine environment, this tutorial will use minikube as the Kubernetes target.In addition to a functional Kubernetes cluster, ensure that the following command-line tools are installed:helm, which we'll use to install the External Secrets Operator Kubernetes resources. Use the helm quickstart guide to install helm.The Doppler command line utility doppler using the Doppler CLI Guide documentation.You will need to register for a Doppler account to create projects and store external secrets.kubectl to interact with Kubernetes. kubectl is available for a wide range of operating systems, and the Kubernetes documentation provides comprehensive installation guides for kubectl and other tools as well.If you choose to use a local Kubernetes sandbox, install minikube and follow the instructions to create a local instance of Kubernetes. This can be as simple as minikube start, but consult the documentation if you need additional assistance.minikube bundles a version-compatible installation of kubectl if you choose to use minikube for this exercise. It's a handy time saver.Once installed, confirm that your environment is ready to go:kubectl version should confirm that kubectl is present and communicating with the Kubernetes API successfully. If you aren't using minikube, simply use the plain kubectl command for the remainder of the tutorial.minikube kubectl -- versionClient Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}The helm and doppler commands should be available:helm versionversion.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.13"}doppler --versionv3.44.0 With the prerequisites installed, let's proceed with building and running our program.Sample ApplicationBefore we start creating secrets, let's begin by illustrating the external secrets workflow with a simple example service. The following guide will assume the use of minikube to build and load an application container image, so if your Kubernetes environment is different, you may need to adapt the instructions to work with your specific container registry strategy.First, perform some initial steps to bootstrap a python Flask application:mkdir -p ~/tmp/secret-sauce cd ~/tmp/secret-sauce git init echo flask > requirements.txt python3 -m venv venv ./venv/bin/pip install -r requirements.txtCreate a file named app.py that contains the following simple web application:from flask import Flask from os import environ app = Flask(__name__) sauce = environ.get('SECRET_SAUCE') @app.route("/") def index(): if sauce: return f"The secret sauce is: {sauce}!" else: return "You'll never find my secret sauce." Define a small Dockerfile that defines how to build a container image for our application:FROM python:3 WORKDIR /usr/src/app COPY requirements.txt ./ RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD [ "python", "-m" , "flask", "run", "--host=0.0.0.0"]You can run this application locally with:./venv/bin/flask runTry accessing the running application at http://localhost:5000 to see a response. The root route (/) will render a static string when no secret is present in the environment but will print the secret when the indicated environment variable is defined. Don't print secrets in production! This application is just a demonstration of how to read the value in a real program.We're ready to load this application into Kubernetes. Assuming that you're following along with minikube, proceed to build the application container in the Kubernetes node under the name "app":minikube image build -t app ....lots of output... Successfully tagged app:latestCreate a new Kubernetes Deployment file named deployment.yaml:apiVersion: apps/v1 kind: Deployment metadata: name: app spec: replicas: 1 selector: matchLabels: app: flask template: metadata: labels: app: flask spec: containers: - name: webapp image: app imagePullPolicy: Never ports: - containerPort: 5000 Take note of a few assumptions in this Deployment:imagePullPolicy has been set to Never because we're running a locally-built image. By default, Kubernetes would attempt to pull the app image, which doesn't exist. As previously mentioned, if you're following this tutorial in an environment other than minikube, you may need to push the container image to a registry and adjust these settings slightly.We've exposed port :5000 which we can access later.Load this Deployment into your running cluster:kubectl apply -f deployment.yamldeployment.apps/app createdFinally, forward the port in another terminal window in order to access the running application in a simple tunnel.kubectl port-forward deployment/app 5000:5000Forwarding from 127.0.0.1:5000 -> 5000 Forwarding from [::1]:5000 -> 5000 Send a request to your Kubernetes application to see it in action:curl http://localhost:5000You'll never find my secret sauce. Fantastic! Note that we've received the response that indicates no secret value has been injected into the environment. How can we add a secret to the application and see the secret sauce?DopplerBy using Doppler, we can achieve a great deal of control over application secrets and manage them at each step of our application's lifecycle, from in our local development environment to within a Kubernetes workload.If you haven't signed up with Doppler, you can do so now. Once you have an account, proceed to use the login command to set up your account locally:doppler loginSetup Doppler for the demo application by defining a doppler-template.yaml file in the root of the application directory. This YAML file defines a template that we can import to create a new Doppler project easily from the command line:projects: - name: secret-sauce description: Kubernetes demo app environments: - slug: dev name: Development configs: - slug: dev - slug: stg name: Staging configs: - slug: stg - slug: prd name: Production configs: - slug: prd secrets: dev: SECRET_SAUCE: tartar stg: SECRET_SAUCE: horseradish prd: SECRET_SAUCE: tzatziki Enter the directory in your shell and run the following doppler command in order to bootstrap your Doppler project. This will create a new project called secret-sauce, set up a few different environments, and load an initial secret value for SECRET_SAUCE:doppler importYou'll see output similar to the following: ID NAME DESCRIPTION CREATED AT secret-sauce secret-sauce Kubernetes demo app 2022-10-11T22:10:36.567Z View the secrets for this project with doppler secrets:doppler secretsIn addition to some default variables that begin with DOPPLER_, you'll also find the secret value we'd like to inject, SECRET_SAUCE: NAME VALUE DOPPLER_CONFIG dev DOPPLER_ENVIRONMENT dev DOPPLER_PROJ