Secret Store

Overview

Secret Manager selection is crucial for securing your application. Different modes can be set up depending on the deployment type. Our system supports four primary secret manager types.

Secret Manager Factory

The SecretManagerFactory is a utility class used to create instances of different types of secret managers. It leverages the Factory design pattern to abstract the creation logic based on the type of secret manager required. The factory supports creating instances of File, GCP, Kubernetes, and Vault Secret Managers. The SECRET_MANAGER_TYPE environment variable plays a crucial role in the SecretManagerFactory for determining the default type of secret manager to be instantiated when no specific type is provided in the method call. Functionality: Default Secret Manager: If the SECRET_MANAGER_TYPE environment variable is set, its value dictates the default type of secret manager that the factory will create. The value of this variable should correspond to one of the types defined in SecretManagerTypes enum (FILE, AWS, GCP, K8S, VAULT, DB). Example Configuration: Setting SECRET_MANAGER_TYPE=GCP in the environment will make the factory create instances of GcpSecretManager by default. If SECRET_MANAGER_TYPE is not set or is set to FILE, the factory defaults to creating instances of FileSecretManager. This environment variable provides flexibility and ease of configuration, allowing different secret managers to be used in different environments or scenarios without code changes.

File Secret Manager

The FileSecretManager is a concrete implementation of the BaseSecretManager for managing secrets stored in the file system. It uses a specified directory (defaulting to ./) to read, write, and delete secret files. Configuration: Set the environment variable SECRET_MANAGER_DIRECTORY to specify the directory where secrets are stored. If not set, defaults to the current directory (./). Usage:

Secrets are stored as files in the specified directory.
Reading a secret involves fetching content from a file.
Writing a secret creates or updates a file with the given content.
Deleting a secret removes the corresponding file.

AWS Secret Manager

The AwsSecretManager integrates with Amazon Web Services’ Secrets Manager service for secure secret management. It provides a robust solution for storing and managing secrets in AWS environments. Configuration: Required environment variables:

AWS_REGION: The AWS region where your secrets are stored
For local development:
- AWS_ACCESS_KEY_ID: Your AWS access key
- AWS_SECRET_ACCESS_KEY: Your AWS secret access key Optional:
AWS_KMS_KEY_ID: The KMS key ID to use for encrypting secrets
AWS_SECRET_MANAGER_TAGS: Comma-separated list of tags to add to the secret in AWS Secrets Manager, e.g. key=value,key2=value2
AWS_SECRET_ROTATION_ENABLED: Set to true to enable automatic rotation of secrets (default: false)
AWS_SECRET_ROTATION_DAYS: Number of days between automatic rotations (default: 30)
AWS_SECRET_ROTATION_LAMBDA_ARN: ARN of the Lambda function to use for secret rotation, required if rotation is enabled

Usage:

Manages secrets using AWS Secrets Manager service
Supports creating, updating, reading, and deleting secrets
Can automatically configure secret rotation policies when creating new secrets

AWS Secret Rotation

Secret rotation is a security best practice that automatically updates secrets at regular intervals. When enabled, Keep will configure newly created secrets with a rotation schedule. To use secret rotation:

Create a Lambda function for rotating your secrets (AWS provides blueprints for common rotation scenarios)
Set AWS_SECRET_ROTATION_ENABLED=true in your environment
Set AWS_SECRET_ROTATION_LAMBDA_ARN to the ARN of your rotation Lambda function
Optionally set AWS_SECRET_ROTATION_DAYS to customize the rotation interval

Example Lambda ARN format: arn:aws:lambda:region:account-id:function:function-name Note: Different secret types (database credentials, API keys, etc.) require different rotation logic. Make sure your Lambda function is appropriate for the type of secrets you’re storing.

Kubernetes Secret Manager

Overview

The KubernetesSecretManager interfaces with Kubernetes’ native secrets system. It manages secrets within a specified Kubernetes namespace and is designed to operate within a Kubernetes cluster.

Configuration

SECRET_MANAGER_TYPE=k8s
K8S_NAMESPACE=keep - environment variable to specify the Kubernetes namespace. Defaults to .metadata.namespace if not set. Assumes Kubernetes configurations (like service account tokens) are properly set up when running within a cluster.
K8S_VERIFY_SSL_CERT=true - environment variable to specify whether to verify the SSL certificate of the Kubernetes API. Defaults to true.

Usage:

Secrets are stored as Kubernetes Secret objects.
Provides functionalities to create, retrieve, and delete Kubernetes secrets.
Handles base64 encoding and decoding as required by Kubernetes.

Environment Variables From Secrets

The Kubernetes Secret Manager integration allows Keep to fetch environment variables from Kubernetes Secrets. For sensitive environment variables, such as DATABASE_CONNECTION_STRING, it is recommended to store as a secret:

Creating Database Connection Secret

# Create the base64 encoded string without newline
CONNECTION_STRING_B64=$(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)

# Create the Kubernetes secret
kubectl create secret generic keep-db-secret \
  --namespace=keep \
  --from-literal=connection_string=$(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)

# Or using a YAML file:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: keep-db-secret
  namespace: keep
type: Opaque
data:
  connection_string: $(echo -n "mysql+pymysql://user:password@host:3306/dbname" | base64)
EOF

Update the helm Values.yaml

After creating the secret, update the values.yaml so the helm chart will inject the secret as env var:

backend:
  enabled: true
  waitForDatabase: true
  databaseConnectionStringFromSecret:
    enabled: true  # Enable using secret for database connection
    secretName: "keep-db-secret"  # Name of the secret we created
    secretKey: "connection_string"  # Key in the secret containing our connection string

Apply with Helm

# If installing for the first time
helm install keep keephq/keep \
  -f values.yaml \
  --namespace keep

# If updating existing installation
helm upgrade keep keephq/keep \
  -f values.yaml \
  --namespace keep

Verify the installation

Check if the secret is properly created:

kubectl get secret keep-db-secret -n keep

Verify the content of the secret is correct:

kubectl get secret keep-db-secret -n keep -o jsonpath='{.data.connection_string}' | base64 -d

Verify the pod using the secret:

kubectl get pod -n keep -l app.kubernetes.io/component=backend -o yaml | grep DATABASE_CONNECTION_STRING -A 5

GCP Secret Manager

The GcpSecretManager utilizes Google Cloud’s Secret Manager service for secret management. It requires setting up with Google Cloud credentials and a project ID. Configuration: Ensure the environment variable GOOGLE_CLOUD_PROJECT is set with your Google Cloud project ID. Usage:

Secrets are managed using Google Cloud’s Secret Manager.
Supports operations to create, access, and delete secrets in the cloud.
Integrates with OpenTelemetry for tracing secret management operations.

Hashicorp Vault Secret Manager

The VaultSecretManager is tailored for Hashicorp Vault, a tool for managing sensitive data. It supports token-based authentication as well as Kubernetes-based authentication for Vault. Configuration:

Set HASHICORP_VAULT_ADDR to the Vault server address. Defaults to http://localhost:8200.
Use HASHICORP_VAULT_TOKEN for token-based authentication.
Set HASHICORP_VAULT_USE_K8S to True and provide HASHICORP_VAULT_K8S_ROLE for Kubernetes-based authentication.

Usage:

Manages secrets in a Hashicorp Vault server.
Provides methods to write, read, and delete secrets from Vault.
Supports different Vault authentication methods including static tokens and Kubernetes service account tokens.

DB Secret Manager

The DbSecretManager is a concrete implementation of the BaseSecretManager for managing secrets stored in the DB. It uses table secret to read, write, and delete secret. Configuration: Ensure table secret exists. Usage:

Secrets are stored in table secret.

Overview

AIOps

Alerts

Incidents

Workflow Automation

Alert Evaluation Engine

Providers

Deployment

Development

Keep CLI

Secret Store

Overview

Secret Manager Factory

File Secret Manager

AWS Secret Manager

AWS Secret Rotation

Kubernetes Secret Manager

Overview

Configuration

Environment Variables From Secrets

Creating Database Connection Secret

Update the helm Values.yaml

Apply with Helm

Verify the installation

GCP Secret Manager

Hashicorp Vault Secret Manager

DB Secret Manager

​Overview

​Secret Manager Factory

​File Secret Manager

​AWS Secret Manager

​AWS Secret Rotation

​Kubernetes Secret Manager

​Overview

​Configuration

​Environment Variables From Secrets

​Creating Database Connection Secret

​Update the helm Values.yaml

​Apply with Helm

​Verify the installation

​GCP Secret Manager

​Hashicorp Vault Secret Manager

​DB Secret Manager

Overview

Secret Manager Factory

File Secret Manager

AWS Secret Manager

AWS Secret Rotation

Kubernetes Secret Manager

Overview

Configuration

Environment Variables From Secrets

Creating Database Connection Secret

Update the helm Values.yaml

Apply with Helm

Verify the installation

GCP Secret Manager

Hashicorp Vault Secret Manager

DB Secret Manager