HashiDays One conference. Three cities. Find a city near you
Terraform
Terraform Home

Deploy Terraform Enterprise to Kubernetes

This topic describes how to deploy Terraform Enterprise to Kubernetes. You should have a deep understanding of Kubernetes before deploying Terraform Enterprise to a production Kubernetes environment.

Overview

You should deploy external service dependencies outside the Kubernetes cluster and scale reliably to accommodate Terraform Enterprise workloads. The following diagram shows the Terraform Enterprise architecture when deployed to Kubernetes-orchestrated containers:

Example of Kubernetes Architecture

Complete the following steps to install Terraform Enterprise:

  1. Complete the prerequisites.
  2. Install the Helm chart and apply your override values.
  3. Complete post installation tasks.

Complete post installation tasks, such creating the initial admin user account.

Prerequisites

Complete the following tasks before attempting to install Terraform Enterprise.

Prepare the deployment environment

Provide a DNS hostname for Terraform Enterprise and the associated TLS certificate. Additionally, you must configure your network so that your host can receive and send traffic. Refer to Prepare the host environment for details about preparing the host environment.

Deploy external storage systems

Deploy the database and other storage devices so that Terraform can connect to them when the application starts. Refer to Data storage settings overview for additional information.

Create the deployment configuration

Create a custom YAML configuration file, for example /tmp/overrides.yaml, to override the default values in the Terraform Enterprise Helm chart. The file contains settings for the operational mode, license, TLS certificates, and network configuration. Add any additional configurations necessary for your environment. Refer to Configuration file overview for additional information.

Externalizing secret values

The Terraform Enterprise deployment configuration requires several secret values, such as keys, passwords, and certificates. You can populate these as part of the Helm overrides.yaml created before installation. However, this requires manual handling of secret values and may leave them exposed on the deployment host without further intervention.

To mitigate this security concern, HashiCorp recommends populating these secret values into a Kubernetes Secret from a centralized secrets management platform using a trusted orchestrator like the Vault Secrets Operator before installing Terraform Enterprise.

You can then reference the Kubernetes Secret in the Terraform Enterprise Helm Chart:

env:
  secretRefs:
    - name: terraform-enterprise-managed-secrets

Automatic environment configuration

There are a collection of environment variables that are predetermined or computed when using the Terraform Enterprise Helm chart. When you configure the variables as .Values.env.variables entries, Terraform Enterprise overwrites them with the predetermined or computed values. Refer to the config-map.yaml template for version of the chart that you are using.

Running as non-root

By default, Terraform Enterprise runs as the terraform-enterprise user. If you want to enforce Terraform Enterprise as a non-root user, you must set the Helm chart values:

Install Terraform Enterprise with Helm

  1. Connect to the host instance.

  2. Log in to the Terraform Enterprise container image registry.

    $ cat <PATH_TO_HASHICORP_LICENSE_FILE> |  docker login --username terraform images.releases.hashicorp.com --password-stdin

  3. Pull the Terraform Enterprise image from the registry.

    $ docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>

  4. Create a custom namespace.

    $ kubectl create namespace <TFE_NAMESPACE>

  5. Create an image pull secret in <TFE_NAMESPACE> to fetch the terraform-enterprise container from the <DOCKER_REGISTRY_URL>. This URL can be images.releases.hashicorp.com, or your internal container registry. If you are using images.releases.hashicorp.com, use terraform as the <DOCKER_REGISTRY_USERNAME> parameter in the following command with --docker-password=$(cat /path/to/terraform.hclic)

    $ kubectl create secret docker-registry terraform-enterprise --docker-server=<DOCKER_REGISTRY_URL> --docker-username=<DOCKER_REGISTRY_USERNAME> --docker-password=<DOCKER_REGISTRY_PASSWORD>  -n <TFE_NAMESPACE>

  6. Add the Hashicorp Helm registry:

    $ helm repo add hashicorp https://helm.releases.hashicorp.com

  7. Render the terraform-enterprise chart with your custom values file <OVERRIDES_FILE>, for example tmp/overrides.yaml.

    $ helm template terraform-enterprise hashicorp/terraform-enterprise –n <TFE_NAMESPACE> --values <OVERRIDES_FILE>

  8. Install terraform-enterprise, this step can take several minutes.

    $ helm install terraform-enterprise hashicorp/terraform-enterprise –n <TFE_NAMESPACE> --values <OVERRIDES_FILE>

  9. Inspect terraform-enterprise pods to verify their successful start.

    $ kubectl get pods -n <TFE_NAMESPACE>

    If Terraform Enterprise pods fail to start, refer to Kubernetes Troubleshooting.

  10. By default, Terraform Enterprise installs a load balancer service. Retrieve the external IP address of this service.

    $ kubectl get services -n <TFE_NAMESPACE>

  11. Set up a DNS record that points to your external IP address to enable routing to your <TFE_HOSTNAME>. A DNS address is required to communicate with Terraform Enterprise, and it is managed outside of Kubernetes and the Terraform Enterprise helm chart or application.

  12. Validate the readiness of the Terraform Enterprise application by querying the health check endpoint.

    $ curl https://tfe.test.hashicorp.com/_health_check

Post installation tasks

Complete the following tasks after the initial installation.

Review startup checks

When you start Terraform Enterprise, several startup checks also run to prevent errors related to invalid configurations or certificates, as well as other issues that could prevent the application from running successfully or safely. Refer to the startup checks reference for additional information.

Create initial admin user

Provision your first administrative user and start using Terraform Enterprise.

Configure the security context

Modify the .securityContext Helm chart value to set the pod security configuration. Modify the .container.securityContext Helm chart value to set the container security configuration. You must also omit the allowPrivilegeEscalation container security context option or set it to true.

Create a custom Helm chart fork

The Terraform Enterprise Helm Chart is designed to meet the needs of the majority of our users. You can fork our Helm chart and adapt it to your organization’s requirements.

If you contact HashiCorp support, include your custom helm chart alongside your support bundle to ensure support has all the information they need.

Custom ingress

You can define an optional ingress resource using the ingress controller. Refer the Terraform Enterprise Helm Chart documentation for additional information about the controller.

Specify values for the ingress section in the deployment configuration. Refer to the example values file in the Terraform Enterprise Helm chart repository for a demonstration of how to enable ingress configuration.

Complete the following steps to set up an custom ingress configuration with Nginx:

  1. Install the nginx controller in a different namespace.
  2. Deploy Terraform Enterprise with Ingress configured in your values file.
  3. Get the address from the ingress resource like so:
$ kubectl get ingress
NAME                   CLASS   HOSTS        ADDRESS         PORTS     AGE
terraform-enterprise   nginx   <hostname>    <ip>           80, 443   60s

Example configurations

The following examples for each cloud-platform are based on cloud native hosted PostgreSQL, storage, or Redis cache services. You can copy an example configuration and modify the values to per your environment. Refer to Configuration Reference for a list of all configuration options.

The examples also depend on the following conditions:

  • Values under .env.variables are set as a ConfigMap and mounted as Terraform Enterprise environment variables.

  • Values under .env.secrets are set as Kubernetes secrets and mounted as Terraform Enterprise environment variables.

  • Extend the env.configMapRefs[] or env.secretRefs[] with your own resources to add additional ConfigMap or Secret resources within your environment configuration.

  • Values marked BASE_64_ENCODED* indicate that the value given must be base 64 encoded. If you are using this certificate configuration to host Terraform Enterprise web traffic, this value must be valid with the env.TFE_HOSTNAME, or match the wildcard pattern.

AWS Elastic Kubernetes Service (EKS)

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
  repository: images.releases.hashicorp.com
  name: hashicorp/terraform-enterprise
  tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g "xxx.us-west-2.rds.amazonaws.com:5432">
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=disable">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # S3 settings. For Server Side Encryption settings, see to the configuration reference.
    TFE_OBJECT_STORAGE_TYPE: s3
    TFE_OBJECT_STORAGE_S3_BUCKET: <S3 bucket name>
    TFE_OBJECT_STORAGE_S3_REGION: <S3 region>
    TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: <To use the pod's credentials to authenticate with AWS? e.g. false>
  secrets:
    TFE_DATABASE_PASSWORD: '<Database password>'
    TFE_OBJECT_STORAGE_S3_ACCESS_KEY_ID: <Required if TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE is false>
    TFE_OBJECT_STORAGE_S3_SECRET_ACCESS_KEY: '<Required if TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE is false>'
    TFE_REDIS_PASSWORD: '<Redis password>'
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: '<Encryption password>'

Google Kubernetes Engine (GKE)

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
 repository: images.releases.hashicorp.com
 name: hashicorp/terraform-enterprise
 tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g 11.22.33.44:5432>
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=require">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # Google Cloud Storage settings.
    TFE_OBJECT_STORAGE_TYPE: google
    TFE_OBJECT_STORAGE_GOOGLE_BUCKET: <Bucket name>
    TFE_OBJECT_STORAGE_GOOGLE_PROJECT: <GCP project ID>
  secrets:
    TFE_DATABASE_PASSWORD: '<Database password>'
    TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS: <BASE_64_ENCODED_SERVICE_ACCOUNT_CREDENTIALS>
    TFE_REDIS_PASSWORD: '<Redis password>'
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: '<Encryption password>'

Azure Kubernetes Service (AKS)

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
 repository: images.releases.hashicorp.com
 name: hashicorp/terraform-enterprise
 tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname (DNS) e.g. terraform.example.com>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g "xxx.postgres.database.azure.com:5432">
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=disable">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # Azure container storage settings.
    TFE_OBJECT_STORAGE_TYPE: azure
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME: <Azure storage account name>
    TFE_OBJECT_STORAGE_AZURE_CONTAINER: <Azure storage container name>
    TFE_OBJECT_STORAGE_AZURE_ENDPOINT: <Azure storage endpoint>
  secrets:
    TFE_DATABASE_PASSWORD: '<Database password>'
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY: '<Azure storage account key>'
    TFE_REDIS_PASSWORD: '<Redis password>'
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: '<Encryption password>'

Using AKS with Workload Identity

If you are using AKS with Workload Identity, the configuration is slightly different. You must set omit the TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY secret, and configure the AKS cluster as an OIDC provider.

serviceAccount:
  enabled: true
  name: "<SERVICE_ACCOUNT_NAME>"
  annotations:
    azure.workload.identity/client-id: "<AZURE_USER_ASSIGNED_IDENTITY_CLIENT_ID>"
  labels:
    azure.workload.identity/use: "true"
pod:
  labels:
    azure.workload.identity/use: "true"
agents:
  rbac:
    enabled: true

The <AZURE_USER_ASSIGNED_IDENTITY_CLIENT_ID> is the client_id from the Azure User Assigned Identity. One can find this value in the Azure portal.

This user assigned identity needs a federated identity credential with the following subject format "system:serviceaccount:${KUBERNETES_NAMESPACE}:${SERVICE_ACCOUNT_NAME}".

The issuer of the federated identity should be the cluster's issuer URL. One can retrieve this URL by running the following command:

$ az aks show --resource-group <RESOURCE_GROUP> --name <CLUSTER_NAME> --query "oidcIssuerProfile.issuerUrl" --output tsv