• Overview
    • Enforce Policy as Code
    • Infrastructure as Code
    • Inject Secrets into Terraform
    • Integrate with Existing Workflows
    • Manage Kubernetes
    • Manage Virtual Machine Images
    • Multi-Cloud Deployment
    • Network Infrastructure Automation
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
  • Registry
  • Tutorials
    • About the Docs
    • Intro to Terraform
    • Configuration Language
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
    • Provider Use
    • Plugin Development
    • Registry Publishing
    • Integration Program
    • Terraform Tools
    • CDK for Terraform
    • Glossary
  • Community
GitHubTerraform Cloud
Download

    Terraform Enterprise Admin

  • Overview
    • Credentials
    • Hardware
      • Supported OS
      • RedHat Linux
      • CentOS Linux
      • Operational Mode
      • PostgreSQL
      • Minio Setup Guide
      • External Vault
    • Network
    • Docker Engine
  • Operational Modes
    • Overview
    • AWS Reference Architecture
    • Azure Reference Architecture
    • GCP Reference Architecture
    • VMware Reference Architecture
    • Pre-Install Checklist
      • 1. Run Installer
      • 2. Configure in Browser
      • Automated Installation
      • Active/Active
      • Initial User Automation
      • Encryption Password
    • Uninstall
    • Configuration
    • Team Membership
    • Attributes
    • Login
      • Sample Auth Request
      • ADFS
      • Azure Active Directory
      • Okta
      • OneLogin
    • Troubleshooting
    • Overview
      • Automated Recovery
      • Upgrades
      • Log Forwarding
      • Monitoring
      • Backups and Restores
      • Admin CLI Commands
      • Terraform Cloud Agents on TFE
      • Demo to Mounted Disk Migration
    • Terraform Cloud Agents on TFE
      • Accessing the Admin Interface
      • General Settings
      • Customization
      • Integration Settings
      • Managing Accounts & Resources
      • Module Sharing
      • Admin API
      • Updating Terraform Enterprise License
    • Terraform Enterprise Logs
    • Overview
    • Architecture Summary
    • Reliability & Availability
    • Capacity & Performance
    • Security Model
    • Overview
      • Overview
      • v202206-1
      • v202205-1
      • v202204-2
      • v202204-1
      • v202203-1
      • v202202-1
      • v202201-2
      • v202201-1
      • Overview
      • v202112-2
      • v202112-1
      • v202111-1
      • v202110-1
      • v202109-2
      • v202109-1
      • v202108-1
      • v202107-1
      • v202106-1
      • v202105-1
      • v202104-1
      • v202103-3
      • v202103-2
      • v202103-1
      • v202102-2
      • v202102-1
      • v202101-1
      • Overview
      • Overview
      • Overview
  • Support
  • Application Usage

  • Overview
  • Plans and Features
  • Getting Started
    • API Docs template
    • Overview
    • Account
    • Agent Pools
    • Agent Tokens
    • Applies
    • Audit Trails
    • Comments
    • Configuration Versions
    • Cost Estimates
    • Feature Sets
    • Invoices
    • IP Ranges
    • Notification Configurations
    • OAuth Clients
    • OAuth Tokens
    • Organizations
    • Organization Memberships
    • Organization Tags
    • Organization Tokens
    • Plan Exports
    • Plans
    • Policies
    • Policy Checks
    • Policy Sets
    • Policy Set Parameters
      • Modules
      • Providers
      • Private Provider Versions and Platforms
      • GPG Keys
    • Runs
      • Run Tasks
      • Stages and Results
      • Custom Integration
    • Run Triggers
    • SSH Keys
    • State Versions
    • State Version Outputs
    • Subscriptions
    • Team Access
    • Team Membership
    • Team Tokens
    • Teams
    • User Tokens
    • Users
    • Variables
    • VCS Events
    • Workspaces
    • Workspace-Specific Variables
    • Workspace Resources
    • Variable Sets
      • Overview
      • Module Sharing
      • Organizations
      • Runs
      • Settings
      • Terraform Versions
      • Users
      • Workspaces
    • Changelog
    • Stability Policy
    • Overview
    • Creating Workspaces
    • Naming
    • Terraform Configurations
      • Overview
      • Managing Variables
      • Overview
      • VCS Connections
      • Access
      • Drift Detection
      • Notifications
      • SSH Keys for Modules
      • Run Triggers
      • Run Tasks
    • Terraform State
    • JSON Filtering
    • Remote Operations
    • Viewing and Managing Runs
    • Run States and Stages
    • Run Modes and Options
    • UI/VCS-driven Runs
    • API-driven Runs
    • CLI-driven Runs
    • The Run Environment
    • Installing Software
    • Users
    • Teams
    • Organizations
    • Permissions
    • Two-factor Authentication
    • API Tokens
      • Overview
      • Microsoft Azure AD
      • Okta
      • SAML
      • Linking a User Account
      • Testing
    • Overview
    • GitHub.com
    • GitHub.com (OAuth)
    • GitHub Enterprise
    • GitLab.com
    • GitLab EE and CE
    • Bitbucket Cloud
    • Bitbucket Server and Data Center
    • Azure DevOps Services
    • Azure DevOps Server
    • Troubleshooting
    • Overview
    • Adding Public Providers and Modules
    • Publishing Private Providers
    • Publishing Private Modules
    • Using Providers and Modules
    • Configuration Designer
  • Migrating to Terraform Cloud
    • Overview
    • Using Sentinel with Terraform 0.12
    • Manage Policies
    • Enforce and Override Policies
    • Mocking Terraform Sentinel Data
    • Working With JSON Result Data
      • Overview
      • tfconfig
      • tfconfig/v2
      • tfplan
      • tfplan/v2
      • tfstate
      • tfstate/v2
      • tfrun
    • Example Policies
    • Overview
    • AWS
    • GCP
    • Azure
      • Overview
      • Service Catalog
      • Admin Guide
      • Developer Reference
      • Example Customizations
      • V1 Setup Instructions
    • Splunk Integration
    • Kubernetes Integration
    • Run Tasks Integration
    • Overview
    • IP Ranges
    • Data Security
    • Security Model
    • Overview
    • Part 1: Overview of Our Recommended Workflow
    • Part 2: Evaluating Your Current Provisioning Practices
    • Part 3: How to Evolve Your Provisioning Practices
    • Part 3.1: From Manual Changes to Semi-Automation
    • Part 3.2: From Semi-Automation to Infrastructure as Code
    • Part 3.3: From Infrastructure as Code to Collaborative Infrastructure as Code
    • Part 3.4: Advanced Workflow Improvements

  • Terraform Cloud Agents

  • Other Docs

  • Intro to Terraform
  • Configuration Language
  • Terraform CLI
  • Terraform Cloud
  • Terraform Enterprise
  • Provider Use
  • Plugin Development
  • Registry Publishing
  • Integration Program
  • Terraform Tools
  • CDK for Terraform
  • Glossary
Type '/' to Search

»Reliability and Availability

This section covers details relating to the reliability and availability of Terraform Enterprise installations. This documentation may be useful to customers evaluating Terraform Enterprise or operators responsible for installing and maintaining Terraform Enterprise.

»Components

Terraform Enterprise consists of several distinct components that each play a role when considering the reliability of the overall system:

  • Application Layer

    • TFE Core - A Rails application at the center of Terraform Enterprise; consists of web frontends and background workers

    • TFE Services - A set of Go services that provide various pieces of key functionality for Terraform Enterprise

    • Terraform Workers - A fleet of isolated execution environments that perform Terraform Runs on behalf of Terraform Enterprise users

  • Coordination Layer

    • Redis - Used for Rails caching and coordination between TFE Core's web and background workers

    • RabbitMQ Used for Terraform Worker job coordination

  • Storage Layer

    • PostgreSQL Database - Serves as the primary store of Terraform Enterprise's application data such as workspace settings and user settings

    • Blob Storage - Used for storage of Terraform state files, plan files, configuration, and output logs

    • HashiCorp Vault - Used for encryption of sensitive data. There are two types of Vault data in Terraform Enterprise - key material and storage backend data.

    • Configuration Data - The information provided and/or generated at install-time (e.g. database credentials, hostname, etc.)

»Operation Modes

This section describes how to set up your Terraform Enterprise deployment to recover from failures in the various operational modes (Mounted Disk, External Services). The operational mode is selected at install time and can not be changed once the install is running.

The below tables explain where each data type in the Storage Layer is stored and the corresponding snapshot and restore procedure. For the data types that use Terraform Enterprise's built-in snapshot and restore function, follow these instructions. For the data types that do not use the built-in functionality, backup and restore is the responsibility of the user.

Data Location

ConfigurationVaultPostgreSQLBlob Storage
Mounted DiskStored in Docker volumes on instanceKey material on host in /var/lib/tfe-vault, storage backend is mounted disk PostgreSQLStored in mounted disksStored in mounted disks
External ServicesStored in Docker volumes on instanceKey material on host in /var/lib/tfe-vault, storage backend is external PostgreSQLStored in external serviceStored in external service
External Vault-Key material in external Vault with user-defined storage backend--

Backup and Restore Responsibility

ConfigurationVaultPostgreSQLBlob Storage
Mounted DiskTerraform EnterpriseTerraform EnterpriseUserUser
External ServicesTerraform EnterpriseTerraform EnterpriseUserUser
External Vault-User--

»Mounted Disk

PostgreSQL Database and Blob Storage use mounted disks for their data. Backup and restore of those volumes is the responsibility of the user, and is not managed by Terraform Enterprise's built-in systems.

Vault Data is stored in PostgreSQL and accordingly lives on the mounted disk. As long as the user has restored the mounted disk successfully, the built-in restore mechanism will restore Vault operations in the event of a failure.

Configuration Data for the installation is stored in Docker volumes on the instance. The built-in snapshot mechanism can package up the Configuration data and store it off the instance, and the built-in restore mechanism can recover the configuration data and restore operation in the event of a failure. Configure snapshot and restore by following the automated recovery instructions.

If the instance running Terraform Enterprise is lost, the use of mounted disks means no state data is lost.

»External Services

In the External Services operation mode, the Application Layer and Coordination Layer execute on a Linux instance, but the Storage Layer is configured to use external services in the form of a PostgreSQL server and an S3-compatible Blob Storage.

The maintenance of PostgreSQL and Blob Storage are handled by the user, which includes backing up and restoring if necessary.

Vault Data is stored in PostgreSQL. As long as PostgreSQL has been restored successfully by the user, the built-in restore mechanism will restore Vault operations in the event of a failure.

Configuration Data for the installation is stored in Docker volumes on the instance. The built-in snapshot mechanism can package up the data and store it off the instance, and the built-in restore mechanism can recover the data and restore operation in the event of a failure. Configure snapshot and restore by following the automated recovery instructions.

If the instance running Terraform Enterprise is lost, the use of external services means no state data is lost.

NOTE: Customers running an optional external Vault cluster are responsible for backing up the Vault data and restoring it if necessary.

»Availability During Upgrades

Upgrades use the installer dashboard. Once an upgrade has been been detected (either online or airgap), the new code is imported. Once ready, all services on the instance are restarted running the new code. The expected downtime is between 30 seconds and 5 minutes, depending on whether database updates have to be applied.

Only application services are changed during the upgrade; data is not backed up or restored. The only data changes that may occur during upgrade are the application of migrations the new version might apply to the PostgreSQL Database.

When an upgrade is ready to start the new code, the system waits for all Terraform runs to finish before continuing. Once the new code has started, the queue of runs is continued in the same order.

github logoEdit this page
  • Overview
  • Docs
  • Extend
  • Privacy
  • Security
  • Press Kit
  • Consent Manager