HashiConf Global Join us for HashiConf Global October 4-6 in Los Angeles & online. Register Now
  • Overview
    • Enforce Policy as Code
    • Infrastructure as Code
    • Inject Secrets into Terraform
    • Integrate with Existing Workflows
    • Manage Kubernetes
    • Manage Virtual Machine Images
    • Multi-Cloud Deployment
    • Network Infrastructure Automation
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
  • Registry
  • Tutorials
    • About the Docs
    • Intro to Terraform
    • Configuration Language
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
    • Provider Use
    • Plugin Development
    • Registry Publishing
    • Integration Program
    • Terraform Tools
    • CDK for Terraform
    • Glossary
  • Community
GitHub
Download
Try Terraform Cloud
    • v202208-2
    • v202208-3 (latest)
    • v202208-1
    • v202207-2
    • v202207-1
    • v202206-1

    Terraform Enterprise

  • Overview
    • API Docs template
    • Overview
    • Account
      • Overview
      • Module Sharing
      • Organizations
      • Runs
      • Settings
      • Terraform Versions
      • Users
      • Workspaces
    • Agent Pools
    • Agent Tokens
    • Applies
    • Audit Trails
    • Assessment Results
    • Comments
    • Configuration Versions
    • Cost Estimates
    • Feature Sets
    • Invoices
    • Notification Configurations
    • OAuth Clients
    • OAuth Tokens
    • Organizations
    • Organization Memberships
    • Organization Tags
    • Organization Tokens
    • Plan Exports
    • Plans
    • Policies
    • Policy Checks
    • Policy Sets
    • Policy Set Parameters
      • Modules
      • Providers
      • Private Provider Versions and Platforms
      • GPG Keys
    • Runs
      • Run Tasks
      • Stages and Results
      • Custom Integration
    • Run Triggers
    • SSH Keys
    • State Versions
    • State Version Outputs
    • Subscriptions
    • Team Access
    • Team Membership
    • Team Tokens
    • Teams
    • User Tokens
    • Users
    • Variables
    • VCS Events
    • Workspaces
    • Workspace-Specific Variables
    • Workspace Resources
    • Variable Sets
    • Changelog
    • Stability Policy
    • Credentials
    • Hardware
      • Supported OS
      • RedHat Linux
      • CentOS Linux
      • Operational Mode
      • PostgreSQL
      • Minio Setup Guide
      • External Vault
    • Network
    • Docker Engine
  • Operational Modes
    • Overview
    • AWS Reference Architecture
    • Azure Reference Architecture
    • GCP Reference Architecture
    • VMware Reference Architecture
    • Pre-Install Checklist
      • 1. Run Installer
      • 2. Configure in Browser
      • Automated Installation
      • Active/Active
      • Initial User Automation
      • Encryption Password
    • Uninstall
    • Overview
      • Automated Recovery
      • Upgrades
      • Log Forwarding
      • Monitoring
      • Backups and Restores
      • Admin CLI Commands
      • Terraform Cloud Agents on TFE
      • Demo to Mounted Disk Migration
    • Terraform Cloud Agents on TFE
      • Accessing the Admin Interface
      • General Settings
      • Customization
      • Integration Settings
      • Managing Accounts & Resources
      • Module Sharing
      • Admin API
      • Updating Terraform Enterprise License
    • Terraform Enterprise Logs
    • Users
    • Teams
    • Organizations
    • Permissions
    • Two-factor Authentication
    • API Tokens
    • Configuration
    • Team Membership
    • Attributes
    • Login
      • Sample Auth Request
      • ADFS
      • Azure Active Directory
      • Okta
      • OneLogin
    • Troubleshooting
    • Overview
    • Creating Workspaces
    • Naming
    • Terraform Configurations
      • Overview
      • Managing Variables
      • Overview
      • VCS Connections
      • Access
      • Notifications
      • SSH Keys for Modules
      • Run Triggers
      • Run Tasks
    • Terraform State
    • JSON Filtering
    • Remote Operations
    • Viewing and Managing Runs
    • Run States and Stages
    • Run Modes and Options
    • UI/VCS-driven Runs
    • API-driven Runs
    • CLI-driven Runs
    • The Run Environment
    • Installing Software
    • Overview
    • GitHub.com (OAuth)
    • GitHub Enterprise
    • GitLab.com
    • GitLab EE and CE
    • Bitbucket Cloud
    • Bitbucket Server and Data Center
    • Azure DevOps Services
    • Azure DevOps Server
    • Troubleshooting
    • Overview
    • Adding Public Providers and Modules
    • Publishing Private Providers
    • Publishing Private Modules
    • Using Providers and Modules
    • Configuration Designer
  • Migrating to Terraform Enterprise
    • Overview
    • Using Sentinel with Terraform 0.12
    • Manage Policies
    • Enforce and Override Policies
    • Mocking Terraform Sentinel Data
    • Working With JSON Result Data
      • Overview
      • tfconfig
      • tfconfig/v2
      • tfplan
      • tfplan/v2
      • tfstate
      • tfstate/v2
      • tfrun
    • Example Policies
    • Overview
    • AWS
    • GCP
    • Azure
      • Overview
      • Service Catalog
      • Admin Guide
      • Developer Reference
      • Example Customizations
      • V1 Setup Instructions
    • Run Tasks Integration
    • Overview
    • Architecture Summary
    • Reliability & Availability
    • Capacity & Performance
    • Security Model
    • Data Security
    • Overview
      • Overview
      • v202208-3
      • v202208-2
      • v202208-1
      • v202207-2
      • v202207-1
      • v202206-1
      • v202205-1
      • v202204-2
      • v202204-1
      • v202203-1
      • v202202-1
      • v202201-2
      • v202201-1
      • Overview
      • v202112-2
      • v202112-1
      • v202111-1
      • v202110-1
      • v202109-2
      • v202109-1
      • v202108-1
      • v202107-1
      • v202106-1
      • v202105-1
      • v202104-1
      • v202103-3
      • v202103-2
      • v202103-1
      • v202102-2
      • v202102-1
      • v202101-1
      • Overview
      • Overview
      • Overview
  • Support

  • Terraform Cloud Agents

  • Other Docs

  • Intro to Terraform
  • Configuration Language
  • Terraform CLI
  • Terraform Cloud
  • Terraform Enterprise
  • Provider Use
  • Plugin Development
  • Registry Publishing
  • Integration Program
  • Terraform Tools
  • CDK for Terraform
  • Glossary
Type '/' to Search

»Terraform Enterprise Admin CLI Commands

The Active/Active operational mode disables the Replicated Admin Console. Instead, it provides admin CLI commands to change the configuration, stop the application safely, and produce support bundles. You must use SSH to log in to a node in the Active/Active cluster to run these commands.

Admin CLI commands are available on installations using the Standalone operational mode.

»Commands

Note that tfe-admin is an alias for replicated admin, and can be used interchangeably.

»support-bundle

tfe-admin support-bundle
tfe-admin support-bundle

This command generates a support bundle for all nodes.

For Standalone (single node) installations the support bundle will be created in /var/lib/replicated/support-bundles.

For Active/Active, the support bundles will be uploaded to the same object store bucket that is used to store Terraform state files. The support bundles for a specific run of the admin command will all be uploaded to a directory with the same JobID, which is a timestamp in RFC3339 format. If you are sending a support bundle to HashiCorp Support, package and send all associated bundles to ensure that we have all the necessary information.

Example upload structure

support-bundles
└── 2020-11-10T02:03:05Z
    ├── 10.0.0.5
    │   └── replicated-support702524260.tar.gz
    └── 10.0.0.6
        └── replicated-support577188727.tar.gz
support-bundles
└── 2020-11-10T02:03:05Z
    ├── 10.0.0.5
    │   └── replicated-support702524260.tar.gz
    └── 10.0.0.6
        └── replicated-support577188727.tar.gz

»node-drain

tfe-admin node-drain
tfe-admin node-drain

This command will quiesce the current node and remove it from service. It will allow current work to complete and safely stop the node from picking up any new jobs from the Redis queue, allowing the application to be safely stopped. Currently, it only affects localhost (it does not support running on one node to drain other nodes).

Note: There is no reverse drain command - a restart is needed to restore the node.

»app-config

tfe-admin app-config -k <KEY> -v <VALUE>
tfe-admin app-config -k <KEY> -v <VALUE>

This command allows you to use the CLI to make real-time application changes, such as capacity_concurrency. You must provide both an allowable <KEY> (setting name) and <VALUE> (new setting value). Run replicatedctl app-config export for a complete list of the current app-config settings.

For the configuration changes to take effect, you must restart the Terraform Enterprise application on each node instance. To restart Terraform Enterprise:

  1. Run replicatedctl app stop to stop the application.
  2. Run replicatedctl app status to confirm the application is stopped.
  3. Run replicatedctl app start to start the application.

Note: You should ensure that any ad hoc changes made in this fashion are captured in the standard node build configuration, as the next time you build/rebuild a node only the configuration stored for that purpose will be in effect and ad hoc changes could be lost.

Hint: Adding a function to your Linux start-up like an alias can give you a short cut to the admin app-config command only requiring a single command and parameters, such as:

# shortcut: tfe-app-config <KEY> <VALUE>
tfe-app-config ()
{
        tfe-admin app-config -k "$1" -v "$2"
}
# shortcut: tfe-app-config <KEY> <VALUE>
tfe-app-config ()
{
        tfe-admin app-config -k "$1" -v "$2"
}

»list-nodes

tfe-admin list-nodes
tfe-admin list-nodes

This command lists the IP addresses of all active nodes in the installation. Nodes send a heartbeat every 5 seconds to signal that they are active. If Terraform Enterprise does not receive a heartbeat from a node within 30 seconds, it considers the node inactive and removes the node from the list.

»rotate-encryption-password

tfe-admin rotate-encryption-password CURRENT_PASSWORD NEW_PASSWORD
tfe-admin rotate-encryption-password CURRENT_PASSWORD NEW_PASSWORD

This command rotates the encryption password in use by Terraform Enterprise.

To prevent sensitive information from being stored in the shell history, temporarily write the current and new encryption passwords to files and read them upon execution, deleting the temporary files when finished:

tfe-admin rotate-encryption-password "$(cat current_password.txt)" "$(cat new_password.txt)"
tfe-admin rotate-encryption-password "$(cat current_password.txt)" "$(cat new_password.txt)"

A successful encryption password rotation will show the following output:

Encryption password successfully rotated!

Updating the `enc_password` application configuration on 2 node(s) to reflect the new encryption password.

You must update any installation or automation processes to reflect the new encryption password!
Encryption password successfully rotated!

Updating the `enc_password` application configuration on 2 node(s) to reflect the new encryption password.

You must update any installation or automation processes to reflect the new encryption password!

An unsuccessful encryption password rotation will show an error:

Error rotating encryption password:
Error:
exit status 1
Output:
Encryption password not rotated!
Error reading previous Vault configuration: failed decrypting unseal key: could not decrypt ciphertext: chacha20poly1305: message authentication failed
Error rotating encryption password:
Error:
exit status 1
Output:
Encryption password not rotated!
Error reading previous Vault configuration: failed decrypting unseal key: could not decrypt ciphertext: chacha20poly1305: message authentication failed

»Other Supporting Commands

There are additional commands available for checking status and troubleshooting directly on nodes. You can use them to confirm successful installation or to check on the status of a running node as part of troubleshooting activities. Also, there are additional command aliases available that allow you to run more abbreviated versions of commands like just support-bundle. Run an alias command with no parameters to see the list of available command aliases.

»Commands

»health-check

tfe-admin health-check
(alias health-check)
tfe-admin health-check
(alias health-check)

This command tests and reports on the status of the major TFE services. Each will be listed as PASS or FAIL. If any are marked as FAIL, your TFE implementation is NOT healthy and additional action must be taken.

»replicated status

replicatedctl system status
replicatedctl system status

Displays status info on the Replicated sub-system. Key values to note are that status values return as "ready". This reports on the status of the system on the node instance that it is run on.

»tfe application status

replicatedctl app status
replicatedctl app status

Displays status info on the TFE application. Key values to note are that State and DesiredState are both "started" and IsTransitioning is false. This reports on the status of the application on the node instance that it is run on.

»Upgrading TFE or Patching TFE Node Instances

The mechanism used to upgrade the TFE node instances is to fully repave the instances (destroy and rebuild entirely). This is another reason why using automation to build the instances is important. Currently, the safest way to perform and upgrade is to shut down all node instances, rebuild one node to validate a successful upgrade, and then scale to additional nodes (currently max 3).

These are the steps required to repave the node instances:

  • Run the node-drain command as described previously on each node to complete active work and stop new work from being processed.
  • Update the instance build configuration such as setting a new ReleaseSequence to upgrade versions and/or make any other alterations such as patching the base image used for building the instances.
  • Follow the instructions in Terraform Enterprise Active/Active to scale down to zero nodes and proceed through scaling up to one node, validating success, and then scaling additional nodes.

If planned and orchestrated efficiently, the total downtime for the repaving will be the amount of time it has taken to build one node as processing will resume as soon as the first node is functional.

  • Overview
  • Docs
  • Extend
  • Privacy
  • Security
  • Press Kit
  • Consent Manager