• Overview
    • Enforce Policy as Code
    • Infrastructure as Code
    • Inject Secrets into Terraform
    • Integrate with Existing Workflows
    • Manage Kubernetes
    • Manage Virtual Machine Images
    • Multi-Cloud Deployment
    • Network Infrastructure Automation
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
  • Registry
  • Tutorials
    • About the Docs
    • Intro to Terraform
    • Configuration Language
    • Terraform CLI
    • Terraform Cloud
    • Terraform Enterprise
    • Provider Use
    • Plugin Development
    • Registry Publishing
    • Integration Program
    • Terraform Tools
    • CDK for Terraform
    • Glossary
  • Community
GitHubTerraform Cloud
Download

    Terraform Enterprise Admin

  • Overview
    • Credentials
    • Hardware
      • Supported OS
      • RedHat Linux
      • CentOS Linux
      • Operational Mode
      • PostgreSQL
      • Minio Setup Guide
      • External Vault
    • Network
    • Docker Engine
  • Operational Modes
    • Overview
    • AWS Reference Architecture
    • Azure Reference Architecture
    • GCP Reference Architecture
    • VMware Reference Architecture
    • Pre-Install Checklist
      • 1. Run Installer
      • 2. Configure in Browser
      • Automated Installation
      • Active/Active
      • Initial User Automation
      • Encryption Password
    • Uninstall
    • Configuration
    • Team Membership
    • Attributes
    • Login
      • Sample Auth Request
      • ADFS
      • Azure Active Directory
      • Okta
      • OneLogin
    • Troubleshooting
    • Overview
      • Automated Recovery
      • Upgrades
      • Log Forwarding
      • Monitoring
      • Backups and Restores
      • Admin CLI Commands
      • Terraform Cloud Agents on TFE
      • Demo to Mounted Disk Migration
    • Terraform Cloud Agents on TFE
      • Accessing the Admin Interface
      • General Settings
      • Customization
      • Integration Settings
      • Managing Accounts & Resources
      • Module Sharing
      • Admin API
      • Updating Terraform Enterprise License
    • Terraform Enterprise Logs
    • Overview
    • Architecture Summary
    • Reliability & Availability
    • Capacity & Performance
    • Security Model
    • Overview
      • Overview
      • v202206-1
      • v202205-1
      • v202204-2
      • v202204-1
      • v202203-1
      • v202202-1
      • v202201-2
      • v202201-1
      • Overview
      • v202112-2
      • v202112-1
      • v202111-1
      • v202110-1
      • v202109-2
      • v202109-1
      • v202108-1
      • v202107-1
      • v202106-1
      • v202105-1
      • v202104-1
      • v202103-3
      • v202103-2
      • v202103-1
      • v202102-2
      • v202102-1
      • v202101-1
      • Overview
      • Overview
      • Overview
  • Support
  • Application Usage

  • Overview
  • Plans and Features
  • Getting Started
    • API Docs template
    • Overview
    • Account
    • Agent Pools
    • Agent Tokens
    • Applies
    • Audit Trails
    • Comments
    • Configuration Versions
    • Cost Estimates
    • Feature Sets
    • Invoices
    • IP Ranges
    • Notification Configurations
    • OAuth Clients
    • OAuth Tokens
    • Organizations
    • Organization Memberships
    • Organization Tags
    • Organization Tokens
    • Plan Exports
    • Plans
    • Policies
    • Policy Checks
    • Policy Sets
    • Policy Set Parameters
      • Modules
      • Providers
      • Private Provider Versions and Platforms
      • GPG Keys
    • Runs
      • Run Tasks
      • Stages and Results
      • Custom Integration
    • Run Triggers
    • SSH Keys
    • State Versions
    • State Version Outputs
    • Subscriptions
    • Team Access
    • Team Membership
    • Team Tokens
    • Teams
    • User Tokens
    • Users
    • Variables
    • VCS Events
    • Workspaces
    • Workspace-Specific Variables
    • Workspace Resources
    • Variable Sets
      • Overview
      • Module Sharing
      • Organizations
      • Runs
      • Settings
      • Terraform Versions
      • Users
      • Workspaces
    • Changelog
    • Stability Policy
    • Overview
    • Creating Workspaces
    • Naming
    • Terraform Configurations
      • Overview
      • Managing Variables
      • Overview
      • VCS Connections
      • Access
      • Drift Detection
      • Notifications
      • SSH Keys for Modules
      • Run Triggers
      • Run Tasks
    • Terraform State
    • JSON Filtering
    • Remote Operations
    • Viewing and Managing Runs
    • Run States and Stages
    • Run Modes and Options
    • UI/VCS-driven Runs
    • API-driven Runs
    • CLI-driven Runs
    • The Run Environment
    • Installing Software
    • Users
    • Teams
    • Organizations
    • Permissions
    • Two-factor Authentication
    • API Tokens
      • Overview
      • Microsoft Azure AD
      • Okta
      • SAML
      • Linking a User Account
      • Testing
    • Overview
    • GitHub.com
    • GitHub.com (OAuth)
    • GitHub Enterprise
    • GitLab.com
    • GitLab EE and CE
    • Bitbucket Cloud
    • Bitbucket Server and Data Center
    • Azure DevOps Services
    • Azure DevOps Server
    • Troubleshooting
    • Overview
    • Adding Public Providers and Modules
    • Publishing Private Providers
    • Publishing Private Modules
    • Using Providers and Modules
    • Configuration Designer
  • Migrating to Terraform Cloud
    • Overview
    • Using Sentinel with Terraform 0.12
    • Manage Policies
    • Enforce and Override Policies
    • Mocking Terraform Sentinel Data
    • Working With JSON Result Data
      • Overview
      • tfconfig
      • tfconfig/v2
      • tfplan
      • tfplan/v2
      • tfstate
      • tfstate/v2
      • tfrun
    • Example Policies
    • Overview
    • AWS
    • GCP
    • Azure
      • Overview
      • Service Catalog
      • Admin Guide
      • Developer Reference
      • Example Customizations
      • V1 Setup Instructions
    • Splunk Integration
    • Kubernetes Integration
    • Run Tasks Integration
    • Overview
    • IP Ranges
    • Data Security
    • Security Model
    • Overview
    • Part 1: Overview of Our Recommended Workflow
    • Part 2: Evaluating Your Current Provisioning Practices
    • Part 3: How to Evolve Your Provisioning Practices
    • Part 3.1: From Manual Changes to Semi-Automation
    • Part 3.2: From Semi-Automation to Infrastructure as Code
    • Part 3.3: From Infrastructure as Code to Collaborative Infrastructure as Code
    • Part 3.4: Advanced Workflow Improvements

  • Terraform Cloud Agents

  • Other Docs

  • Intro to Terraform
  • Configuration Language
  • Terraform CLI
  • Terraform Cloud
  • Terraform Enterprise
  • Provider Use
  • Plugin Development
  • Registry Publishing
  • Integration Program
  • Terraform Tools
  • CDK for Terraform
  • Glossary
Type '/' to Search

»Terraform Enterprise Log Forwarding

Terraform Enterprise supports forwarding its logs to one or more external destinations, a process called log forwarding. Log forwarding provides increased observability, assistance complying with log retention requirements, and information during troubleshooting.

Note: For Terraform Enterprise versions prior to v202109-1, please refer to the previous version of this page with logspout information.

»Requirements

Log forwarding requires:

  • Terraform Enterprise running on an instance using systemd-journald. Execute systemctl status systemd-journald to check if the systemd-journald service is started and enabled.
  • A version of Docker that supports the journald logging driver. Execute docker info --format '{{.Plugins.Log}}' to check if the journald plugin is listed.
  • Network connectivity between Terraform Enterprise and the external destination(s) where logs should be forwarded.

»Enable Log Forwarding

Log forwarding is disabled by default. To enable log forwarding, set the log_forwarding_enabled Terraform Enterprise application setting to the value 1.

tfe-admin app-config -k log_forwarding_enabled -v 1
tfe-admin app-config -k log_forwarding_enabled -v 1

Note: For Terraform Enterprise versions before v202203-1, use this command for standalone installations

replicatedctl app-config set log_forwarding_enabled --value 1
replicatedctl app-config set log_forwarding_enabled --value 1

When log forwarding is enabled, the Terraform Enterprise application settings show the following for log_forwarding_enabled:

    "log_forwarding_enabled": {
        "value": "1"
    },
    "log_forwarding_enabled": {
        "value": "1"
    },

»Configure External Destinations

The log_forwarding_config Terraform Enterprise application setting must contain valid Fluent Bit [OUTPUT] configuration specifying supported external destination(s) where Terraform Enterprise should forward logs. The default configuration does not forward any logs.

Since the Terraform Enterprise application settings are stored as JSON strings, we recommend first creating a fluent-bit.conf file with the valid Fluent Bit [OUTPUT] configuration and then using that file to configure the log_forwarding_config application setting. This method ensures that the configuration is stored in the application settings exactly how it appears in the fluent-bit.conf file.

For a Standalone installation of Terraform Enterprise:

replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"
replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

For an Active/Active installation of Terraform Enterprise:

tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"
tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"

Once configured, the Terraform Enterprise application settings show the log_forwarding_config setting in escaped JSON string format:

    "log_forwarding_config": {
        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"
    },
    "log_forwarding_config": {
        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"
    },

That escaped JSON string renders to the following:

# Match all logs and do not forward them anywhere.
[OUTPUT]
    Name null
    Match *
# Match all logs and do not forward them anywhere.
[OUTPUT]
    Name null
    Match *

To forward logs to multiple external destinations, use multiple [OUTPUT] directives.

# Forward all logs to Datadog.
[OUTPUT]
    Name datadog
    Match *
    ...

# Forward all logs to Fluent Bit or Fluentd.
[OUTPUT]
    Name forward
    Match *
    ...
# Forward all logs to Datadog.
[OUTPUT]
    Name datadog
    Match *
    ...

# Forward all logs to Fluent Bit or Fluentd.
[OUTPUT]
    Name forward
    Match *
    ...

Note: Do not use an [OUTPUT] directive with the stdout Fluent Bit output plugin. Doing this creates a loop that continuously emits logs!

»Restart Terraform Enterprise

Once log forwarding is enabled and configured, you need to restart Terraform Enterprise for the changes to take effect. Please refer to the HashiCorp Help Center for details on how to restart Terraform Enterprise.

»Supported External Destinations

You can only forward logs to one of the supported external destinations below. Each supported external destination contains example configuration for convenience.

»Amazon CloudWatch

Sending to Amazon CloudWatch is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon CloudWatch. Refer to the cloudwatch_logs Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name               cloudwatch_logs
    Match              *
    region             us-east-1
    log_group_name     example-log-group
    log_stream_name    example-log-stream
    auto_create_group  On
[OUTPUT]
    Name               cloudwatch_logs
    Match              *
    region             us-east-1
    log_group_name     example-log-group
    log_stream_name    example-log-stream
    auto_create_group  On

Note: In Terraform Enterprise installations using AWS external services, Fluent Bit will have access to the same AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables that are used for object storage.

»Amazon S3

Sending to Amazon S3 is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon S3. Refer to the s3 Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                          s3
    Match                         *
    bucket                        example-bucket
    region                        us-east-1
    total_file_size               250M
    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz
    s3_key_format_tag_delimiters  .-
[OUTPUT]
    Name                          s3
    Match                         *
    bucket                        example-bucket
    region                        us-east-1
    total_file_size               250M
    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz
    s3_key_format_tag_delimiters  .-

Note: In Terraform Enterprise installations using AWS external services, Fluent Bit will have access to the same AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables that are used for object storage.

»Azure Blob Storage

This example configuration forwards all logs to Azure Blob Storage. Refer to the azure_blob Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name                   azure_blob
    match                  *
    account_name           example-account-name
    shared_key             example-access-key
    path                   logs
    container_name         example-container-name
    auto_create_container  on
    tls                    on
[OUTPUT]
    name                   azure_blob
    match                  *
    account_name           example-account-name
    shared_key             example-access-key
    path                   logs
    container_name         example-container-name
    auto_create_container  on
    tls                    on

»Azure Log Analytics

This example configuration forwards all logs to Azure Log Analytics. Refer to the azure Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name         azure
    match        *
    Customer_ID  example-log-analytics-workspace-id
    Shared_Key   example-access-key
[OUTPUT]
    name         azure
    match        *
    Customer_ID  example-log-analytics-workspace-id
    Shared_Key   example-access-key

»Datadog

This example configuration forwards all logs to Datadog. Refer to the datadog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name        datadog
    Match       *
    Host        http-intake.logs.datadoghq.com
    TLS         on
    compress    gzip
    apikey      example-api-key
    dd_service  terraform_enterprise
    dd_source   docker
    dd_tags     environment:development,owner:engineering
[OUTPUT]
    Name        datadog
    Match       *
    Host        http-intake.logs.datadoghq.com
    TLS         on
    compress    gzip
    apikey      example-api-key
    dd_service  terraform_enterprise
    dd_source   docker
    dd_tags     environment:development,owner:engineering

»Forward

This example configuration forwards all logs to a listening Fluent Bit or Fluentd instance. Refer to the forward Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name   forward
    Match  *
    Host   fluent.example.com
    Port   24224
[OUTPUT]
    Name   forward
    Match  *
    Host   fluent.example.com
    Port   24224

»Google Cloud Platform Cloud Logging

Sending to Google Cloud Platform Cloud Logging is only supported when Terraform Enterprise is located within GCP due to how Fluent Bit reads GCP credentials.

This example configuration forwards all logs to Google Cloud Platform Cloud Logging (formerly known as Stackdriver). Refer to the stackdriver Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name       stackdriver
    Match      *
    location   us-east1
    namespace  terraform_enterprise
    node_id    example-hostname
    resource   generic_node
[OUTPUT]
    Name       stackdriver
    Match      *
    location   us-east1
    namespace  terraform_enterprise
    node_id    example-hostname
    resource   generic_node

Note: In Terraform Enterprise installations using GCP external services, Fluent Bit will have access to the GOOGLE_SERVICE_CREDENTIALS environment variable that points to a file containing the same GCP Service Account JSON credentials that are used for object storage.

»Splunk Enterprise HTTP Event Collector (HEC)

This example configuration forwards all logs to Splunk Enterprise via the HTTP Event Collector (HEC) interface. Refer to the splunk Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name          splunk
    Match         *
    Host          example-splunk-hec-endpoint
    Port          8088
    Splunk_Token  example-splunk-token
[OUTPUT]
    Name          splunk
    Match         *
    Host          example-splunk-hec-endpoint
    Port          8088
    Splunk_Token  example-splunk-token

»Syslog

This example configuration forwards all logs to a Syslog-compatible endpoint. Refer to the syslog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                 syslog
    Match                *
    host                 example-syslog-host
    port                 514
    mode                 tcp
    syslog_message_key   message
    syslog_severity_key  PRIORITY
    syslog_hostname_key  _HOSTNAME
    syslog_appname_key   SYSLOG_IDENTIFIER
    syslog_procid_key    _PID
[OUTPUT]
    Name                 syslog
    Match                *
    host                 example-syslog-host
    port                 514
    mode                 tcp
    syslog_message_key   message
    syslog_severity_key  PRIORITY
    syslog_hostname_key  _HOSTNAME
    syslog_appname_key   SYSLOG_IDENTIFIER
    syslog_procid_key    _PID

»Audit Logs

Terraform Enterprise emits its audit logs along with its application logs. Currently, log forwarding can forward either all Terraform Enterprise logs or no logs at all. To distinguish audit logs from application logs, audit log entries contain the string [Audit Log].

Here's an example audit log entry formatted for readability:

2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]
{
  "resource":"run",
  "action":"create",
  "resource_id":"run-nL77p69bsesoF3RK",
  "organization":"example-org",
  "organization_id":"org-pveSPvxocni226Fn",
  "actor":"example-user",
  "timestamp":"2021-08-31T04:58:30Z",
  "actor_ip":"19.115.231.192"
}
2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]
{
  "resource":"run",
  "action":"create",
  "resource_id":"run-nL77p69bsesoF3RK",
  "organization":"example-org",
  "organization_id":"org-pveSPvxocni226Fn",
  "actor":"example-user",
  "timestamp":"2021-08-31T04:58:30Z",
  "actor_ip":"19.115.231.192"
}

If you have a requirement to split audit logs from application logs, we recommend forwarding all Terraform Enterprise logs to a log aggregation system, filtering the audit logs based on the [Audit Log] string, and forwarding just the audit logs to the desired destination.

»Log Rotation

Log forwarding uses the journald Docker logging driver to send Terraform Enterprise logs to systemd-journald. This can cause increased disk utilization for the /var/log/journal path.

To limit disk utilization, configure the SystemMaxFileSize and SystemMaxFiles settings within /etc/systemd/journald.conf.

The following configuration tells systemd-journald to use up to 7GB of disk space by limiting the size a log file to 1024MB and keeping up to 7 files at any given time:

[Journal]
SystemMaxFileSize=1024M
SystemMaxFiles=7
[Journal]
SystemMaxFileSize=1024M
SystemMaxFiles=7

To apply these changes, restart systemd-journald:

sudo systemctl restart systemd-journald
sudo systemctl restart systemd-journald
github logoEdit this page
  • Overview
  • Docs
  • Extend
  • Privacy
  • Security
  • Press Kit
  • Consent Manager