»Terraform Enterprise Log Forwarding

Terraform Enterprise supports forwarding its logs to one or more external destinations, a process called log forwarding. Log forwarding provides increased observability, assistance complying with log retention requirements, and information during troubleshooting.

»Requirements

Log forwarding requires:

• Terraform Enterprise running on an instance using systemd-journald. Execute systemctl status systemd-journald to check if the systemd-journald service is started and enabled.
• A version of Docker that supports the journald logging driver. Execute docker info --format '{{.Plugins.Log}}' to check if the journald plugin is listed.
• Network connectivity between Terraform Enterprise and the external destination(s) where logs should be forwarded.

»Enable Log Forwarding

Log forwarding is disabled by default. To enable log forwarding, set the log_forwarding_enabled Terraform Enterprise application setting to the value 1.

For a Standalone installation of Terraform Enterprise:

replicatedctl app-config set log_forwarding_enabled --value 1

replicatedctl app-config set log_forwarding_enabled --value 1

For an Active/Active installation of Terraform Enterprise:

tfe-admin app-config -k log_forwarding_enabled -v 1

tfe-admin app-config -k log_forwarding_enabled -v 1

When log forwarding is enabled, the Terraform Enterprise application settings show the following for log_forwarding_enabled:

    ...
    "log_forwarding_enabled": {
        "value": "1"
    },
    ...

    ...    "log_forwarding_enabled": {        "value": "1"    },    ...

»Configure External Destinations

The log_forwarding_config Terraform Enterprise application setting must contain valid Fluent Bit [OUTPUT] configuration specifying supported external destination(s) where Terraform Enterprise should forward logs. The default configuration does not forward any logs.

Since the Terraform Enterprise application settings are stored as JSON strings, we recommend first creating a fluent-bit.conf file with the valid Fluent Bit [OUTPUT] configuration and then using that file to configure the log_forwarding_config application setting. This method ensures that the configuration is stored in the application settings exactly how it appears in the fluent-bit.conf file.

For a Standalone installation of Terraform Enterprise:

replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

For an Active/Active installation of Terraform Enterprise:

tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"

tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"

Once configured, the Terraform Enterprise application settings show the log_forwarding_config setting in escaped JSON string format:

    ...
    "log_forwarding_config": {
        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"
    },
    ...

    ...    "log_forwarding_config": {        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"    },    ...

# Match all logs and do not forward them anywhere.
[OUTPUT]
    Name null
    Match *

# Match all logs and do not forward them anywhere.[OUTPUT]    Name null    Match *

To forward logs to multiple external destinations, use multiple [OUTPUT] directives.

# Forward all logs to Datadog.
[OUTPUT]
    Name datadog
    Match *
    ...

# Forward all logs to Fluent Bit or Fluentd.
[OUTPUT]
    Name forward
    Match *
    ...

# Forward all logs to Datadog.[OUTPUT]    Name datadog    Match *    ...
# Forward all logs to Fluent Bit or Fluentd.[OUTPUT]    Name forward    Match *    ...

»Restart Terraform Enterprise

Once log forwarding is enabled and configured, you need to restart Terraform Enterprise for the changes to take effect. Please refer to the HashiCorp Help Center for details on how to restart Terraform Enterprise.

»Supported External Destinations

You can only forward logs to one of the supported external destinations below. Each supported external destination contains example configuration for convenience.

»Amazon CloudWatch

Sending to Amazon CloudWatch is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon CloudWatch. Refer to the cloudwatch_logs Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name               cloudwatch_logs
    Match              *
    region             us-east-1
    log_group_name     example-log-group
    log_stream_name    example-log-stream
    auto_create_group  On

[OUTPUT]    Name               cloudwatch_logs    Match              *    region             us-east-1    log_group_name     example-log-group    log_stream_name    example-log-stream    auto_create_group  On

»Amazon S3

Sending to Amazon S3 is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon S3. Refer to the s3 Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                          s3
    Match                         *
    bucket                        example-bucket
    region                        us-east-1
    total_file_size               250M
    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz
    s3_key_format_tag_delimiters  .-

[OUTPUT]    Name                          s3    Match                         *    bucket                        example-bucket    region                        us-east-1    total_file_size               250M    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz    s3_key_format_tag_delimiters  .-

»Azure Blob Storage

This example configuration forwards all logs to Azure Blob Storage. Refer to the azure_blob Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name                   azure_blob
    match                  *
    account_name           example-account-name
    shared_key             example-access-key
    path                   logs
    container_name         example-container-name
    auto_create_container  on
    tls                    on

[OUTPUT]    name                   azure_blob    match                  *    account_name           example-account-name    shared_key             example-access-key    path                   logs    container_name         example-container-name    auto_create_container  on    tls                    on

»Azure Log Analytics

This example configuration forwards all logs to Azure Log Analytics. Refer to the azure Fluent Bit output plugin documentation for more information.

[OUTPUT]
    name         azure
    match        *
    Customer_ID  example-log-analytics-workspace-id
    Shared_Key   example-access-key

[OUTPUT]    name         azure    match        *    Customer_ID  example-log-analytics-workspace-id    Shared_Key   example-access-key

»Datadog

This example configuration forwards all logs to Datadog. Refer to the datadog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name        datadog
    Match       *
    Host        http-intake.logs.datadoghq.com
    TLS         on
    compress    gzip
    apikey      example-api-key
    dd_service  terraform_enterprise
    dd_source   docker
    dd_tags     environment:development,owner:engineering

[OUTPUT]    Name        datadog    Match       *    Host        http-intake.logs.datadoghq.com    TLS         on    compress    gzip    apikey      example-api-key    dd_service  terraform_enterprise    dd_source   docker    dd_tags     environment:development,owner:engineering

»Forward

This example configuration forwards all logs to a listening Fluent Bit or Fluentd instance. Refer to the forward Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name   forward
    Match  *
    Host   fluent.example.com
    Port   24224

[OUTPUT]    Name   forward    Match  *    Host   fluent.example.com    Port   24224

»Google Cloud Platform Cloud Logging

Sending to Google Cloud Platform Cloud Logging is only supported when Terraform Enterprise is located within GCP due to how Fluent Bit reads GCP credentials.

This example configuration forwards all logs to Google Cloud Platform Cloud Logging (formerly known as Stackdriver). Refer to the stackdriver Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name       stackdriver
    Match      *
    location   us-east1
    namespace  terraform_enterprise
    node_id    example-hostname
    resource   generic_node

[OUTPUT]    Name       stackdriver    Match      *    location   us-east1    namespace  terraform_enterprise    node_id    example-hostname    resource   generic_node

»Splunk Enterprise HTTP Event Collector (HEC)

This example configuration forwards all logs to Splunk Enterprise via the HTTP Event Collector (HEC) interface. Refer to the splunk Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name          splunk
    Match         *
    Host          example-splunk-hec-endpoint
    Port          8088
    Splunk_Token  example-splunk-token

[OUTPUT]    Name          splunk    Match         *    Host          example-splunk-hec-endpoint    Port          8088    Splunk_Token  example-splunk-token

»Syslog

This example configuration forwards all logs to a Syslog-compatible endpoint. Refer to the syslog Fluent Bit output plugin documentation for more information.

[OUTPUT]
    Name                 syslog
    Match                *
    host                 example-syslog-host
    port                 514
    mode                 tcp
    syslog_message_key   message
    syslog_severity_key  PRIORITY
    syslog_hostname_key  _HOSTNAME
    syslog_appname_key   SYSLOG_IDENTIFIER
    syslog_procid_key    _PID

[OUTPUT]    Name                 syslog    Match                *    host                 example-syslog-host    port                 514    mode                 tcp    syslog_message_key   message    syslog_severity_key  PRIORITY    syslog_hostname_key  _HOSTNAME    syslog_appname_key   SYSLOG_IDENTIFIER    syslog_procid_key    _PID

»Audit Logs

Terraform Enterprise emits its audit logs along with its application logs. Currently, log forwarding can forward either all Terraform Enterprise logs or no logs at all. To distinguish audit logs from application logs, audit log entries contain the string [Audit Log].

Here's an example audit log entry formatted for readability:

2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]
{
  "resource":"run",
  "action":"create",
  "resource_id":"run-nL77p69bsesoF3RK",
  "organization":"example-org",
  "organization_id":"org-pveSPvxocni226Fn",
  "actor":"example-user",
  "timestamp":"2021-08-31T04:58:30Z",
  "actor_ip":"19.115.231.192"
}

2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]{  "resource":"run",  "action":"create",  "resource_id":"run-nL77p69bsesoF3RK",  "organization":"example-org",  "organization_id":"org-pveSPvxocni226Fn",  "actor":"example-user",  "timestamp":"2021-08-31T04:58:30Z",  "actor_ip":"19.115.231.192"}

If you have a requirement to split audit logs from application logs, we recommend forwarding all Terraform Enterprise logs to a log aggregation system, filtering the audit logs based on the [Audit Log] string, and forwarding just the audit logs to the desired destination.