Skip to main content

Migrate From Storage-Based to Pub/Sub-Based Google Cloud Audit Log Integration Using Terraform

This topic describes how you can use Terraform to migrate your existing Google Cloud Storage-based audit log integration to a Pub/Sub-based audit log integration for audit log monitoring.

Lacework recommends this migration procedure because it ensures audit log monitoring coverage for your Google Cloud organization or project during the migration.

The migration involves the following four steps. All the steps are required to ensure that there is no gap in delivery of audit log data from Google Cloud to the Lacework platform during the migration.

  1. Collect details of existing Storage-based audit log integration
  2. Delete Terraform files for the existing Storage-based audit log integration
  3. Create Pub/Sub-based audit log integration
  4. Mark existing Storage-based audit log integration for migration
note

You can also use the Lacework Console to manually migrate your Storage-based integration to a Pub/Sub-based integration. For more information, see Migrate From Storage-Based to Pub/Sub-Based Google Cloud Audit Log Integration - Manual Configuration.

Important

If you do not want audit log monitoring coverage for your Google Cloud organization or project during the migration, you can skip this migration procedure and perform the following four steps. Note that this can result in a brief gap in delivery of audit log data from Google Cloud to the Lacework platform.

  1. Delete the Terraform files for your Storage-based audit log integration. For more information, see Delete Terraform Files for the Existing Storage-Based Audit Log Integration.
  2. Create a Pub/Sub-based audit log integration using one of the following methods:
  3. Do one of the following to delete your Storage-based audit log integration:
    • Use the lacework cloud-account delete Lacework CLI command to delete the integration.
    • In the Lacework Console, go to Settings > Integrations > Cloud Accounts and delete the integration.
  4. (Optional) Delete the sink and storage bucket for the Storage-based audit log integration. For more information, see Delete the Sink and Storage Bucket for the Storage-based Integration.

Prerequisites​

  • Ensure that you have the required privileges in Google Cloud. For more information, see Required Roles for Google Cloud Configuration and Audit Log Integrations.

  • Google Cloud Console - Administrator access to Google Cloud Console.

  • Lacework Console - Org admin or Account admin access to the Lacework Console.

  • gcloud CLI - To configure resources in Google Cloud.

  • Lacework CLI - To generate and execute the Terraform code for the integration.

  • Terraform - Lacework Terraform projects support Terraform versions ~> 0.14, ~> 0.15, ~> 1.0, and ~> 1.1.

  • In the Lacework Console, go to Settings > Integrations > Cloud Accounts and verify that the Storage-based audit log integration is not disabled.

    Important

    Do not disable the Storage-based audit log integration before or during the migration process. Once the migration is completed, the Storage-based audit log integration will be deleted automatically.

Collect Details of Existing Storage-Based Audit Log Integration​

You can reuse the project and service account that you created for the Storage-based audit log integration for the Pub/Sub-based audit log integration. Do the following to collect the project and service account details from the Lacework Console.

  1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts.

  2. Select the row for the Storage-based integration. A Storage-based integration has the provider as GCP and type as Audit Log (Storage).

    The Cloud Account page displays the integration details.

    • For an organization-level Storage-based integration:

      • The Account field displays the ID of the organization for which you created the Storage-based integration.

      • The Client Email field displays the email ID of the service account you created for the Storage-based integration. The service account email ID is in the format: my-service-account@my-project-name.iam.gserviceaccount.com.

        The service account email ID contains the name of the project in which the service account exists. For example, if the email ID is my-service-account@my-project-name.iam.gserviceaccount.com, my-project-name is the name of the project in which the service account exists. In the Google Cloud Console, open the page for the project and note the Project ID for use in the procedures below.

      • The ID field displays the ID of the integration.

    • For a project-level Storage-based integration:

      • The Account field displays the ID of the project in which you configured the resources for the Storage-based integration.
      • The Client Email field displays the email ID of the service account you created for the Storage-based integration. The service account email ID is in the format: my-service-account@my-project-name.iam.gserviceaccount.com.
      • The ID field displays the ID of the integration.

  3. Note the organization ID, project ID, service account email ID, and integration ID for use in the procedures below.

Delete Terraform Files for the Existing Storage-Based Audit Log Integration​

By default, the Terraform files for the Storage-based integration are created in the ~/lacework/gcp directory when you run the lacework generate cloud-account gcp Lacework CLI command.

Lacework recommends that you delete the main.tf and terraform.tfstate Terraform files for your Storage-based integration to ensure that they are not used to accidentally recreate the Storage-based integration or cause Terraform state conflicts.

Create Pub/Sub-Based Audit Log Integration​

In this procedure, you will create a Pub/Sub topic and subscription to record audit log events and add a Pub/Sub-based audit log integration to the Lacework platform.

  1. Do one of the following:

    • For an organization-level Pub/Sub-based audit log integration, run the following command: 

      lacework generate cloud-account gcp  \
        --audit_log --use_pub_sub --audit_log_integration_name AuditLogIntegName \
        --organization_integration         \
        --organization_id OrganizationId   \
        --project_id ProjectId             \
        --service_account_credentials PathToServiceAccountKeyFile \
        --output OutputDirectoryPath \
        --noninteractive

    • For a project-level Pub/Sub-based audit log integration, run the following command:

      lacework generate cloud-account gcp  \
        --audit_log --use_pub_sub --audit_log_integration_name AuditLogIntegName \
        --project_id ProjectId   \
        --service_account_credentials PathToServiceAccountKeyFile \
        --output OutputDirectoryPath \
        --noninteractive

    Where:

    Warning
    • Lacework recommends that you specify an output directory that is different from the one where the Terraform files for a Google Cloud configuration or audit log integration are stored. If you specify an output directory that contains the Terraform files for an existing Google Cloud configuration or audit log integration, all the resources created for that integration in Google Cloud will be deleted.
    • If you do not specify the --output OutputDirectoryPath option, the Terraform files will be created in the ~/lacework/gcp directory. If the ~/lacework/gcp directory contains the Terraform files for an existing Google Cloud configuration or audit log integration, all the resources created for that integration in Google Cloud will be deleted.
    tip

    If you have a CI/CD pipeline, ensure that you run the Terraform for the Pub/Sub-based audit log integration in a directory that does not contain Terraform files for an existing Google Cloud configuration or audit log integration.

  2. Navigate to the output directory that you specified in the --output OutputDirectoryPath option.

  3. Run terraform plan and review the changes that will be applied.

  4. Once satisfied with the changes that will be applied, run terraform apply to execute Terraform.

Mark Existing Storage-Based Audit Log Integration for Migration​

After you create the Pub/Sub-based audit log integration, you must mark the existing Storage-based audit log integration for migration. When you mark a Storage-based integration for migration, the Lacework Platform ensures that all the audit log messages in the storage bucket for the integration are ingested, and then safely deletes the integration.

  1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts and verify that the Storage-based audit log integration is not disabled.

  2. Run the following Lacework CLI command:

    lacework cloud-account migrate IntegrationID

    Where IntegrationID is the integration ID you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure.

Delete the Sink and Storage Bucket for the Storage-based Integration (Optional)​

To reduce your Google Cloud storage costs, you can delete the log sink and storage bucket for the Storage-based integration.

  1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts.
  2. Ensure that the Storage-based audit log integration that you marked for migration is not displayed on the Cloud accounts page. It can take up to five hours for an integration that is marked for migration to be deleted.
  3. To delete a sink, see the instructions in Manage Sinks.
  4. To delete a storage bucket, see the instructions in Delete a Bucket.

Create Custom LQL Policies for the Pub/Sub-Based Audit Log Integration​

The Pub/Sub-based audit log integration does not support the default Google Cloud audit log policies. You must use the LW_ACT_GCP_ACTIVITY Lacework Query Language (LQL) datasource to create custom LQL policies for Pub/Sub-based audit log integrations.

Last updated on