Migrate From Storage-Based to Pub/Sub-Based Google Cloud Audit Log Integration - Manual Configuration

This topic describes how you can use the Lacework Console to manually migrate your Google Cloud Storage-based audit log integration to a Pub/Sub-based audit log integration for audit log monitoring.

Lacework recommends this migration procedure because it ensures audit log monitoring coverage for your Google Cloud organization or project during the migration.

The migration involves the following four steps. All the steps are required to ensure that there is no gap in delivery of audit log data from Google Cloud to the Lacework platform during the migration.

1. Collect details of existing Storage-based audit log integration
2. Create Pub/Sub topic and subscription
3. Create Pub/Sub-based audit log integration
4. Mark existing Storage-based audit log integration for migration

note

You can also use Terraform to migrate your Storage-based integration to a Pub/Sub-based integration. For more information, see Migrate From Storage-Based to Pub/Sub-Based Google Cloud Audit Log Integration Using Terraform.

Important

If you do not want audit log monitoring coverage for your Google Cloud organization or project during the migration, you can skip this migration procedure and perform the following three steps. Note that this can result in a brief gap in delivery of audit log data from Google Cloud to the Lacework platform.

1. Create a Pub/Sub-based audit log integration using one of the following methods:

   • Guided integration method described in Create an Audit Log (PubSub)+Configuration Integration.

     Caution: The guided integration method will overwrite the Terraform files in the ~/lacework/gcp directory. If this directory contains the Terraform files for an existing Google Cloud configuration or audit log integration, all the resources created for that integration in Google Cloud will be deleted. Hence, Lacework recommends that you delete the main.tf and terraform.tfstate files in the ~/lacework/gcp directory before you use the guided integration method.

   • Manual integration method described in Pub/Sub-based Google Cloud Audit Log Integration - Manual Configuration.

2. In the Lacework Console, go to Settings > Integrations > Cloud Accounts and delete your Storage-based audit log integration.
3. (Optional) Delete the sink and storage bucket for the Storage-based audit log integration. For more information, see Delete the Sink and Storage Bucket for the Storage-based Integration.

Prerequisites

• Ensure that you have the required privileges in Google Cloud. For more information, see Required Roles for Google Cloud Configuration and Audit Log Integrations.

• Google Cloud Console - Administrator access to Google Cloud Console.

• Lacework Console - Org admin or Account admin access to the Lacework Console.

• Lacework CLI

• In the Lacework Console, go to Settings > Integrations > Cloud Accounts and verify that the Storage-based audit log integration is not disabled.

  Important

  Do not disable the Storage-based audit log integration before or during the migration process. Once the migration is completed, the Storage-based audit log integration will be deleted automatically.

Collect Details of Existing Storage-Based Audit Log Integration

You can reuse the project and service account that you created for the Storage-based audit log integration for the Pub/Sub-based audit log integration. Do the following to collect the project and service account details from the Lacework Console.

1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts.

2. Select the row for the Storage-based integration. A Storage-based integration has the provider as GCP and type as Audit Log (Storage).

   The Cloud Account page displays the integration details.

   • For an organization-level Storage-based integration:

     • The Account field displays the ID of the organization for which you created the Storage-based integration.

     • The Client Email field displays the email ID of the service account you created for the Storage-based integration. The service account email ID is in the format: my-service-account@my-project-name.iam.gserviceaccount.com.

       The service account email ID contains the name of the project in which the service account exists. For example, if the email ID is my-service-account@my-project-name.iam.gserviceaccount.com, my-project-name is the name of the project in which the service account exists. In the Google Cloud Console, open the page for the project and note the Project ID for use in the procedures below.

     • The ID field displays the ID of the integration.

   • For a project-level Storage-based integration:

     • The Account field displays the ID of the project in which you configured the resources for the Storage-based integration.
     • The Client Email field displays the email ID of the service account you created for the Storage-based integration. The service account email ID is in the format: my-service-account@my-project-name.iam.gserviceaccount.com.
     • The ID field displays the ID of the integration.

3. Note the project ID, service account email ID, and integration ID for use in the procedures below.

Create Pub/Sub Topic and Subscription

In this procedure, you will create a Pub/Sub topic and subscription to record audit log events.

1. Create a Pub/Sub topic in the project you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure. Follow the steps in Create a topic.

2. Create a subscription for the Pub/Sub topic. Follow the steps in Add a subscription.

3. Create a log sink, select the Pub/Sub topic as the sink destination, and add inclusion and exclusion filters.

   1. Do one of the following:

      • For a project-level Pub/Sub audit log integration, create a log sink in the project you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure, select the Pub/Sub topic as the sink destination, and add inclusion and exclusion filters using the instructions in Create a sink.
      • For an organization-level Pub/Sub audit log integration, create an aggregated log sink at the organization level, select the Pub/Sub topic as the sink destination, and add inclusion and exclusion filters using the instructions in Create an aggregated sink.

   2. Add the following inclusion filter to the log sink:

      (protoPayload.@type=type.googleapis.com/google.cloud.audit.AuditLog)

   3. Add the following exclusion filters to the log sink:

      (protoPayload.serviceName="k8s.io") AND (protoPayload.serviceName="login.googleapis.com") AND (protoPayload.methodName="storage.objects")

4. Grant the roles/pubsub.publisher role to the sink's writer identity using the instructions in Set destination permissions.

5. Grant the following roles to the service account you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure.

   • roles/monitoring.viewer role on the project you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure to the service account.
   • roles/pubsub.subscriber role on the Pub/Sub subscription to the service account.

Create Pub/Sub-based Audit Log Integration

Create the Pub/Sub-based audit log integration using the instructions in Create the Google Cloud Audit Log Integration on the Lacework Console.

Mark Existing Storage-Based Audit Log Integration for Migration

After you create the Pub/Sub-based audit log integration, you must mark the existing Storage-based audit log integration for migration. When you mark a Storage-based integration for migration, the Lacework Platform ensures that all the audit log messages in the storage bucket for the integration are ingested, and then safely deletes the integration.

1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts and verify that the Storage-based audit log integration is not disabled.

2. Run the following Lacework CLI command:

   lacework cloud-account migrate IntegrationID

   Where IntegrationID is the integration ID you identified in the Collect Details of Existing Storage-Based Audit Log Integration procedure.

Delete the Sink and Storage Bucket for the Storage-based Integration (Optional)

To reduce your Google Cloud storage costs, you can delete the log sink and storage bucket for the Storage-based integration.

1. In the Lacework Console, go to Settings > Integrations > Cloud Accounts.
2. Ensure that the Storage-based audit log integration that you marked for migration is not displayed on the Cloud accounts page. It can take up to five hours for an integration that is marked for migration to be deleted.
3. To delete a sink, see the instructions in Manage Sinks.
4. To delete a storage bucket, see the instructions in Delete a Bucket.

Create Custom LQL Policies for the Pub/Sub-Based Audit Log Integration

The Pub/Sub-based audit log integration does not support the default Google Cloud audit log policies. You must use the LW_ACT_GCP_ACTIVITY Lacework Query Language (LQL) datasource to create custom LQL policies for Pub/Sub-based audit log integrations.