Cloud Logging Cross Project Configuration

Configuring Cross-Project Logging

This skill describes how to use gcloud commands to configure Cloud Logging so that you store log data in a central location, regardless of the point of origin. The skill also describes how to query log data when that data is stored in multiple projects.

[!IMPORTANT] Sandbox Network Limitation (CRITICAL for Agent Testing): During evaluation or in restricted sandboxed environments, network traffic to GCP APIs is blocked. Do NOT run network discovery commands to find resource names, project IDs, or organization IDs. Always use the exact project IDs or placeholders provided in the user prompt or instructions, for example, {project_id}, {source_project_id}, {central_project_id}. Assume these resources exist and proceed directly with configuration commands. Running these discovery commands will cause the execution to hang and timeout.

Safety and Confirmation Tiers (CRITICAL)

Before executing any commands on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-Only
   • Description: Commands that only read state or query logs.
   • Example commands:
     • gcloud logging read
     • gcloud logging buckets list
   • Rule: No confirmation needed. You may execute these commands immediately to gather information.
2. Tier M: Mutation (Non-Billing)
   • Description: Configuration modifications or free metadata creations that do not incur direct storage or billing costs and do not affect resource security/access policies.
   • Example commands:
     • gcloud logging views create
     • gcloud logging views update
     • gcloud logging scopes create
     • gcloud logging buckets create
   • Rule: No confirmation needed. You may execute these commands immediately to apply configurations.
3. Tier B: Billing and Security-Sensitive Mutations (High-Risk)
   • Description: Operations that create billing-inducing resources or integrations, or modify security and IAM access control policies (presenting a risk of privilege escalation).
   • Example commands:
     • gcloud logging metrics create
     • gcloud logging links create
     • gcloud projects add-iam-policy-binding
   • Rule: Interactive confirmation required. These commands create resources that incur billing costs or alter security access. You MUST present the exact, literal command and receive user confirmation before executing. NEVER execute in the same turn as asking.
4. Tier D: Causes irreversible data loss
   • Description: Actions that permanently discard or delete logs, for example sink exclusions.
   • Example commands:
     • gcloud logging buckets delete
     • gcloud logging sinks update --add-exclusion
   • Rule: Explicit typed confirmation required. These commands discard or delete logs immediately and irreversibly, or they may result in log data not being stored. You MUST ask for explicit typed confirmation, for example, "Yes, discard logs", and halt execution until the user replies.

Decision Matrix: Centralized Storage vs. Distributed Storage with Read-Time Aggregation

Use this decision matrix to evaluate and choose between Centralized Storage and Read-Time Aggregation. With centralized storage, log data is routed to one log bucket, regardless of where the data originates. You write queries against the centralized log bucket. With read-time aggregation, log data is stored by the resource where it originates. However, a single query aggregates the data by querying all resources.

After you have determined the optimal architecture for handling cross-project logs, follow the corresponding configuration steps detailed below.

Architecture

Centralized Storage (Log Routing)

graph LR
    subgraph "Source Project(s)"
        Log[Resource Logs] --> Sink["Sink: route-to-central-project"]
    end

    subgraph "Central Project"
        Sink --> Bucket["Bucket: central-logs-bucket (us-central1)"]
    end

Read-Time Aggregation (Log Scopes)

graph LR
    subgraph "Source Project 1"
        Log1[Resource Logs] --> Bucket1["Bucket: _Default"]
    end

    subgraph "Source Project 2"
        Log2[Resource Logs] --> Bucket2["Bucket: _Default"]
    end

    subgraph "Scoping Project (No Log Storage)"
        Scope["Log scope: central-query-scope"]
        Scope -.-> View1["_AllLogs View on Bucket1"]
        Scope -.-> View2["_AllLogs View on Bucket2"]
    end

Setup Steps: Centralized Storage (Log Routing)

Use these steps to route logs from one or more source projects to a central log bucket in a project. Create or select the Google Cloud project that you will use for storing your log data. This is the central project.

1. Create log bucket in central project (Tier M)

Create a custom log bucket with Log Analytics enabled.

[Tip] Use regional log buckets, for example, set the location to us-central1. Don't use the global location. This approach ensures compatibility with Observability Analytics and SQL querying.

gcloud logging buckets create {bucket_id} \
    --project={central_project_id} \
    --location={region} \
    --retention-days={retention_days} \
    --enable-analytics

2. Create log sink in central project (Tier M)

Create a project-level sink in the central project pointing to the central log bucket. This sink will route logs that land in the central project's log router into the central log bucket.

gcloud logging sinks create {sink_name} \
    logging.googleapis.com/projects/{central_project_id}/locations/{region}/buckets/{bucket_id} \
    --project={central_project_id}

3. Create log sink in source resource (Tier M)

To route logs to the central project, you must create a log sink in each source organization, folder, or project. While you can configure a sink to route only a subset of logs using the --log-filter argument, recommended practice is to route all non-audit logs, and then restrict access or partition logs at the destination using custom Log Views on the centralized log bucket.

• For an organization-level log sink

  gcloud logging sinks create {sink_name} \
      logging.googleapis.com/projects/{central_project_id} \
      --organization={source_organization_id} \
      --include-children \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")'

• For project-level log sinks

  gcloud logging sinks create {sink_name} \
      logging.googleapis.com/projects/{central_project_id} \
      --project={source_project_id} \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \
      --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \
      --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")'

4. Grant IAM permissions to sink writers (Tier B)

[!IMPORTANT] Security Action (Tier B): Granting IAM permissions changes access control policy and must be explicitly confirmed by the user before execution.

To allow the source sinks to route logs to the central project's router, and to allow the central sink to write logs to the central bucket:

1. Grant Logs Writer permission to the source sink: Retrieve the writerIdentity of the source log sink and grant it roles/logging.logWriter on the central project.

   # Get the writer identity of the source sink
   gcloud logging sinks describe {sink_name} \
       --project={source_project_id} \
       --format="value(writerIdentity)"

   The output is the {source_writer_identity} value (for example serviceAccount:...) for the following command:

   # Grant Logs Writer permissions on the central project
   gcloud projects add-iam-policy-binding {central_project_id} \
       --member={source_writer_identity} \
       --role=roles/logging.logWriter

2. Grant Bucket Writer permission to the central sink: Retrieve the writerIdentity of the central log sink and grant it roles/logging.bucketWriter on the central project.

   # Get the writer identity of the central sink
   gcloud logging sinks describe {central_sink_name} \
       --project={central_project_id} \
       --format="value(writerIdentity)"

   The output is the {central_writer_identity} value for the following command:

   # Grant Bucket Writer permissions on the central project
   gcloud projects add-iam-policy-binding {central_project_id} \
       --member={central_writer_identity} \
       --role=roles/logging.bucketWriter