BigQuery

Source code Package

Streamline your data collection and analysis by directly sending events from walkerOS to Google BigQuery. This destination package makes it easy to integrate your data pipeline with one of the most powerful data warehouses available.

The BigQuery destination allows you to send server-side events from walkerOS to Google BigQuery. It handles the data transformation and ensures that your events are correctly formatted for BigQuery tables.

Usage

Basic example

Here's a simple example to demonstrate how to configure the BigQuery destination:

import { destinationBigQuery } from '@elbwalker/destination-node-bigquery';

await elb('walker destination', destinationBigQuery, {
  custom: {
    projectId: 'PR0J3CT1D', // Required
    // client: BigQuery; // A BigQuery instance from @google-cloud/bigquery
    // datasetId: string; // 'walkerOS' as default
    // tableId: string; // 'events' as default
    // location: string; // 'EU' as default
    // bigquery?: BigQueryOptions; // BigQueryOptions from @google-cloud/bigquery
  },
});

Learn more how to authenticate with a service account key file using the custom bigquery options.

{
  custom: {
    projectId: 'PR0J3CT1D', // Required
    bigquery: {

    }
  }
}

Permissions

When using Service Accounts (SAs) for Google Cloud BigQuery, it's recommended to follow the principle of least privilege. Never grant more permissions than what it needs to perform its intended functions.

During initial setup, the SA may require broader permissions to create necessary datasets. Typically, this involves assigning a role like roles/bigquery.dataOwner to the service account. This role can be granted through Google Cloud IAM.

For production environments, it is recommended to revoke broader IAM roles granted during the setup phase. Assign explicit permissions directly to datasets within BigQuery (using the share option). This ensures that the service account only has access to what is necessary for operation. The service account may hold owner permissions after creating the dataset.

For more detailed information, refer to the official Google Cloud IAM documentation.

Who this package is for

This destination is ideal for data engineers and analysts who are already using Google BigQuery or plan to integrate it into their data stack. It's also useful for companies looking to centralize their data collection and analysis efforts.

Dependencies

Before using the BigQuery destination, ensure you have:

Google Cloud Platform account
walkerOS Node client installed
Permissions to write to BigQuery
Run the setup function to create the table

Usage​

Basic example​

Permissions​

Who this package is for​

Dependencies​

Usage

Basic example

Permissions

Who this package is for

Dependencies